Your premise about predictability is wrong. First of all, humans are also unpredictable. We just had a case where two pilots started a brawl in the cabin and almost took the airplane down with them. Second, and most importantly, the code generated by LLMs is just as deterministic as the ones by humans. So if you think we can reasonably test code made by humans, then we can also test code made by AI, regardless of whether its writer is deterministic in nature or not (ie, "predictable").
Human error rates have already been beaten, even in precisely the domains we are discussing here. Just look at safety data from Waymo for example [1].
Human error rates have already been beaten, even in precisely the domains we are discussing here. Just look at safety data from Waymo for example [1].
[1] https://waymo.com/safety/impact/