> you're ignoring the distinct possibility that [...] LLMs could simply stop writing bugs
That just doesn't seem like a reasonable possibility to me, given what we're seeing from and learning about these models. They have no capacity to be aware and adhere to an overarching vision, which is why they inevitably corrode architectural coherence over time [1]. The only reason these agents are able to still create usable code is because harnesses take advantage of model's ability to find bugs, giving us the famous loop, adding duct tape on top of duct tape, ratcheting up the complexity, until the adversarial model isn't able to find any more issues. Thus, the bugs are pushed out into higher complexity dimensions, temporarily invisible until a stronger model comes along which can detect them.
[1] see for example the "Decoherence Trap" article discussed here yesterday: https://maevyn.codes/decoherence
> There are plenty of examples of bugs in formally verified software.
Of course. Which is why humans will always need to be involved in the software creation process. We have to specify what we want. It's true that we ourselves often don't know what we want, in fact this is a big part of what software development is about: requirements gathering and continuously recondisering and reworking them. But it would be beyond naive to imagine that software might one day know better than us humans what it is we actually want, no?
That just doesn't seem like a reasonable possibility to me, given what we're seeing from and learning about these models. They have no capacity to be aware and adhere to an overarching vision, which is why they inevitably corrode architectural coherence over time [1]. The only reason these agents are able to still create usable code is because harnesses take advantage of model's ability to find bugs, giving us the famous loop, adding duct tape on top of duct tape, ratcheting up the complexity, until the adversarial model isn't able to find any more issues. Thus, the bugs are pushed out into higher complexity dimensions, temporarily invisible until a stronger model comes along which can detect them.
[1] see for example the "Decoherence Trap" article discussed here yesterday: https://maevyn.codes/decoherence
> There are plenty of examples of bugs in formally verified software.
Of course. Which is why humans will always need to be involved in the software creation process. We have to specify what we want. It's true that we ourselves often don't know what we want, in fact this is a big part of what software development is about: requirements gathering and continuously recondisering and reworking them. But it would be beyond naive to imagine that software might one day know better than us humans what it is we actually want, no?