To me this sounds like an old cobbler complaining that machines aren't producing good shoes if left unsupervised and that the old process of making shoes completely by hand is far superior.
So what he is telling us? That agents are not infaillable and they are not capable to one shot complex software and they do not produce perfect code?
We know what and the solution is to use agents for what they are good at and work around their limitations and we have a human in the loop.
>not some RLVR shit that comments out the failing test and tells you all the tests are now passing
That's what harnesses should be about: detect when the agent is misbehaving and force it to take the right approach.
This example in particular should be easy to solve if we generated the tests before coding and we have a workflow or state machine that doesn't allow the agent to disable tests and doesn't allow it to reach the next stage unless all tests are passing.
So what he is telling us? That agents are not infaillable and they are not capable to one shot complex software and they do not produce perfect code?
We know what and the solution is to use agents for what they are good at and work around their limitations and we have a human in the loop.
>not some RLVR shit that comments out the failing test and tells you all the tests are now passing
That's what harnesses should be about: detect when the agent is misbehaving and force it to take the right approach.
This example in particular should be easy to solve if we generated the tests before coding and we have a workflow or state machine that doesn't allow the agent to disable tests and doesn't allow it to reach the next stage unless all tests are passing.