There's been a huge amount of improvement in coding agent effectiveness since they ran that experiment. In a more recent follow up experiment, METR found 20% speed up from AI assistance and says they believe that is likely an underestimate of the impact. https://metr.org/blog/2026-02-24-uplift-update/
They are working on making a new measurement approach that will be more accurate.
They are working on making a new measurement approach that will be more accurate.