It shows a remarkably consistent curve for AI completing increasingly difficult coding tasks over time. In fact, the curve is exponential, where the X axis is time and the Y axis is task difficulty as measured by how long a human would take to perform the task. The current value for 80% success rate is only 45 minutes, but if it continues to follow the exponential curve, it will only take 3 years and change to get to a full 40 hour human work week's worth of work. The 50% success rate graph is also interesting, as it's similarly exponential and is currently at 6 hours.
Of course, progress could fall off as LLMs hit various scaling limits or as the nature of the difficulty changes. But I for one predicted that progress would fall off before, and was wrong. (And there is nothing saying that progress can't speed up.)
On the other hand, I do find it a little suspicious that so many eggs are in the one basket of METR, prediction-wise.
https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com...
It shows a remarkably consistent curve for AI completing increasingly difficult coding tasks over time. In fact, the curve is exponential, where the X axis is time and the Y axis is task difficulty as measured by how long a human would take to perform the task. The current value for 80% success rate is only 45 minutes, but if it continues to follow the exponential curve, it will only take 3 years and change to get to a full 40 hour human work week's worth of work. The 50% success rate graph is also interesting, as it's similarly exponential and is currently at 6 hours.
Of course, progress could fall off as LLMs hit various scaling limits or as the nature of the difficulty changes. But I for one predicted that progress would fall off before, and was wrong. (And there is nothing saying that progress can't speed up.)
On the other hand, I do find it a little suspicious that so many eggs are in the one basket of METR, prediction-wise.