> I will just point out the benefit is not as obvious as you think. Developers have consistently overestimated LLM
I think there are two different claims here:
- developers overestimate productivity gains, which is a solid finding in many of these studies. Skepticism of extremely large productivity gains is warranted and I flatly disbelieve "10x uplift" claims.
- LLMs give no productivity uplift at all, which is much harder to defend. A repeat of the famous METR RCT study did find evidence of improved productivity, and this seems to align with the experience of many experts I trust.
I think there are two different claims here:
- developers overestimate productivity gains, which is a solid finding in many of these studies. Skepticism of extremely large productivity gains is warranted and I flatly disbelieve "10x uplift" claims.
- LLMs give no productivity uplift at all, which is much harder to defend. A repeat of the famous METR RCT study did find evidence of improved productivity, and this seems to align with the experience of many experts I trust.