The effectiveness of a Copilot-like tool trialed at FB showed that 8% of code contributed by participants was sourced from suggestions, but the latter study made no promise about coding velocity: https://arxiv.org/abs/2305.12050. In my own experience the time taken to review machine-generated suggestions often eats into developer time.
This is not a critique of LLMs in general — I’ve found ChatGPT really great for kicking off greenfield projects in well-known languages and frameworks.
The effectiveness of a Copilot-like tool trialed at FB showed that 8% of code contributed by participants was sourced from suggestions, but the latter study made no promise about coding velocity: https://arxiv.org/abs/2305.12050. In my own experience the time taken to review machine-generated suggestions often eats into developer time.
This is not a critique of LLMs in general — I’ve found ChatGPT really great for kicking off greenfield projects in well-known languages and frameworks.