>
I do not fully understand where the disconnect comes from. I do believe people that say that they get crap from LLMs, and I suspect I can guess some of the reasons for that, but I cannot be sure.Same here. There is an introspective comment below by khazhoux asking the question from the opposite experience. Why do developers end up on different sides of this coin?
LLM capability is obviously important variable. But those with bad experiences report using top tier models, so that isn't it. Likewise, level of experience can be ruled out. Each side of the coin have experienced developers reporting.
A lot of the other variables are not reported but I can go from my own experience.
Greenfield vs brownfield? The first is indeed usually easier but I get great results updating large code bases with decades of development. And I suspect developing greenfield with LLM is easier as the code is better written from the start.
Rarity of the subject matter? In principle, yet another XYZ app or web page is well represented in the training data so the LLM should have an easy go at making similar. In the past with low-end LLMs, this was noticeable. Working in relatively obscure programming languages produced just garbage. But, that was prior to the inflection point. And today I get good results working on relatively arcane scientific/mathematical algorithm as I do when I ask the LLM to barf out some web app.
Different levels of testing and validation and expectation? Even best LLMs make mistakes. The better LLMs make fewer and expose fewer still to the human as they create internal tests. Maybe people have different expectations and those with bad experience expect perfection from a single prompt? Perhaps, but any experienced developer knows bugs are a part of software development.
What else?
Whatever the answer, it's not a simple one and it may be a confluence of many smaller things. Maybe different developers have a multi-variant function in their brain that pops out a different quality-of-software score and also a different threshold that divides good from bad and all we see in their reports is that binary conclusion.
Same here. There is an introspective comment below by khazhoux asking the question from the opposite experience. Why do developers end up on different sides of this coin?
LLM capability is obviously important variable. But those with bad experiences report using top tier models, so that isn't it. Likewise, level of experience can be ruled out. Each side of the coin have experienced developers reporting.
A lot of the other variables are not reported but I can go from my own experience.
Greenfield vs brownfield? The first is indeed usually easier but I get great results updating large code bases with decades of development. And I suspect developing greenfield with LLM is easier as the code is better written from the start.
Rarity of the subject matter? In principle, yet another XYZ app or web page is well represented in the training data so the LLM should have an easy go at making similar. In the past with low-end LLMs, this was noticeable. Working in relatively obscure programming languages produced just garbage. But, that was prior to the inflection point. And today I get good results working on relatively arcane scientific/mathematical algorithm as I do when I ask the LLM to barf out some web app.
Different levels of testing and validation and expectation? Even best LLMs make mistakes. The better LLMs make fewer and expose fewer still to the human as they create internal tests. Maybe people have different expectations and those with bad experience expect perfection from a single prompt? Perhaps, but any experienced developer knows bugs are a part of software development.
What else?
Whatever the answer, it's not a simple one and it may be a confluence of many smaller things. Maybe different developers have a multi-variant function in their brain that pops out a different quality-of-software score and also a different threshold that divides good from bad and all we see in their reports is that binary conclusion.