I've developed a new mental model of the LLM codebase automation solutions. These are effectively identical to outsourcing your product to someone like Infosys. From an information theory perspective, you need to communicate approximately the same amount of things in either case.
Tweaking claude.md files until the desired result is achieved is similar to a back and forth email chain with the contractor. The difference being that the contractor can be held accountable in our human legal system and can be made to follow their "prompt" very strictly. The LLM has its own advantages, but they seem to be a subset since the human contractor can also utilize an LLM.
Those who get a lot of uplift out of the models are almost certainly using them in a cybernetic manner wherein the model is an integral part of an expert's thinking loop regarding the program/problem. Defining a pile of policies and having the LLM apply them to a codebase automatically is a significantly less impactful use of the technology than having a skilled human developer leverage it for immediate questions and code snippets as part of their normal iterative development flow.
If you've got so much code that you need to automate eyeballs over it, you are probably in a death spiral already. The LLM doesn't care about the terrain warnings. It can't "pull up".
Tweaking claude.md files until the desired result is achieved is similar to a back and forth email chain with the contractor. The difference being that the contractor can be held accountable in our human legal system and can be made to follow their "prompt" very strictly. The LLM has its own advantages, but they seem to be a subset since the human contractor can also utilize an LLM.
Those who get a lot of uplift out of the models are almost certainly using them in a cybernetic manner wherein the model is an integral part of an expert's thinking loop regarding the program/problem. Defining a pile of policies and having the LLM apply them to a codebase automatically is a significantly less impactful use of the technology than having a skilled human developer leverage it for immediate questions and code snippets as part of their normal iterative development flow.
If you've got so much code that you need to automate eyeballs over it, you are probably in a death spiral already. The LLM doesn't care about the terrain warnings. It can't "pull up".