> Prove it.
Anyone can make up a anecdote. In my case an agent was able to do a task in 10 min vs almost an hour without documentation. Citing an anecdote as “proof” isn’t good enough.
Here is my proof:
Hai, N. L., Nguyen, D. M., & Bui, N. D. Q. (2024). On the Impacts of Contexts on Repository-Level Code Generation. arXiv. https://doi.org/10.48550/arxiv.2406.11927
Cited by: 46
Jimenez, C. E., Yang, J., Wettig, A., et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv. https://doi.org/10.48550/arxiv.2310.06770
Cited by: 4998
Zhang, F., Chen, B., Zhang, Y., et al. (2023). RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/2023.emnlp-main.151
Cited by: 698
> The code is the documentation.
No because it doesn’t capture the intent or the rationale why something was designed a certain way. Like why does it do some weird logic in this spot? What and why is being tested (a lot of tests are not straightforward)?
> Your two statements are contradictory here. Docs should be good enough so the llm doesn't need to read the codebase, but it should just summarize everything? You can't have both.
They are not contradictory. There’s this thing called a table of contents that summarizes large sections into a sentence or less, and links to said sections. Each section both large and small can also have a little paragraph that serves as a summary for the content to come. Sections can even have their own mini TOC. This enables an AI to be able to skip the parts it doesn’t need.
> All the LLM generated docs are causing friction
Ok, if you have a LLM generate your docs and you don’t vet it and make it your own, it doesn’t help you. Because again, how can you prove that you actually know the code base? Unless all you do is work on the same part of the same application day in and day out? Maybe the programs you’re working on is small or simple? How can you ever quickly ramp up when changing projects? This is assuming that you’re not just pairing vibe coding with manual monkey testing, or are you even validating anything when you’re not recording your goals in writing? Most of the cost in application development lies in maintenance and that is also where documentation benefits the most
Anyone can make up a anecdote. In my case an agent was able to do a task in 10 min vs almost an hour without documentation. Citing an anecdote as “proof” isn’t good enough.
Here is my proof:
Hai, N. L., Nguyen, D. M., & Bui, N. D. Q. (2024). On the Impacts of Contexts on Repository-Level Code Generation. arXiv. https://doi.org/10.48550/arxiv.2406.11927 Cited by: 46
Jimenez, C. E., Yang, J., Wettig, A., et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv. https://doi.org/10.48550/arxiv.2310.06770 Cited by: 4998
Zhang, F., Chen, B., Zhang, Y., et al. (2023). RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/2023.emnlp-main.151 Cited by: 698
> The code is the documentation.
No because it doesn’t capture the intent or the rationale why something was designed a certain way. Like why does it do some weird logic in this spot? What and why is being tested (a lot of tests are not straightforward)?
> Your two statements are contradictory here. Docs should be good enough so the llm doesn't need to read the codebase, but it should just summarize everything? You can't have both.
They are not contradictory. There’s this thing called a table of contents that summarizes large sections into a sentence or less, and links to said sections. Each section both large and small can also have a little paragraph that serves as a summary for the content to come. Sections can even have their own mini TOC. This enables an AI to be able to skip the parts it doesn’t need.
> All the LLM generated docs are causing friction
Ok, if you have a LLM generate your docs and you don’t vet it and make it your own, it doesn’t help you. Because again, how can you prove that you actually know the code base? Unless all you do is work on the same part of the same application day in and day out? Maybe the programs you’re working on is small or simple? How can you ever quickly ramp up when changing projects? This is assuming that you’re not just pairing vibe coding with manual monkey testing, or are you even validating anything when you’re not recording your goals in writing? Most of the cost in application development lies in maintenance and that is also where documentation benefits the most