engineercodex

We all know this isn’t replacing doctors. We do know that when applied correctly, this will assist doctors and other clinical specialists in their work. With doctor burnout at all time highs, I think stuff like this is amazing.

I could see that, and I wouldn't want LLMs just generating tests willy-nilly with no human oversight. I don't have any trust in the reasoning ability of LLMs at all.

Rather, I prefer to view LLMs as a junior dev that submits PRs that pass presubmits and other verifiable, programmatic checks. A human dev then reviews the PR manually. In this case, the LLM + its processing is used to make sure that no BS is sent out of review - only potential improvements.

You're right. I've edited my wording to be more realistic about the value of the test. I believe you're right that the test is not an outlier in terms of value provided.

Some of my comments within the article are more aspirational than realistic in this case, and I've made edits to reflect that.

I want to clarify that I view this LLM as a junior dev that submits PRs that pass presubmits and other verifiable, programmatic checks. A human dev then reviews the PR manually. In this case, the LLM + its processing is used to make sure that no BS is sent out of review - only potential improvements.

In no scenario should it's auto-generated code be auto-submitted into the codebase. That becomes a nightmare really fast.

Good point! I used exponential to emphasize the nature of the value of a certain test compared to the rest of the generated tests, but you're right it's not the right word. I updated the article to remove the usage. :)

Quality has to be backed my metrics or else it will never happen.

PS. Big fan of this person’s recent blog posts. He’s been on a roll!

I inserted these because I personally like reading related discussions and articles of topics at hand. Not sure how this is a negative :/

Ouch - half of the article (the "Actionable Takeaways" section) was my own commentary. The summary was for those who didn't want to necessarily parse through the entire paper's PDF.

Happy to listen to any constructive feedback if you have any, though!

Hey! Thanks for your comment - I'm the one who wrote this article. I wasn't trying to say that the paper authors talked about "unexpected edge cases" or "thinking outside the box." I edited the post to be more clear that some of these takeaways are my own opinions.

This article is less of a summary of a paper and rather commentary on what the results of the paper entails. After all, Hacker News is meant for discussion :)

I will say though that I do believe that I still stand by the "exponentially more valuable" portion. I think the fact that LLMs can fluke their way into "hitting a jackpot" in terms of test coverage is exactly why they're so valuable. When you have something constantly trying out different combinations, if it hits even one jackpot, like in the paper, it's extremely valuable to the team. It's a case that could have been either non-obvious or simply too tedious to write a test for manually. I think there's tremendous value in that, especially speaking as someone who has spend way too much time simply figuring out how to test something within a Big Tech codebase (F/G) when I already knew what to test.

This makes so much more sense now. Thanks for investigating

I don’t really understand this comment.

I didn’t “slap” any date on top of the article. 2 Oct 2023 is the date I published my article. (This is how blogs on the Internet work.)

The first line of the article says that this all happened in 2012. I’m not sure I could get any clearer there.

The “GPT distillation” phrase is pretty rude. Anyone reading the article can see the difference between it and the presentation. Every “hand-drawn” image in the article is created by me in Excalidraw, and any images from the presentation are sourced with a very clear link to the presentation.

If you have any constructive feedback, I am happy to listen and take it into account.

Otherwise, I’m not really a fan of the rudeness. Thanks.

Correct. This is a retrospective of an artifact from the early 2010s. The architecture is much different now.

I’m the writer of this. I linked to this presentation very clearly throughout the article as a source.

This was an article meant to resurface learnings from something that happened a decade ago, with added images and a clear distillation so that somebody doesn’t have to watch the 45 minute presentation to understand what happened.

I’m sorry you didn’t like it.

But I hope you can see the value I’m trying to provide here, as someone myself who doesn’t really have the time to sit through hour-long presentations to learn something.

Try brilliant.org or swequiz.com

Watch some YouTube videos or read some CS articles.

If you have a project idea, break down the next steps into tiny actionable tasks and draw out how you would implement certain features.