orisho
Joined 7 years ago
Joined 7 years ago
When a test case fails, the relevant part of the plan is surfaced in the error. I find this helps Claude stay on track for longer - I've been able to do 12h most times and even up to 48h unattended (11h of API time) with good results.
Then whenever I do check in, I ask it to update the HTML with current state in an append only fashion (sort of like it's writing a blog), and then based on that, we iterate on the end to end test (I think of it as a "test harness") - update the test cases and error messages.
I've been able to build some truly large projects this way, both greenfield and up to spec (for example, a video game I've always wanted to play), and brownfield while staying within the conventions and design of the codebase, and with very little attention required on my part.
I imagine this style of commenting is partially responsible for model improvement in SWE work, a trained trait through feedback, since it makes the comments actually useful. Not to you or me, but to an LLM.
Some of these errors are obvious to a human, that's why we find the comments unnecessary. But if everyone's using LLMs and said model tends to make this error, then they are very useful. It's like a comment you would leave for another human to help them avoid a footgun. It's just a LLM footgun, not a human one.
However, LLM generated code will often, at least in my experience, avoid raising any errors at all, in any case. This is undesirable, because some errors should result in a complete failure - for example, errors which are not transient or environment related but a bug. And in any case, a LLM will prefer turning these single file errors into warnings, though the way I see it, they are errors. They just don't need to abort the process, but errors nonetheless.
I live in such a country and have parents with older phones who can't use esims, so the value is obvious to me. :)
Another poster touched on another important point: it's important for this to be changeable independent of the code. The reason for this is actually kind of subtle. Obviously, you don't wanna have to need to rebuild in order to regenerate permissions. But the real reason, imo, is that it should be easy to parse for a human, easy to locate for a human, and also easy to parse and adjust for a machine, that might determine a permission is no longer necessary, or who is trying to build a dependency graph in order to determine who to wake up during an incident. That means it should go into configuration that is versioned and deployed alongside the code, but not in the code.
If you make this hard to understand and change, people will just copy it, and the you're back to square one. It's gotta be the easiest thing to do the right thing, because at scale, people are gonna do the easiest thing.
I feel like I'm kinda going on at length about this, so instead I'm gonna leave you with a link to a blog I wrote about the same concepts, if you wanna read more. It's about Kubernetes network policies, but really the same concepts apply to all kinds of access.
https://otterize.com/blog/network-policies-are-not-the-right...