bradleygibbs

Interesting repo!

No followers and 1 star, so, I'm assuming that you mean that process is what people are merging on, not necessarily this particular repo?

There are some similarities between this and what we've built (as described in the article I posted). One thing I noticed is that this workflow is based largely on agent skills, which are guidance/advisory. Sometimes agents follow them. Sometimes they don't.

We prefer deterministic runners for the process overall. We've also found that passing instructions to agents in their initial task context just prior to spawning them makes them much more likely to adhere to those instructions.

Deterministic everywhere possible. Firmly bounded everywhere else. The design conformance contract should be frozen before starting code generation and the evaluator must exist independent of the build loop.

Brad

Did I use AI to help write this article? Absolutely. I use AI 10-14 hours day, seven days a week to do all sorts of things, including trying to get a code generation and verification app off the ground. It would be unnatural for me not to use AI to research and help think through this article.

Do I know how to use Pangram, too? Absolutely. I got my Pangram merit badge months ago.

So, why didn't I make sure I got a 100% human-written score on Pangram before putting this out there?

I spent weeks reading what others had to say on these topics--Lilian Weng, Addy Osmani, Han Lee, Simon Willison, and a number of arxiv papers. I felt I had some things to add to the conversation. A slight twist here and there, maybe some more detailed information about how to implement these ideas than I'd seen elsewhere. For example, I see people writing about the model eating the harness, but very few people countering with "but AI failure modes".

I feel like there are a lot of holes and gaps in what I've been able to piece together from others' writings, largely because each article is only covering a portion of the entire process. The reader is left to fill in the blanks and connect the processes.

I have firsthand experience filling those gaps and holes I wanted to share. I felt this needed to be a single article, against the advice of multiple AI writing partners, because that's where the devil lives, in the details. Without a single piece covering the process from end-to-end, the reader is left to fill in the gaps themselves by triangulating multiple excellent articles. I've done that work. I wanted to share what I've found with others to try to start meaningful conversations about best practices for code generation, code verification, and learning loops that can optimize the entire process.

So, call it slop because Pangram told you you could. But I spent ~20 hours just researching and writing the article, not including the 100's of hours I spent writing code generation and verification pipelines.

I remain eager to engage with anyone on the contents of the article.

Brad

Author here. This is a detailed (8,000+ word, plus 2 companion pieces) walk-through of building loops for code generation and verification, complete with artifacts and detailed explanations to allow you to implement any of these techniques.

We walk through building: - an evaluation contract, evaluator and stopping conditions to minimize bugs - establishing an environment and priming AI to produce production-quality tests and code - detecting and correcting common AI failure modes - building a learning loop to monitor and improve all phases.

I'd love to get your feedback!

Brad