iamwil

Depends on the code. For things I've already written in specific domains on specific stacks, writing by hand is moderately quicker. I'd never been a faster coder--more of a well-designed and architecturally sound coder.

But the scope of the stuff I'm building now is beyond what I've done before, so yes, it is better and faster for the agent to do it. Software with agents is an industrial process now, rather than an artisanal one, so it's a better fit for some kinds of things than others.

Peter Naur in 1985 wrote that programming is the act of building a working theory of the system in our minds, and this often takes form as tacit knowledge in the head of programmers. Even if you handed the complete codebase and perfect documentation to a completely new team, they would often need to go back to the old team and ask questions.

And before agents, we often built this working theory of the system by writing the code by hand. It helps us figure out the "grain" of the codebase so we have a better idea of how to extend it.

So what happens when agents are writing the code? The main avenue with which we built the working theory of the system in our mind is now taken away, and we're left feeling like our understanding of the system is blinded.

I ended up more or less changing my entire workflow. Four main principles:

1. Interrogation/interview-based design and specification

2. Make the decisions that matter and leave the rest to the agent.

3. Use agent to generate explainers, quizzes, and simulations.

4. Use verifiers and guardrails in the coding loop.

Interview-based design and spec

When I tell the agent what I want to build or extend, there's naturally ambiguity. But the agent knows what is ambiguous to it, so it can ask me. /grill-me is one way to do this. You'd think this would be commonplace now, but it's actually not. Some people are still wondering why they're getting bad results. It's because they're not resolving the ambiguity.

Making the decisions that matter

Not every decision matters in a product. Agents are fast at writing code and pretty good at some of the nitty gritty details, so I let it make those decisions. Also, if a particular subsystem is commonplace and undifferentiating, such as login, then I also let it make architectural decisions for that subsystem.

For decisions that are product differentiators, where that differentiation is supported by technology, I make decisions on the stack, the architecture, and the approach. This is the design phase. Because agents tend to make the mainstream decisions, they almost always take a midwit approach which doesn't leverage the technology in a way that makes a difference for the customer.

Some try to build this understanding from code reviews. I personally don't think it's the way to go, because it's not motivating. I find that I'm more motivated to learn how the system works today, because I want to change how it works tomorrow--and it sticks in my head better too (no different than pre-agents).

Generating explainers, quizzes, and simulations

As I'm doing design for an improvement and I don't know how a part of the system works, I can ask questions of the agent in chat. If I still don't get it, I can ask for diagrams (either ASCII or mermaid). But if it's an entire subsystem, I have a skill in which I can ask for visual and interactive explainers. It'll generate a markdown page (or html) that has these interactive components that help demonstrate a concept.

To help make the concept stick, there are quizzes after each section and at the end.

With the computational simulations, I generate a throwaway tool in which I can explore the state space. This doesn't happen as often, as I try to keep state spaces small, and for those explainers tend to work well.

With the design simulations, a same-model agent and a cross-model agent and I will talk about an explainer, and we'll name things that we understand or notice about the design and the intended direction we want to extend it. Then, we'll use subagents to generate options and then discuss the pros and cons. I find this is a good way for me to understand the dead-ends of a design.

Using Verifiers and Guardrails

I define guardrails as mechanistic checks that doesn't change with the feature. such as linters, typecheckers, and pre-commit hooks are all guardrails. Verifiers are mechanistic checks that will be different with each feature, such as unit/integration tests.

For more complicated states or concurrent parts, I'll ask the agent to use formal methods to write a proof. Then use the proof to generate a trace (a series of steps that demonstrate a happy or unhappy path). Then I run that trace in unit tests (kinda like property-based testing--so I guess this can be called trace-based testing), and the code has to adhere to the trace.

I also wrote a design verifier, where the requirements in the spec can be converted into a series of mechanically checkable rules. Then during testing, the verifier fires up the headless browser and uses the DOM measurements of the page to see if it matches up to the rules. This was completely impractical before agents.

All of this so a) I can be sure that the agent is doing what it's supposed to with the decisions that it's making b) I can be out of the coding loop, so that I'm not up at 2am begging, "you fixed this, but you broke that. pls fix." The agent can check for itself.

---

If any of this sounds helpful, lemme know. I've been wondering if other engineers care about understanding the code that's being written, or if people just don't care anymore. I've spoken to a few, and it seems to be split. And it may just be a matter of degree and detail. Some like it at a higher level. Some like it nitty gritty.

But then, how do you keep the theory in your own mind enough to steer the agents to extend the program? I've been asking around, and different people seem to have different ways, colored by the way they work.

How do you coordinate their work? Do you just have one orchestrator/mayor that you talk to, and it coordinates the rest, and the rest talk amongst themselves? How do you ensure they're doing the right thing or efficiently?

How, if at all, do you keep the architecture or a working theory of the code in your head?

This is kind of a cool concept. It's like Space Golfing.

Another alternative is to make it like a turn-based Scorched Earth (Mother of all games), where you can either

1. launch asteroid size projectiles across star systems in a field of stars and black holes to it each other.

2. launch projectiles that then place black holes (perhaps either by timeout or signal--taking into account the speed of light!) to change the field.

It's a good start!

Hrm, I don't think so with a cursory search on NewGrounds. I can't remember the name.

How often you have to take algorithmic complexity into account depends on what level of the stack you're at, but even at the application level, I think understanding algorithmic complexity is still important.

I was working at Company X around 2013, and we had a Rails ecommerce site. Trivial in the algorithmic sense. Had a contractor that pushed a change that somehow made everything crawl. When we did profiling together, it turned out he had written an O(n^2) algorithm when looking up country codes or something. Sometimes, there's no off-the-shelf library, and it's the minimum a dev should know not to do.

I'm not saying it should be a maniacal focus, but it should be taken into account. Of course, it's with judgement. We get shittier and shittier software if it's not taken into account at all.

I remember playing a crap web game in college, kinda like gunslinger, where you try to shoot people who turn around with a gun (and not those without). When I graduated college, Ian--one of my good friends at work--ended up being one of those guys in the game! Turns out that game I was playing was built by Tom Fulp, and he was using his friends (like Ian) as characters in his games pre-Newgrounds. All that messing around with crap games on the internet paid off. He had started Newgrounds while in college, and had quit to do it.

Met Tom once, and he was a cool guy, but he didn't believe in the stuff that they taught in CS. He had a long rant about how learning string diff algorithms were useless. Anyway, it turns out later on, he went back to college to finish his degree. When his classmate(s) found out who he was, this one kid was so excited to sit next to him. "Do you know who he is?!" he would exclaim to his professor, to Tom's embarrassment. I wonder if he had to do string diff algos when he went back to finish college.

Working on an interactive explainer for code. When agents write most of the code, I find I'm losing the working theory of the code/architecture. Code review had typically served a couple of purposes, two of which are catching errors and disseminating codebase knowledge amongst the team. I find that adversarial agent reviews are good at catching errors, and that I prefer to learn about the architecture before/while I'm designing the next feature, rather than after.

I've also been designing a APL/BQN-like language that uses Korean as a way to compact and compose the meanings of the operations.

Plato thought writing was a lie:

> If men learn this, it will implant forgetfulness in their souls; they will cease to exercise memory because they rely on that which is written...And it is no true wisdom that you offer your disciples, but only its semblance, for by telling them of many things without teaching them you will make them seem to know much, while for the most part they know nothing, and as men filled, not with wisdom, but with the conceit of wisdom, they will be a burden to their fellows.

Basically, reading writing (without instruction) will only make people seem like they know stuff, by parroting others, when in fact, they know not much of anything.

Any specific Sony bots you're recommend looking at and studying?