mrothroc

Joined 8 months ago

Karma
195
Submissions
58
Member since
Mar 3
Decades of building event-driven distributed systems. Currently researching AI agent pipeline reliability (Trust Topology).

https://michael.roth.rocks github.com/mrothroc

View on Hacker News

Recent submissions

Comment · 3 days ago
I've been doing multi-agent development for a while now and I've been very happy with the outcome. I disagree with the approach of creating a fleet of "persona" agents and just letting them rip. Instead, I use a standard SDLC pipeline with stages that produce artifacts and gates that evaluate them.

The gates have both deterministic and stochastic (LLM as a judge) components. The blend depends on the artifact: prose leans heavily on LLMs, where code has a lot of tests.

I work by having a discussion with claude code about what I'm trying to do and how it might be built. I ask it how its proposal fits into the existing bounded contexts and if it is reusing the existing patterns and libraries. Once it can give me an approach I like, I have it create a plan. This uses a skill and it has guidance on the structure and contents of the plan.

It is mostly prose, so the deterministic checks just look for the sections in the correct order. The LLM reviewer is a different model entirely to avoid bias and it is agentic, so it can (and does) explore the code and is prompted to verify the plan is correct both in the abstract and in the context of the code.

Once approved, it moves to breaking the plan into one or "releases" which are batches of tasks. Another gate, then to coding, which is where it fans out to multiple implementing agents. Another gate.

As I said, very standard SDLC. I get asked this question a lot, so I wrote it up with a guide on how to do it on the second to last page. https://michael.roth.rocks/research/543-hours

Comment · 5 days ago
That approval thing is interesting, for sure. It reminds me of how experts are very deliberate in narrating their thinking for juniors while they work, so juniors can learn the why behind the how.

There are definitely companies that are being short-sighted on this. I've seen this come back and bite them (usually, not always).

I'd argue that if the company doesn't provide this, then people should rely on their community. I'm lucky enough to live in a place that has lots of meetups and we talk about this. Absent that, there are plenty of active discussions online, as this very thread shows.

The bonus of working via the community is that's where you can find a new opportunity if the company doesn't take care of their people.

Comment · 5 days ago
"junior developers didn't have to develop the skill to understand assembly" well that's certainly fair.

At the same time, what I saw during that time was that organizations that actively sought to help juniors transition their skills to match the tools did better than those who just left everyone to wander around in the dark.

Comment · 5 days ago
How do I personally have the skill? I built it the old fashioned way: decades of doing this, starting with vi and hand coding assembly, through IDEs and high level languages, and now with distributed tracing on ephemeral serverless deployments.

How do juniors develop it? Yeah, that's the tough part. Those years spent slogging through very tedious work got me here. You have to exercise to build muscle. Maybe we need to find some way to let people exercise the skills without the hands-on slog? I'm honestly not sure, how are you seeing it handled where you are?

Comment · 5 days ago
Sure! Make sequence diagrams, or get the agents to do it as part of what they deliver. That's always been very helpful for me to develop a "boxes and lines" view. The boxes are right there across the top and the lines go between them!
Comment · 5 days ago
I agree there are differences in the teams, but it sure feels like there is a lot of overlap in the skill I need to deliver software, regardless of who is creating the code.

You raise another point though: agents are not human. The things that drive agentic loops are not at all the things that motivate humans to do good work.

That's probably where the human-management skill ends in this new world. The techniques that get high quality from humans do not apply to agents.

Comment · 5 days ago
This matches my own experience, pre-AI. As I've moved from IC to having a team over the years, more code was being written than I could possibly understand at the depth I would have if I'd written it myself.

The skill I learned is exactly as you describe: understand the system. For me, this means "boxes and lines", the components and how they interact. I've encouraged developers on my team to make sequence diagrams part of their deliverable, because it helps both of us put the code in the context of the system.

The boxes can be opaque because I know the boundaries and expected inputs and outputs. This tells me when I need to go in a box: is it sending a malformed message? I can trace that back into the code.

I think that's the skill we have to cultivate now that coding agents produce far more than we can review line-by-line. The system view is the map, and the boundaries tell us where to go hunting inside the box.

Comment · 20 days ago
Fred Brooks famously advised us to plan to throw one away. That was in the days of waterfall engineering.

The reason he said that is because the best way to learn how to properly build a complex system is by going through the process of building it. Sure, implementation is a bottleneck, but the real problem is that the experience of development is the best teacher.

Personally, I try to bring this lesson into my agentic coding practice by doing spikes and trial implementations first. This is where the AIs really deliver value: they can quickly build something out that touches a lot of the architecture so both it and I can get the proper shape of it. Only after that do I unleash it, constrained by what we learned.

Comment · 2 months ago
Several comments here touch on the core problem: agents are writing more code than we can review. Seniors have never had enough time to review, and the prolific output from coding agents is making it worse. Moreover, the code is almost always good. So you're reviewing a tsunami of pretty good code, which means you get review fatigue and the whole thing just becomes theater.

For me, this means the checks have to be more than just "I looked at it". There are two things that happen before I ever see it: first, as others have mentioned, I have a model from a different family review it with well-defined criteria. Same-family reviewers share bias, so it must be a different one. Second, I have a core set of deterministic gates (like lint, but also unit tests) that must run. In either case, failures go back to the coding agent.

And only then do I bother. But I really don't read everything. If it is bog-standard CRUD operations, the agents are pretty good at that, especially if they are using mature packages. I focus on critical things, like how it enforces permissions.

This works well for me, though it leaves one major issue untouched: whether this is worth building. The agents tend to be a bit overeager, so I have to do a lot of work up front to make sure the output will add value. The gates can only check the artifact, not my intent.

Comment · 3 months ago
Relevant passage from Jarred's post:

"At the time of writing, about 4% of Bun's Rust code sits inside an unsafe block (~13,000 unsafe keywords across ~27,000 lines / ~780,000 lines), and 78% of those blocks are a single line — a pointer that came from C++, or one call into a C library. I expect this number to go down over time as we refactor from a faithful Zig port (which had no greppable unsafe keyword) to idiomatic Rust, but we are going to continue using C & C++ libraries like JavaScriptCore so it will always have more unsafe than pure Rust projects."

Comment · 3 months ago
Comment · 3 months ago
Agents are fast and powerful, and that is a double-edged sword. The volume you can produce is breathtaking, but they make far too much for you to review every line of output. Personally, I focus my limited attention on the places that give the biggest levers and put deterministic guards throughout.

First, recognize that most of what you want done is actually a sequence of steps, and those steps produce intermediate artifacts. And you can either review or gate artifacts as they flow through the process. The earlier you look, the more leverage you have.

I make software, which follows these steps: make a plan with tasks, make a design for each task, code each task, run the tests. I spend a lot of time on the plan, because mistakes here expand drastically once they hit code.

But it's not just all on me: I write scripts and use tools to make guarantees about the artifacts. For a plan, for example, I have a script that makes sure it has the correct structure. I also use another agent to review it according to my personal specification.

Every artifact has this, because the reliability of the final product of the pipeline is actually composed from all those checks.

Comment · 3 months ago
You have to have a multi-layered approach. This is the "Swiss cheese" model of prevention: individual layers may have holes, but if none of the holes in the stack line up, nothing makes it all the way through.

I approach this by thinking of verification surfaces. Each one is a layer.

For SQL, I have a tool that the agent can call to run queries. (I actually keep a range of tools: for known queries the SQL is baked into the tool and only results come back; I also have a general SQL tool. Prefer the former; deploy the latter only if you must.)

The general SQL tool itself is multi-layered. The raw SQL produced by the agent is a verification surface, as are the parameters used in the tool call.

There are plenty of existing utilities that validate SQL and define its claims, scope, and if it mutates in a structured way. My tool runs those against the SQL as a verification surface and it hard gates some requests.

It also verifies the parameters. It does this in two ways: at a shallow level, it just makes sure they are valid and sane, so it can fail fast if the LLM does the wrong thing.

But it also does a deeper check, where it calls out to supporting microservices to verify a) permissions embedded in the request and b) consistency of the request with the rest of the system.

Since you asked about challenges in prod: if you are not careful, the LLM can write wildly inefficient queries. Some of those made it through and I had to go back and tweak the prompts to give enough context to have it tune properly.

My suggestion: think through the verification surfaces, figure out what you can verify deterministically, and use that as gates in your tool.

I actually did a little writeup of the "verification surface" here, seems relevant: https://michael.roth.rocks/blog/verification-surface/

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton