nseskin

I’m curious how you handle design docs when requirements change significantly during implementation. At some point, updates and corrections can start describing a somewhat different project from the one that originally went through review.

Is there a rule of thumb for when it’s enough to update the existing document, and when the design has changed enough to justify a new design doc and another review?

Great work! I really like projects that have such a clearly defined scope and stay completely focused on it.

What happens when AWS interrupts a Spot instance? I don't see an ASG in the architecture — does the controller detect the terminated worker and provision a replacement automatically?

I think this article demonstrates a broader problem with constraint following in LLMs. Agents often satisfy the most obvious and easily verifiable part of a task, but lose track of the constraint that actually determines success.

Testing makes this especially clear: “use fuzzing” turns into generating random bytes, and “use formal verification” into proving a property that isn’t particularly useful. Formally, the technique is applied, but its actual purpose is lost.

This may also explain why different testing techniques produce similar results: the problem may be less about knowing a particular technique and more about the agent’s ability to keep its actual goal as the primary constraint.

Are there currently any per-app CPU or memory limits? If not, could an app with a memory leak or high CPU usage affect the entire server and other applications? Are there any plans to enforce resource limits using Podman/cgroups?