yetanotherjosh
Joined 16 years ago
Joined 16 years ago
But, "coding" is, except that solving it means using techniques that are still barely understood and barely described today (something I'm hoping to be able to communicate better myself about). It is now possible (at least for most consumer software, I would not say applications where human life or extreme risk is involved) to work exclusively in the domain of natural language requirements, natural language architectural/design decisions, and natural language test cases, and deliver product of equal or better software quality than average human code authors could have produced. Even in a language that you plausibly don't actually know how to code in, because the language itself can often be separable from the requirements and verification process.
If someone who doesn't know a language or framework can plausibly produce better software using it, and faster, than a team of people who do know the language/framework (and I would strongly argue that is more than plausible now with models like Astra and Fable) then it's not just "coding is less expensive."
It's like saying computers make complex calculations less expensive. That's true, but it's missing the real paradigm shift.
Natural language test cases still define the expectations both at the product and architectural level and are essential for triangulating the agents on successful outcomes.
A requirements and verification approach with agents is not waterfall any more than TDD is waterfall. Does thinking ahead and doing some planning equal waterfall? Does describing how a feature works to an end user, and making some key technical decisions, before you build it, mean waterfall? Does having some sense of what you're building first mean waterfall? With agents, you can specify (with PRD and technical specs) what you THINK it should do, and in minutes or hours or at most days, have the result, which you then learn from and iterate. If you didn't fully think it through, the agents will do one of three things: 1) decide for you, which you learn from 2) stop and ask, which you learn from 3) introduce bugs, which you learn from.
It's extremely iterative.
Just to make this clear: if you can define a really good PRD and sophisticated technical specs, and a strong set of tests cases to pass, at the right level of architectural granularity, plus adversarial code review processes that triangulate and weed out most mistakes, SOTA agents can write the code autonomously, at or above the quality level of most human coding teams. I call that "solved" but only if you meet those context requirements. Which is still hard, not solved, at that layer.
Solving writing is not a good analogy IMO. Writing is for human consumption, and cannot be wrapped in objective requirements and verification processes. Certain forms of writing perhaps could be (can't think of one at the moment but I don't doubt some exist), and those forms might be good analogies for being "solvable" or "solved."
I need an explainer. Each of these products is carving out a particular niche, or competing directly for someone else's niche with better X or Y, and I'd love to see some analysis of the landscape.
1. Model routing. Do you regret giving a task to Fable when it could have literally been done by Sonnet?
This is the one thing that I can understand.
2. Targeted code + context search. We think embedding the whole repo is dumb. We also think sticking the whole context (or compressed context) in chat is dumb. So we do faster and better greps over both.
Is this just a different system prompt? Why would other harnesses not be able to do it this way given instructions to do so?
3. Aggressive context hygiene. Tool output is bounded, stale screenshots disappear, we don’t re-read files…the garbage never floods the model.
Existing harnesses already don't re-read files that have not changed. And content that is already in the context window is cached tokens and doesn't add significant latency, what am I missing? Removing context from history invalidates KV cache.
4. Efficient turns. Batch independent investigation, make one surgical edit, then perform one focused verification. Internal measurement showed 16% fewer round trips and 27% lower cost.
Isn't this just a specific sub-agent strategy, also addressable with prompting?
I'm genuinely confused about what is mechanically different in this harness that could not be accomplished with a prompt/skill in another harness.
Or is your definition of "open source" mean that a small indie dev should be able to reproduce the entire pipeline? Because that would disqualify more than just LLMs, but also hardware platforms like Arduino where you need to pay for manufacturing to get the underlying stuff built... is Arduino "open source?"
"The current behavior paper" -> The behavior in the running system that was previously described as papered over.
"Marker transport over-claim" -> The inaccurate review finding on the object's sentinel flag in the API response.
I suppose the cryptic/invented language problem is about token efficiency? But this sort of token efficiency is extremely difficult to deal with when it comes to conversation with a human about complex system. It might be efficient inside reasoning blocks, but when the model generates the final turn text, it should avoid this, as it's brutally inefficient due to the time spent wondering what each uniquely coined phrase means and having to ask for constant clarifications, which then you have to wait for another turn, eating up time and context while it burns more xhigh reasoning just thinking about how to explain its own awful language.
Codex is a harness, while GPT-5.5 is a model. The last codex-branded model was 5.3. Codex as a harness ships as a CLI, a desktop app, and a web product (and I'm not at all sure how similar the underlying harness is between them.)
Is the bug here supposed to be with the CLI harness, or the model? Does it also happen in pi, opencode, etc while running GPT-5.5?
To be fair I appreciate the effort of running and sharing the test. It will hopefully lead to better ones. But this is not a great test. Super interesting to think about what would constitute a better test.
For one, I think the agent would have to be expected to have productive interaction through the email channel, in a way the user depends on it generally working for some real world use case / value prop. In other words, needing emails to actually have the agent really do work, respond with results, etc. Also, most requests should be legit and the real attacks should be intelligently disguised, not pitiful/joke-level spam (although those would be arguably realistic to have in the stream, but, perhaps only as deflection so that the real attack is mischaracterized.)