magnio
Joined 6 years ago
Joined 6 years ago
What a clusterfuck
The first tool execution runtime in harnesses are direct tool calls with JSON or XML, such as the Read and Edit tools. As an escape hatch, we have Bash tool that allows arbitrary code execution on the host running the agent. The downsides of using bash (on the host) as the main tool execution runtime are:
- Syntax and obvious errors only surface at runtime
- Unergonomic orchestration of parallel and background tasks
- Verbose command output cluttering context
- Dependent on the host environment, packages versions, etc.
- No security measures by default.
To me the last point is the biggest inherent weakness, usually mitigated by creating a dedicated unprivileged user or running bash in a sandbox.
Note that direct tool calling is kind of the polar opposite on these points: syntax errors are caught early, orchestration can be done with some wrapping tools, command output is controlled, and most importantly they are more sandboxed. On the flip side, they obviously have way less power, necessitating Bash tool in the first place.
Codemode is the middle ground between these two extremes. It actually can be derived simply by one idea: what if we replace Bash by another language that can be checked for obvious errors, i.e. type checked?
Everything else falls out from there:
- Any language would do, but I think TypeScript fits the balance between safety, speed, conciseness, and popularity in training data.
- If we use TypeScript, might as well run it in a sandbox as JS runtimes have been designed with this in mind for 20 years
- Orchestration comes for free from the JS runtime. It's not more powerful, just more ergonomic.
- Since the tools are controlled by the harness and not dependent on the host, cloud agent becomes easier.
- With this in place, MCP are not very different from a tool provided to this sandboxed runtime.
Overall I find the benefits compelling enough, but we'll see if the heavily-RLed models these days will use it effectively.
It's so funny and somber to see programmers having an existential crisis when they get a glimpse of what work is like for business managers, the demographics many programmers detest.
I am also guilty of holding the business majors in contempt back in college, and now here I am, doing what they are doing in office in a much more indifferent and unenjoyable manner. At least I don't get into trouble with HR from calling my agents a stupid fuck (yet).
What if we create a benchmark that works like this and assigns ELO scores? Models fight head-to-head by writing a question, a bug, or an incomplete implementation, which the opponent has to answer, fix, or finish.
(LLM writing rant below)
---
> That alone tells you something: Users had a real need, and the ecosystem filled the gap.
> This sounds straightforward, but it solves a real operational problem.
> None of these change the world. All of them make day-to-day data workflows better.
> The easy thing to do here is list planner changes and call it done. But the more useful takeaway is this: Postgres keeps getting better at recognizing the shape of common queries and doing less unnecessary work.
> [Proceed to list planner changes]
If Orwell were alive today, he might declare himself illiterate in English and learn Klingon just to avoid having to read these.
Yeah I sympathize with people saying software developers should not be called engineers too.