rbehrends
Joined 15 years ago
Joined 15 years ago
Example problems are:
- https://hub.harborframework.com/tasks/terminal-bench-science...
- https://hub.harborframework.com/tasks/terminal-bench-science...
I have no doubt that these are very relevant task for people in those respective fields, but I suspect they have next to no relevance to some 99.9% of model users. Hence why especially smaller labs with smaller models didn't really train for such use cases and you can see it.
Personally, it's not something that matters at all to me, as I don't work in cheminformatics or bioinformatics.
This is also arguably more of a test of a model's knowledge base than reasoning ability.
But most importantly, there is also the question to which extent the aggregate score then remains a meaningful result when you mix up too many too disparate benchmarks.
The coding agents subindex, in contrast, looks at least superficially plausible [1].
[1] https://artificialanalysis.ai/agents/coding-agents#coding-ag...
These days, where an agent can just write you an AVL tree, red-black tree, or B-tree library in under a minute, it probably matters less.
That especially the small flash models that are primarily optimized for coding tasks would perform worse at non-coding tasks is not per se surprising to me. The introduction of Terminal-Bench Science 0.1 in particular hurts their aggregate scores [2].
The general question remains of how meaningful aggregating the results of very heterogeneous tasks into a single numeric score is. (Aside from general questions about individual benchmark validity.)
Note: the cyber index is also interesting. While refusals make it somewhat opaque, even if you optimistically count all refusals for Opus and Fable as successes, the results are ... fascinating [3].
I'll add that I really want to know how Opus 4.x and Codex 5.x perform on the new index. Once we have that information, this is going to give us a better understanding of how meaningful the scores are.
[1] https://artificialanalysis.ai/#coding-agents-tabs
[2] https://artificialanalysis.ai/evaluations/terminal-bench-sci...
The recent update to SlopCodeBench that includes Astra and Opus 5.5 (including the insight that higher reasoning levels may not improve the situation) seems to match my intuition. [1]
[1] https://github.com/humanlayer/advanced-context-engineering-f...
My usual workflow (not counting throwaway code) is to use agents as virtual pair programmers, which gives me a good balance of quality and velocity.
Each interaction with the agent remains individually comprehensible and makes progress towards one of the following goals:
- Understanding the problem/solution space.
- Implementing a solution.
- Understanding the solution, especially any agent-written parts and interaction with existing code.
- Evaluating the solution (tests, code analysis, what-if scenarios, consider edge cases).
This does not just involve the model generating code, but also explaining or analyzing it as needed.
This process allows me to retain a mental model in lockstep with developing the written code and have some assurance both of its functionality and non-functional requirements. It also fits in nicely with most existing team-level workflows.
But it's also something that I can do with pretty much any of the modern models.
First, it does not really reflect my experience. I find modern models to be largely interchangeable when it comes to common software engineering tasks, including some of the more recent open weight ones. The strongest models become interesting for Ph.D. level of work for me, which is not something I deal with regularly anymore.
None of them are perfect; they all have their strengths and shortcomings, but I can reasonably work with all of them and it honestly doesn't matter much anymore. I do sometimes wonder what exactly people are doing to see such a difference; at a minimum, their workflows must be very different from mine.
Second, you're arguing based on subscription pricing. But Anthropic makes the bulk of its revenue (and has its biggest margins) from enterprise/API pricing, where they are the most expensive provider by far. Anthropic isn't going to make a lot of money off your $20 subscription. With a subscription, you currently get more usage per dollar out of Anthropic's than OpenAI's plans, which approximately offsets the difference in API pricing.
Example: https://gist.github.com/rbehrends/4d9febe5c5c008961b309dd4a6...
The "Install From ACP Registry ..." allow you to use pi, OpenCode, Kilo Code, Cline, Cursor, Copilot (and a ton of other agents that I've never heard of) if you want.
Whether and under what circumstances they actually enforce this these days, I do not know. Regardless, if your preferred harness does not support that authentication flow (which most don't), then it doesn't really help you even if you could. For what it's worth, I do not consider the Claude Code ACP adapter an adequate solution.