systima
Joined 8 months ago
Joined 8 months ago
1) We have added a repository reproducing the methods.
2) We have commented on output quality in the post (in addition to tokens/cost).
3) Per a previous update, we now handle Fable 5 in the post.
After reading PUSH_AX's valid comment: ``` This is like saying contractor (A) asked for $33,000 to undertake the work and contractor (B) asked for $7,000 Are we measuring and caring about the right thing? ``` We will update the post to include:
1) A more in-depth task. 2) Qualitative results comparison. 3) As soon as possible, a reproduction of the inputs and outputs.
Suspect that many have covered the "Comparing agentic coding tools" angle before, and that the differentiator is depth of analysis + conclusions.
Cost, mainly. The runs went through a Claude Max subscription rather than metered API billing, and pinning an older stable snapshot kept run-to-run comparisons clean and cheap. The fixed harness payload (system prompt plus tool schemas), so the headline numbers shouldn't change too much.
That said, happy to re-run the matrix on Fable and publish the diff; payload figures should barely move, tool-calling behaviour might.
Gateway:
Meridian (github.com/rynfar/meridian); proxy that bridges the Claude Code SDK to a standard Anthropic endpoint so a Claude Max subscription can drive OpenCode-et-al.
It's the auth route for all agent traffic on the machine, not something built for the benchmark.
I recall the Beeper Mini debacle not so long ago, and fear that this may be a house built on sand.