Comment

Comment by ahmadyan

There are a few benchmark, where they run different harness with the same model, e.g. [artficial analysis](https://artificialanalysis.ai/agents/coding-agents) but honestly they are easily game-able.

For example, consider something like SWEBench, where a test should pass. Harness or a skill can just tell the agent keep trying until the test pass. So same model can have a lot of behavior changes between the harnesses.

In my experience (i own a multi-agent IDE with 50+ supported harness) folks stick to first party, usually either openai or anthropic, grok, agy; if they are willing to pay for the subscription the value is unbeatable. And they don't risk trying different harnesses.

parent ahmadyan 25 days ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton