Comment by tosh

I can only recommend to regularly measure how many tokens a harness+model combination uses for a certain task

There are huge token efficiency/bloat differences between agents while working on the same tasks, using the same model, in the same environment

Yesterday I ran 10 agentic tasks using GPT 5.6 Sol in an ubuntu 26.04 vm a couple of times with different harnesses and got vastly different token usage.

  +-------------+-----------+-----------+-----------+-----------+--------+
  | Harness     | API total | Input     | Cached    | Uncached  | Output |
  +-------------+-----------+-----------+-----------+-----------+--------+
  | smol        |   172,807 |   142,334 |     8,704 |   133,630 | 30,473 |
  | Pi          |   427,211 |   392,767 |   137,216 |   255,551 | 34,444 |
  | OpenCode    | 1,564,429 | 1,523,957 | 1,204,736 |   319,221 | 40,472 |
  | Codex       | 3,005,744 | 2,953,154 | 2,649,344 |   303,810 | 52,590 |
  | Hermes      | 3,856,611 | 3,808,231 | 3,167,232 |   640,999 | 48,380 |
  | Claude Code | 5,073,137 | 5,029,969 | 4,587,008 |   442,961 | 43,168 |
  +-------------+-----------+-----------+-----------+-----------+--------+

https://x.com/__tosh/status/2083593799872237680

I'm not surprised that Claude Code is not optimized for an OpenAI model but I was still quite shocked re how much of a difference the harness makes.

Disclaimer: I'm working on 'smol' which is a minimalist harness but it's really nothing special, just a minimal system prompt, no skills files, only tool is shell

Do not underestimate how much popular harnesses are spamming the context window. The context window is very important.