I'm the CEO of Neuralwatt, an inference cloud that bills on measured GPU energy. I wrote the post.
The harness, alongside the model, decides how much prefill work gets wasted. Across clients, uncached input ranged from 1.6% (deepseek-harness) to 11.1%. A mostly uncached request uses roughly twice the energy of a cached one of the same size.
Past 128k tokens, a turn that re-reads over half its prompt costs about as much prefill as 600 normal turns at that depth.
Every cache break in our lab came from a harness editing something it had already sent, such as changing the date in the system prompt.
The harness, alongside the model, decides how much prefill work gets wasted. Across clients, uncached input ranged from 1.6% (deepseek-harness) to 11.1%. A mostly uncached request uses roughly twice the energy of a cached one of the same size.
Past 128k tokens, a turn that re-reads over half its prompt costs about as much prefill as 600 normal turns at that depth.
Every cache break in our lab came from a harness editing something it had already sent, such as changing the date in the system prompt.
Happy to discuss the methodology.