ccgibson

I agree with you, accuracy of answers and the ability to complete task is the most important thing. But once a system can achieve a goal, throughput and cost come into consideration. In this regard, I think cache rates help make things both faster and cheaper.

One of the main things I've noticed is that several of the techniques are well intentioned for other reasons. For example, when we debug low cache rates with users probably culprit #1 is usually some memory system. Many of those memory systems are trading a stable prefix for the ability to augment the current turns request with some new memory data. For some -- this may be the right trade-off, but the impact is slower responses times due to the extra prefill compute.

So when a memory system promises better LLM performance the other trade-offs aren't necessarily well understood (at the expense of caching).

Features like OMP trying to clean up history to shrink context I think are also well intentioned. But I found these trade-offs interesting hence wanting to share.

I'm the CEO of Neuralwatt, an inference cloud that bills on measured GPU energy. I wrote the post.

The harness, alongside the model, decides how much prefill work gets wasted. Across clients, uncached input ranged from 1.6% (deepseek-harness) to 11.1%. A mostly uncached request uses roughly twice the energy of a cached one of the same size.

Past 128k tokens, a turn that re-reads over half its prompt costs about as much prefill as 600 normal turns at that depth.

Every cache break in our lab came from a harness editing something it had already sent, such as changing the date in the system prompt.

Happy to discuss the methodology.

To add a bit more to what @scottcha is saying: overall GPU load has a fairly significant impact on the energy per result. Energy per result is inversely related, since the idle TDP of these servers is significant the more the energy gets spread the more efficient the system becomes. I imagine Anthropic is able to harness that efficiency since I imagine their servers are far from idle :)