I mean, memory prices are high. They ain't getting cheaper. And KVCache is held in memory. The more people are using cache, the more memory you need. It's not free to cache it.
Plus there's complicated cluster-wide KVCache distribution schemes, built around RDMA networking, etc etc.
By far the most expensive hardware cost now of inference is the memory cost. KVCache is still using it. Though obv GPU compute is using more energy
It's not just like... "stick it in Redis, all done!"
Plus there's complicated cluster-wide KVCache distribution schemes, built around RDMA networking, etc etc.
By far the most expensive hardware cost now of inference is the memory cost. KVCache is still using it. Though obv GPU compute is using more energy
It's not just like... "stick it in Redis, all done!"