Comment by kristianp

They've shown the calculation for number of concurrent requests limited by kv cache size. I guess for each new request loaded into the queue, the kv cache for that context will need to be loaded from a supporting storage server, that will need to have a lot of ram to store the active caches of multiple requests?