Comment by Jabdoa2

By running vllm at work I learned that you will usually outperform benchmarks which you will find on the internet. Mostly by running proper parameter tuning. So you get impressive numbers. Still in reality those numbers do not tell the whole truth. We found that real usage is more 60k-140k prompt with a few k of output for realistic agentic coding workloads. Most of the time that will give you half to a third of decode throughput vs the benchmark. It also becomes much more about cache aware routing and kv cache management (how to offload and restore). Because prefilling 60-140k will simply take too long for interactive work.