Comment by max-privatevoid

llama.cpp has done this since forever. I've used other engines including vLLM here and there, and I was quite surprised that many of them don't have this feature. Glad vLLM has it now.