Got it, I'll test more deployment params when I benchmark models for vision tasks.
Do you have any tips for Gemma 4 31B in particular? I quite like the model but I feel like I'm underutilizing my rented GPU hard due to skill issues. Throughput should be way higher than this, right?
---
Gemma 4 31B NVFP4, vLLM 0.29, single B200 (modal), FlashInfer, fp8 KV, prefix caching, 32k ctx
Workload: ~6k-token shared prefix (~98% cache hit) + short input, ~250 tokens out,
~250 seqs running. Get ~5-6k output tok/s, flat from 128 to 384 concurrency.
Spec decode (DFlash, n-gram) didn't give a good boost which is annoying because I feel like this task should be easy for a draft model to predict.
Individual request latency doesn't matter I just need as much throughput as possible. Its only active for a few hours when I need it.
Do you have any tips for Gemma 4 31B in particular? I quite like the model but I feel like I'm underutilizing my rented GPU hard due to skill issues. Throughput should be way higher than this, right?
---
Gemma 4 31B NVFP4, vLLM 0.29, single B200 (modal), FlashInfer, fp8 KV, prefix caching, 32k ctx
Workload: ~6k-token shared prefix (~98% cache hit) + short input, ~250 tokens out, ~250 seqs running. Get ~5-6k output tok/s, flat from 128 to 384 concurrency.
Spec decode (DFlash, n-gram) didn't give a good boost which is annoying because I feel like this task should be easy for a draft model to predict.
Individual request latency doesn't matter I just need as much throughput as possible. Its only active for a few hours when I need it.