Yes, but in practice it’s more interesting than that
The massively lopsided compute makes speculative execution techniques (MTP/DFlash etc) way more effective, which makes modern MoE models quite usable
I’m running Qwen 3.8 Flash Next with Nota AI’s conservative NVFP4 quant at 30tk/s lowest decode, usually more around 45-55 for coding, and with prompt processing at a solid 2000tk/s
And that’s without very optimised kernels or anything, mostly standard vLLM
The massively lopsided compute makes speculative execution techniques (MTP/DFlash etc) way more effective, which makes modern MoE models quite usable
I’m running Qwen 3.8 Flash Next with Nota AI’s conservative NVFP4 quant at 30tk/s lowest decode, usually more around 45-55 for coding, and with prompt processing at a solid 2000tk/s
And that’s without very optimised kernels or anything, mostly standard vLLM