What would you prefer, 30t/s 50k output tokens needed for solution, or 300t/s, 500k output tokens needed for solution?
This is why "tokens per second" alone is a deeply flawed comparator.
This is also why discussions of Chinese models focus on tokens per second- they struggle to compete on intelligence so they dramatically increase effort levels to compensate, meaning they require massive total output tokens.
For example, flash 1017 isn't benchmarked yet, but 0731 required 15.4M tokens to complete the artificial analysis benchmark, and Sol 6.1 high required just 2.8M.
Sol 6.1 uses 2.8M at 64 tok/s = 12h 9m
Flash 0731 reported 149 token/s at 15.4M needed = 28h 42m
And in my example in the first line, scenario A (30 t/s) completes 6 minutes faster than scenario B, despite 10X slower inference.
This is why "tokens per second" alone is a deeply flawed comparator.
This is also why discussions of Chinese models focus on tokens per second- they struggle to compete on intelligence so they dramatically increase effort levels to compensate, meaning they require massive total output tokens.
For example, flash 1017 isn't benchmarked yet, but 0731 required 15.4M tokens to complete the artificial analysis benchmark, and Sol 6.1 high required just 2.8M.
Sol 6.1 uses 2.8M at 64 tok/s = 12h 9m
Flash 0731 reported 149 token/s at 15.4M needed = 28h 42m
And in my example in the first line, scenario A (30 t/s) completes 6 minutes faster than scenario B, despite 10X slower inference.