In the context of local LLMs on limited hardware I've ran to the exact same conclusion: "tok/s" isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals _in the minimum amount of time_.
Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that they generated many more tokens compared to a Qwen model of same caliber to answer the same thing.
So even though it had better generated tok/s, because so many more were generated, the clock time was longer.
And this compounds over mutli-turns: more generated token means more context used in the next turn (until some compaction or something runs)
Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that they generated many more tokens compared to a Qwen model of same caliber to answer the same thing.
So even though it had better generated tok/s, because so many more were generated, the clock time was longer.
And this compounds over mutli-turns: more generated token means more context used in the next turn (until some compaction or something runs)