Comment by bastawhiz

You can run a model at 1000t/s on a low and android phone. The output will be dogshit bad, but it'll pump tokens like nobody's business. The problem with what you're proposing is that there's no baseline definition of what those tokens are (for what purpose? of what quality?). And pinning that to a certain model or level of correctness means your benchmark becomes almost immediately obsolete.