abdik

We often find that models that wins on clean speech are often not the one that wins on your terms, so test with your own vocabulary rather than a headline number.

Wider picture: we rebuilt the speech-to-speech test around a hard 20-turn concierge call, and today only the closed models finish it cleanly. Board: https://benchmarks.speko.ai/s2s

shared some write-up: https://benchmarks.speko.ai/blog/can-s2s-replace-the-cascade

And jakswa's Gemma datapoint matches what we measure, the fastest rows on our LLM board are the small models like Gemma 4 is performing incredibly well for voice agents

production phone agents are a different shape today: the call terminates server-side, three models plus turn-taking under one latency budget, and per-language quality still swings a lot from our tests

You are right about human input for naturalness, that one we did not automate away with yet. We run blind A/B listening rounds with native speakers.

On fast dumb models answering while a smarter one takes over: we are experimenting with exactly that split - a small fast model holds the conversation while a larger one works behind it. Today it runs as two pinned routes, not one packaged API. Most turns in a phone call do not need a frontier model, and the fastest models on our LLM board are all small, so this is where routing earns its keep. We publish benchmarks on LLMs here: https://benchmarks.speko.ai/llm