telenardo

I tested Wispr on the same 525 dictation clips a month apart (May and again in June) and there's a clear regression: Word Error Rate rose from 9.0% to 11.2% (lower is better). It got worse in 8 of 9 categories I track.

Full results and the raw transcripts attached above.

Important context: this only measures a single, American‑accented voice on synthetic prompts. Wispr supports many languages and accents, and I’d assume their updates have boosted overall accuracy. But, they regressed for my use‑case: plain old boring, unaccented American English.

How I tested: I have a harness that drives a dictation app by playing an audio file through a virtual input device and captures the app’s pasted output. I ran this on 525 audio files through Wispr Flow a month ago. Yesterday, I ran the same clips through again. The clips are recordings of me whispering AI-generated sentences.

Results: There’s a clear regression on word accuracy. Words that were correct ~1 month ago are now incorrect. Phonetically, they’re similar, but just off.

May: "Brian, refactor this auth middleware to stop leaking tokens and debug logs." (near perfect)

June: "Ryan refactored his odd metalware to stop leaking tokens in deep bugs and backlogs." (garbled)

So no, you're not imagining it, it has gotten worse. At least if you speak plain, unaccented American English like me!

Disclosure: I work on a competitor which is why I had the May eval ready. Take this with a grain of salt and do your own testing!

I tried to benchmark Google’s new on-device dictation app (Eloquent) and basically couldn’t. It drops about half of my dictations.

Background: Google shipped a new fully‑local dictation app yesterday with proprietary new models, so I was excited to benchmark it against the leading open models (Qwen3‑ASR, NVIDIA Parakeet V3, etc).

I have a harness that drives a dictation app by playing an audio file through a virtual input device and captures the app’s pasted output, so I can compare different apps on the same clips. I also have ~1,500 manually corrected clips from my daily engineering work.

What happened: I couldn’t get a clean eval, because ~half of dictations come back missing a large number of words. A clip of with ~20+ words routinely returns just 5-10 words. I assumed my harness was broken, so I used the app manually, speaking slowly and clearly into the mic. Same thing: roughly half the time, I only get a small fraction of what I actually said.

When Eloquent did return a complete transcript (15 of 50 tests), its accuracy was actually competitive ~24% WER vs ~21% for Qwen3-ASR on the same clips. The problem isn't the recognition. It's that for most dictations, you don't get your words back at all!

My theory: The transcriber is a chat‑style AI model, and chat models sometimes reply about your audio instead of transcribing it.

To test this, I ran Gemma 3n (Google's open model from the same family) directly on the same clips bypassing the Eloquent app. On 11 / 44 attempts it responded something like “I’m sorry, I can’t transcribe this,” instead of producing a transcript. Gemma had the same ~60 % word error rate as Eloquent. My guess is that Eloquent’s model has the same issue, the app just hides it.

Has anyone been able to get good results with this app? Or are others seeing this issue?

Disclosure: I build a competitive local dictation app, so not a neutral party!

It's not set up for that, no, though it's theoretically possible!

The issues I see are: - Transcription models use beam search to choose the most likely words at each step, taking into account the surrounding words. The accuracy will drop a lot if you pick each top word individually as it’s spoken. The surrounding context matters a lot. - To that point, transcription models do get things wrong (i.e. "best" instead of "test"). The LLM post-processing can help here, by taking in the top-N hypotheses from the transcription mode and determining which makes the most sense (i.e. "run the tests", not "run the bests"), adding another layer of semantic understanding. Again, the surrounding context really matters here.

Do you need each word to stream individually? Or would it be sufficient for short phrases to stream?

The MLX inference is so fast that you could accomplish something like the latter by releasing and re-pressing the shortcut every 5-10 words. It so fast it honestly feels like streaming. In practice, I tend to do something like this anyway, because I find it easier to review shorter transcripts!

Not yet, but if all goes according to plan, you should be able to soon!

(that this is all a bit tangential)

I would agree!

It's hard to find unoccupied shortcuts these days! I don't use shortcuts on the numbers often, so I set that as a default. But yes, it's easily configurable in settings so you can choose something that works for your workflows.

Sarp! Good to hear from you! I hope life has been good since the Instagram days. Yes, I've noticed the multi-resizing issue with cmd + 8 - I'll look into it this week. Regarding the cmd + 0 toggle, I think I can probably make that work too. We can set it up so you can set your dismiss shortcut. Then, you can choose the same as the launch shortcut, making it a toggle. I'll also take a look at that this week.

Thanks for the feedback! Fixed these - though the site title seems to be cached...

Addressed towards the end of the post:

"Why not Linux or Windows? Gotta start somewhere! If the reception is positive, we’ll work hard to add further support."

As you can see from the commit log, we have 3 people working on this. So we're quite limited in what we can take on. That said, our belief holds and we'd love to support Linux and Windows.

I had "MacOS" in my original title, but HN limits titles to 80 characters!