daanzu

With an open system/engine, you can train your own personal speech model. For kaldi-active-grammar (https://github.com/daanzu/kaldi-active-grammar), you can do so without all that much difficulty, although the process/documentation could certainly use improvement.

I bootstrapped my personal speech model by retaining the commands from me using WSR. My voice is quite abnormal, and it took only 10 hours of speech data to train a model orders of magnitude more accurate than any generic model I've ever used. And of course, I retain much of my usage now with Kaldi, so my model improves more and more over time. A virtuous flywheel!

Yep, I have been using my Kaldi backend through Dragonfly exclusively ever since I got v0.1.0 working.

I bootstrapped writing it initially using the Dragonfly WSR (windows speech recognition) backend, because that gave me the best accuracy out of the available options at the time. All of my development of it since the initial working version has been done using each previous version, so now it is basically bootstrapped itself. My productivity skyrocketed once I switched to Kaldi, due to being able to use my custom trained speech model just for my voice for orders of magnitude better accuracy, plus dramatically lower latency. (And it freed me from being dependent on closed software out of my control.)

I bootstrapped my personal speech model by retaining the commands from me using WSR. My voice is quite abnormal, and it took only 10 hours of speech data to train a model dramatically more accurate than any generic model I've ever used. And of course, I retain much of my usage now with Kaldi, so my model improves more and more over time. A virtuous flywheel!

"Everything other than talon has terrible latency": False! I develop kaldi-active-grammar (https://github.com/daanzu/kaldi-active-grammar), a free and open source speech recognition backend, which has extremely low latency. You can adjust how aggressive the VAD (voice activity detection) is to suit your preference, but the speech engine latency can be almost negligible, especially for voice commands (vs prose dictation). However, I agree that "most existing speech recognition engines were not designed with the kind of latency you want for quick one syllable commands", and that low latency is pivotal to being productive with voice commands. I also agree with your other points.

I have been coding entirely by voice for approximately 10 years now (by hand long before that). Most of that time I have been using the Dragonfly (https://github.com/dictation-toolbox/dragonfly) library to construct my own customized voice coding system. The library is highly flexible and open source, allowing you to easily customize everything to suit what you need to be productive. It is perhaps the power user analogue to Dragon Naturally Speaking. With it, you can certainly be highly productive coding by voice. However, it does require work to setup and customize to suit you, so it isn't really for the "general population" of computer users to just sit down and use. With regard to accuracy of speech recognition, being open allows you to (with sufficient motivation) to train a custom acoustic speech model that recognizes your voice specifically extremely well.

Regarding the software packages you referenced: Yes, Dragon is trash that I want nothing to do with, because of its inefficient interface, its complete inability to accurately understand my voice, and its generally shoddy software quality. Voice Computer (which I hadn't seen before) is therefore eliminated as well, though it doesn't look terrible as a front end to Dragon to better use the OS GUI-accessibility info. Many people like Talon, but I demand something open, which I can modify to suit my needs.

Background: I develop kaldi-active-grammar (https://github.com/daanzu/kaldi-active-grammar), a free and open source speech recognition backend usable by Dragonfly, itself entirely by voice. There's also a community of voice coders using Dragonfly and other tools that build on top of it, such as Caster (https://github.com/dictation-toolbox/Caster).

I have been coding entirely by voice for approximately 10 years now (by hand long before that). Most of that time I have been using the Dragonfly (https://github.com/dictation-toolbox/dragonfly) library to construct my own customized voice coding system. The library is highly flexible and open source, allowing you to easily customize everything to suit what you need to be productive. It is perhaps the power user analogue to Dragon Naturally Speaking. With it, you can certainly be highly productive coding by voice. In fact, I develop kaldi-active-grammar (https://github.com/daanzu/kaldi-active-grammar), a free and open source speech recognition backend usable by Dragonfly, itself entirely by voice. There's also a community of voice coders using Dragonfly and other tools that build on top of it, such as Caster (https://github.com/dictation-toolbox/Caster).

A great demonstration of coding by voice.

I want to point out that the Dragonfly library has backends other than the closed/commercial Dragon, and there has been significant progress expanding beyond it recently. I have been developing kaldi-active-grammar [0] for the past couple years as a fully open source, modifiable, cross-platform, and free alternative to Dragon when used with Dragonfly. While KaldiAG misses some of the user-friendly niceties of Dragon, these are least helpful when coding by voice. And KaldiAG has benefits such as lower latency that improves coding by voice tremendously.

I created KaldiAG because I didn't trust relying on closed source software for something so crucial to my productivity, where a decision by an outside party determines whether I can function. As a bonus, open source means I can make it work better to fit my needs than closed source ever could.

You raise good points. For what it's worth, I think all "invalidated" samples are still included in the distribution (invalidated.tsv), with the number of up and down votes for each (but not the reasoning).

Google is certainly doing some great work with this, both Project Euphonia and other research [0]. However, as far as I know, the Euphonia dataset is closed and only usable by Google. A Common Voice disordered speech dataset would (presumably) be open to all, allowing independent projects and research. (I would love to have access to such a dataset.)

[0] https://ai.googleblog.com/2019/08/project-euphonias-personal...

Gathering, collecting, and publishing such a dataset would be great, and would certainly much improve the baseline speech recognition for people with disordered speech, but it can only help so much without personalizing to a specific individual. This is true for anybody, but more so for disordered speech. This is an area where I think "generic" solutions will inevitably struggle, even if they are somewhat specialized on "generic disarthritic" speech.

However, this means that the gains to be had from personalized training are greater for disordered speech than for "average" speech. I develop kaldi-active-grammar [0], which specializes the Kaldi speech recognition engine for real-time command & control with many complex grammars. I am also working on making it easier to train personalized speech models, and to fine tune generic models with training for an individual. I have posted basic numbers on some small experiments [1]. Such personalized training can be time consuming (depending on how far one wants to take it), but as my parent comment says, disabled people may need to rely more on ASR, which means they have that much more to gain by investing the time for training.

Nevertheless, a Common Voice disordered speech dataset would be quite helpful, both for research, and for pre-training models that can still be personalized with further training. It is good to see (in my sibling comment) that it is being discussed.

[1] https://github.com/daanzu/kaldi-active-grammar/blob/master/d...

I develop kaldi-active-grammar [0]. The Kaldi engine itself is state of the art and open source, but is focused on research rather than usability. My project has a simple interface and comes with a pretty good open source speech model.

However, kaldi-active-grammar specializes in real time command and control, with advanced features that don't really apply to your use case. Vosk [1] is probably a simpler, better fit for you. It likewise uses Kaldi and can use my models, and offers some others of its own as well.

Neither are particularly focused on transcription per se, but they are open.

[1] https://github.com/alphacep/vosk-api

Just to be clear, the Dragonfly speech recognition command and control framework has multiple "backends" (speech recognition engines), including my Kaldi one. Probably the most used one currently is the Dragon Naturally Speaking backend.

The Kaldi engine, being developed primarily for research in speech recognition, can support a huge variety of "models". I think the consensus general best for most use cases (particularly for real time, low latency, streaming use) currently would be considered to be the "nnet3 chain" models, which are what my kaldi-active-grammar uses/supports.

For me at least, dictation is actually the more straining mode of speech recognition, as compared to using my command grammars. With dictation, you might say anything, so the computer is given wide leeway in what to recognize, and so you must speak as clearly as possible. With commands (especially a nice simple command grammar), however, what you can say is greatly restricted, which allows you the freedom to speak indistinctly and still be understood by the computer. This can even be magnified by personalized training of the speech recognition model.

When using commands at my computer, I frequently find myself muttering and grunting things that even I think to myself (that is utterly un-understandable), yet the computer understands just fine. Dictating for prolonged periods can be tiring for me, but I can happily code by voice commands all night.

More info in my other comments here: https://news.ycombinator.com/item?id=23507363 https://news.ycombinator.com/item?id=23507829

I agree with everything you said, but I would add that a critical component of voice command and control is strict grammars. There is so much structure and context in what we speak, and being able to limit what can be recognized to only what can be reasonably spoken (based on the current context) can allow massive increases in accuracy. (EDIT: ah, you edited to add a mention of this as well.)

And one shameless plug deserves another! Vosk is a great project, but my kaldi-active-grammar [0] (mentioned in another comment here) also uses the same Kaldi engine, but extends it and is designed specifically for this use case. It supports defining many grammars, in any combination, and activating/deactivating them at will instantly per-utterance. I think it's probably a better fit as a backend for your project than vosk. My work focuses on the backend technology, so it would be great to have more front ends using it to put it within users' reach (so to speak).

Windows Speech Recognition is far from the best, so perhaps your trouble could be partly caused by how you had to speak in order to be understood, rather than the command style? I used to use WSR to code by voice, and it was far more laborious than my current setup.

I develop kaldi-active-grammar [0]. The Kaldi engine is state of the art for command and control. Although I don't have the data and resources for training a model like Microsoft/Nuance/Google, being an open rather than closed system allows me to train models that are far more personalized than the large commercial/generic ones you are used to. For example, see the video of me using it [1], where I can speak in a relaxed manner without having to over enunciate and strain my voice.

Gathering the data for such training does take some time, but the results can be huge [2]. Performing the actual training is currently complicated; I am working on making it portable and more turnkey, but it's not ready yet. However, I am running test training for some people. Contact me if you want me to use you as a guinea pig.