Hi HN, I'm the author. This runs an open-vocabulary English speech recognizer live on an ESP32-P4: you speak into the board's mic and the words appear on its screen as you talk. No network, no NPU: just the chip's two RISC-V cores and their SIMD extension (PIE).
The model is Kroko (by Banafo), a 65.7M-parameter streaming Zipformer2 transducer. As released it is 71 MB in int8, which does not fit the board's 32 MB of PSRAM, so I quantized the weights to 2 and 4 bits (26.65 MB, groups of 128 with a Hadamard rotation, the bit width chosen per matrix) and wrote my own engine: integer arithmetic throughout, with the hot loops hand-written in PIE assembly.
Numbers, measured on all of LibriSpeech:
test-clean 4.34% WER, test-other 10.26% (the original int8 model on a PC: 3.94% and 9.17%).
1.28 s of audio processed in 0.90 s (0.70x real time), both cores at 400 MHz.
A word is final about 2 s after it is spoken.
One thing I care about: the chip computes exactly the same bits as the engine's PC build, so every accuracy number above is the chip's, not a simulation. Getting there meant replacing libm (newlib and glibc disagree in the last bit of log/exp/cos) with deterministic math on both sides.
There are also hotwords (phrases to listen for, from a text file on the SD card) and six voice-control demos driven by simple word rules, no LLM.
Limits: English only; it needs the chip revision v3 (P4X) with 32 MB PSRAM; accuracy is measured on clean read speech, noisy rooms not yet. You can flash it from the browser; the weights are on Hugging Face.
Video: https://www.youtube.com/watch?v=oe-2MowdsSs
Happy to answer questions about the quantization, the kernels or the streaming.
The model is Kroko (by Banafo), a 65.7M-parameter streaming Zipformer2 transducer. As released it is 71 MB in int8, which does not fit the board's 32 MB of PSRAM, so I quantized the weights to 2 and 4 bits (26.65 MB, groups of 128 with a Hadamard rotation, the bit width chosen per matrix) and wrote my own engine: integer arithmetic throughout, with the hot loops hand-written in PIE assembly.
Numbers, measured on all of LibriSpeech:
test-clean 4.34% WER, test-other 10.26% (the original int8 model on a PC: 3.94% and 9.17%).
1.28 s of audio processed in 0.90 s (0.70x real time), both cores at 400 MHz.
A word is final about 2 s after it is spoken.
One thing I care about: the chip computes exactly the same bits as the engine's PC build, so every accuracy number above is the chip's, not a simulation. Getting there meant replacing libm (newlib and glibc disagree in the last bit of log/exp/cos) with deterministic math on both sides.
There are also hotwords (phrases to listen for, from a text file on the SD card) and six voice-control demos driven by simple word rules, no LLM.
Limits: English only; it needs the chip revision v3 (P4X) with 32 MB PSRAM; accuracy is measured on clean read speech, noisy rooms not yet. You can flash it from the browser; the weights are on Hugging Face.
Video: https://www.youtube.com/watch?v=oe-2MowdsSs
Happy to answer questions about the quantization, the kernels or the streaming.