vvolhejn
Joined 5 years ago
Karma
99
Submissions
25
Member since
Sep 13
Recent submissions
Comment · 4 months ago
Comment · 4 months ago
Václav here from the team, we're happy to answer questions :)
The most surprising part to me is the auto-recovery behavior we mention at the end of the blog post, since any other model I've seen always stays diverged once it goes off the rails once. But MIRA really doesn't like to be out-of-distribution. To be completely honest we're not entirely sure why this happens.
Comment · 9 months ago
> Is this just sort of expected for these models? Should users of this expect only truncation or can hallucinated bits happen too?
Basically, yes, sort of expected: we don't have detailed enough control to precent it fully. We can measure how much it happens and train better models, but no 100% guarantee. The bigger the model, the less this happens, but this one is tiny, so it's not the sharpest tool in the shed. Hallucinated bits can theoretically happen but I haven't observed it with this model yet.
Comment · 9 months ago
Václav from Kyutai here. Thanks for the bug report! A workaround for now is to chunk the text into smaller parts where the model is more reliable. We already do some chunking in the Python package. There is also a more fancy way to do this chunking in a way that ensures that the stitched-together parts continue well (teacher-forcing), but we haven't implemented that yet.
Comment · 9 months ago
Václav from Kyutai here. Yes the original naming scheme was from Les Miserables, glad you noticed! We just stuck to Alba because that's the real name of the voice actor that provided the voice sample to us (see https://huggingface.co/kyutai/tts-voices), the other ones are either from pre-existing datasets or given anonymously.
Comment · 1 year ago
I had no idea this existed, the internet is amazing
Comment · 1 year ago
There's a great blog post from Sander Dieleman about exactly this - why do we need a two step pipeline, in particular for images and audio?
https://sander.ai/2025/04/15/latents.html
For text, there are a few papers that train the tokenization and language model end-to-end, see: https://arxiv.org/abs/2305.07185
Comment · 1 year ago
I train for 1M steps (batch size 64, block size 2048), which is enough for the model to more-or-less converge.
It's also a tiny model for LLM standards, with 150M parameters. The goal wasn't really to reach state of the art but to show how the performance of a single language model architecture can be vastly different when you just change the tokenizer.
Comment · 1 year ago
Author here, thanks for the kind words! I think such a physics-based codec is unlikely to happen: in general, machine learning is always moving from handcrafted domain-specific assumptions to leaving as much as possible to the model. The more assumptions you bake in, the smaller the space of sounds you can model, so the quality is capped. Basically, modern ML is just about putting the right data into transformers.
That being said, having a more constrained model can also lead to some really cool stuff. The DDSP paper learns how to control a synthesizer to mimic instruments: https://arxiv.org/abs/2001.04643
You could probably do something similar for a speech model. The result would not sound as good but you could get away with much fewer parameters, because much of the modelling work is done by the assumptions you put in.
Compare also KokoroTTS, a tiny TTS that's so tiny because it uses a handcrafted system to turn text into phonemes, and then just synthesizes from those phonemes: https://huggingface.co/spaces/hexgrad/Kokoro-TTS
Comment · 1 year ago
merci, will fix tomorrow
Comment · 1 year ago
I use Descript to edit videos/podcasts and it works great for this kind of thing! It transcribes your audio and then you can edit it as if you were editing text.
Comment · 1 year ago
I had a part about this but I took it out: for compression, you could keep the embeddings unquantized and it would still compress quite well, depending on the embedding dimension and the number of quantization levels.
But categorical distributions are better for modelling. It's a little difficult to explain here without using diagrams. The intuition is that if you try to have a model predict the next embedding and not the next token, you can't model multimodal distributions - you'll end up predicting the mean of the possible continuations and not the mode, which is not what you want.
Check out Section 5.3 and Figure 6 from PixelRNN, where they discuss this phenomenon: https://arxiv.org/pdf/1601.06759
At the bottom of the blog, I link two articles that do make continuous embeddings work. One of them is the Kyutai paper Continuous Audio Language Models: https://arxiv.org/abs/2509.06926
Comment · 1 year ago
I don't know about linear models, but this kind of hierarchical modelling is quite a common idea in speech research. For example, OpenAI's Jukebox (2020) [1], which uses a proto-neural audio codec, has three levels of encoding that get coarser and coarser. They use a language model to predict continuations in the coarsest level and then have models to upscale to the finer levels and finally back to audio.
The recent MiMo-audio bunches tokens into "patches" of four timesteps and has the model predict those. [2]
[1] https://arxiv.org/abs/2005.00341
[2] https://github.com/XiaomiMiMo/MiMo-Audio/blob/main/MiMo-Audi...
Comment · 1 year ago
Author here. Speech-to-text is more or less solved, it's easy to automatically get captions including precise timestamps. For training Moshi, Kyutai's audio LLM, my colleagues used whisper-timestamped to transcribe 7 million hours of audio.
See Section 4.2 in the Moshi paper: https://arxiv.org/pdf/2410.00037
[1] https://mira-wm.com/blog-post/#hidden-information