The idea of getting a speed up in language models from using diffusion is compelling, but it just doesn’t seem like discrete diffusion models work as good at discrete non-diffusion. Which kind of makes sense, tokens don’t have implicit continuous representations that are easily noised. You have to learn a continuous embedding and figure out how to noise that, which is ideal for distillation (Ie the Gemma models, drafter heads) but not necessarily for pretraining.
There’s been more recent work on continuous space diffusion models for language this year. Sander Dielman has a good blog post on that.
But if I’d predict where diffusion lands in LLMs, it’ll be used in looped models like Astra. Once reasoning is happening in hidden states, we’re in a good continuous domain, perfect for diffusion. We’re going to end up swapping “looping” for predicting the models hidden states at the next “timestep” with diffusion. And that way the time to generate traces (no longer human intelligible though) will become 10x faster.
There’s been more recent work on continuous space diffusion models for language this year. Sander Dielman has a good blog post on that.
But if I’d predict where diffusion lands in LLMs, it’ll be used in looped models like Astra. Once reasoning is happening in hidden states, we’re in a good continuous domain, perfect for diffusion. We’re going to end up swapping “looping” for predicting the models hidden states at the next “timestep” with diffusion. And that way the time to generate traces (no longer human intelligible though) will become 10x faster.