Comment by mmastrac

What's missing from these discussions is the real weakness of diffusion models: you can have two positions where two tokens need to be coordinated, but both spots are teetering between two valid states. They don't always collapse to the matching pair.

I only learned this the hard way reimplementing diffusiongemma. I had ideas on how to fix it but no cluster to train and experiment, hah.