It's mostly a matter of increasingly complex and sophisticated sampling methodologies to sift for the valuable synthetic tokens.
There have also been some interesting results about models that use the same data converging to very similar behavior, but I can't find the specific reference at the moment. That one tracks intuitively with the notion that the model is building a latent representation of everything it sees.
- https://arxiv.org/abs/2609.37891
- https://arxiv.org/pdf/2508.10975
- https://www.datologyai.com/blog/eating-the-rainbow-a-primer-...
As well as recent model releases that used large fractions of synthetic data (Arcee Trinity):
- https://arxiv.org/pdf/2602.17004v1
It's mostly a matter of increasingly complex and sophisticated sampling methodologies to sift for the valuable synthetic tokens.
There have also been some interesting results about models that use the same data converging to very similar behavior, but I can't find the specific reference at the moment. That one tracks intuitively with the notion that the model is building a latent representation of everything it sees.