I'd really be curious to see how much of Muse and similar 2026 models are actually from scraping, especially from this hypothetical unsorted random sites.
My understanding is that most of the gains after the early GPT-3 "add more weights" era have come from post-training from synthetic data, especially through reinforcement learning.
I couldn't quickly find any public statements about the actual breakdown of training data, other than almost every lab saying that post-training size was "scaling" or growing "by an order of magnitude" from their previous major release.
My understanding is that most of the gains after the early GPT-3 "add more weights" era have come from post-training from synthetic data, especially through reinforcement learning.
I couldn't quickly find any public statements about the actual breakdown of training data, other than almost every lab saying that post-training size was "scaling" or growing "by an order of magnitude" from their previous major release.