If true, does this imply the end of the scaling laws? I'm guessing all these players were expecting a certain level of model improvement for their compute budget. If they're now disappointed, this suggests their estimates were off?
Or is this a disconnect between pertaining loss and downstream performance on real-world tasks?
Or is this a disconnect between pertaining loss and downstream performance on real-world tasks?