I'm a big believer that smaller models will take hold. The throwing paint at the wall strategy of having 100B+ models and random data won't scale. It's possible it will take new model architectures to get there. But having more fine-tuned control over model parameters is a pattern that will emerge. Then smaller, domain specific models can be joined together to form larger models, if full generalization is needed.
I have an article on building micromodels that discusses some of this: https://neuml.hashnode.dev/train-a-language-model-from-scrat...