Whether scaling laws hold or not, is up for debate.
What isn't up for debate, is that:
1) Giant serverfarms are expensive
2) People want on premises/on machine solutions
3) LoRA Tuning of small models continues to excel
4) Thus specialized models continue to evolve at a fast pace
5) Open source foundation models on which tuning can be done are accelerating by the week
6) Performance doesn't matter once a model is "good enough" for a given task
The long and short of it is this: LLMs are a fantastic technology, and people want to have it. And they want to have it local, private, and as cheap as possible. Just one example, how many devs are out there who would love having a
local LLM model integrated into their IDE? The answer: Yes.
And again I draw your attention to point 6): For such use-cases, It doesn't matter if a gargantuan model performs a bit better ... if the small model is good enough for the task, then it wins, purely by being private, offline and is free to use.
We are already at a point, where small models are competitive for specialized tasks, and aren't performing bad at all as general models either. What will it take for bigger models to out-perform them so enourmously that customers simply have no other choice than to use them? A 5x increase in size? 10x? 100x? Where do we run models with trillions of params? What will that cost, and how available will that be?
What isn't up for debate, is that:
The long and short of it is this: LLMs are a fantastic technology, and people want to have it. And they want to have it local, private, and as cheap as possible. Just one example, how many devs are out there who would love having a local LLM model integrated into their IDE? The answer: Yes.And again I draw your attention to point 6): For such use-cases, It doesn't matter if a gargantuan model performs a bit better ... if the small model is good enough for the task, then it wins, purely by being private, offline and is free to use.
We are already at a point, where small models are competitive for specialized tasks, and aren't performing bad at all as general models either. What will it take for bigger models to out-perform them so enourmously that customers simply have no other choice than to use them? A 5x increase in size? 10x? 100x? Where do we run models with trillions of params? What will that cost, and how available will that be?