> Given consistent trends of exponential performance improvements over many years and across many industries, it would be extremely surprising if these improvements suddenly stopped.
I'm sure people were saying that about commercial airline speeds in the 1970's too.
But a lot of technologies turn out to be S-shaped, not purely exponential, because there are limiting factors.
With LLM's at the moment, the limiting factors might turn out to be training data, cost, or inherent limits of the transformer approach and the fact that LLM's fundamentally cannot learn outside of their context window. Or a combination of all of these.
The tricky thing about S curves is, you never know where you are on them until the slowdown actually happens. Are we still only in the beginning of the growth part? Or the middle where improvement is linear rather than exponential? And then the growth starts slowing...
I'm sure people were saying that about commercial airline speeds in the 1970's too.
But a lot of technologies turn out to be S-shaped, not purely exponential, because there are limiting factors.
With LLM's at the moment, the limiting factors might turn out to be training data, cost, or inherent limits of the transformer approach and the fact that LLM's fundamentally cannot learn outside of their context window. Or a combination of all of these.
The tricky thing about S curves is, you never know where you are on them until the slowdown actually happens. Are we still only in the beginning of the growth part? Or the middle where improvement is linear rather than exponential? And then the growth starts slowing...