The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer.
I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop and took all the rewards.
It's likely that GPT-4 is on the optimal spot between cost and performance and there won't be significant improvements on performance in a near future. I guess the next task would be more on efficiency, which has a significant implication on its productionization.
I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop and took all the rewards.
It's likely that GPT-4 is on the optimal spot between cost and performance and there won't be significant improvements on performance in a near future. I guess the next task would be more on efficiency, which has a significant implication on its productionization.