Comment by jug

There has actually been research that found that there are strong diminishing returns in terms of at least expanding parameter sizes. While I think there are still breakthroughs to be made in terms of window sizes and workarounds like Mixture of Experts, I'm not sure how much farther we will get here in the long term in terms of raw performance of the LLM itself. FWIW, Sam Altman agrees and has a surprisingly similar quote as found here "I think we’re at the end of the era where it’s gonna be these giant models, and we’ll make them better in other ways": https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wo...