> It turns out that the larger these models get, the more unexpected emergent capabilities they have, so I'm mostly in the camp of thinking AGI is just a matter of time and resources.
AI research has a long history of people saying this. Whenever there is a new fundamental improvement, it looks like you can just keep getting better results by throwing more resources at it. However, eventually we end up reaching a point where throwing more resources at it stops meaningfully improving performance.
LLMs have an additional problem related to training data. We are already throwing all the data we can get our hands on at them. However, unlike most other AI systems we have developed, LLMs are actively polluting their data pool, so this intitial generation of LLMs are probably going to have the best data set of any that we ever develop. Of course, today's data will continue to be available, but will loose value as it ages.
AI research has a long history of people saying this. Whenever there is a new fundamental improvement, it looks like you can just keep getting better results by throwing more resources at it. However, eventually we end up reaching a point where throwing more resources at it stops meaningfully improving performance.
LLMs have an additional problem related to training data. We are already throwing all the data we can get our hands on at them. However, unlike most other AI systems we have developed, LLMs are actively polluting their data pool, so this intitial generation of LLMs are probably going to have the best data set of any that we ever develop. Of course, today's data will continue to be available, but will loose value as it ages.