A great demonstration of why in-distribution learning is not enough for AGI.
The largest GPT-3 model "GPT-3 175B" is trained with 499 Billion tokens (one token is maybe equal to 4 characters in text).
Human reading/talking/listening equivalent of 200 pages of text per day for 80 years would be just 13GB of raw data or 3B tokens. You could also make an estimate using 39 bits/s as the normal information rate humans can absorb and get the same order of magnitude estimate.
It's not completely wrong to say that we are figuring out how far interpolation from data can go as a learning method. Even the most advanced deep learning is in-distribution learning and system 1 thinking (Kahneman's term). Just run through data until the model can interpolate accurately between data points.
We must figure out how do learn models that allow out-of-distribution learning or these models fail the Turing test after a few questions.
The largest GPT-3 model "GPT-3 175B" is trained with 499 Billion tokens (one token is maybe equal to 4 characters in text).
Human reading/talking/listening equivalent of 200 pages of text per day for 80 years would be just 13GB of raw data or 3B tokens. You could also make an estimate using 39 bits/s as the normal information rate humans can absorb and get the same order of magnitude estimate.
It's not completely wrong to say that we are figuring out how far interpolation from data can go as a learning method. Even the most advanced deep learning is in-distribution learning and system 1 thinking (Kahneman's term). Just run through data until the model can interpolate accurately between data points.
We must figure out how do learn models that allow out-of-distribution learning or these models fail the Turing test after a few questions.