There's a deeper more troubling problem being exposed here - deep learning systems are at least an order of magnitude less data efficient than the systems they hope to replicate.
GPT-3 175B is trained with 499 Billion tokens[1]. Let's assume token = word for the sake of this argument[2]. The average adult person reads at a rate of 238wmp[3]. Then a human who reads 24hrs/day from birth until their 18th birthday would read a total of 2.2B billion words[4], or 0.45% of the words GPT-3 was trained on.
Human's simply do much more with much less. So what gives? I don't disagree that we still haven't reached the end of what scaling can do, but there is a creeping suspicion that we've gotten something fundamentally wrong on the way there.
1. https://lambdalabs.com/blog/demystifying-gpt-3/
2. GPT-based models use BPE and while we would dive into the actual dictionary of tokens and make a word-token relationship, we both agree that although this isn't a 1-to-1 relationship it won't change the conclusion https://huggingface.co/docs/transformers/tokenizer_summary
3. https://psyarxiv.com/xynwg/
4. 238*60*24*365*18=2,251,670,400
GPT-3 175B is trained with 499 Billion tokens[1]. Let's assume token = word for the sake of this argument[2]. The average adult person reads at a rate of 238wmp[3]. Then a human who reads 24hrs/day from birth until their 18th birthday would read a total of 2.2B billion words[4], or 0.45% of the words GPT-3 was trained on.
Human's simply do much more with much less. So what gives? I don't disagree that we still haven't reached the end of what scaling can do, but there is a creeping suspicion that we've gotten something fundamentally wrong on the way there.
1. https://lambdalabs.com/blog/demystifying-gpt-3/
2. GPT-based models use BPE and while we would dive into the actual dictionary of tokens and make a word-token relationship, we both agree that although this isn't a 1-to-1 relationship it won't change the conclusion https://huggingface.co/docs/transformers/tokenizer_summary
3. https://psyarxiv.com/xynwg/
4. 238*60*24*365*18=2,251,670,400