There were rumors floating around that GPT-4 was going to be a 100 trillion parameter model. Those rumors seemed ridiculous in hindsight, but this announcement makes me rethink how ridiculous it really was. 100 Terabytes of GPU memory is exactly what you need to train that class of model.
However, I’m not even sure enough text data exists in the world to saturate 100T parameters. Maybe if you generated massive quantities of text with GPT-4 and used that dataset as your pre-training data. Training on the entirety of the internet then becomes just another fine tuning step. The bulk of the training could be on some 400TB dataset of generated text.
However, I’m not even sure enough text data exists in the world to saturate 100T parameters. Maybe if you generated massive quantities of text with GPT-4 and used that dataset as your pre-training data. Training on the entirety of the internet then becomes just another fine tuning step. The bulk of the training could be on some 400TB dataset of generated text.