Right, but GPT-2 was the name of the particular ML
architecture they were studying the properties of; not the name of any specific model trained on that architecture.
There was a pre-trained GPT-2 model offered for download. The whole "interesting thing" they were publishing about, was that models trained under the GPT-2 ML architecture were uniquely-good at transfer learning, and so any pre-trained GPT-2 model of sufficient size, would be extremely useful as a "seed" for doing your own model training on top of.
They built one such model, but that model was not, itself, "GPT-2."
Keep in mind, the training data for that model is open; you can download it yourself and reproduce the offered base-model from it if you like. That's because GPT-2 (the architecture) was formal academic computer science: journal papers and all. The particular pre-trained model, and its input training data, were just published as experimental data.
It is under that lens, that I call GPT-3 "GPT-2++." It's a different model, but it's the same science. The model was never OpenAI's "product." The science itself was/is.
Certainly, the SaaS pre-trained model named "GPT-3" is qualitatively different than the downloadable pre-trained base-model people refer to as "GPT-2." But so are all the various trained models people have built by training GPT-2 the architecture with their own inputs. The whole class of things trained on that architecture are fundamentally all "GPT-2 models." And so "GPT-3" is just one such "GPT-2 model." Just a really big, surprisingly-useful one.
There was a pre-trained GPT-2 model offered for download. The whole "interesting thing" they were publishing about, was that models trained under the GPT-2 ML architecture were uniquely-good at transfer learning, and so any pre-trained GPT-2 model of sufficient size, would be extremely useful as a "seed" for doing your own model training on top of.
They built one such model, but that model was not, itself, "GPT-2."
Keep in mind, the training data for that model is open; you can download it yourself and reproduce the offered base-model from it if you like. That's because GPT-2 (the architecture) was formal academic computer science: journal papers and all. The particular pre-trained model, and its input training data, were just published as experimental data.
It is under that lens, that I call GPT-3 "GPT-2++." It's a different model, but it's the same science. The model was never OpenAI's "product." The science itself was/is.
Certainly, the SaaS pre-trained model named "GPT-3" is qualitatively different than the downloadable pre-trained base-model people refer to as "GPT-2." But so are all the various trained models people have built by training GPT-2 the architecture with their own inputs. The whole class of things trained on that architecture are fundamentally all "GPT-2 models." And so "GPT-3" is just one such "GPT-2 model." Just a really big, surprisingly-useful one.