> This means the ability to train 50 GPT-4 scale models every 90 days or 200 such models per year.
Not anywhere close to that.
Those 350k GPUs you talk about aren't linked together. They also definitely aren't all H100s.
To train a GPT-4 scale model you need a single cluster, where all the GPUs are tightly linked together. At the scale of 20k+ GPUs, the price you pay in networking to link those GPUs is basically almost the same as the price of those GPUs themselves. It's really hard and expensive to do.
FB has maybe 2 such clusters, not more than that. And I'm somewhat confident one of those cluster is an A100 cluster.
Not anywhere close to that.
Those 350k GPUs you talk about aren't linked together. They also definitely aren't all H100s.
To train a GPT-4 scale model you need a single cluster, where all the GPUs are tightly linked together. At the scale of 20k+ GPUs, the price you pay in networking to link those GPUs is basically almost the same as the price of those GPUs themselves. It's really hard and expensive to do.
FB has maybe 2 such clusters, not more than that. And I'm somewhat confident one of those cluster is an A100 cluster.
So they can train maybe 6 GPT-4 every 90 days.