ioedward

8 H100s would have enough VRAM to finetune a 70B model.

Not at all, PyTorch uses sub-classing (nn.Module) and other Pythonic features heavily that are not supported with Mojo.

Trying the fuse feature, seemed the most interesting:

> spaghetti ≈ trumpet

> Both spaghetti and a trumpet can be difficult to eat without making a mess—spaghetti with its long, slippery noodles, and a trumpet with its wide, flared bell.

ok google

The technical term is GPGPU, general-purpose computing on GPUs, but I like to call them GPUs for short.

Nvidia's enterprise GPUs are surprisingly unreliable. Working on a 128 GPU A100 cluster on AWS, 1 would fail every few days. I didn't have any insight on whether it was a hardware or software failure.

You also need the optimizer (e.g. Adam)'s state, which is usually double the parameter's size. So if using fp16, one parameter takes up 6 bytes in memory.

TPUs are mostly hoarded by Google Research (including Deepmind) and Ads. Very few are being used by external people.

The twitter thread is worth reading, Yann responds to some of the questions raised here.

No it won't. Large language models are trained on 1,000 - 50,000 GPUs. No one's going to buy hundreds of Mac pros to mount them in a datacenter for training ML models.

Normally people split up the model across multiple GPUs, i.e. model/tensor parallelism.