The general rule of thumb that I'm familiar with is that you need about 80 bytes of VRAM per parameter when you are doing training. Inference is different and a lot more efficient, and LoRA is also different and more efficient, but training a base model requires a LOT of memory.
A machine like this would top out below 2 trillion parameters using the training algorithms that I'm familiar with.
A machine like this would top out below 2 trillion parameters using the training algorithms that I'm familiar with.