Comment by yayr

each billion parameters using 16 bit floats requires around 2 GB of GPU or TPU RAM. ChatGPT is expected to have around 1000 billion. Good open source LLMs have around 7-20 billion currently. Consumer GPUs currently max out at 24 GB. You can now quantize the model to e.g. 4 bits instead of 32 per parameter and do other compressions, but still there is quite a limit what you can do with 24 GB of RAM. The Apple unified memory approach may be a path forward to increase that... so one box gives you access to the small models, for a GPT4 like model you'd need (for inference and if you had the model and tools) probably 100 of those 4090s or 25 of H100 with 96 GBs I guess to fit in 2 TB of model data.