Comment Comment by tgrowazay LLM speed is roughly <memory_bandwidth> / <model_size> tok/s.DDR4 tops out about 27GbsDDR5 can do around 40GbsSo for 70B model at 8 bit quant, you will get around 0.3-0.5 tokens per second using RAM alone. parent tgrowazay 8 months ago
DDR4 tops out about 27Gbs
DDR5 can do around 40Gbs
So for 70B model at 8 bit quant, you will get around 0.3-0.5 tokens per second using RAM alone.