Not really. Kimi has 104B active parameters, so afaik that means you’d only need ~64gb vram to run an 8bit quantization of this, and potentially as little as 32gb to run a 4bit quantization. The rest, the MoE experts, can be in system ram… or if you’re willing to take the speed hit, offloaded to ssd. But yes you’d need a lot of system ram… like 1.5TB.
But that’s all kinda moot because I was referring to smaller open models, ones you could easily run on $1500 of hardware.
But that’s all kinda moot because I was referring to smaller open models, ones you could easily run on $1500 of hardware.