Im less interested in the memory than the memory bandwidth. The current system with 128GB can load pretty big models, but its meaningless unless you want to wait 40 minutes per prompt.
I've had best results with Qwen3.6-35B-A3B, which uses 40GB of memory, but only uses 3 billion parameters per token which helps with throughput.
Until memory bandwidth significantly improves I just can't see myself wanting to use all that memory. Unless it's just to keep a wide variety of models in memory.
I've had best results with Qwen3.6-35B-A3B, which uses 40GB of memory, but only uses 3 billion parameters per token which helps with throughput.
Until memory bandwidth significantly improves I just can't see myself wanting to use all that memory. Unless it's just to keep a wide variety of models in memory.