Comment Comment by gentile OP could be doing a lot better with a MoE model (uses some RAM as well).I'm doing Qwen 3.6 35BA3B-NVFP4 (a larger ~20GB model) in 8GB VRAM and 20 GB RAM (I also get more than 16k context, and more t/s). parent gentile 25 days ago
I'm doing Qwen 3.6 35BA3B-NVFP4 (a larger ~20GB model) in 8GB VRAM and 20 GB RAM (I also get more than 16k context, and more t/s).