Comment Comment by carlos_rpn You can run a 4bit Qwen3.8 27B on a 8GB GPU, though on my RX 6650 XT it will only reach about 3 tokens/s.If you can give the GPU units enough RAM to load the whole thing, it may be a little faster.I'm not sure why would you want to run something at 3-4 tokens/s though. At least try a MoE model. I get about 30 tokens/s from those. parent carlos_rpn 2 months ago
If you can give the GPU units enough RAM to load the whole thing, it may be a little faster.
I'm not sure why would you want to run something at 3-4 tokens/s though. At least try a MoE model. I get about 30 tokens/s from those.