GPU bandwidth limits the amount of tokens/sec that you get.
I 've mostly been running QWEN-3.6-35B-A3B at Q8 and QWEN-3.5-122B-A10B at Q4 with Q8 kv-cache on llama.cpp (use vulkan version). Also use MTP GGUF's you get ~ 50% speed up when predicting 2 or 3 tokens.
Haven't tested the new laguna S 2.1 yet.
Another thing I liked was that I had enough ram to also run an embedding open model for LLM Wiki apps.
Overall I think it's cheaper and better (intelligence wise) to get a $20 codex sub and always run gpt-5.6-luna than getting a desktop for local inference. Unless you have explicit needs for it (whether it is just experimentation or you have personal things you 'd rather keep on your computers or w/e). OpenCode go at $10 is also a great option.
GPU bandwidth limits the amount of tokens/sec that you get.
I 've mostly been running QWEN-3.6-35B-A3B at Q8 and QWEN-3.5-122B-A10B at Q4 with Q8 kv-cache on llama.cpp (use vulkan version). Also use MTP GGUF's you get ~ 50% speed up when predicting 2 or 3 tokens.
Haven't tested the new laguna S 2.1 yet.
Another thing I liked was that I had enough ram to also run an embedding open model for LLM Wiki apps.
Overall I think it's cheaper and better (intelligence wise) to get a $20 codex sub and always run gpt-5.6-luna than getting a desktop for local inference. Unless you have explicit needs for it (whether it is just experimentation or you have personal things you 'd rather keep on your computers or w/e). OpenCode go at $10 is also a great option.