IMHO Qwen are shipping the best OSS models you can run locally on consumer GPUs right now, getting great results from both qwen2.5-coder:32b and qwq:32b running at ~18 tok/s and running on older NVidia A4000 GPUs. Definitely my first choices for local workloads.
It's also great to see qwq's open chain of thought baked in a OSS LLM so you can see it reason with itself in real-time, it's the kind of secret sauce that proprietary LLMs like o1 would prefer to keep hidden to try build a moat around.
We've got a lot to thank Meta and Qwen for in continually releasing improving high quality OSS models which also encourages others to follow. High quality OSS models are the best thing keeping the cost of LLMs down, you can get unbelievable value on OpenRouter with qwen2.5-coder:32b at $0.08/$0.18 M/tok qwq:32 available at $0.15/$0.60 M/tok which is more than 18x cheaper than Anthropic's latest budget Haiku 3.5 model at $0.80/$4 M/tok (4x price hike over Haiku 3.0).
It's also great to see qwq's open chain of thought baked in a OSS LLM so you can see it reason with itself in real-time, it's the kind of secret sauce that proprietary LLMs like o1 would prefer to keep hidden to try build a moat around.
We've got a lot to thank Meta and Qwen for in continually releasing improving high quality OSS models which also encourages others to follow. High quality OSS models are the best thing keeping the cost of LLMs down, you can get unbelievable value on OpenRouter with qwen2.5-coder:32b at $0.08/$0.18 M/tok qwq:32 available at $0.15/$0.60 M/tok which is more than 18x cheaper than Anthropic's latest budget Haiku 3.5 model at $0.80/$4 M/tok (4x price hike over Haiku 3.0).