Perplexity looks bad and unsloth just did some good quants but they don’t perform as good on my asymmetric old dual 3090 box.
However, actual quality evals show literally zero degradation in “hard math” questions against full hosted model at a cost of 1.5x - 2x CoT tokens. I expect other quants to suffer from this too. This behavior seems to decrease as precision increases. I also suspect it’s the specific tuning a what experts I’m picking.
Perplexity looks bad and unsloth just did some good quants but they don’t perform as good on my asymmetric old dual 3090 box.
However, actual quality evals show literally zero degradation in “hard math” questions against full hosted model at a cost of 1.5x - 2x CoT tokens. I expect other quants to suffer from this too. This behavior seems to decrease as precision increases. I also suspect it’s the specific tuning a what experts I’m picking.