You have to understand that Strata and HyperQwen are giving you good looking tokens per second but are not giving you anything close to the bf16 model’s capability (they quantize, worse they quantize KV caches, they approximate expert routing etc.)
The models are so impressive that getting 70-80% of their capability can be helpful, but beware, you are using a significantly dumber model.
The models are so impressive that getting 70-80% of their capability can be helpful, but beware, you are using a significantly dumber model.