Comment by djoldman

"I can't wait to run this new sota model locally. I'll just use the quantized version that is certain to be better than [other model I'm running]."

This is fast becoming one of my top old-man-yells-at-clouds pet peeves.

Reported performance metrics are ONLY good for the exact model weights.

Quantizing a model, or changing it in any way, requires new evaluation to know how well it performs.

Quantizing a good model doesn't mean the quantized version is good.