I don't believe that quantization comes for free. Someone made the observation that llama3 models quantize "worse" than llama2 models do -- they suffer in quality from quantization far more.
My intuition is that a model which is undertrained suffers less from quantization, because the training process has not utilized each weight to its full potential. One of the key findings with llama, and why it punches above its weight for its size, is that they trained it for longer on a much larger dataset then was "optimal" according to the literature up to that point.
Putting two and two together, it seems that:
small model, lots of data, long training > large model + quantization
That basically, quantization is a lossy shortcut to the tail of training long. Amount and quality of data is, as always, the most important part about all of this.
My intuition is that a model which is undertrained suffers less from quantization, because the training process has not utilized each weight to its full potential. One of the key findings with llama, and why it punches above its weight for its size, is that they trained it for longer on a much larger dataset then was "optimal" according to the literature up to that point.
Putting two and two together, it seems that:
small model, lots of data, long training > large model + quantization
That basically, quantization is a lossy shortcut to the tail of training long. Amount and quality of data is, as always, the most important part about all of this.