Comment by Eisenstein

We are talking about dynamically quantizing KV cache, not the model weights.