Comment by RandyOrion

For people who want to quantize kv caches for longer context, basically, if the model checkpoint you use didn't trained to use quantized kv caches (and all most all of model checkpoints you can access didn't), you shouldn't quantize kv caches during inference. A major benefit to do quantization-aware training/distillation on official checkpoints is to let the model to be familiar with quantized activations/weights/kv caches.