Comment by trucks-refinish

I would give it significantly more context, if you don't have enough room you can quantize the kv cache with some flags that I don't remember ottomh.

I would go ahead and give it at least 64k but the more the better really!

I run the kv cache at q8 for keys and q4 for values, but going below q8 is generally not recommended.