When trying to deploy llms in with larger context windows constrained environments 2 things start to hurt:
a) increased memory footprint for longer KV cache
b) increased decode speed due to longer context window.
this paper addresses a) only, which is useful, but we are still left with b) (right?)
When trying to deploy llms in with larger context windows constrained environments 2 things start to hurt: a) increased memory footprint for longer KV cache b) increased decode speed due to longer context window. this paper addresses a) only, which is useful, but we are still left with b) (right?)