Comment

Comment by verdverm

you cannot do this with direct byte copy, you have to reprocess all input to rebuild the cache in model B, it's exactly like waiting hours to send a new message to an existing session and your cache has expired

tl;dr - you will not get the 90% discount the first time you change models and send a message

this is why I am saying you cannot "transfer" kv cache, it's just like changing models in your favorite harness and that requires reprocessing the input from scratch to build a kv cache for the model

parent verdverm 3 months ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton