Comment Comment by charcircuit >and shouldn't charge more and more the longer the conversation getsThe cost of generating token N is O(N) with KV cache so it's unrealistic to expect to not be charged more the longer the conversation is if you are looking for the minimum price. parent charcircuit 3 months ago
The cost of generating token N is O(N) with KV cache so it's unrealistic to expect to not be charged more the longer the conversation is if you are looking for the minimum price.