I was wondering how feasible it is to modify the context, instead of just appending - e.g. to remove malformed tool calls and their error messages that would pollute the context otherwise.
Sure, you'd have to rebuild the KV cache and take a performance loss, but that might be acceptable if it happens rarely enough.
I was wondering how feasible it is to modify the context, instead of just appending - e.g. to remove malformed tool calls and their error messages that would pollute the context otherwise.
Sure, you'd have to rebuild the KV cache and take a performance loss, but that might be acceptable if it happens rarely enough.