That sounds really quite interesting- one question pops up, this sounds like it could produce a lot of non-cache input tokens. How much impact does this have, and are you mitigating it?
Interested in adding support to my own 3code- https://3code.capocasa.dev- which aims to reduce cost by using more compact system prompts and a clever compaction variant.
Interested in adding support to my own 3code- https://3code.capocasa.dev- which aims to reduce cost by using more compact system prompts and a clever compaction variant.