Thank you! We have built out the cache system -- we do both simple caching (matching the request strings 100%) and also do semantic caching (returning a cache hit for semantically similar requests). More here - https://portkey.ai/docs/product/ai-gateway-streamline-llm-in...
The caching part isn't open source yet, but part of our internal workers. Would be very cool to open source it!
The caching part isn't open source yet, but part of our internal workers. Would be very cool to open source it!