Congrats, this looks neat, and surely great to have more TS products in the ecosystem.
One plugin or feature that I will like to see in an AI gateway:
*Cache* per unique request.
So if I send the same request (system, messages, temperature, etc.), I will have the option to pull if from a cache (if it was already populated) and skip the LLM generation. This is much faster and cheaper - especially during development and testing.
One plugin or feature that I will like to see in an AI gateway: *Cache* per unique request. So if I send the same request (system, messages, temperature, etc.), I will have the option to pull if from a cache (if it was already populated) and skip the LLM generation. This is much faster and cheaper - especially during development and testing.