Comment by aikin-nivedit

Bunch of optimisations for large data, including: Smart Batching, Prompt Caching, Custom Routing Algorithms and hardware planning for batch.