Comment by robrenaud

Are there inference products focused on say, optimizing frequently used internal contexts, so that they are compact in the kv cache? Perhaps there rep could be optimized over hundreds+ of distinct real use cases?

Maybe your law firm doesn't want your data going outside your business. But you obviously want high quality analysis.