pcvetkovski

Sub-50ms on text-to-speech running on an LLM is commendable.

We recently shipped text-to-speech and speech-to-text support inside Finsight (Maxint). We tapped into the platform’s native speech capabilities, which were integrated with the user’s preferred LLM inference endpoint (including local on-device models). This approach enabled us to eliminate latency and preserve user’s privacy, delivering human-like conversational experience.

Since your approach involves running the model directly, did you run into memory bandwidth contention or audio buffer underruns during bursty generation, when both the LLM and TTS models are executing concurrently?

How does your VM handle local state and performance, when the user attempts to fetch, join and render data from multiple endpoints?

We use Dart and Go (front/backend) to allow users to render widgets/mini-apps on Maxint (Finsight). And hence, the speed (virtually instant) and overall quality of the output mostly depends on the LLM/inference endpoint that they choose.

In general, most users are happy with the existing features, however more advanced users have a playground to ship enhancements on-the-fly.