Sub-50ms on text-to-speech running on an LLM is commendable.
We recently shipped text-to-speech and speech-to-text support inside Finsight (Maxint). We tapped into the platform’s native speech capabilities, which were integrated with the user’s preferred LLM inference endpoint (including local on-device models). This approach enabled us to eliminate latency and preserve user’s privacy, delivering human-like conversational experience.
Since your approach involves running the model directly, did you run into memory bandwidth contention or audio buffer underruns during bursty generation, when both the LLM and TTS models are executing concurrently?
We recently shipped text-to-speech and speech-to-text support inside Finsight (Maxint). We tapped into the platform’s native speech capabilities, which were integrated with the user’s preferred LLM inference endpoint (including local on-device models). This approach enabled us to eliminate latency and preserve user’s privacy, delivering human-like conversational experience.
Since your approach involves running the model directly, did you run into memory bandwidth contention or audio buffer underruns during bursty generation, when both the LLM and TTS models are executing concurrently?