koljab

Didn't find many coqui finetunes too so far. I have David Attenborough and Snoop Dogg finetunes on my huggingface, quality is medium.

Coqui can to 17 languages. The problem with RealtimeVoiceChat repo is turn detection, the model I use to determine if a partial sentence indicates turn change is trained on english corpus only.

Yes, you're absolutely right. I'll provide UV and conda support soon, especially for Windows. I'm using python 3.10 still, maybe that's the issue. You can always mail me your current problem or log an issue, I really care about opening up the repo for as many ppl to use as possible.

That would be absolutely awesome. But I doubt it, since they released a shitty version of that amazing thing they put online. I feel they aren't planning to give us their top model soon.

Maybe possible, I did not look into that much for Coqui XTTS. What i know is that the quantized versions for Orpheus sound noticably worse. I feel audio models are quite sensitive to quantization.

LLM and TTS latency get's determined and logged at the start. It's around 220ms for the LLM returning the first synthesizable sentence fragment (depending on the length of the fragment, which is usually something between 3 and 10 words). Then around 80ms of TTS until the first audio chunk is delivered. STT with base.en you can neglect, it's under 5 ms, VAD same. Turn detection model also adds around 20 ms. I have zero clue if and how fast this runs on a Mac.

With the current 24b LLM model it's 24 GB. I have no clue how far down you can go with the GPU is using smaller models, you can set the model in server.py. Quite sure 16 GB will work but at some point it will probably fail.

Did not look into that one. Looks quite good, I will try that soon.

Yes, I tested it. I'm not that sure what they created there. It adds some noticable latency compared towards using raw websockets. Imho it's not supposed to, but it did it nevertheless in my tests.

That would be quite easy to integrate. RealtimeSTT already has wakeword support for both pvporcupine and openwakewords.

It's not aware. The information that it had been interrupted would be something we can easily add to the next user chat request. Where exactly is harder, because at least for Coqui XTTSv2 we don't have TTS wordstamps (we do have them for Kokoro though). So adding the information where it had been interrupted would be easily possible when using Kokoro as TTS system. With Coqui we'd need to add another transcription on the tts output including word timestamps. That would cost more compute than a normal transcription and word timestamps aren't perfectly accurate. Yet directly after an interruption there's not that much concurrent need for compute (like in the end of turn detection phase where a lot of stuff is happening). So I guess with a bit of programming work this could be integrated.

It's in fact using Silero via RealtimeSTT. RealtimeSTT tells when silence starts. Then a binary sentence classification model is used on the realtime transcription text which infers blazingly fast (10ms) and returns a probability between 0 and 1 indicating if the current spoken sentence is considered "complete". The turn detection component takes this information to calculate the silence waiting time until "turn is over".

This character prompt has undergone so many iterations with LLMs it's not funny anymore. "Make her act more bold." - "She again talked about her character description, prevent that!"

I doubt TTS will be fast enough for realtime without a Nvidia GPU