Author here. I wanted to start a discussion about where the decision to take the floor should live.
I wrote this article about my attempt to use an external classifier to separate backchanneling and floor-taking intents, but it seems to have too much latency, so it becomes unusable both ways: using built-in VAD makes the experience choppy because any input interrupts the speech, and a custom one takes too much time before it processes the input and interrupt.
So how do you guys solve this problem in production?
(Don't suggest waiting for the `gpt-live-x` family. Waiting is killing.
I wrote this article about my attempt to use an external classifier to separate backchanneling and floor-taking intents, but it seems to have too much latency, so it becomes unusable both ways: using built-in VAD makes the experience choppy because any input interrupts the speech, and a custom one takes too much time before it processes the input and interrupt.
So how do you guys solve this problem in production? (Don't suggest waiting for the `gpt-live-x` family. Waiting is killing.