nicktikhonov

Thanks! let me know how you find it, I'm looking to make useful improvements daily.

From what I've seen, it's really easy to get PersonaPlex stuck in a death spiral - talking to itself, stuttering and descending deeper and deeper into total nonsense. Useless for any production use case. But I think this kind of end-to-end model is needed to correctly model conversations. STT/TTS compresses a lot of information - tone, timing, emotion out of the input data to the model, so it seems obvious that the results will always be somewhat robotic. Excited to see the next iteration of these models!

Yep. Seems like caching more broadly is something worth exploring next if I were to do a pt2.

Yep. I've been learning Chinese for the past 3 months, so the name was a fold-in inspiration from my other hobby :)

Glad to hear! I built my blog on top of NextJS - it basically just renders .mdx files with contentlayer. One of the things I discovered is that you can easily vibe-code these explainer widgets. Seems like a perfect use case for vibe coding - each is a simple react component and I can keep iterating until I get it working just the way I like. And super easy to interleave with content. Seems like this could be an obvious feature addition to all the blogging platforms.

You're probably right, at least at scale this could help

One thing you can get the LLM to do is to call a "skip turn" tool, which will basically trigger the system to wait without saying anything. Then all it will take is clever prompting to get the desired result.

I feel like you could get pretty far with a raspberry pi and microphone/speaker. I think the hard part is running a model that can detect a "Hey agent" on-device, so that it can run 24/7 and hand off to the orchestrator when it catches a real question/query.

This is fascinating, thanks for sharing! I wonder why amazon/google/apple didn't hop on the voice assistant/agent train in the last few years. All 3 have existing products with existing users and can pretty much define and capture the category with a single over-the-air update.

I'd say it was a collaboration. I had to hand-hold Claude quite a bit in the early stages, especially with architecture, and find the right services to get the outcome I wanted. But if you care most about where the code came from - it was probably 85-90% LLM, and that's fantastic given that the result is as performant as anything you'll be able to find out of the box.

100% - I thought about that shortly after writing this up. One way to make this work is to have a tiny, lower latency model generate that first reply out of a set of options, then aggressively cache TTS responses to get the latency super low. Responses like "Hmm, let me think about that..." would be served within milliseconds.

Gross

Very cool! starred and on my reading list. Would love to chat and share notes, if you'd like