mitchsayre

I now fully believe that the models will soon be compact enough to work even on older mobile devices. I work on lightweight text-to-speech models. After training on distilled datasets the models sound basically the same as any closed-source speech API model, they just need a ton of data to train on. Other researchers are seeing similar gains with other types of models and its only a matter of time before one drops thats commercially viable. Once this happens, the innovative apps and games will begin shipping AI as a feature that drives the user experience forward, rather than the thing you price the entire product around.

Just emailed you but I'll reply here as well in case anyone comes across this thread and finds it useful later.

-TTS: I am actively working on this at Wfloat and just released a 30M param model with 20 voices, emotion, and intensity control that supports running on even legacy 2017 phones. -ASR: I think this is relatively in a good spot, the current ones small enough to fit on-device just mess up more at transcribing -LLM: For sure the main bottleneck. I know a bunch of people are working on this one. The problem with LLMs is just that they have to be so big to actually know how to do anything.

The main thing stopping me from using Proton Meet is I don't like that the booking pages that come with Proton Calendar only show in 24-hour time.

Do you think Pollen is applicable to distributed AI inference? I think it could work for realtlime Voice Agents running directly on mobile hardware.

There are speech-to-speech LLMs that are big and do pure audio in audio out. But you can also make voice agents that use multiple smaller models cascadded. ASR for transcription, LLM for response text, TTS for speech, interrupt detection. If you try to load ASR, LLM, and TTS models that actually do a good job onto the same mobile device all at once, you can't get it to be realtime. But if you run them in a distributed setup, where each device has only one model loaded and streams its output to the next task device, you might achieve realtime performance while using stronger models for each task.

Does this sound possible, or am I misunderstanding how Pollen works?

This is really cool. Do you have any plans to make a version that runs on Android or iOS eventually?