
Jon Taylor
@JonPTaylor • 1,759 subscribers
🧑💻 Voice AI @trydaily // ᓚᘏᗢ @pipecat_ai 🎹 I have too many synths
Videos

Pipecat 1.16 adds PerceptionReader, something I'm really excited about. How someone speaks can matter just as much as what they say, and voice agents have mostly been missing that. We’ve been testing it with Google Decision 1, released yesterday. Stream it audio and you get a pretty accurate read on the prosody, tone and sarcasm of what the user said. Try talking to it at this demo here: We've exploring many different ways of measuring expressivity... more to share on that soon! ➡️ Pipecat Docs: ➡️ Google Decision 1:
Jon Taylor17,381 views • 5 days ago

New week, new tiny model for Pipecat. Pipecat Voice Reader listens to how something is said: tone of voice, energy, laughter, speaking rate and potential sarcasm. It's a fine-tuned version of LAION’s affect-trained Whisper encoder, in two sizes (base or tiny), and runs on CPU in well under 50 ms per turn. In a Pipecat pipeline it runs beside the text tone reader, which reads what was said in the context of the previous exchange, providing broader context to the harness and metering. This is one area where a cascaded pipeline has a real edge over speech-to-speech... each turn gets explicit, calibrated readings of both the words and the delivery, with context of the single turn and the entire conversation. Available in two weights (base and tiny):
Jon Taylor16,466 views • 14 days ago

Wow, TypeSafe AI Jev is a game changer for voice-driven UI. Responsiveness really matters for an experience to 'feel' right, and this certainly does the job! Latency here is not optimal given I'm in London and calling out to us-west. But still, super snappy. In this video, Jamcat is using Pipecat to orchestrate a real-time handover between a command agent (Jev) that drives the UI, and voice agent (PhoneLLM) for handling generations and (not in the video) discussing ideas and session setup with the user.
Jon Taylor47,943 views • 2 months ago

Phonon from Gradium is an open TTS model in 31M and 40M sizes, perfect for on-device! It’s voice promptable too, via Gradium’s Voice Designer. I love experimenting with voice and mobile. Here is Little Voices, an app where you can ask the angel and devil on your shoulder anything. It's a Pipecat AI app in embedded Python on iPhone. • Apple SpeechAnalyzer for speech-to-text • Apple Foundation Models for the LLM • Gradium Phonon 31M for TTS. • Pipecat running in Python on the iPhone • Pipecat Audience (for who speaks next. It works out whether you’re talking to the angel or the devil.) • No network or inference costs! 📖 Repo here:
Jon Taylor17,407 views • 28 days ago

Weeks on and I'm still finding awesome ways to use TypeSafe AI Jev in Pipecat. This demo is ONE bot pipeline, with a classifier front-running the LLM to determine who the user is talking to. Pipecat then orchestrates a handover between two 'character' workers. Oh, and of course, Opus 5.5 one-shotted this idea. Code might be ugly:
Jon Taylor18,319 views • 1 month ago
No more content to load