
Jon Taylor
@JonPTaylor • 1,759 subscribers
🧑💻 Voice AI @trydaily // ᓚᘏᗢ @pipecat_ai 🎹 I have too many synths
Videos

Pipecat 1.16 adds PerceptionReader, something I'm really excited about. How someone speaks can matter just as much as what they say, and voice agents have mostly been missing that. We’ve been testing it with Google Decision 1, released yesterday. Stream it audio and you get a pretty accurate read on the prosody, tone and sarcasm of what the user said. Try talking to it at this demo here: We've exploring many different ways of measuring expressivity... more to share on that soon! ➡️ Pipecat Docs: ➡️ Google Decision 1:
Jon Taylor17,381 Aufrufe • vor 5 Tagen

New week, new tiny model for Pipecat. Pipecat Voice Reader listens to how something is said: tone of voice, energy, laughter, speaking rate and potential sarcasm. It's a fine-tuned version of LAION’s affect-trained Whisper encoder, in two sizes (base or tiny), and runs on CPU in well under 50 ms per turn. In a Pipecat pipeline it runs beside the text tone reader, which reads what was said in the context of the previous exchange, providing broader context to the harness and metering. This is one area where a cascaded pipeline has a real edge over speech-to-speech... each turn gets explicit, calibrated readings of both the words and the delivery, with context of the single turn and the entire conversation. Available in two weights (base and tiny):
Jon Taylor16,466 Aufrufe • vor 14 Tagen

Wow, TypeSafe AI Jev is a game changer for voice-driven UI. Responsiveness really matters for an experience to 'feel' right, and this certainly does the job! Latency here is not optimal given I'm in London and calling out to us-west. But still, super snappy. In this video, Jamcat is using Pipecat to orchestrate a real-time handover between a command agent (Jev) that drives the UI, and voice agent (PhoneLLM) for handling generations and (not in the video) discussing ideas and session setup with the user.
Jon Taylor47,943 Aufrufe • vor 2 Monaten

Phonon from Gradium is an open TTS model in 31M and 40M sizes, perfect for on-device! It’s voice promptable too, via Gradium’s Voice Designer. I love experimenting with voice and mobile. Here is Little Voices, an app where you can ask the angel and devil on your shoulder anything. It's a Pipecat AI app in embedded Python on iPhone. • Apple SpeechAnalyzer for speech-to-text • Apple Foundation Models for the LLM • Gradium Phonon 31M for TTS. • Pipecat running in Python on the iPhone • Pipecat Audience (for who speaks next. It works out whether you’re talking to the angel or the devil.) • No network or inference costs! 📖 Repo here:
Jon Taylor17,407 Aufrufe • vor 28 Tagen

Weeks on and I'm still finding awesome ways to use TypeSafe AI Jev in Pipecat. This demo is ONE bot pipeline, with a classifier front-running the LLM to determine who the user is talking to. Pipecat then orchestrates a handover between two 'character' workers. Oh, and of course, Opus 5.5 one-shotted this idea. Code might be ugly:
Jon Taylor18,319 Aufrufe • vor 1 Monat

Here is an example Pipecat project for trying PhoneLLM Alpha 1. Deploy the model to a Modal endpoint with one click. Client from the video in repo too (with all that sweet sweet terminal-ish aura.) Next up, a TTS that can pronounce Merve Tan's name correctly? 😅
Jon Taylor21,122 Aufrufe • vor 2 Monaten
Keine weiteren Inhalte verfügbar