Загрузка видео...

Не удалось загрузить видео

На главную

New week, new tiny model for Pipecat. Pipecat Voice Reader listens to how something is said: tone of voice, energy, laughter, speaking rate and potential sarcasm. It's a fine-tuned version of LAION’s affect-trained Whisper encoder, in two sizes (base or tiny), and runs on CPU in well under 50...

16,466 просмотров • 14 дней назад •via X (Twitter)

Комментарии: 25

Фото профиля Pablo Giménez
Pablo Giménez14 дней назад

The "great, just great" detector, finally 😄 How does sarcasm hold up across languages? Spanish sarcasm is basically a national sport.

Фото профиля Jon Taylor
Jon Taylor14 дней назад

Being a British, game on! 😅 Right now, this is English only. Since it’s so easy and fast to fine-tune these weights, it simply comes down to well annotated data.

Фото профиля Pablo Giménez
Pablo Giménez14 дней назад

British vs Spanish sarcasm is a benchmark I'd pay to watch 😄 And yes, labeling is the hard part. Humans disagree on half the sarcastic lines too.

Фото профиля AJ
AJ14 дней назад

@JonPTaylor Wonder if sarcasm is universal or if it differs between languages

Фото профиля Pablo Giménez
Pablo Giménez14 дней назад

Differs a lot. In Spanish it often rides on a stretched vowel ("claaaro"), in English more on a flat, deadpan pitch. Same intent, different melody. A model trained on one will miss it in the other.

Фото профиля AJ
AJ14 дней назад

Claaaaaro 😁 I remember this research from Hume finding the sounds people use to express emotions are largely universal (with some cultural differences)

Фото профиля Pablo Giménez
Pablo Giménez14 дней назад

Makes sense. The sounds travel (a laugh, a sigh), the melody on the words doesn't. That's why non-verbal tags port across languages and prosody has to be learned per language.

Фото профиля Sebastian Sosa
Sebastian Sosa14 дней назад

this is the right shape: a tiny model reading how it's said beside the one deciding what to say. ours does a read in ~0.2s per turn. next to an agent that's nothing, inside a live conversation it's the difference between answering the words and answering the person

Фото профиля Callan Hardin
Callan Hardin14 дней назад

Tone on the CPU is a great fit for StarStranded, my Godot game where NPCs run on an offline local model. Voice input already uses whisper.cpp base.en on the CPU since the GPU is busy with the NPC model. Does the tiny Voice Reader run alongside it on the same audio chunks?

Фото профиля Jon Taylor
Jon Taylor14 дней назад

It could do, yes. The purpose of keeping these models small is on-device consideration. We have an Unreal engine demo in the Pipecat org that demos an approach - I'll have to update it to use this model too!

Фото профиля Callan Hardin
Callan Hardin14 дней назад

An Unreal demo is a good sign it can live inside a game loop. My game is in Godot, so I'd wire it in as a sidecar next to llama-server and whisper.cpp. Does the demo run the model in-process, or as a separate local service the engine talks to?

Фото профиля Hazem Hegazy
Hazem Hegazy14 дней назад

Is it only for emotions or it can detect other aspects as well ?

Фото профиля Bagzhan Karl
Bagzhan Karl13 дней назад

I like keeping the voice reading beside the text reading. If the words sound positive but the delivery doesn’t, that disagreement seems worth preserving rather than collapsing it into one label.

Фото профиля Bin Chen
Bin Chen14 дней назад

That's a great innovation. Besides the text, the context and the mood of the speaker is super important for AI to decide the next step. Inspiring!

Фото профиля Base RSS
Base RSS14 дней назад

Cascaded pipelines often add more latency than speech-to-speech. The edge depends more on your latency budget than the architecture.

Фото профиля Mohammed Choukairi
Mohammed Choukairi14 дней назад

This is getting good

Фото профиля ML Steve
ML Steve14 дней назад

Tiny models are terribly underrated.

Фото профиля BreezeOg
BreezeOg14 дней назад

tiny model catching sarcasm from tone

Фото профиля Alano
Alano14 дней назад

Sarcasm is context-dependent, so uncertainty is worth preserving in the output.

Фото профиля John Kovacs
John Kovacs14 дней назад

under 50ms on cpu is what makes this usable. the part i like is you get the tone reading as something you can log per turn, so when a call goes sideways you can actually see the turn where the user's energy dropped. with speech to speech that's buried inside the model

Фото профиля Khushal
Khushal14 дней назад

finally something that can tell when i say 'no worries' and mean worries

Фото профиля Jon Taylor
Jon Taylor14 дней назад

So many worries. We also need to train a “passive aggressive” head on all these models too 😅

Фото профиля Khushal
Khushal14 дней назад

a 'per my last email' detector would sell itself

Фото профиля Omkar Ray
Omkar Ray12 дней назад

tone is the thing almost every voice eval skips right now, so a small model for it is great. does it hold up when people switch languages mid sentence, like hindi and english together?

Фото профиля sudhir
sudhir14 дней назад

we are in the golden age of voice tech for sure

Похожие видео

Voice AI turn taking is a solved problem. The single most common complaint about voice AI, today, is that agents interrupt too often. But the voice agents I build for myself now respond quickly and interrupt me less often than the people I talk to every day. (I actually measured this.) Mark Backman made a Pipecat AI PR two weeks ago that was the last piece of the puzzle for turn taking so good that I no longer ever think about it. The approach combines three layers of processing: 1. Voice activity detection, with a short (200ms) trigger. 2. A native audio turn detection model that's small, fast, and runs on CPU. This model captures audio nuances like inflection and filler sounds that don't get transcribed. 3. A prompt mixin for the conversation LLM that decides turn completion based on conversation context. None of these are new. We've been using VAD for a long time. We trained the first version of the Pipecat Smart Turn native audio model in December 2024. And we've been experimenting with prompt-based large model turn detection (sometimes called "selective refusal") for more than a year. Now, the Smart Turn model and the SOTA LLMs we're using in voice agents have both gotten so good that using them together feels like we've finally "solved" turn detection. Mark also figured out how to elegantly apply a "single-token tagging" technique to this problem. We sometimes use single-token tagging in place of tool calling, when we need a near-zero latency programmatic trigger. Mark's Pipecat mixin defines three single-token characters and prompts the LLM to output exactly one of them at the beginning of every response. - ✓ means the agent should respond normally (immediately) - ○ is a "short incomplete" - the agent should wait 5 seconds - ◐ is a "long incomplete" - the agent should wait 10 seconds The wait times, and the details of the prompt, are configurable, of course. Watch the video to see me talk to an agent that handles all my various pauses and inflections, plus phrases like "let me think," pretty much the way a person would handle them, in terms of response latency. Also, in the second half of the video, I ask the agent to adjust its response pattern because I'm going to tell it a phone number. This kind of "in-context" adjustment of response wait times is really useful. The LLM in the video is GTP-4.1. We've tested the prompt and single-token adherance with GPT-4.1, Gemini 2.5 Flash, Anthropic Claude Sonnet 4.5, and AWS Nova 2 Pro. Note that older models in all these families (and, in general, smaller open weights models) aren't able to reliably output these single-token tags. But the new models we're using these days are pretty amazing.

kwindla

27,086 просмотров • 9 месяцев назад

This is the ChatGPT moment for Voice AI. A ten-second sample can show that a synthetic voice sounds natural. But it tells little about whether that voice can stay coherent across an entire conversation. Real callers ask follow-up questions, change their requirements, and interrupt before the agent finishes speaking. The voice must respond quickly without resetting its tone and pacing after every turn. This creates two memory problems. - The LLM needs to remember what the caller said. - The speech model needs to retain how the agent has been speaking. A transcript handles the first problem, but not the second. It preserves the words, not their delivery or how much of the previous response the caller actually heard. Deepgram's Flux TTS handles this by maintaining acoustic state inside a persistent streaming session. Each response begins with the speech history from earlier turns, and the first audio arrives in under 200 ms. Flux also handles interruptions through playback tracking. The application sends the point where playback stopped, and the API reports which words the caller heard. That gives the LLM an accurate conversation history for its next response. In a blind listening evaluation across roughly 14,400 paired comparisons, Flux's best voice recorded a 73.4% naturalness win rate. Cartesia Sonic 3.5 recorded 72.0%. Build with Flux TTS here: I recorded a conversation with a car dealership agent about hybrid and electric cars. I changed my requirements and asked follow-up questions to hear how the voice behaved across the full exchange. The recording is below, and I worked with the Deepgram team on this post to put it together.

Avi Chawla

42,683 просмотров • 1 месяц назад