Video yükleniyor...
Video Yüklenemedi
Voice AI turn taking is a solved problem. The single most common complaint about voice AI, today, is that agents interrupt too often. But the voice agents I build for myself now respond quickly and interrupt me less often than the people I talk to every day. (I actually... show more
27,086 görüntüleme • 9 ay önce •via X (Twitter)
64 Yorum

Docs for this in Pipecat: Mark's original PR: The `UserTurnCompletionLLMServiceMixin` was shipped in Pipecat 0.0.101

Agree the stack is converging. Where we still see failures is accents + disfluencies + overlap in real devices/rooms. That’s why we’ve been focusing at Silencio on collecting in‑the‑wild speech across niche dialects—turn detection gets harder the moment you leave clean US‑English

Interruptions are quite annoying but my biggest gripe is that neither chatgpt nor claude have a voice mode with their most powerful models. I’d rather wait for a good response than get an instant response from a dumb model.

Agreed. I pretty much always use Gemini 2.5 Flash, GPT-4.1, or Claude 4.5 Sonnet for both my own agents and for enterprise voice AI use cases. The models need to be able to keep track of multi-turn context, do complex instruction following, and make tool calls reliably.

@hrishio Why not gemini flash 3 preview, out of interest?

@hrishio Right now the Gemini 3 Flash TTFT is ~1200ms. Gemini 2.5 Flash is half that (twice as fast). The latency difference really matters.

@hrishio Fair enough. Didn't realise it was that high. How about groq models if ttft is that important?

Groq is now owned by NVIDIA, so we'll see if new models land on Groq. We generally need to use SOTA models for most voice use cases, because accurate instruction following and tool calling are so important. There aren't any open weights models that are good enough at both open-ended conversation and tool calling. I'm excited about the upcoming release of Nemotron 3 Super that NVIDIA has said is coming soon, though!

Yeah, the interruption problem is mostly engineering now: solid VAD + endpointer tuning, plus a “barge-in budget” so it errs on letting the user finish. The real differentiator is how gracefully it recovers when it guessed wrong without making the convo feel brittle.

> Yeah, the interruption problem is mostly engineering now I think there are a *lot* of things in the AI agents space like this, now: hard problems that are tractable with good, pragmatic, engineering work. The models are going to keep getting better and keep subsuming some of this framework-level work down into just "what the models do." But that takes time, and even as the models get better we generally want to do more than any generation of models can do natively. I'll be very happy when speech-to-speech and STT models handle turn detection entirely internally. But we don't need to wait for that to happen to build *great* voice agents!

Incredible work! Pipecat is leading the way here

This is pretty impressive! I was thinking about using a semantic vad for reachy mini but didn’t find anything easy to port, this seems like it would help 🙌

The Smart Turn native audio model has gotten quite good, and the latest versions are now small enough to run on CPU in pretty much any environment. All of the training data is open source (and the training code, etc). We're also always working on collecting more human data and improving the synthetic data pipeline for the Smart Turn model. (We augment human data with synthetic speech data.) Happy to work on that together if there are gaps you would like to see filled in the training sets!

Just looked into this, quite cool! So you use it on top of silero ? Happy to collab as well, we won’t gather data with reachy but we have compute and expertise 🤗

Definitely didn't mean that we should gather any data from end users! We're really, really careful about the data in the Smart Turn data sets being clean and properly acquired. We work with a number of the data collection/labeling companies to gather data for specific languages/accents/scenarios, etc. They contribute because of the marketing value of helping a high-profile, useful, open source model. (And sometimes we pay small amounts to fill data set gaps.) The typical configuration of Smart Turn is to segment the audio with Silero set to `min_silence_duration_ms = 200` (~7 non-speech frames). The Smart Turn model sees up to 16 seconds of audio from the current user turn, including the 200ms tail before the Silero EOS event. We trained the model to overfit to a relatively bi-modal COMPLETE/INCOMPLETE response distribution. Unlike some of the other (good) work in this domain from Krisp, Soniox, Deepgram, and others, we do not use the silence duration as input to the training, or try to produce a useful confidence score in the "mid range." The theory is that turn detection is a data poor domain, and the main job of this model is to gate inference calls to a big, expensive LLM. So we're trying not to make a tiny model running on CPU do impossible things. But the trade-offs here are really interesting, and it's fun to think about how to improve this kind of model in various ways.

Wonderful work. Have you tried using a separate smaller LLM for turn completion detection (i.e., outputting the single token) to lower latency and prevent drifting the main LLM’s attention to turn completion detection (instead of say a hard reasoning task in the user message)?

Yes, we did this with parallel inference pipelines in the past. So you always run the main voice inference but gate output on another model running in parallel making the turn detection decision. There are two issues with doing it that way: 1) you basically need the big model anyway, because small models aren't as good at this classification as you'd think they would be, and 2) running two inference calls has rate limits implications in larger-scale applications. Now that the SOTA models are good enough to do this as an initial, single-token decision in the context of other complex prompting, we've shifted to just doing it this way.

Thank you, that makes sense! 🙏 Will definitely give pipecat a try soon.

This is amazing work, adding the full weight of the LLM to decide whether to respond or wait is pretty smart. On the other hand, how can we support assistant interrupting the user? This is not useful for voice agents, but for conversational voice chats humans usually respond with 0 latency or even negative latency by interrupting the prior speaker. This seems crucial for a truly natural chat experience. I guess this could be done a similar way to the token tagging approach, except you run it at an interval while the speaker is speaking?

wait you actually measured the interruption rate against real people? that's such a good way to validate it. curious what latency you're hitting to get that natural back-and-forth feel

P50 total voice-to-voice is just under 1500ms. There is a wrinkle I'm still thinking about. The Smart Turn model tends to be conservative about INCOMPLETE classifications if I've paused at least once in the current utterance, so it's a little bit of a bi-modal distribution. ~75% of responses are under 1500ms. 25% are ~3s because of Smart Turn false INCOMPLETE classifications. I'd like to reduce those cases. But, arguably, when I've paused at least once in the middle of an utterance, the model should be more conservative about emitting a COMPLETE. In practice, as long as the model responds very fast for simple utterances, I don't actually notice if the response time is 3s when I've rambled for a while. :-)

This is so cool!

The single-token tag idea is elegant. Latency-aware turn taking is a big deal.

Wow - this is really good!

Everyone is sleeping on Voice AI. It IS the killer use case for engagement with AI in the most positive and productive of ways.

This is so clever

Wow.. amazing demo for turn detection. Is this setup ready for production grade voice agents usage at scale. I’m working on a poc using pipecat, but started getting issues with VAD in later for multi turn conversations

Yes, this setup is being used in production at scale. What are the VAD issues you're having? If it's tied to conversation length, it's definitely something weird, because VAD should basically reset each turn. It could be CPU-related or memory pressure if there's something that runs each turn that doesn't get cleaned up.

woah, its probably the best turn detection I listened to. one quick question tho, you said you used gpt4.1 model, is it also the same for smart turn? As if it is the case it will perform good in multilingual as well!

The Smart Turn model supports 23 languages! And gpt-4.1 and Gemini 2.5 Flash are both very good in a pretty wide variety of languages, in text mode. So for non-English, the biggest thing is picking a transcription (STT) model that is great in your target language.

Given your strategic partnership with vapi, will this soon become available OOB far Vapi voice agents?

@kwindla if someone wants to attach this promptmixin LLM with smart turn v3.2 how would they go about it?

Here's the example I ran to record the video:

I’ve already solved most of these on iOS and with powerful models.

I’ve been using the realtime api from OAI for a cooking assistant ( with decent results. I’m curious how this compares particularly on production. Are you ruling out OAI realtime due to cost or the fact that the underlying model is not good enough?

Definitely not

This works with all SOTA STT (like Soniox ) and TTS ?

Yes!

How is your latency when using GPT-4.1 with tts? I am still not happy with LLM + TTS combinations conversation just don’t sound natural enough compared to realtime models. But multi turn context and tool calling is not reliable enough on voice in voice out models

Hey @sventy, my AI intent-radar just flagged a highly qualified buyer specifically looking for a local-first voice/meeting tool on Hacker News today. Tried to send you the direct link but your DMs are closed. Where can I drop it for you on the house?

cool! is this with webrtc?

Yes! This is the Pipecat SmallWebRTCTransport, which is a zero-dependency, peer-to-peer, WebRTC connection between my browser and the Pipecat agent. Pipecat supports SmallWebRTCTransport, telephony (SIP and PSTN), Daily Global Cloud WebRTC, WebSockets of various kinds, local transport in-process on a machine, and some weirder and more specialized network options, too.

nicee! I will dive deeper into this! I was testing this is really neat!

Is the smart turn taking model multilingual?

23 languages!

whose voice is this? sounds like Sati from three body problem

What I did to record the video was just run the `examples/foundational/22-filter-incomplete-turns.py` single-file example from the Pipecat source code repo. The example uses the "British Reading Lady" voice from @cartesia.

@cartesia Thanks! I played with pipecat about 2 years ago when it was early. things are getting better at an accelerating speed. I thought it was her voice.

Can we use this with Pipecat Flow? I tried to use it, but agent audio output wasn't working for some reason.

Tried very impressive, single word responses don't seem to work though e.g. "yep". Is this a prompting issue or is there another way of solving this?

Amazing. What do you guys use for VAD? Silero?

Yes, generally Silero VAD. Works well in both 16khz and 8khz, proven over time, relatively robust to noisy input, efficient on CPU. And for use cases where background speech is an issue, we usually use the @krispHQ VIVA noise reduction models.

ran 28 voice agents, 5k+ sessions. turn detection got solid but curious about CPU overhead running the native audio model alongside the LLM on concurrent calls. we hit 3.2s latency on voicemail detection at 70-150 calls/day, separate classifier helped

The Smart Turn v3.2 model is very efficient on CPU. We don't have any issues in production. 3.2s latency on voice mail detection sounds high, but of course it depends on exactly what you're doing in that detection step. Voice mail detection is still something we see people struggle with. My general advice is to use a transcript only, and not to try to detect the "beep." Are you using the pipecat voicemail detection classifier?

not using pipecat, rolled our own with a simple transcript-based classifier. the beep detection rabbit hole was a mistake, burned 2 weeks on it. transcript-only approach ended up way cleaner. will check v3.2, didnt know it was that cpu-efficient

@thewhychain Totally know the feeling. We spent some time on beep detection. It's definitely something you can try a small model to do reliably, in theory. But in practice, there's so much variation, and there's no actual way to get a good enough set of training data.

yeah that's the core of it. what's your ground truth even? 'the call connected' vs 'that was a beep' is ambiguous in 30% of cases. we tried labeling 500 clips, inter-rater agreement was terrible. transcript timestamps just worked.

The default `voicemail_response_delay` in the Pipecat voicemail detector service is 2s. Which means you do end up at about 3s of delay time when a human says "hello" and then waits. We did a *lot* of testing and it's hard to go below that and not have the bot mis-classify or start talking just because there's a pause in the voicemail message.

makes sense. the pause-in-message case was our biggest misfire too, bot would kick in mid-greeting. we just flagged voicemails as a category and routed to human, stopped trying to handle them gracefully

Can you use this for “instant” translation in real world condition?

Todd Kuehnl Nice work, it is trickier then it seems at first, when I added STT to Anvil-OS last year, I approached it the same way but it took some tweaking for sure.

Oh this is really cool - been experimenting with my own home rolled solution but this project seems to tick many of the right boxes out of the gate.

Hey Kwin, this is very clever. But did you get any mileage out of simpler methods like heuristic gates that just match "hold on, one sec, let me think..." or a small classifier model? Trying to think of useful alternatives when models capable of single-token adherence may not be reliable. (at the cost of in-context adjustments ofc).

I haven't liked any of the results I've gotten from simple heuristics, but could be a skill issue on my part! I wonder if we should just collect a bunch of data and fine-tune a small, fast, text model! :-)
