Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Voice AI can pass a Turing test. For about a minute. That's a generated clip, though. Have a human actually talk back and the number collapses to six or seven seconds, roughly where generated voice sat three years ago. One reason: real conversation isn't turn-based. Around 20% of the... time more than one person is speaking, and laughter drives a lot of that overlap, since you laugh at a joke while it's still being told. We also adjust our pacing toward whoever we're talking to without noticing we're doing it. Voice models struggle with all of this. Full-duplex voice, where a model listens and speaks at the same time, is still extremely early. So an agent can know your joke is funny and still have to wait until you've finished before it laughs, by which point the timing has killed it. Miso Labs CEO & Co-Founder, Aoden Teo, describes a second consequence: agents get pushed toward almost "psychotically emotive" behavior. If they can only talk once you've stopped, they need some other way to show they were listening. You finish your sentence, and the thing goes "Hmm?". You've heard it. Underneath that sits an architecture problem. Voice models have to respond fast, which constrains how large they can be, and fast means something different here than it does in text. Working with an LLM like Claude, you care how quickly it finishes your code, more than how quickly it starts. Voice inverts that. Nobody needs 10 hours of audio generated in two seconds, because nobody can listen to 10 hours of audio in two seconds; what matters is reaction time. Most architectural decisions trade latency against throughput, and Aoden expects voice to keep moving away from LLM-style designs toward ones built around very low latency. Miso Labs is already pushing on it. Miso-1 got 3,000+ stars on GitHub and 5 million views on X, and they record data in their own LA studio because the internet doesn't contain every kind of audio a voice model might need. Nobody has released a podcast of someone reading millions and millions of email addresses, and people still want voice models that can read email addresses aloud, so teams end up generating some very strange training data themselves. The clip isn't the hard part. The hard part starts when you talk back. "The most emotive foundation models for voice" 🎙️Aoden Teo, CEO & Co-Founder, Miso Labs on Fondo.com START 1:03 Miso-1: 3K+ GitHub stars + 5M X views 1:59 Why emotiveness matters for games, UGC + interactive products 3:06 Measuring progress in voice AI with longer Turing tests 4:01 Why interactive conversation is harder than generating convincing clips 5:08 Full-duplex voice, interruptions + why laughter matters 6:04 Latency vs. throughput - and why voice differs from LLMs 7:09 Miso's LA recording studio + the challenge of voice training data 9:02 Talking teddy bears, UGC, anime + unexpected voice AI use cases 10:19 From serious chess player to math obsession to building Miso Labs 12:11 The surprise YC interviewshow more

David J Phillips

20,871 subscribers

36,612 Aufrufe • vor 22 Tagen •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

2 Kommentare

Profilbild von Reya Khanna

Reya Khannavor 19 Tagen

that laugh timing detail stuck with me, it's how conversations actually sound day to day

Profilbild von Killian Dunne

Killian Dunnevor 22 Tagen

Is Miso-1 turn-based or are you already doing full duplex?

Ähnliche Videos

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...

kwindla

331,175 Aufrufe • vor 28 Tagen

Learn to build conversational AI voice agents in "Building AI Voice Agents for Production", created in collaboration with LiveKit and RealAvatar, and taught by dsa (Co-founder & CEO of LiveKit), Shayne (Developer Advocate, LiveKit), and Nedelina Teneva (Head of AI at RealAvatar, an AI Fund portfolio company). Voice agents combine speech and reasoning capabilities to enable real-time conversations. They're already being used to support customer service, to improve accessibility in healthcare, for entertainment applications, and for talk therapy. In this course, you’ll learn to build voice agents that listen, reason, and respond naturally. You’ll follow the architecture used to create the "AI Andrew" Avatar, a collaborative project between and RealAvatar that responds to users in what sounds like my voice. You’ll build a voice agent from scratch and deploy it to the cloud, enabling support for many simultaneous users. What you’ll learn: - Understand the fundamentals of voice agents, including key components like speech-to-text (STT), text-to-speech (TTS), and LLMs, and how latency is introduced at each layer. - Explore voice agent architectures and the trade-offs between modular pipelines and speech-to-speech APIs. - Explore how platforms like LiveKit mitigate latency issues with optimized networking infrastructure and low-latency communication protocols. - Learn how to connect client devices to voice agents using WebRTC—and why it outperforms HTTP and WebSocket for low-latency audio streaming. - Incorporate voice activity detection (VAD), end-of-turn detection, and context management to detect turns, handle interruptions, and manage conversational flow. - Understand the trade-offs between latency, quality, and cost in an example in which you build a voice agent and change its voice. - Equip your agent with metrics to measure latency at each stage of the voice pipeline and learn the key levers you can pull to make your agent faster and more responsive. The voice agents built in this course also incorporate voice technology from , a supporting contributor to the project. By the end of this course, you'll have learned the components of an AI voice agent pipeline, combined them into a system with low-latency communication, and deployed them on cloud infrastructure so it scales to many users. I’m looking forward to seeing what voice agents you build from this course! Please sign up here:

Andrew Ng

87,868 Aufrufe • vor 1 Jahr

Cerebras inference is very fast. So fast that it changes how we think about configuring our LLMs for voice agent use cases. Kimi K2.6 is a 1T parameter reasoning model that Cerebras serves at 650 - 1,000 tokens per second (end-to-end throughput), with time to first token metrics as low as 150ms (latency). These numbers are two to three times faster than other similarly capable models. The biggest lever we get from this kind of speed is that we can use the model in reasoning mode, and still have excellent "time to first non-thinking token." This solves a big pain point we have in 2026 for voice agent use cases. Almost all recent innovation in post-training has focused on making models good at reasoning ("test time compute"). This is great, but it makes the user-facing model latency much, much slower. Which is a problem for conversational voice agents. We can run Kimi K2.6 with reasoning turned on, and get responses faster than other models produce with reasoning disabled. On my 30-turn voice agent benchmark, Kimi K2.6 with reasoning enabled ties GPT 5.1 and Haiku 4.5 with reasoning disabled, and is still about 200ms seconds faster! On my primary task agent benchmark, Kimi K2.6 is now the #2 model. It ranks just behind Gemini 3.5 Flash in "high" reasoning mode, and tied with GLM 5, Sonnet 4.6, and GPT 5.4 with reasoning set to "low." But Kimi K2.6 completes each turn in the agent loop in under 500ms. The other four models are all at least 3x slower. (Models only qualify for this benchmark if they can complete task turns at a P50 <4s.) A couple of other things that this speed buys us, for production voice agents: - Tool calls happen fast enough that we don't have to work around tool call latency in our pipeline design. - We can prompt the model to output structured data at the beginning of a response, followed by plain text for voice generation. This opens up possibilities like asking the model to do complex classification/generation tasks that influence the rest of the pipeline. For example, the model could create a detailed style prompt for a steerable TTS model, for each individual conversation turn. And, of course, you can use Kimi K2.6 with reasoning turned off. Cerebras calls this "instant" mode. Here's a video of a Cerebras Kimi K2.6 voice agent with voice-to-voice response time, measured at the client, under 500ms. This is the true response latency as perceived by the user, including all network and audio codec overhead, transcription and turn detection, Kimi K2.6 token generation, and voice generation. 500ms is, effectively, instant. So the Cerebras naming for this mode is a propos. :-)

Cerebras inference is very fast. So fast that it changes how we think about configuring our LLMs for voice agent use cases. Kimi K2.6 is a 1T parameter reasoning model that Cerebras serves at 650 - 1,000 tokens per second (end-to-end throughput), with time to first token metrics as low as 150ms (latency). These numbers are two to three times faster than other similarly capable models. The biggest lever we get from this kind of speed is that we can use the model in reasoning mode, and still have excellent "time to first non-thinking token." This solves a big pain point we have in 2026 for voice agent use cases. Almost all recent innovation in post-training has focused on making models good at reasoning ("test time compute"). This is great, but it makes the user-facing model latency much, much slower. Which is a problem for conversational voice agents. We can run Kimi K2.6 with reasoning turned on, and get responses faster than other models produce with reasoning disabled. On my 30-turn voice agent benchmark, Kimi K2.6 with reasoning enabled ties GPT 5.1 and Haiku 4.5 with reasoning disabled, and is still about 200ms seconds faster! On my primary task agent benchmark, Kimi K2.6 is now the #2 model. It ranks just behind Gemini 3.5 Flash in "high" reasoning mode, and tied with GLM 5, Sonnet 4.6, and GPT 5.4 with reasoning set to "low." But Kimi K2.6 completes each turn in the agent loop in under 500ms. The other four models are all at least 3x slower. (Models only qualify for this benchmark if they can complete task turns at a P50 <4s.) A couple of other things that this speed buys us, for production voice agents: - Tool calls happen fast enough that we don't have to work around tool call latency in our pipeline design. - We can prompt the model to output structured data at the beginning of a response, followed by plain text for voice generation. This opens up possibilities like asking the model to do complex classification/generation tasks that influence the rest of the pipeline. For example, the model could create a detailed style prompt for a steerable TTS model, for each individual conversation turn. And, of course, you can use Kimi K2.6 with reasoning turned off. Cerebras calls this "instant" mode. Here's a video of a Cerebras Kimi K2.6 voice agent with voice-to-voice response time, measured at the client, under 500ms. This is the true response latency as perceived by the user, including all network and audio codec overhead, transcription and turn detection, Kimi K2.6 token generation, and voice generation. 500ms is, effectively, instant. So the Cerebras naming for this mode is a propos. :-)

kwindla

40,593 Aufrufe • vor 4 Monaten

Most voice models are great at cloning voices, but they still manage to erase the person. I have an accent. Voice models don't know how to preserve it. Whenever I clone my voice, models get the words right, but mess up the rhythm badly. They flatten my vowels, and everything drifts toward a generic American accent. I'm attaching one example from Mirage. This is the first model I have heard of that can reproduce access as-is. • My accent stays intact, even when speaking English and Spanish • My pacing and cadence feel intentional, not averaged • The expressivity of my voice comes through Best part: Mirage only needs a short sample of your voice to clone it. I think this solves the hard part of voice cloning: intelligibility is easy, but identity is what makes a huge difference. Listen to the attached sample, and you'll notice the difference if you compare it with any other model out there. If you want to try this out, check out this link: Thanks to the Mirage team for collaborating with me on this post.

Most voice models are great at cloning voices, but they still manage to erase the person. I have an accent. Voice models don't know how to preserve it. Whenever I clone my voice, models get the words right, but mess up the rhythm badly. They flatten my vowels, and everything drifts toward a generic American accent. I'm attaching one example from Mirage. This is the first model I have heard of that can reproduce access as-is. • My accent stays intact, even when speaking English and Spanish • My pacing and cadence feel intentional, not averaged • The expressivity of my voice comes through Best part: Mirage only needs a short sample of your voice to clone it. I think this solves the hard part of voice cloning: intelligibility is easy, but identity is what makes a huge difference. Listen to the attached sample, and you'll notice the difference if you compare it with any other model out there. If you want to try this out, check out this link: Thanks to the Mirage team for collaborating with me on this post.

Santiago

32,044 Aufrufe • vor 9 Monaten

One of the stranger assumptions in AI is that the same model architecture should work equally well for thinking and interacting. A voice model has a weird job, it cannot just produce the right answer. It has to keep up with a person while the sequence keeps getting longer. In voice AI "speech" may not the right abstraction for the hard part, rather continuous state could be it. Here, Cartesia's founder talking how they came into the space from sequence modeling rather than speech research, which is probably why they focused so heavily on state space models. --- (Full video on “The Neon Show” YT channel, link in comment)

One of the stranger assumptions in AI is that the same model architecture should work equally well for thinking and interacting. A voice model has a weird job, it cannot just produce the right answer. It has to keep up with a person while the sequence keeps getting longer. In voice AI "speech" may not the right abstraction for the hard part, rather continuous state could be it. Here, Cartesia's founder talking how they came into the space from sequence modeling rather than speech research, which is probably why they focused so heavily on state space models. --- (Full video on “The Neon Show” YT channel, link in comment)

Rohan Paul

213,154 Aufrufe • vor 3 Tagen

Remember when AI couldn't draw a hand? Seven fingers, knuckles pointing backwards. And the AI spaghetti videos. That was three years ago. Images are done now. Video is close enough that you scrolled past AI ads this week and clocked exactly zero of them. Code writes itself and there are like 40 coding agents. AI voice spent that entire stretch sounding like the lady voice in a 2014 GPS. Flat, evenly spaced and every sentence landing with the same weight, like it's reading from a phone book. Here's why it stayed broken. Bad images are funny. You screenshot the seven fingers, it goes viral for being bad, someone fixes it. Bad audio is just boring. It doesn't fail spectacularly, so it never got that pressure. The bigger problem was the scoring. The whole industry graded AI voices on whether you could make out the words. So the models learned to over pronounce everything, hitting every syllable like a newsreader. Perfectly clear but robotic. Everyone was chasing a score that had nothing to do with sounding human. Meanwhile a small open-source team was doing something harder. Their lead researcher, an ex-NVIDIA engineer, went all in on an approach the rest of the field had written off. Two years early. No funding announcements or launch tour. He just put the whole thing on GitHub for free. It's sitting at 50,000+ stars now. Then they ran the test everyone else avoided. For 10 days they piped real users through their model and every big competitor with the listener never told which was which. Thousands of real people, real scripts. Whichever voice you actually preferred, they logged it. Theirs came out on top. It beat ElevenLabs about 6 times out of 10, head to head. It beat OpenAI's voice model 8 times out of 10. The gap was widest on the breathing, the pauses, the little hesitations, which is exactly the stuff that makes a voice sound like a person instead of a machine reading. They ran on real users rather than a lab, which is more than most of these claims can say. That's Fish Audio. This week they shipped S2.1 Pro: - Clone anyone's voice from 15 seconds of audio - Fast enough to hold a live conversation - 83 languages, one model - Type [whisper] or [sigh] mid-sentence and it does it - Around 70% cheaper than ElevenLabs - Free to download and run yourself Voice was the last thing on the list. Around 20 people with a free repo got there before other billion dollar companies did.

Remember when AI couldn't draw a hand? Seven fingers, knuckles pointing backwards. And the AI spaghetti videos. That was three years ago. Images are done now. Video is close enough that you scrolled past AI ads this week and clocked exactly zero of them. Code writes itself and there are like 40 coding agents. AI voice spent that entire stretch sounding like the lady voice in a 2014 GPS. Flat, evenly spaced and every sentence landing with the same weight, like it's reading from a phone book. Here's why it stayed broken. Bad images are funny. You screenshot the seven fingers, it goes viral for being bad, someone fixes it. Bad audio is just boring. It doesn't fail spectacularly, so it never got that pressure. The bigger problem was the scoring. The whole industry graded AI voices on whether you could make out the words. So the models learned to over pronounce everything, hitting every syllable like a newsreader. Perfectly clear but robotic. Everyone was chasing a score that had nothing to do with sounding human. Meanwhile a small open-source team was doing something harder. Their lead researcher, an ex-NVIDIA engineer, went all in on an approach the rest of the field had written off. Two years early. No funding announcements or launch tour. He just put the whole thing on GitHub for free. It's sitting at 50,000+ stars now. Then they ran the test everyone else avoided. For 10 days they piped real users through their model and every big competitor with the listener never told which was which. Thousands of real people, real scripts. Whichever voice you actually preferred, they logged it. Theirs came out on top. It beat ElevenLabs about 6 times out of 10, head to head. It beat OpenAI's voice model 8 times out of 10. The gap was widest on the breathing, the pauses, the little hesitations, which is exactly the stuff that makes a voice sound like a person instead of a machine reading. They ran on real users rather than a lab, which is more than most of these claims can say. That's Fish Audio. This week they shipped S2.1 Pro: - Clone anyone's voice from 15 seconds of audio - Fast enough to hold a live conversation - 83 languages, one model - Type [whisper] or [sigh] mid-sentence and it does it - Around 70% cheaper than ElevenLabs - Free to download and run yourself Voice was the last thing on the list. Around 20 people with a free repo got there before other billion dollar companies did.

Rez Karim

18,446 Aufrufe • vor 1 Monat

Did Google just mass-murdered every voice AI startup irrelevant? Google just shipped Gemini 3.1 Flash Live. Real-time voice AND vision. Sub-second latency. 90+ languages. And the one thing no voice AI has cracked until now: it filters background noise in real-world environments. Not demo environments. Real ones. Traffic outside. TV left on. Drill running next door. That's the wall every voice AI startup has been quietly hitting for three years. The demos worked. The deployments didn't. Because the real world is loud and messy and the models couldn't tell your voice from the noise around it. Google just called that problem solved. And it's not sitting in a research paper. It's already running in production apps. Stitch is using it to let designers talk to their canvas in real-time. Ato is using it to give elderly people a companion who actually listens. Wit's End built an RPG game master with it that delivers theatrical dialogue indistinguishable from a human voice actor. Three live products. Three different industries. All shipping now. Every voice AI startup just woke up to a new baseline. The question isn't whether to build with this. The question is how fast.

Did Google just mass-murdered every voice AI startup irrelevant? Google just shipped Gemini 3.1 Flash Live. Real-time voice AND vision. Sub-second latency. 90+ languages. And the one thing no voice AI has cracked until now: it filters background noise in real-world environments. Not demo environments. Real ones. Traffic outside. TV left on. Drill running next door. That's the wall every voice AI startup has been quietly hitting for three years. The demos worked. The deployments didn't. Because the real world is loud and messy and the models couldn't tell your voice from the noise around it. Google just called that problem solved. And it's not sitting in a research paper. It's already running in production apps. Stitch is using it to let designers talk to their canvas in real-time. Ato is using it to give elderly people a companion who actually listens. Wit's End built an RPG game master with it that delivers theatrical dialogue indistinguishable from a human voice actor. Three live products. Three different industries. All shipping now. Every voice AI startup just woke up to a new baseline. The question isn't whether to build with this. The question is how fast.

Vaibhav Sisinty

59,376 Aufrufe • vor 6 Monaten

What if your voice AI could interrupt you the moment it figured out your question - sometimes even before you finished asking it? Last week, I sat down with Neil, CEO of Gradium and co-founder of Kyutai , to talk about the future of speech-to-speech models and why he believes today's cascaded voice systems will soon look "archaic and brittle." Some highlights from our conversation: 🎯 How Kyutai built Moshi—a full duplex conversational AI with "negative latency"—in 6 months with just 4-6 people (while big tech teams had 10-20x the resources) 🧠 Why speech-to-speech models lose intelligence compared to their text counterparts (and what's being done about it) 📱 Pocket TTS: The first voice cloning model that runs on your phone's CPU—not GPU, CPU 🤖 Why robotics and spatial audio represent the next frontier (hint: current voice systems completely break in these environments) 👶 The efficiency gap: Babies learn to speak fluently from <5,000 hours of audio. Current models train on millions of hours. We're doing something wrong. My favorite vision from Neil? The first truly contrarian AI that interrupts you mid-sentence to tell you why you're wrong. Not just more natural conversation—but actually useful for testing ideas and playing devil's advocate. Full episode and detailed blog post linked in the comments 👇 What's your take - will speech-to-speech replace cascaded systems, or will modularity keep cascaded architectures dominant even as naturalness improves?

What if your voice AI could interrupt you the moment it figured out your question - sometimes even before you finished asking it? Last week, I sat down with Neil, CEO of Gradium and co-founder of Kyutai , to talk about the future of speech-to-speech models and why he believes today's cascaded voice systems will soon look "archaic and brittle." Some highlights from our conversation: 🎯 How Kyutai built Moshi—a full duplex conversational AI with "negative latency"—in 6 months with just 4-6 people (while big tech teams had 10-20x the resources) 🧠 Why speech-to-speech models lose intelligence compared to their text counterparts (and what's being done about it) 📱 Pocket TTS: The first voice cloning model that runs on your phone's CPU—not GPU, CPU 🤖 Why robotics and spatial audio represent the next frontier (hint: current voice systems completely break in these environments) 👶 The efficiency gap: Babies learn to speak fluently from <5,000 hours of audio. Current models train on millions of hours. We're doing something wrong. My favorite vision from Neil? The first truly contrarian AI that interrupts you mid-sentence to tell you why you're wrong. Not just more natural conversation—but actually useful for testing ideas and playing devil's advocate. Full episode and detailed blog post linked in the comments 👇 What's your take - will speech-to-speech replace cascaded systems, or will modularity keep cascaded architectures dominant even as naturalness improves?

Brooke Hopkins

13,000 Aufrufe • vor 7 Monaten

Introducing Lightning V3 - it beats every model we tested against. ElevenLabs, Cartesia, OpenAI. Lightning sets a new SOTA with V3 in conversational text-to-speech. → Highest MOS score for conversational TTS at 3.9 → ~76% win rate vs gpt-4o-mini-tts on naturalness → 15 languages with mid-sentence code-switching → Built from scratch for voice agents, not read-aloud Every TTS model sounds clean in a demo. You type a sentence and you get beautiful audio. Voice agents don't work that way. They stream. They're generating audio in real-time chunks with half the context missing. That's where everything breaks. A great reading voice and a great conversational voice are fundamentally different things. A conversational voice has to sound like it's thinking - with the pauses, the rhythm shifts, the reactions. It has to handle the way real people actually talk, including switching languages mid-sentence. That's what V3 does. V3.1 also ships voice cloning. 5 to 15 seconds of audio, no fine-tuning, production-grade clone across 15 languages. Blog link in the comments.

Introducing Lightning V3 - it beats every model we tested against. ElevenLabs, Cartesia, OpenAI. Lightning sets a new SOTA with V3 in conversational text-to-speech. → Highest MOS score for conversational TTS at 3.9 → ~76% win rate vs gpt-4o-mini-tts on naturalness → 15 languages with mid-sentence code-switching → Built from scratch for voice agents, not read-aloud Every TTS model sounds clean in a demo. You type a sentence and you get beautiful audio. Voice agents don't work that way. They stream. They're generating audio in real-time chunks with half the context missing. That's where everything breaks. A great reading voice and a great conversational voice are fundamentally different things. A conversational voice has to sound like it's thinking - with the pauses, the rhythm shifts, the reactions. It has to handle the way real people actually talk, including switching languages mid-sentence. That's what V3 does. V3.1 also ships voice cloning. 5 to 15 seconds of audio, no fine-tuning, production-grade clone across 15 languages. Blog link in the comments.

Sudarshan Kamath

71,298 Aufrufe • vor 6 Monaten

Today we're announcing $280M in Series B funding at a $2B valuation, led by our long-time investor and partner, Menlo Ventures. When we announced our Series A last May, most conversations I had about voice began with someone explaining why they doubted it. I don't have those conversations anymore. People tell me instead how much time they save talking instead of typing and what they want us to build next. In a little over a year, voice has gone from something people were questioning to something they rely on, and it happened faster than we expected. That shift is why we've raised a new round of funding. This funding represents a deeper investment in our products, lab, models, and our team. Alongside the funding, we're announcing a preview of our first proprietary speech model, Canto. Canto is a 2B parameter speech model trained for the places people actually talk: loud rooms, windy streets, a toddler in the background, a second language mixed into the first. Models like this will change how we all interact with devices, and Canto is the first in a rapid line of them. Each one will be larger and more capable than the last, and each should improve Flow's dictation in a way you can feel the day it ships. Larger speech models also open the door to the vision we dreamed up 5 years ago: using your voice as the primary way you interact with devices. That still has to be invented. A voice interface people rely on all day, in every app, doesn't exist anywhere yet. It's time for that to change. To Sahaj Garg, our team, and every person who gave Wispr Flow a chance - this one's for you.

Today we're announcing $280M in Series B funding at a $2B valuation, led by our long-time investor and partner, Menlo Ventures. When we announced our Series A last May, most conversations I had about voice began with someone explaining why they doubted it. I don't have those conversations anymore. People tell me instead how much time they save talking instead of typing and what they want us to build next. In a little over a year, voice has gone from something people were questioning to something they rely on, and it happened faster than we expected. That shift is why we've raised a new round of funding. This funding represents a deeper investment in our products, lab, models, and our team. Alongside the funding, we're announcing a preview of our first proprietary speech model, Canto. Canto is a 2B parameter speech model trained for the places people actually talk: loud rooms, windy streets, a toddler in the background, a second language mixed into the first. Models like this will change how we all interact with devices, and Canto is the first in a rapid line of them. Each one will be larger and more capable than the last, and each should improve Flow's dictation in a way you can feel the day it ships. Larger speech models also open the door to the vision we dreamed up 5 years ago: using your voice as the primary way you interact with devices. That still has to be invented. A voice interface people rely on all day, in every app, doesn't exist anywhere yet. It's time for that to change. To Sahaj Garg, our team, and every person who gave Wispr Flow a chance - this one's for you.

Tanay Kothari

615,867 Aufrufe • vor 1 Monat

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses *three* NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses three NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.

kwindla

274,861 Aufrufe • vor 8 Monaten

Elon Musk on local AI in Tesla Optimus: “Well, we think we can put a lot of intelligence locally in the robot and it certainly needs to be enough intelligence that if the robot gets disconnected, like if it's a bad cellular signal or there isn't Wi-Fi, you know, Optimus can't just get stuck. It needs to have enough local intelligence that it can still do useful things, even if it loses the connection, kind of like the car. The car does not need any cellular or Wi-Fi connection to be able to drive safely. Now, I guess you can think of like Optimus needs kind of a manager to tell it what to do, broadly speaking, like if otherwise it's going to keep doing the same thing it did before. so you know i think you need kind of a an orchestration ai which uh you know grok would be good for orchestration um and and then for you know for optimist's voice you know having um a low latency intelligent voice ai grok is actually very good for that so if you when I talk to Optimus and have kind of a you know, a Grok level conversation you kind of need to connect to a Grok level AI for that. But I would expect the amount of interaction apart from like the voice stuff and asking complicated questions of the robot that necessarily needs a large AI model to answer the Gronk would probably have about as much interaction with Optimus as a manager would have with the people on their team. “

Elon Musk on local AI in Tesla Optimus: “Well, we think we can put a lot of intelligence locally in the robot and it certainly needs to be enough intelligence that if the robot gets disconnected, like if it's a bad cellular signal or there isn't Wi-Fi, you know, Optimus can't just get stuck. It needs to have enough local intelligence that it can still do useful things, even if it loses the connection, kind of like the car. The car does not need any cellular or Wi-Fi connection to be able to drive safely. Now, I guess you can think of like Optimus needs kind of a manager to tell it what to do, broadly speaking, like if otherwise it's going to keep doing the same thing it did before. so you know i think you need kind of a an orchestration ai which uh you know grok would be good for orchestration um and and then for you know for optimist's voice you know having um a low latency intelligent voice ai grok is actually very good for that so if you when I talk to Optimus and have kind of a you know, a Grok level conversation you kind of need to connect to a Grok level AI for that. But I would expect the amount of interaction apart from like the voice stuff and asking complicated questions of the robot that necessarily needs a large AI model to answer the Gronk would probably have about as much interaction with Optimus as a manager would have with the people on their team. “

Brian Roemmele

26,740 Aufrufe • vor 5 Monaten

Text-to-speech is moving way too fast. Just a few days ago, I tweeted about PersonaPlex-7B, NVIDIA's new open source TTS ( And today, Qwen just open-sourced Qwen3-TTS 🤯 It’s a revolutionary text-to-speech model built for control. Not just about generating speech, but about shaping how it sounds directly from language. You can guide the pace, the tone, and the expressiveness straight from text, without touching audio graphs or hand-tuning parameters. That’s the real shift! What makes Qwen3-TTS stand out is how practical it already is: → voice cloning from just a few seconds of audio → voice creation without any reference sample → support for 10 languages out of the box → end-to-end latency down to ~97ms → works in both streaming and non-streaming setups The models come in two sizes (0.6B and 1.7B), so you can trade off quality and hardware cost depending on your setup. You can work with curated voices, designed voices, or cloned ones, and it integrates cleanly with vLLM for production use. It also ships as a simple Python package you can pip install. If you’re building real-time voice systems, this removes a lot of friction! 100% free and open source. I put the repo in the 🧵↓

Text-to-speech is moving way too fast. Just a few days ago, I tweeted about PersonaPlex-7B, NVIDIA's new open source TTS ( And today, Qwen just open-sourced Qwen3-TTS 🤯 It’s a revolutionary text-to-speech model built for control. Not just about generating speech, but about shaping how it sounds directly from language. You can guide the pace, the tone, and the expressiveness straight from text, without touching audio graphs or hand-tuning parameters. That’s the real shift! What makes Qwen3-TTS stand out is how practical it already is: → voice cloning from just a few seconds of audio → voice creation without any reference sample → support for 10 languages out of the box → end-to-end latency down to ~97ms → works in both streaming and non-streaming setups The models come in two sizes (0.6B and 1.7B), so you can trade off quality and hardware cost depending on your setup. You can work with curated voices, designed voices, or cloned ones, and it integrates cleanly with vLLM for production use. It also ships as a simple Python package you can pip install. If you’re building real-time voice systems, this removes a lot of friction! 100% free and open source. I put the repo in the 🧵↓

Charly Wargnier

59,144 Aufrufe • vor 8 Monaten

"Voice agents are built to be helpful. The problem is they can be too helpful and cross the boundary." I sat down with Sumanyu Sharma 🍫, Founder and CEO of Hamming, at our AI Engineer 🔜 Paris 🇫🇷 booth. Hamming stress-tests and monitors voice agents so the bank or hospital running one catches failures before a customer does. Fresh after his talk “I Monitored Crime Audio. Voice Agents Scare Me More,” we got into: 0:00 how Sumanyu went from listening to 911 calls at Citizen to founding Hamming AI 0:39 why he's more nervous about voice agents than he was about crime 1:51 the first voice-agent security product 3:16 simulating messy reality: accents, background noise, code-switching 4:11 voice agents testing voice agents 5:47 why taking actions is table stakes, and what makes it dangerous 11:07 hot take: everyone rushed to B2B, more founders should build consumer Thanks for coming to our Browserbase booth, Sumanyu Sharma 🍫! Watch the full episode here:

"Voice agents are built to be helpful. The problem is they can be too helpful and cross the boundary." I sat down with Sumanyu Sharma 🍫, Founder and CEO of Hamming, at our AI Engineer 🔜 Paris 🇫🇷 booth. Hamming stress-tests and monitors voice agents so the bank or hospital running one catches failures before a customer does. Fresh after his talk “I Monitored Crime Audio. Voice Agents Scare Me More,” we got into: 0:00 how Sumanyu went from listening to 911 calls at Citizen to founding Hamming AI 0:39 why he's more nervous about voice agents than he was about crime 1:51 the first voice-agent security product 3:16 simulating messy reality: accents, background noise, code-switching 4:11 voice agents testing voice agents 5:47 why taking actions is table stakes, and what makes it dangerous 11:07 hot take: everyone rushed to B2B, more founders should build consumer Thanks for coming to our Browserbase booth, Sumanyu Sharma 🍫! Watch the full episode here:

justine

18,670 Aufrufe • vor 2 Monaten

Voice AI turn taking is a solved problem. The single most common complaint about voice AI, today, is that agents interrupt too often. But the voice agents I build for myself now respond quickly and interrupt me less often than the people I talk to every day. (I actually measured this.) Mark Backman made a Pipecat AI PR two weeks ago that was the last piece of the puzzle for turn taking so good that I no longer ever think about it. The approach combines three layers of processing: 1. Voice activity detection, with a short (200ms) trigger. 2. A native audio turn detection model that's small, fast, and runs on CPU. This model captures audio nuances like inflection and filler sounds that don't get transcribed. 3. A prompt mixin for the conversation LLM that decides turn completion based on conversation context. None of these are new. We've been using VAD for a long time. We trained the first version of the Pipecat Smart Turn native audio model in December 2024. And we've been experimenting with prompt-based large model turn detection (sometimes called "selective refusal") for more than a year. Now, the Smart Turn model and the SOTA LLMs we're using in voice agents have both gotten so good that using them together feels like we've finally "solved" turn detection. Mark also figured out how to elegantly apply a "single-token tagging" technique to this problem. We sometimes use single-token tagging in place of tool calling, when we need a near-zero latency programmatic trigger. Mark's Pipecat mixin defines three single-token characters and prompts the LLM to output exactly one of them at the beginning of every response. - ✓ means the agent should respond normally (immediately) - ○ is a "short incomplete" - the agent should wait 5 seconds - ◐ is a "long incomplete" - the agent should wait 10 seconds The wait times, and the details of the prompt, are configurable, of course. Watch the video to see me talk to an agent that handles all my various pauses and inflections, plus phrases like "let me think," pretty much the way a person would handle them, in terms of response latency. Also, in the second half of the video, I ask the agent to adjust its response pattern because I'm going to tell it a phone number. This kind of "in-context" adjustment of response wait times is really useful. The LLM in the video is GTP-4.1. We've tested the prompt and single-token adherance with GPT-4.1, Gemini 2.5 Flash, Anthropic Claude Sonnet 4.5, and AWS Nova 2 Pro. Note that older models in all these families (and, in general, smaller open weights models) aren't able to reliably output these single-token tags. But the new models we're using these days are pretty amazing.

Voice AI turn taking is a solved problem. The single most common complaint about voice AI, today, is that agents interrupt too often. But the voice agents I build for myself now respond quickly and interrupt me less often than the people I talk to every day. (I actually measured this.) Mark Backman made a Pipecat AI PR two weeks ago that was the last piece of the puzzle for turn taking so good that I no longer ever think about it. The approach combines three layers of processing: 1. Voice activity detection, with a short (200ms) trigger. 2. A native audio turn detection model that's small, fast, and runs on CPU. This model captures audio nuances like inflection and filler sounds that don't get transcribed. 3. A prompt mixin for the conversation LLM that decides turn completion based on conversation context. None of these are new. We've been using VAD for a long time. We trained the first version of the Pipecat Smart Turn native audio model in December 2024. And we've been experimenting with prompt-based large model turn detection (sometimes called "selective refusal") for more than a year. Now, the Smart Turn model and the SOTA LLMs we're using in voice agents have both gotten so good that using them together feels like we've finally "solved" turn detection. Mark also figured out how to elegantly apply a "single-token tagging" technique to this problem. We sometimes use single-token tagging in place of tool calling, when we need a near-zero latency programmatic trigger. Mark's Pipecat mixin defines three single-token characters and prompts the LLM to output exactly one of them at the beginning of every response. - ✓ means the agent should respond normally (immediately) - ○ is a "short incomplete" - the agent should wait 5 seconds - ◐ is a "long incomplete" - the agent should wait 10 seconds The wait times, and the details of the prompt, are configurable, of course. Watch the video to see me talk to an agent that handles all my various pauses and inflections, plus phrases like "let me think," pretty much the way a person would handle them, in terms of response latency. Also, in the second half of the video, I ask the agent to adjust its response pattern because I'm going to tell it a phone number. This kind of "in-context" adjustment of response wait times is really useful. The LLM in the video is GTP-4.1. We've tested the prompt and single-token adherance with GPT-4.1, Gemini 2.5 Flash, Anthropic Claude Sonnet 4.5, and AWS Nova 2 Pro. Note that older models in all these families (and, in general, smaller open weights models) aren't able to reliably output these single-token tags. But the new models we're using these days are pretty amazing.

kwindla

27,016 Aufrufe • vor 7 Monaten

Ok this is SO COOL 🤯🤯🤯 I just made an AI agent that calls restaurants to make reservations for me. How it works: 1. You punch in the name of the restaurant and desired time and number of guests. 2. It calls the restaurant (in an incredibly real sounding voice) and makes the reservation. 3. If they don't have the time available, it texts you some alternate times (and can call back to make them). If they do, it books it and puts it on your calendar. If they ask, it can even provide allergies and other details, and it's powered by Claude so it can handle pretty much any line of conversation. It's also trained to tell the person it's my AI assistant if they ask or get weirded out by the voice. See attached video and photo. I built it using Lindy. Image 1: my prompt to Lindy Video: Call that resulted Image 2: My calendar 30 seconds later So useful. Took me about 10 minutes to put it all together.

Ok this is SO COOL 🤯🤯🤯 I just made an AI agent that calls restaurants to make reservations for me. How it works: 1. You punch in the name of the restaurant and desired time and number of guests. 2. It calls the restaurant (in an incredibly real sounding voice) and makes the reservation. 3. If they don't have the time available, it texts you some alternate times (and can call back to make them). If they do, it books it and puts it on your calendar. If they ask, it can even provide allergies and other details, and it's powered by Claude so it can handle pretty much any line of conversation. It's also trained to tell the person it's my AI assistant if they ask or get weirded out by the voice. See attached video and photo. I built it using Lindy. Image 1: my prompt to Lindy Video: Call that resulted Image 2: My calendar 30 seconds later So useful. Took me about 10 minutes to put it all together.

Andrew Wilkinson

169,248 Aufrufe • vor 1 Jahr

Mistral just released Voxtral TTS, an open-weight voice model that tries to match ElevenLabs while keeping it free. A 3-billionn param model that fits on a laptop and runs 6X faster than real-time speech. It is split into 3 parts that handle different jobs. - The 3.4B part is the main language brain that understands the text and plans what should be spoken. - The 390M part turns that plan into speech features like rhythm, pronunciation, and tone. - The 300M part converts those speech features into the final compressed audio signal you can play. That split helps because each part only does the job it is best at, which can make the whole system faster and lighter than one giant model doing everything alone. Mistral claims about 90ms to first audio, about 6x faster than real-time speech output, about 3GB RAM after quantization, 9 languages, and voice copying from 5s of audio, even across languages. Its internal tests report 62.8% preference over ElevenLabs Flash on default voices and 69.9% on custom voices, but the bigger point is that companies can keep speech data on their own machines. IMO, enterprise buyers usually care most about control, cost, and latency, So this model should give it great push for voice-based AI apps/products.

Mistral just released Voxtral TTS, an open-weight voice model that tries to match ElevenLabs while keeping it free. A 3-billionn param model that fits on a laptop and runs 6X faster than real-time speech. It is split into 3 parts that handle different jobs. - The 3.4B part is the main language brain that understands the text and plans what should be spoken. - The 390M part turns that plan into speech features like rhythm, pronunciation, and tone. - The 300M part converts those speech features into the final compressed audio signal you can play. That split helps because each part only does the job it is best at, which can make the whole system faster and lighter than one giant model doing everything alone. Mistral claims about 90ms to first audio, about 6x faster than real-time speech output, about 3GB RAM after quantization, 9 languages, and voice copying from 5s of audio, even across languages. Its internal tests report 62.8% preference over ElevenLabs Flash on default voices and 69.9% on custom voices, but the bigger point is that companies can keep speech data on their own machines. IMO, enterprise buyers usually care most about control, cost, and latency, So this model should give it great push for voice-based AI apps/products.

Rohan Paul

15,672 Aufrufe • vor 6 Monaten

Rime founder, Lily Clifford on why voice is about to become the primary way humans talk to AI: "For the first time since the era of the mainframe, our main way of interacting with compute is essentially a terminal window again. The fascinating thing about Claude Code and Codex is how it has made software development essentially something that happens behind the scenes, with very little hands-on keyboard time." "When you're running 12 agents at the same time, you're getting pinged all the time, and it starts to feel really weird." "Voice is the primary way we've been communicating for hundreds of thousands of years, and indeed is probably the thing that separates us from our human ancestors and human cousins like the Neanderthals." "The one thing that is just psychologically true about voice as opposed to our visual system is that we really can only be paying attention to one thing at a time in voice." "I personally don't like typing. I think typing is something that is at this point anachronistic. In the same way that my three-year-old niece probably will not learn how to drive, I don't have a prediction for when AI will become the dominant way we interface with compute, but I definitely think it will be one of the dominant ways in the very near term." "My 160 words per minute personal benchmark means almost nothing." lily clifford

Rime founder, Lily Clifford on why voice is about to become the primary way humans talk to AI: "For the first time since the era of the mainframe, our main way of interacting with compute is essentially a terminal window again. The fascinating thing about Claude Code and Codex is how it has made software development essentially something that happens behind the scenes, with very little hands-on keyboard time." "When you're running 12 agents at the same time, you're getting pinged all the time, and it starts to feel really weird." "Voice is the primary way we've been communicating for hundreds of thousands of years, and indeed is probably the thing that separates us from our human ancestors and human cousins like the Neanderthals." "The one thing that is just psychologically true about voice as opposed to our visual system is that we really can only be paying attention to one thing at a time in voice." "I personally don't like typing. I think typing is something that is at this point anachronistic. In the same way that my three-year-old niece probably will not learn how to drive, I don't have a prediction for when AI will become the dominant way we interface with compute, but I definitely think it will be one of the dominant ways in the very near term." "My 160 words per minute personal benchmark means almost nothing." lily clifford

MTS

367,615 Aufrufe • vor 2 Monaten