Loading video...

Video Failed to Load

Go Home

Build a conversational voice bot with 1 second voice-to-voice latency with Modal, Pipecat AI, and open models. Modal works seamlessly with WebRTC, WebSockets, and tunneling to squash latency to an absolute minimum.

modal's profile picture

Modal

25,259 subscribers

29,559 views • 11 months ago •via X (Twitter)

EducationScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Instant Voice Cloning 3 seconds of audio is all you need. Text-to-voice AI generated voices with <1 second latency. 🧵 A thread
1:19

Instant Voice Cloning 3 seconds of audio is all you need. Text-to-voice AI generated voices with <1 second latency. 🧵 A thread

Linus ✦ Ekenstam

576,641 views • 3 years ago

Learn to build conversational AI voice agents in "Building AI Voice Agents for Production", created in collaboration with LiveKit and RealAvatar, and taught by dsa (Co-founder & CEO of LiveKit), Shayne (Developer Advocate, LiveKit), and Nedelina Teneva (Head of AI at RealAvatar, an AI Fund portfolio company). Voice agents combine speech and reasoning capabilities to enable real-time conversations. They're already being used to support customer service, to improve accessibility in healthcare, for entertainment applications, and for talk therapy. In this course, you’ll learn to build voice agents that listen, reason, and respond naturally. You’ll follow the architecture used to create the "AI Andrew" Avatar, a collaborative project between and RealAvatar that responds to users in what sounds like my voice. You’ll build a voice agent from scratch and deploy it to the cloud, enabling support for many simultaneous users. What you’ll learn: - Understand the fundamentals of voice agents, including key components like speech-to-text (STT), text-to-speech (TTS), and LLMs, and how latency is introduced at each layer. - Explore voice agent architectures and the trade-offs between modular pipelines and speech-to-speech APIs. - Explore how platforms like LiveKit mitigate latency issues with optimized networking infrastructure and low-latency communication protocols. - Learn how to connect client devices to voice agents using WebRTC—and why it outperforms HTTP and WebSocket for low-latency audio streaming. - Incorporate voice activity detection (VAD), end-of-turn detection, and context management to detect turns, handle interruptions, and manage conversational flow. - Understand the trade-offs between latency, quality, and cost in an example in which you build a voice agent and change its voice. - Equip your agent with metrics to measure latency at each stage of the voice pipeline and learn the key levers you can pull to make your agent faster and more responsive. The voice agents built in this course also incorporate voice technology from , a supporting contributor to the project. By the end of this course, you'll have learned the components of an AI voice agent pipeline, combined them into a system with low-latency communication, and deployed them on cloud infrastructure so it scales to many users. I’m looking forward to seeing what voice agents you build from this course! Please sign up here:
2:41

Learn to build conversational AI voice agents in "Building AI Voice Agents for Production", created in collaboration with LiveKit and RealAvatar, and taught by dsa (Co-founder & CEO of LiveKit), Shayne (Developer Advocate, LiveKit), and Nedelina Teneva (Head of AI at RealAvatar, an AI Fund portfolio company). Voice agents combine speech and reasoning capabilities to enable real-time conversations. They're already being used to support customer service, to improve accessibility in healthcare, for entertainment applications, and for talk therapy. In this course, you’ll learn to build voice agents that listen, reason, and respond naturally. You’ll follow the architecture used to create the "AI Andrew" Avatar, a collaborative project between and RealAvatar that responds to users in what sounds like my voice. You’ll build a voice agent from scratch and deploy it to the cloud, enabling support for many simultaneous users. What you’ll learn: - Understand the fundamentals of voice agents, including key components like speech-to-text (STT), text-to-speech (TTS), and LLMs, and how latency is introduced at each layer. - Explore voice agent architectures and the trade-offs between modular pipelines and speech-to-speech APIs. - Explore how platforms like LiveKit mitigate latency issues with optimized networking infrastructure and low-latency communication protocols. - Learn how to connect client devices to voice agents using WebRTC—and why it outperforms HTTP and WebSocket for low-latency audio streaming. - Incorporate voice activity detection (VAD), end-of-turn detection, and context management to detect turns, handle interruptions, and manage conversational flow. - Understand the trade-offs between latency, quality, and cost in an example in which you build a voice agent and change its voice. - Equip your agent with metrics to measure latency at each stage of the voice pipeline and learn the key levers you can pull to make your agent faster and more responsive. The voice agents built in this course also incorporate voice technology from , a supporting contributor to the project. By the end of this course, you'll have learned the components of an AI voice agent pipeline, combined them into a system with low-latency communication, and deployed them on cloud infrastructure so it scales to many users. I’m looking forward to seeing what voice agents you build from this course! Please sign up here:

Andrew Ng

87,965 views • 1 year ago

🔊🔛🔥 ... Groq Inc launched voice generation today. GroqCloud now has realtime transcription, LLMs, *and* text-to-speech. You can build super-responsive, ultra low-latency voice agents end-to-end entirely on Groq! At Daily, we're big fans of Groq's fast, low-latency inference. Pipecat AI supports all the Groq models, including the new voice model. Lately, we've been obsessively playing a voice chat game that Mark Backman wrote. (My high score is 6, so far.) Here's Mark, with Groq's `Celeste-PlayAI` voice.
0:47

🔊🔛🔥 ... Groq Inc launched voice generation today. GroqCloud now has realtime transcription, LLMs, *and* text-to-speech. You can build super-responsive, ultra low-latency voice agents end-to-end entirely on Groq! At Daily, we're big fans of Groq's fast, low-latency inference. Pipecat AI supports all the Groq models, including the new voice model. Lately, we've been obsessively playing a voice chat game that Mark Backman wrote. (My high score is 6, so far.) Here's Mark, with Groq's `Celeste-PlayAI` voice.

kwindla

31,843 views • 1 year ago

Learn how to build an audiobook voice agent using Gradium and Pipecat AI Gradium's TTS handles the narration and Pipecat's built-in WebRTC transport delivers the audio to the browser.
4:40

Learn how to build an audiobook voice agent using Gradium and Pipecat AI Gradium's TTS handles the narration and Pipecat's built-in WebRTC transport delivers the audio to the browser.

Gradium

11,247 views • 4 months ago

Today, we're introducing PlayHT2.0 - our new Conversational Text-to-Voice AI Model that's trained and built to generate humanlike conversations across languages with <1s latency. Sign up for beta access -
1:19

Today, we're introducing PlayHT2.0 - our new Conversational Text-to-Voice AI Model that's trained and built to generate humanlike conversations across languages with <1s latency. Sign up for beta access -

PlayAI

52,378 views • 3 years ago

The future of artificial intelligence is physical. Physical Intelligence runs robotic control inference on Modal with >2x lower latency than the lag between your brain and your finger.
0:20

The future of artificial intelligence is physical. Physical Intelligence runs robotic control inference on Modal with >2x lower latency than the lag between your brain and your finger.

Modal

95,966 views • 6 months ago

How to build the world's fastest voice AI bot: - Self-host speech-to-text, LLM inference, and text-to-speech all together in the same container/cluster. - Route audio over the internet using WebRTC and edge networking. - Configure timings for voice activity detection, phrase endpointing, and other parts of the pipeline to optimize for latency. (There are trade-offs to doing this!) Here's a LLama 3 voice bot that has voice-to-voice response times of ~500ms. We used Deepgram's STT and TTS for this bot, and everything is hosted on cerebriumai's serverless GPU infrastructure.
4:18

How to build the world's fastest voice AI bot: - Self-host speech-to-text, LLM inference, and text-to-speech all together in the same container/cluster. - Route audio over the internet using WebRTC and edge networking. - Configure timings for voice activity detection, phrase endpointing, and other parts of the pipeline to optimize for latency. (There are trade-offs to doing this!) Here's a LLama 3 voice bot that has voice-to-voice response times of ~500ms. We used Deepgram's STT and TTS for this bot, and everything is hosted on cerebriumai's serverless GPU infrastructure.

kwindla

282,890 views • 2 years ago

Oh my…FlashLabs releases Chroma 1.0, first open-source real-time speech-to-speech model with personalized voice cloning. Native speech-to-speech with <150ms latency & voice cloning from seconds of audio. Finally, an open alternative to OpenAI Realtime.
1:57

Oh my…FlashLabs releases Chroma 1.0, first open-source real-time speech-to-speech model with personalized voice cloning. Native speech-to-speech with <150ms latency & voice cloning from seconds of audio. Finally, an open alternative to OpenAI Realtime.

Alvaro Cintas

46,151 views • 8 months ago

xAI is cooking 🔥 They just launched Grok Voice, a platform for building AI voice agents. - You can build voice agents in under 2 minutes - No code required - Human-like conversations with sub-second latency - Supports 25+ languages - 80+ built-in voices or clone your own - Works with Gmail, Google Calendar, Outlook, Notion, MCPs, and more - Starts at $0.05/min and includes a free phone number
0:46

xAI is cooking 🔥 They just launched Grok Voice, a platform for building AI voice agents. - You can build voice agents in under 2 minutes - No code required - Human-like conversations with sub-second latency - Supports 25+ languages - 80+ built-in voices or clone your own - Works with Gmail, Google Calendar, Outlook, Notion, MCPs, and more - Starts at $0.05/min and includes a free phone number

AshutoshShrivastava

144,824 views • 3 months ago

Developers are building great voice experiences with the Realtime API. See how Lemonade uses it to power AI Maya, their friendly and engaging voice agent. The Realtime API's automatic voice detection and low latency enable them to offer 24/7 multilingual phone support 📞
4:50

Developers are building great voice experiences with the Realtime API. See how Lemonade uses it to power AI Maya, their friendly and engaging voice agent. The Realtime API's automatic voice detection and low latency enable them to offer 24/7 multilingual phone support 📞

OpenAI Developers

126,348 views • 1 year ago

New course: Add voice to your AI agents and applications, built with Vocal Bridge (disclosure: an AI Fund portfolio company) and taught by its CEO Ashwyn Sharma. Voice applications historically required making a hard tradeoff: using fast voice-to-voice models that sacrifice reliability, or accurate speech-to-text pipelines that add latency. This course teaches you how to build voice agents that are both reliable and fast. You'll build three types of voice-enabled applications: a voice-interactive game where voice commands and mouse clicks work together over a single channel, an agent that gains a voice in about 10 lines of code without touching its prompts or tools, and an agent that places outbound phone calls using a make_phone_call function. Skills you'll gain: - Add a voice layer to an existing agent without rewriting your prompts, RAG pipeline, or tools - Give an agent the ability to place outbound calls and stream transcripts back live - Set up voice evaluation to score calls, catch regressions, and improve quality before deployment Join and add voice to your agents without overhauling your architecture:
2:27

New course: Add voice to your AI agents and applications, built with Vocal Bridge (disclosure: an AI Fund portfolio company) and taught by its CEO Ashwyn Sharma. Voice applications historically required making a hard tradeoff: using fast voice-to-voice models that sacrifice reliability, or accurate speech-to-text pipelines that add latency. This course teaches you how to build voice agents that are both reliable and fast. You'll build three types of voice-enabled applications: a voice-interactive game where voice commands and mouse clicks work together over a single channel, an agent that gains a voice in about 10 lines of code without touching its prompts or tools, and an agent that places outbound phone calls using a make_phone_call function. Skills you'll gain: - Add a voice layer to an existing agent without rewriting your prompts, RAG pipeline, or tools - Give an agent the ability to place outbound calls and stream transcripts back live - Set up voice evaluation to score calls, catch regressions, and improve quality before deployment Join and add voice to your agents without overhauling your architecture:

Andrew Ng

97,401 views • 3 months ago

GLM-5, the latest frontier open model from Z.ai, is available now on Modal. We partnered with to release an endpoint that will be free for a limited time.
0:45

GLM-5, the latest frontier open model from Z.ai, is available now on Modal. We partnered with to release an endpoint that will be free for a limited time.

Modal

58,785 views • 8 months ago

I built the best memory for voice agents while sponsoring @YCombinator hackathon. Introducing: Voice AI + Memory! We built a deep supermemory integration for Pipecat AI, so now even your agents have great memory With user profiles, these are almost-instant latency as well :)
2:11

I built the best memory for voice agents while sponsoring @YCombinator hackathon. Introducing: Voice AI + Memory! We built a deep supermemory integration for Pipecat AI, so now even your agents have great memory With user profiles, these are almost-instant latency as well :)

Dhravya Shah

83,405 views • 7 months ago

Conversational AI is here. Build AI agents that can speak in minutes with low latency, full configurability, and seamless scalability.
1:56

Conversational AI is here. Build AI agents that can speak in minutes with low latency, full configurability, and seamless scalability.

ElevenLabs

770,731 views • 1 year ago

Give your voice agents a face with Decart • ElevenLabs generates speech stream • Decart animates the video avatar in realtime with lip sync • Pipecat orchestrates the pipeline and handles WebRTC transport Open-source demo code below.
4:10

Give your voice agents a face with Decart • ElevenLabs generates speech stream • Decart animates the video avatar in realtime with lip sync • Pipecat orchestrates the pipeline and handles WebRTC transport Open-source demo code below.

ElevenLabs Developers

29,269 views • 11 months ago

📣 You can now have a conversation with Meta AI using voice. It’s super fast, connected to the web, natural and conversational and even comes with celebrity voice options from Awkwafina, Kristen Bell, John Cena, and more. What voice speaks to you? (pun intended 😆)
1:30

📣 You can now have a conversation with Meta AI using voice. It’s super fast, connected to the web, natural and conversational and even comes with celebrity voice options from Awkwafina, Kristen Bell, John Cena, and more. What voice speaks to you? (pun intended 😆)

Ahmad Al-Dahle

112,760 views • 2 years ago

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...
3:28

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...

kwindla

333,621 views • 1 month ago