正在加载视频...

视频加载失败

返回首页

Build a conversational voice bot with 1 second voice-to-voice latency with Modal, Pipecat AI, and open models. Modal works seamlessly with WebRTC, WebSockets, and tunneling to squash latency to an absolute minimum.

modal's profile picture

Modal

25,259 subscribers

29,559 次观看 • 9 个月前 •via X (Twitter)

教育科学技术
LIVE

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Instant Voice Cloning 3 seconds of audio is all you need. Text-to-voice AI generated voices with <1 second latency. 🧵 A thread
1:19

Instant Voice Cloning 3 seconds of audio is all you need. Text-to-voice AI generated voices with <1 second latency. 🧵 A thread

Linus ✦ Ekenstam

576,497 次观看 • 3 年前

Learn to build conversational AI voice agents in "Building AI Voice Agents for Production", created in collaboration with LiveKit and RealAvatar, and taught by dsa (Co-founder & CEO of LiveKit), Shayne (Developer Advocate, LiveKit), and Nedelina Teneva (Head of AI at RealAvatar, an AI Fund portfolio company). Voice agents combine speech and reasoning capabilities to enable real-time conversations. They're already being used to support customer service, to improve accessibility in healthcare, for entertainment applications, and for talk therapy. In this course, you’ll learn to build voice agents that listen, reason, and respond naturally. You’ll follow the architecture used to create the "AI Andrew" Avatar, a collaborative project between and RealAvatar that responds to users in what sounds like my voice. You’ll build a voice agent from scratch and deploy it to the cloud, enabling support for many simultaneous users. What you’ll learn: - Understand the fundamentals of voice agents, including key components like speech-to-text (STT), text-to-speech (TTS), and LLMs, and how latency is introduced at each layer. - Explore voice agent architectures and the trade-offs between modular pipelines and speech-to-speech APIs. - Explore how platforms like LiveKit mitigate latency issues with optimized networking infrastructure and low-latency communication protocols. - Learn how to connect client devices to voice agents using WebRTC—and why it outperforms HTTP and WebSocket for low-latency audio streaming. - Incorporate voice activity detection (VAD), end-of-turn detection, and context management to detect turns, handle interruptions, and manage conversational flow. - Understand the trade-offs between latency, quality, and cost in an example in which you build a voice agent and change its voice. - Equip your agent with metrics to measure latency at each stage of the voice pipeline and learn the key levers you can pull to make your agent faster and more responsive. The voice agents built in this course also incorporate voice technology from , a supporting contributor to the project. By the end of this course, you'll have learned the components of an AI voice agent pipeline, combined them into a system with low-latency communication, and deployed them on cloud infrastructure so it scales to many users. I’m looking forward to seeing what voice agents you build from this course! Please sign up here:
2:41

Learn to build conversational AI voice agents in "Building AI Voice Agents for Production", created in collaboration with LiveKit and RealAvatar, and taught by dsa (Co-founder & CEO of LiveKit), Shayne (Developer Advocate, LiveKit), and Nedelina Teneva (Head of AI at RealAvatar, an AI Fund portfolio company). Voice agents combine speech and reasoning capabilities to enable real-time conversations. They're already being used to support customer service, to improve accessibility in healthcare, for entertainment applications, and for talk therapy. In this course, you’ll learn to build voice agents that listen, reason, and respond naturally. You’ll follow the architecture used to create the "AI Andrew" Avatar, a collaborative project between and RealAvatar that responds to users in what sounds like my voice. You’ll build a voice agent from scratch and deploy it to the cloud, enabling support for many simultaneous users. What you’ll learn: - Understand the fundamentals of voice agents, including key components like speech-to-text (STT), text-to-speech (TTS), and LLMs, and how latency is introduced at each layer. - Explore voice agent architectures and the trade-offs between modular pipelines and speech-to-speech APIs. - Explore how platforms like LiveKit mitigate latency issues with optimized networking infrastructure and low-latency communication protocols. - Learn how to connect client devices to voice agents using WebRTC—and why it outperforms HTTP and WebSocket for low-latency audio streaming. - Incorporate voice activity detection (VAD), end-of-turn detection, and context management to detect turns, handle interruptions, and manage conversational flow. - Understand the trade-offs between latency, quality, and cost in an example in which you build a voice agent and change its voice. - Equip your agent with metrics to measure latency at each stage of the voice pipeline and learn the key levers you can pull to make your agent faster and more responsive. The voice agents built in this course also incorporate voice technology from , a supporting contributor to the project. By the end of this course, you'll have learned the components of an AI voice agent pipeline, combined them into a system with low-latency communication, and deployed them on cloud infrastructure so it scales to many users. I’m looking forward to seeing what voice agents you build from this course! Please sign up here:

Andrew Ng

87,628 次观看 • 1 年前

🔊🔛🔥 ... Groq Inc launched voice generation today. GroqCloud now has realtime transcription, LLMs, *and* text-to-speech. You can build super-responsive, ultra low-latency voice agents end-to-end entirely on Groq! At Daily, we're big fans of Groq's fast, low-latency inference. Pipecat AI supports all the Groq models, including the new voice model. Lately, we've been obsessively playing a voice chat game that Mark Backman wrote. (My high score is 6, so far.) Here's Mark, with Groq's `Celeste-PlayAI` voice.
0:47

🔊🔛🔥 ... Groq Inc launched voice generation today. GroqCloud now has realtime transcription, LLMs, *and* text-to-speech. You can build super-responsive, ultra low-latency voice agents end-to-end entirely on Groq! At Daily, we're big fans of Groq's fast, low-latency inference. Pipecat AI supports all the Groq models, including the new voice model. Lately, we've been obsessively playing a voice chat game that Mark Backman wrote. (My high score is 6, so far.) Here's Mark, with Groq's `Celeste-PlayAI` voice.

kwindla

31,791 次观看 • 1 年前

Today, we're introducing PlayHT2.0 - our new Conversational Text-to-Voice AI Model that's trained and built to generate humanlike conversations across languages with <1s latency. Sign up for beta access -
1:19

Today, we're introducing PlayHT2.0 - our new Conversational Text-to-Voice AI Model that's trained and built to generate humanlike conversations across languages with <1s latency. Sign up for beta access -

PlayAI

52,378 次观看 • 3 年前

The future of artificial intelligence is physical. Physical Intelligence runs robotic control inference on Modal with >2x lower latency than the lag between your brain and your finger.
0:20

The future of artificial intelligence is physical. Physical Intelligence runs robotic control inference on Modal with >2x lower latency than the lag between your brain and your finger.

Modal

95,966 次观看 • 3 个月前

How to build the world's fastest voice AI bot: - Self-host speech-to-text, LLM inference, and text-to-speech all together in the same container/cluster. - Route audio over the internet using WebRTC and edge networking. - Configure timings for voice activity detection, phrase endpointing, and other parts of the pipeline to optimize for latency. (There are trade-offs to doing this!) Here's a LLama 3 voice bot that has voice-to-voice response times of ~500ms. We used Deepgram's STT and TTS for this bot, and everything is hosted on cerebriumai's serverless GPU infrastructure.
4:18

How to build the world's fastest voice AI bot: - Self-host speech-to-text, LLM inference, and text-to-speech all together in the same container/cluster. - Route audio over the internet using WebRTC and edge networking. - Configure timings for voice activity detection, phrase endpointing, and other parts of the pipeline to optimize for latency. (There are trade-offs to doing this!) Here's a LLama 3 voice bot that has voice-to-voice response times of ~500ms. We used Deepgram's STT and TTS for this bot, and everything is hosted on cerebriumai's serverless GPU infrastructure.

kwindla

282,617 次观看 • 2 年前

Oh my…FlashLabs releases Chroma 1.0, first open-source real-time speech-to-speech model with personalized voice cloning. Native speech-to-speech with <150ms latency & voice cloning from seconds of audio. Finally, an open alternative to OpenAI Realtime.
1:57

Oh my…FlashLabs releases Chroma 1.0, first open-source real-time speech-to-speech model with personalized voice cloning. Native speech-to-speech with <150ms latency & voice cloning from seconds of audio. Finally, an open alternative to OpenAI Realtime.

Alvaro Cintas

46,151 次观看 • 6 个月前

xAI is cooking 🔥 They just launched Grok Voice, a platform for building AI voice agents. - You can build voice agents in under 2 minutes - No code required - Human-like conversations with sub-second latency - Supports 25+ languages - 80+ built-in voices or clone your own - Works with Gmail, Google Calendar, Outlook, Notion, MCPs, and more - Starts at $0.05/min and includes a free phone number
0:46

xAI is cooking 🔥 They just launched Grok Voice, a platform for building AI voice agents. - You can build voice agents in under 2 minutes - No code required - Human-like conversations with sub-second latency - Supports 25+ languages - 80+ built-in voices or clone your own - Works with Gmail, Google Calendar, Outlook, Notion, MCPs, and more - Starts at $0.05/min and includes a free phone number

AshutoshShrivastava

144,037 次观看 • 1 个月前

Developers are building great voice experiences with the Realtime API. See how Lemonade uses it to power AI Maya, their friendly and engaging voice agent. The Realtime API's automatic voice detection and low latency enable them to offer 24/7 multilingual phone support 📞
4:50

Developers are building great voice experiences with the Realtime API. See how Lemonade uses it to power AI Maya, their friendly and engaging voice agent. The Realtime API's automatic voice detection and low latency enable them to offer 24/7 multilingual phone support 📞

OpenAI Developers

126,135 次观看 • 1 年前

New course: Add voice to your AI agents and applications, built with Vocal Bridge (disclosure: an AI Fund portfolio company) and taught by its CEO Ashwyn Sharma. Voice applications historically required making a hard tradeoff: using fast voice-to-voice models that sacrifice reliability, or accurate speech-to-text pipelines that add latency. This course teaches you how to build voice agents that are both reliable and fast. You'll build three types of voice-enabled applications: a voice-interactive game where voice commands and mouse clicks work together over a single channel, an agent that gains a voice in about 10 lines of code without touching its prompts or tools, and an agent that places outbound phone calls using a make_phone_call function. Skills you'll gain: - Add a voice layer to an existing agent without rewriting your prompts, RAG pipeline, or tools - Give an agent the ability to place outbound calls and stream transcripts back live - Set up voice evaluation to score calls, catch regressions, and improve quality before deployment Join and add voice to your agents without overhauling your architecture:
2:27

New course: Add voice to your AI agents and applications, built with Vocal Bridge (disclosure: an AI Fund portfolio company) and taught by its CEO Ashwyn Sharma. Voice applications historically required making a hard tradeoff: using fast voice-to-voice models that sacrifice reliability, or accurate speech-to-text pipelines that add latency. This course teaches you how to build voice agents that are both reliable and fast. You'll build three types of voice-enabled applications: a voice-interactive game where voice commands and mouse clicks work together over a single channel, an agent that gains a voice in about 10 lines of code without touching its prompts or tools, and an agent that places outbound phone calls using a make_phone_call function. Skills you'll gain: - Add a voice layer to an existing agent without rewriting your prompts, RAG pipeline, or tools - Give an agent the ability to place outbound calls and stream transcripts back live - Set up voice evaluation to score calls, catch regressions, and improve quality before deployment Join and add voice to your agents without overhauling your architecture:

Andrew Ng

87,140 次观看 • 1 个月前

I built the best memory for voice agents while sponsoring Y Combinator hackathon. Introducing: Voice AI + Memory! We built a deep supermemory integration for Pipecat AI, so now even your agents have great memory With user profiles, these are almost-instant latency as well :)
2:11

I built the best memory for voice agents while sponsoring Y Combinator hackathon. Introducing: Voice AI + Memory! We built a deep supermemory integration for Pipecat AI, so now even your agents have great memory With user profiles, these are almost-instant latency as well :)

Dhravya Shah

83,207 次观看 • 5 个月前

Conversational AI is here. Build AI agents that can speak in minutes with low latency, full configurability, and seamless scalability.
1:56

Conversational AI is here. Build AI agents that can speak in minutes with low latency, full configurability, and seamless scalability.

ElevenLabs

770,091 次观看 • 1 年前

GLM-5, the latest frontier open model from Z.ai, is available now on Modal. We partnered with to release an endpoint that will be free for a limited time.
0:45

GLM-5, the latest frontier open model from Z.ai, is available now on Modal. We partnered with to release an endpoint that will be free for a limited time.

Modal

58,785 次观看 • 5 个月前

📣 You can now have a conversation with Meta AI using voice. It’s super fast, connected to the web, natural and conversational and even comes with celebrity voice options from Awkwafina, Kristen Bell, John Cena, and more. What voice speaks to you? (pun intended 😆)
1:30

📣 You can now have a conversation with Meta AI using voice. It’s super fast, connected to the web, natural and conversational and even comes with celebrity voice options from Awkwafina, Kristen Bell, John Cena, and more. What voice speaks to you? (pun intended 😆)

Ahmad Al-Dahle

112,760 次观看 • 1 年前

Give your voice agents a face with Decart • ElevenLabs generates speech stream • Decart animates the video avatar in realtime with lip sync • Pipecat orchestrates the pipeline and handles WebRTC transport Open-source demo code below.
4:10

Give your voice agents a face with Decart • ElevenLabs generates speech stream • Decart animates the video avatar in realtime with lip sync • Pipecat orchestrates the pipeline and handles WebRTC transport Open-source demo code below.

ElevenLabs Developers

29,269 次观看 • 9 个月前

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses *three* NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.
1:24

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses *three* NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.

kwindla

274,562 次观看 • 6 个月前

Is this the tiniest little voice agent yet?! My ElevenLabs voice clone running on an esp32 microcontroller via Pipecat AI and WebRTC! 🔥 Story time: I recently caught up with Danilo Campos who is building the awesome DeskHog (seriously, check it out!) at PostHog and he told me about M5Stack, who is building modular open-source hardware. So on a recent trip to Shenzhen, I went to visit their office and got myself a little AtomS3R and atomic echo base thinking: "How hard can it be to make this little buddy conversational?!" Ohh boy was I wrong. This thing was wrecking my brain and LLMs are currently absolutely useless here also, so I was just completely lost. But I tend to be lucky with timing, so just days later kwindla posted that the Pipecat AI team was working on a pipecat-esp32 WebRTC client. What are the chances?! So within days, the awesome Sean DuBois and Pipecat team had a PR open to make the pipecat-esp32 client run on the AtomS3R 🤯 You gotta love the open source community 🫶
3:13

Is this the tiniest little voice agent yet?! My ElevenLabs voice clone running on an esp32 microcontroller via Pipecat AI and WebRTC! 🔥 Story time: I recently caught up with Danilo Campos who is building the awesome DeskHog (seriously, check it out!) at PostHog and he told me about M5Stack, who is building modular open-source hardware. So on a recent trip to Shenzhen, I went to visit their office and got myself a little AtomS3R and atomic echo base thinking: "How hard can it be to make this little buddy conversational?!" Ohh boy was I wrong. This thing was wrecking my brain and LLMs are currently absolutely useless here also, so I was just completely lost. But I tend to be lucky with timing, so just days later kwindla posted that the Pipecat AI team was working on a pipecat-esp32 WebRTC client. What are the chances?! So within days, the awesome Sean DuBois and Pipecat team had a PR open to make the pipecat-esp32 client run on the AtomS3R 🤯 You gotta love the open source community 🫶

Thor 雷神 ⚡️

36,687 次观看 • 1 年前