Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Further tinkering with my little French tutor app. This version is using the Gemini Multimodal Live API. The speech understanding in Gemini is quite something. In this video you can see Gemini correcting my pronunciation. (Very patiently.) The language tutor use case really highlights the strengths of a next-generation... show more

kwindla

9,989 subscribers

13,005 views • 1 year ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

12 Comments

kwindla1 year ago

Code is here: Here are the docs for Pipecat's open source client SDKs for javascript, React, iOS, Android, React Native, and C++: The Pipecat clients all support WebRTC, WebSocket, and HTTP network transports.

Nexus 👾1 year ago

Nexus combines CMC’s market insights with Zerion and Debank’s wallet tracking—all in one easy-to-use dashboard. Simplify your crypto journey today, completely free 🚀 Try it now!

The Canaanite1 year ago

bravo, excellente prononciation monsieur 🤣

عای‌ خان1 year ago

Amazing. I need to code up a Maths tutor for my kids for those inevitable times where my patience sadly failed. 😅

Emanuel Perez1 year ago

Whoa

Samuel Ekpe1 year ago

interesting

Mitja Martini1 year ago

Thanks for sharing this very nice use case! I've been using ChatGPT with my daughter to help her learning Latin, not in voice mode, though :)

Nicolas MICHEL1 year ago

it is funny, the machine has an African accent. This will lead to some funny situations if you start speaking in public with this accent 😂

Jake1 year ago

cool any other interesting uncommon use cases? I feel like the voice use cases are going to blow up this year

kwindla1 year ago

> any other interesting uncommon use cases? Check out the voice interaction experiments that @trudypainter has been posting!

Dan Goodman 🍊1 year ago

Can you have it take in audio but output text in pipecat?

kwindla1 year ago

> Can you have it take in audio but output text in pipecat? Yes!

Related Videos

Gemini 2.0 drops the beat. Watch the video all the way through — I had four legit "no way it did that" reactions when Jon Taylor sent this to me. "It looks like it's only hitting on the first beat of each bar." This is Jon collaborating with Gemini to create a song in Ableton Live. Jon is using the Multimodal Live API to stream audio and video to Gemini and have a conversation about the song he's creating.

Gemini 2.0 drops the beat. Watch the video all the way through — I had four legit "no way it did that" reactions when Jon Taylor sent this to me. "It looks like it's only hitting on the first beat of each bar." This is Jon collaborating with Gemini to create a song in Ableton Live. Jon is using the Multimodal Live API to stream audio and video to Gemini and have a conversation about the song he's creating.

kwindla

86,132 views • 1 year ago

Better/faster/cheaper voice AI turn detection with Gemini 2.0 The code that determines when the agent should respond to the user is some of the most important code in your voice AI agent. The technical terms for this job are "turn detection" or "phrase endpointing." If the voice AI responds before the user has finished their thought, the conversation is choppy and unproductive. If the AI waits too long, the conversation is slow and frustrating. There are a number of ways to approach this. You can: 1. Use a fast "voice activity detection" model to detect pauses in speech. Respond when the user pauses. 2. Use a specialized phrase endpointing model that operates on transcribed text, pattern-matching on text semantics. 3. Train a specialized phrase endpointing model that operates directly on audio. 4. Leverage the native audio capabilities of a SOTA LLM like Gemini 2.0. We've benchmarked all four of these, and Gemini 2.0 currently beats other approaches. Using Gemini is also cheaper than transcribing the audio separately using a transcription service or model. Here's a short video showing Gemini phrase endpointing in two scenarios. First, correctly handling pauses in natural conversation. Second, requesting a phone number (which is a common activity in a use case like customer support). You can see the Completeness check lines in the terminal output, printed each time Gemini processes a chunk of audio.

Better/faster/cheaper voice AI turn detection with Gemini 2.0 The code that determines when the agent should respond to the user is some of the most important code in your voice AI agent. The technical terms for this job are "turn detection" or "phrase endpointing." If the voice AI responds before the user has finished their thought, the conversation is choppy and unproductive. If the AI waits too long, the conversation is slow and frustrating. There are a number of ways to approach this. You can: 1. Use a fast "voice activity detection" model to detect pauses in speech. Respond when the user pauses. 2. Use a specialized phrase endpointing model that operates on transcribed text, pattern-matching on text semantics. 3. Train a specialized phrase endpointing model that operates directly on audio. 4. Leverage the native audio capabilities of a SOTA LLM like Gemini 2.0. We've benchmarked all four of these, and Gemini 2.0 currently beats other approaches. Using Gemini is also cheaper than transcribing the audio separately using a transcription service or model. Here's a short video showing Gemini phrase endpointing in two scenarios. First, correctly handling pauses in natural conversation. Second, requesting a phone number (which is a common activity in a use case like customer support). You can see the Completeness check lines in the terminal output, printed each time Gemini processes a chunk of audio.

kwindla

64,190 views • 1 year ago

Multilingual voice AI tutor using the new Cartesia Sonic-2 voice model and Google DeepMind Gemini 2.0 Flash native audio understanding ...

Multilingual voice AI tutor using the new Cartesia Sonic-2 voice model and Google DeepMind Gemini 2.0 Flash native audio understanding ...

kwindla

18,793 views • 1 year ago

Just vibe coded with Gemini 3: an AI sparring partner to help you practice pitching skills. general chatbot is "too nice" to do that quite shocked by the low-latency audio stream by Gemini live api Google Gemini , vibe building a audio native app is no longer a dream

Just vibe coded with Gemini 3: an AI sparring partner to help you practice pitching skills. general chatbot is "too nice" to do that quite shocked by the low-latency audio stream by Gemini live api Google Gemini , vibe building a audio native app is no longer a dream

Jay Ding

12,289 views • 6 months ago

Gemini 3 Pro is the best model in the world for multimodal understanding. One of its most exciting capabilities is document understanding and reasoning. This means you can convert information in any format and into the medium that works best for you. Gemini 3 also has leading multilingual capabilities, enabling it to process, reason and even capture cultural relevance across a variety of languages. For example, here Gemini 3 is translating handwritten recipes in Korean and English to build a digital family cookbook in different languages.

Gemini 3 Pro is the best model in the world for multimodal understanding. One of its most exciting capabilities is document understanding and reasoning. This means you can convert information in any format and into the medium that works best for you. Gemini 3 also has leading multilingual capabilities, enabling it to process, reason and even capture cultural relevance across a variety of languages. For example, here Gemini 3 is translating handwritten recipes in Korean and English to build a digital family cookbook in different languages.

Google AI

36,696 views • 8 months ago

The new Sonic-3 voice model from launched today. The big additions are increased emotional range, emotion steering, and laughter tags. For example: I wrote some quick demo code, prompting Gemini Flash to know about the emotion tags. You can hear the results in the video, and see the emotion tags in the Pipecat developer console. Code is below.

The new Sonic-3 voice model from launched today. The big additions are increased emotional range, emotion steering, and laughter tags. For example: I wrote some quick demo code, prompting Gemini Flash to know about the emotion tags. You can hear the results in the video, and see the emotion tags in the Pipecat developer console. Code is below.

kwindla

12,350 views • 9 months ago

Today we’re taking a big step on the path toward AGI and releasing Gemini 3— our most intelligent model yet. With Gemini 3, you can bring any idea to life. It is state-of-the-art in reasoning, the best model in the world for multimodal understanding, and our best agentic and vibe coding model.

Today we’re taking a big step on the path toward AGI and releasing Gemini 3— our most intelligent model yet. With Gemini 3, you can bring any idea to life. It is state-of-the-art in reasoning, the best model in the world for multimodal understanding, and our best agentic and vibe coding model.

Google AI

492,973 views • 8 months ago

Talk to Gemini on your phone ... Deploy your own Gemini Multimodal Live voice agent that you can call on the phone (or that can call you) in 5 minutes.

Talk to Gemini on your phone ... Deploy your own Gemini Multimodal Live voice agent that you can call on the phone (or that can call you) in 5 minutes.

kwindla

38,680 views • 1 year ago

(1/5) Gemini 3, our most intelligent model, is landing in Google Search today – starting with AI Mode. Excited that this is the first time we’re shipping a new Gemini model in Search on day one! 🚀 In Search, Gemini 3 with generative layouts will make it easy to get a rich understanding of anything on your mind. It has state-of-the-art reasoning, deep multimodal understanding and advanced agentic capabilities. That allows the model to shine when you ask it to explain advanced concepts or ideas – it reasons and can code interactive visuals in real-time. It can tackle your toughest questions like advanced science.

(1/5) Gemini 3, our most intelligent model, is landing in Google Search today – starting with AI Mode. Excited that this is the first time we’re shipping a new Gemini model in Search on day one! 🚀 In Search, Gemini 3 with generative layouts will make it easy to get a rich understanding of anything on your mind. It has state-of-the-art reasoning, deep multimodal understanding and advanced agentic capabilities. That allows the model to shine when you ask it to explain advanced concepts or ideas – it reasons and can code interactive visuals in real-time. It can tackle your toughest questions like advanced science.

Robby Stein

94,877 views • 8 months ago

With Gemini 3, you can take an image of anything, like a board game, a floor plan, or a doodle, and make it an interactive experience. This is because of Gemini 3’s deep multimodal understanding. The model is able to analyze the content of the image, distill the most compelling parts, and then translate the image’s details into a functional, interactive experience.

With Gemini 3, you can take an image of anything, like a board game, a floor plan, or a doodle, and make it an interactive experience. This is because of Gemini 3’s deep multimodal understanding. The model is able to analyze the content of the image, distill the most compelling parts, and then translate the image’s details into a functional, interactive experience.

Google AI

98,281 views • 8 months ago

How does an AI model actually learn to see? 🤖 Learn about the tech behind native multimodality, how models reason over visual data like documents and video, and the future of proactive AI assistants with Logan Kilpatrick and Gemini Model Behavior Product Lead, Ani Baddepudi. ↓ Timestamps: 01:12 Why Gemini is natively multimodal 02:23 The technology behind multimodal models 05:15 Video understanding with Gemini 2.5 09:25 Deciding what to build next 13:23 Building new product experiences with multimodal AI 17:15 The vision for proactive assistants 24:13 Improving video usability with variable FPS and frame tokenization 27:35 What’s next for Gemini’s multimodal development 31:47 Deep dive on Gemini’s document understanding capabilities 37:56 The teamwork and collaboration behind Gemini 40:56 What’s next with model behavior

How does an AI model actually learn to see? 🤖 Learn about the tech behind native multimodality, how models reason over visual data like documents and video, and the future of proactive AI assistants with Logan Kilpatrick and Gemini Model Behavior Product Lead, Ani Baddepudi. ↓ Timestamps: 01:12 Why Gemini is natively multimodal 02:23 The technology behind multimodal models 05:15 Video understanding with Gemini 2.5 09:25 Deciding what to build next 13:23 Building new product experiences with multimodal AI 17:15 The vision for proactive assistants 24:13 Improving video usability with variable FPS and frame tokenization 27:35 What’s next for Gemini’s multimodal development 31:47 Deep dive on Gemini’s document understanding capabilities 37:56 The teamwork and collaboration behind Gemini 40:56 What’s next with model behavior

Google AI

58,703 views • 1 year ago

Can AI chain tools together to complete complex tasks in real-time? Gemini 2.0 can. Here's a React demo using the new Multimodal Live API to chain code execution + real-time Google Search. I'm amazed at how fast this is.

Can AI chain tools together to complete complex tasks in real-time? Gemini 2.0 can. Here's a React demo using the new Multimodal Live API to chain code execution + real-time Google Search. I'm amazed at how fast this is.

Addy Osmani

17,782 views • 1 year ago

Gemini 2.0 with Code Execution is actually INSANE🔥 90% don't even know about this feature. - Run Python code in real-time - Handles complex data analysis & makes charts instantly - Works in both API & AI Studio - Use Multimodal Live API and Reasoning model Here's how you can use 👇 1/4 Example 1: Real-time analysis & visualization

Gemini 2.0 with Code Execution is actually INSANE🔥 90% don't even know about this feature. - Run Python code in real-time - Handles complex data analysis & makes charts instantly - Works in both API & AI Studio - Use Multimodal Live API and Reasoning model Here's how you can use 👇 1/4 Example 1: Real-time analysis & visualization

AshutoshShrivastava

92,641 views • 1 year ago

We’re expanding the Gemini 3 family with the launch of Gemini 3 Flash. This model: — Combines Gemini 3’s Pro-grade reasoning with Flash-level latency, efficiency, and cost — Delivers frontier-level performance on PHD-level reasoning and knowledge benchmarks — Is our most impressive model for agentic workflows Watch Gemini 3 Flash handle hundreds of function calling options at low latency, compiling global recipes from 100 ingredients and 100 kitchen tools.

We’re expanding the Gemini 3 family with the launch of Gemini 3 Flash. This model: — Combines Gemini 3’s Pro-grade reasoning with Flash-level latency, efficiency, and cost — Delivers frontier-level performance on PHD-level reasoning and knowledge benchmarks — Is our most impressive model for agentic workflows Watch Gemini 3 Flash handle hundreds of function calling options at low latency, compiling global recipes from 100 ingredients and 100 kitchen tools.

Google AI

229,722 views • 7 months ago

Watch this demo from Thor 雷神 ⚡️ to see Gemini 3.1 Flash Live in action as a real-time DJ. The model uses function calling (to the Gemini API) to generate custom 30-second clips using Lyria 3️⃣ Start your own studio session in Google AI Studio:

Watch this demo from Thor 雷神 ⚡️ to see Gemini 3.1 Flash Live in action as a real-time DJ. The model uses function calling (to the Gemini API) to generate custom 30-second clips using Lyria 3️⃣ Start your own studio session in Google AI Studio:

Google AI Developers

11,216 views • 3 months ago

Gemini 3.5 Live Translate! We just shipped a real-time babel fish. - 70+ languages, 2,000+ language pairs. - Natural translated speech, works in noisy environments. - Stays in sync with the speaker, no lag, no awkward pauses. - Auto-detects the language being spoken. Available today in Google Translate (Android & iOS), the Gemini API (Public Preview), and Google Meet (Private Preview). I genuinely think this is the beginning of the end of language barriers. Anyone can now speak and understand anyone.

Gemini 3.5 Live Translate! We just shipped a real-time babel fish. - 70+ languages, 2,000+ language pairs. - Natural translated speech, works in noisy environments. - Stays in sync with the speaker, no lag, no awkward pauses. - Auto-detects the language being spoken. Available today in Google Translate (Android & iOS), the Gemini API (Public Preview), and Google Meet (Private Preview). I genuinely think this is the beginning of the end of language barriers. Anyone can now speak and understand anyone.

Philipp Schmid

17,419 views • 1 month ago

Gemini 3 is indeed impressive. In my limited testing with it, it feels like it is able to take on larger, complex prompts with ease and low failure rate. I think one of the biggest strengths that Gemini 3 has is its ability to generate coherent looking 3D voxel or SVG art, which makes it super easy to create game prototypes without having to implement art at all. Made this Heroes of might and magic-inspired overworld prototype with mini map and basic UI in less than 2 hours. Really excited to test this more. Demis Hassabis and Logan Kilpatrick and the Gemini team, you guys really cooked this time!

Gemini 3 is indeed impressive. In my limited testing with it, it feels like it is able to take on larger, complex prompts with ease and low failure rate. I think one of the biggest strengths that Gemini 3 has is its ability to generate coherent looking 3D voxel or SVG art, which makes it super easy to create game prototypes without having to implement art at all. Made this Heroes of might and magic-inspired overworld prototype with mini map and basic UI in less than 2 hours. Really excited to test this more. Demis Hassabis and Logan Kilpatrick and the Gemini team, you guys really cooked this time!

Danny Limanseta

119,970 views • 8 months ago

Super exciting… Google Maps is getting a huge Gemini-powered upgrade! We’re bringing together Maps’s fresh, reliable information about the real world with the conversational intelligence of Gemini. It’s our next big step in building the map of the future. 🧵 (1/3)

Super exciting… Google Maps is getting a huge Gemini-powered upgrade! We’re bringing together Maps’s fresh, reliable information about the real world with the conversational intelligence of Gemini. It’s our next big step in building the map of the future. 🧵 (1/3)

Nick Fox

203,746 views • 9 months ago

Really happy to see the interest around our “Hands-on with Gemini” video. In our developer blog yesterday, we broke down how Gemini was used to create it. We gave Gemini sequences of different modalities — image and text in this case — and had it respond by predicting what might come next. Devs can try similar things when access to Pro opens on 12/13 🚀. The knitting demo used Ultra⚡ All the user prompts and outputs in the video are real, shortened for brevity. The video illustrates what the multimodal user experiences built with Gemini could look like. We made it to inspire developers. When you’re building an app, you can get similar results (there’s always some variability with LLMs) by prompting Gemini with an instruction that allows the user to "configure" the behavior of the model, like inputting “you are an expert in science …” before a user can engage in the same kind of back and forth dialogue. Here’s a clip of what this looks like in AI Studio with Gemini Pro. We’ve come a long way since Flamingo 🦩 & PALI, looking forward to seeing what people build with it!

Really happy to see the interest around our “Hands-on with Gemini” video. In our developer blog yesterday, we broke down how Gemini was used to create it. We gave Gemini sequences of different modalities — image and text in this case — and had it respond by predicting what might come next. Devs can try similar things when access to Pro opens on 12/13 🚀. The knitting demo used Ultra⚡ All the user prompts and outputs in the video are real, shortened for brevity. The video illustrates what the multimodal user experiences built with Gemini could look like. We made it to inspire developers. When you’re building an app, you can get similar results (there’s always some variability with LLMs) by prompting Gemini with an instruction that allows the user to "configure" the behavior of the model, like inputting “you are an expert in science …” before a user can engage in the same kind of back and forth dialogue. Here’s a clip of what this looks like in AI Studio with Gemini Pro. We’ve come a long way since Flamingo 🦩 & PALI, looking forward to seeing what people build with it!

Oriol Vinyals

180,966 views • 2 years ago