Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Further tinkering with my little French tutor app. This version is using the Gemini Multimodal Live API. The speech understanding in Gemini is quite something. In this video you can see Gemini correcting my pronunciation. (Very patiently.) The language tutor use case really highlights the strengths of a next-generation...

kwindla's profile picture

kwindla

9,989 subscribers

13,005 Aufrufe • vor 1 Jahr •via X (Twitter)

BildungWissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

12 Kommentare

Profilbild von kwindla
kwindlavor 1 Jahr

Code is here: Here are the docs for Pipecat's open source client SDKs for javascript, React, iOS, Android, React Native, and C++: The Pipecat clients all support WebRTC, WebSocket, and HTTP network transports.

Profilbild von Nexus 👾
Nexus 👾vor 1 Jahr

Nexus combines CMC’s market insights with Zerion and Debank’s wallet tracking—all in one easy-to-use dashboard. Simplify your crypto journey today, completely free 🚀 Try it now!

Profilbild von The Canaanite
The Canaanitevor 1 Jahr

bravo, excellente prononciation monsieur 🤣

Profilbild von عای‌ خان
عای‌ خانvor 1 Jahr

Amazing. I need to code up a Maths tutor for my kids for those inevitable times where my patience sadly failed. 😅

Profilbild von Emanuel Perez
Emanuel Perezvor 1 Jahr

Whoa

Profilbild von Samuel Ekpe
Samuel Ekpevor 1 Jahr

interesting

Profilbild von Mitja Martini
Mitja Martinivor 1 Jahr

Thanks for sharing this very nice use case! I've been using ChatGPT with my daughter to help her learning Latin, not in voice mode, though :)

Profilbild von Nicolas MICHEL
Nicolas MICHELvor 1 Jahr

it is funny, the machine has an African accent. This will lead to some funny situations if you start speaking in public with this accent 😂

Profilbild von Jake
Jakevor 1 Jahr

cool any other interesting uncommon use cases? I feel like the voice use cases are going to blow up this year

Profilbild von kwindla
kwindlavor 1 Jahr

> any other interesting uncommon use cases? Check out the voice interaction experiments that @trudypainter has been posting!

Profilbild von Dan Goodman 🍊
Dan Goodman 🍊vor 1 Jahr

Can you have it take in audio but output text in pipecat?

Profilbild von kwindla
kwindlavor 1 Jahr

> Can you have it take in audio but output text in pipecat? Yes!

Ähnliche Videos

Gemini 2.0 drops the beat. Watch the video all the way through — I had four legit "no way it did that" reactions when Jon Taylor sent this to me. "It looks like it's only hitting on the first beat of each bar." This is Jon collaborating with Gemini to create a song in Ableton Live. Jon is using the Multimodal Live API to stream audio and video to Gemini and have a conversation about the song he's creating.
4:15

Gemini 2.0 drops the beat. Watch the video all the way through — I had four legit "no way it did that" reactions when Jon Taylor sent this to me. "It looks like it's only hitting on the first beat of each bar." This is Jon collaborating with Gemini to create a song in Ableton Live. Jon is using the Multimodal Live API to stream audio and video to Gemini and have a conversation about the song he's creating.

kwindla

86,132 Aufrufe • vor 1 Jahr

Better/faster/cheaper voice AI turn detection with Gemini 2.0 The code that determines when the agent should respond to the user is some of the most important code in your voice AI agent. The technical terms for this job are "turn detection" or "phrase endpointing." If the voice AI responds before the user has finished their thought, the conversation is choppy and unproductive. If the AI waits too long, the conversation is slow and frustrating. There are a number of ways to approach this. You can: 1. Use a fast "voice activity detection" model to detect pauses in speech. Respond when the user pauses. 2. Use a specialized phrase endpointing model that operates on transcribed text, pattern-matching on text semantics. 3. Train a specialized phrase endpointing model that operates directly on audio. 4. Leverage the native audio capabilities of a SOTA LLM like Gemini 2.0. We've benchmarked all four of these, and Gemini 2.0 currently beats other approaches. Using Gemini is also cheaper than transcribing the audio separately using a transcription service or model. Here's a short video showing Gemini phrase endpointing in two scenarios. First, correctly handling pauses in natural conversation. Second, requesting a phone number (which is a common activity in a use case like customer support). You can see the Completeness check lines in the terminal output, printed each time Gemini processes a chunk of audio.
1:23

Better/faster/cheaper voice AI turn detection with Gemini 2.0 The code that determines when the agent should respond to the user is some of the most important code in your voice AI agent. The technical terms for this job are "turn detection" or "phrase endpointing." If the voice AI responds before the user has finished their thought, the conversation is choppy and unproductive. If the AI waits too long, the conversation is slow and frustrating. There are a number of ways to approach this. You can: 1. Use a fast "voice activity detection" model to detect pauses in speech. Respond when the user pauses. 2. Use a specialized phrase endpointing model that operates on transcribed text, pattern-matching on text semantics. 3. Train a specialized phrase endpointing model that operates directly on audio. 4. Leverage the native audio capabilities of a SOTA LLM like Gemini 2.0. We've benchmarked all four of these, and Gemini 2.0 currently beats other approaches. Using Gemini is also cheaper than transcribing the audio separately using a transcription service or model. Here's a short video showing Gemini phrase endpointing in two scenarios. First, correctly handling pauses in natural conversation. Second, requesting a phone number (which is a common activity in a use case like customer support). You can see the Completeness check lines in the terminal output, printed each time Gemini processes a chunk of audio.

kwindla

64,190 Aufrufe • vor 1 Jahr

Multilingual voice AI tutor using the new Cartesia Sonic-2 voice model and Google DeepMind Gemini 2.0 Flash native audio understanding ...
0:48

Multilingual voice AI tutor using the new Cartesia Sonic-2 voice model and Google DeepMind Gemini 2.0 Flash native audio understanding ...

kwindla

18,793 Aufrufe • vor 1 Jahr

Just vibe coded with Gemini 3: an AI sparring partner to help you practice pitching skills. general chatbot is "too nice" to do that quite shocked by the low-latency audio stream by Gemini live api Google Gemini , vibe building a audio native app is no longer a dream
5:45

Just vibe coded with Gemini 3: an AI sparring partner to help you practice pitching skills. general chatbot is "too nice" to do that quite shocked by the low-latency audio stream by Gemini live api Google Gemini , vibe building a audio native app is no longer a dream

Jay Ding

12,289 Aufrufe • vor 7 Monaten

Gemini 3 Pro is the best model in the world for multimodal understanding. One of its most exciting capabilities is document understanding and reasoning. This means you can convert information in any format and into the medium that works best for you. Gemini 3 also has leading multilingual capabilities, enabling it to process, reason and even capture cultural relevance across a variety of languages. For example, here Gemini 3 is translating handwritten recipes in Korean and English to build a digital family cookbook in different languages.
0:37

Gemini 3 Pro is the best model in the world for multimodal understanding. One of its most exciting capabilities is document understanding and reasoning. This means you can convert information in any format and into the medium that works best for you. Gemini 3 also has leading multilingual capabilities, enabling it to process, reason and even capture cultural relevance across a variety of languages. For example, here Gemini 3 is translating handwritten recipes in Korean and English to build a digital family cookbook in different languages.

Google AI

36,696 Aufrufe • vor 8 Monaten

Talk to Gemini on your phone ... Deploy your own Gemini Multimodal Live voice agent that you can call on the phone (or that can call you) in 5 minutes.
2:12

Talk to Gemini on your phone ... Deploy your own Gemini Multimodal Live voice agent that you can call on the phone (or that can call you) in 5 minutes.

kwindla

38,680 Aufrufe • vor 1 Jahr

Today we’re taking a big step on the path toward AGI and releasing Gemini 3— our most intelligent model yet. With Gemini 3, you can bring any idea to life. It is state-of-the-art in reasoning, the best model in the world for multimodal understanding, and our best agentic and vibe coding model.
0:51

Today we’re taking a big step on the path toward AGI and releasing Gemini 3— our most intelligent model yet. With Gemini 3, you can bring any idea to life. It is state-of-the-art in reasoning, the best model in the world for multimodal understanding, and our best agentic and vibe coding model.

Google AI

492,973 Aufrufe • vor 8 Monaten

The new Sonic-3 voice model from launched today. The big additions are increased emotional range, emotion steering, and laughter tags. For example: I wrote some quick demo code, prompting Gemini Flash to know about the emotion tags. You can hear the results in the video, and see the emotion tags in the Pipecat developer console. Code is below.
1:32

The new Sonic-3 voice model from launched today. The big additions are increased emotional range, emotion steering, and laughter tags. For example: I wrote some quick demo code, prompting Gemini Flash to know about the emotion tags. You can hear the results in the video, and see the emotion tags in the Pipecat developer console. Code is below.

kwindla

12,350 Aufrufe • vor 9 Monaten

(1/5) Gemini 3, our most intelligent model, is landing in Google Search today – starting with AI Mode. Excited that this is the first time we’re shipping a new Gemini model in Search on day one! 🚀 In Search, Gemini 3 with generative layouts will make it easy to get a rich understanding of anything on your mind. It has state-of-the-art reasoning, deep multimodal understanding and advanced agentic capabilities. That allows the model to shine when you ask it to explain advanced concepts or ideas – it reasons and can code interactive visuals in real-time. It can tackle your toughest questions like advanced science.
1:13

(1/5) Gemini 3, our most intelligent model, is landing in Google Search today – starting with AI Mode. Excited that this is the first time we’re shipping a new Gemini model in Search on day one! 🚀 In Search, Gemini 3 with generative layouts will make it easy to get a rich understanding of anything on your mind. It has state-of-the-art reasoning, deep multimodal understanding and advanced agentic capabilities. That allows the model to shine when you ask it to explain advanced concepts or ideas – it reasons and can code interactive visuals in real-time. It can tackle your toughest questions like advanced science.

Robby Stein

94,877 Aufrufe • vor 8 Monaten

With Gemini 3, you can take an image of anything, like a board game, a floor plan, or a doodle, and make it an interactive experience. This is because of Gemini 3’s deep multimodal understanding. The model is able to analyze the content of the image, distill the most compelling parts, and then translate the image’s details into a functional, interactive experience.
0:47

With Gemini 3, you can take an image of anything, like a board game, a floor plan, or a doodle, and make it an interactive experience. This is because of Gemini 3’s deep multimodal understanding. The model is able to analyze the content of the image, distill the most compelling parts, and then translate the image’s details into a functional, interactive experience.

Google AI

98,335 Aufrufe • vor 8 Monaten

How does an AI model actually learn to see? 🤖 Learn about the tech behind native multimodality, how models reason over visual data like documents and video, and the future of proactive AI assistants with Logan Kilpatrick and Gemini Model Behavior Product Lead, Ani Baddepudi. ↓ Timestamps: 01:12 Why Gemini is natively multimodal 02:23 The technology behind multimodal models 05:15 Video understanding with Gemini 2.5 09:25 Deciding what to build next 13:23 Building new product experiences with multimodal AI 17:15 The vision for proactive assistants 24:13 Improving video usability with variable FPS and frame tokenization 27:35 What’s next for Gemini’s multimodal development 31:47 Deep dive on Gemini’s document understanding capabilities 37:56 The teamwork and collaboration behind Gemini 40:56 What’s next with model behavior
44:17

How does an AI model actually learn to see? 🤖 Learn about the tech behind native multimodality, how models reason over visual data like documents and video, and the future of proactive AI assistants with Logan Kilpatrick and Gemini Model Behavior Product Lead, Ani Baddepudi. ↓ Timestamps: 01:12 Why Gemini is natively multimodal 02:23 The technology behind multimodal models 05:15 Video understanding with Gemini 2.5 09:25 Deciding what to build next 13:23 Building new product experiences with multimodal AI 17:15 The vision for proactive assistants 24:13 Improving video usability with variable FPS and frame tokenization 27:35 What’s next for Gemini’s multimodal development 31:47 Deep dive on Gemini’s document understanding capabilities 37:56 The teamwork and collaboration behind Gemini 40:56 What’s next with model behavior

Google AI

58,703 Aufrufe • vor 1 Jahr

Can AI chain tools together to complete complex tasks in real-time? Gemini 2.0 can. Here's a React demo using the new Multimodal Live API to chain code execution + real-time Google Search. I'm amazed at how fast this is.
2:17

Can AI chain tools together to complete complex tasks in real-time? Gemini 2.0 can. Here's a React demo using the new Multimodal Live API to chain code execution + real-time Google Search. I'm amazed at how fast this is.

Addy Osmani

17,804 Aufrufe • vor 1 Jahr

Gemini 2.0 with Code Execution is actually INSANE🔥 90% don't even know about this feature. - Run Python code in real-time - Handles complex data analysis & makes charts instantly - Works in both API & AI Studio - Use Multimodal Live API and Reasoning model Here's how you can use 👇 1/4 Example 1: Real-time analysis & visualization
2:22

Gemini 2.0 with Code Execution is actually INSANE🔥 90% don't even know about this feature. - Run Python code in real-time - Handles complex data analysis & makes charts instantly - Works in both API & AI Studio - Use Multimodal Live API and Reasoning model Here's how you can use 👇 1/4 Example 1: Real-time analysis & visualization

AshutoshShrivastava

92,641 Aufrufe • vor 1 Jahr

We’re expanding the Gemini 3 family with the launch of Gemini 3 Flash. This model: — Combines Gemini 3’s Pro-grade reasoning with Flash-level latency, efficiency, and cost — Delivers frontier-level performance on PHD-level reasoning and knowledge benchmarks — Is our most impressive model for agentic workflows Watch Gemini 3 Flash handle hundreds of function calling options at low latency, compiling global recipes from 100 ingredients and 100 kitchen tools.
1:06

We’re expanding the Gemini 3 family with the launch of Gemini 3 Flash. This model: — Combines Gemini 3’s Pro-grade reasoning with Flash-level latency, efficiency, and cost — Delivers frontier-level performance on PHD-level reasoning and knowledge benchmarks — Is our most impressive model for agentic workflows Watch Gemini 3 Flash handle hundreds of function calling options at low latency, compiling global recipes from 100 ingredients and 100 kitchen tools.

Google AI

229,722 Aufrufe • vor 7 Monaten

Watch this demo from Thor 雷神 ⚡️ to see Gemini 3.1 Flash Live in action as a real-time DJ. The model uses function calling (to the Gemini API) to generate custom 30-second clips using Lyria 3️⃣ Start your own studio session in Google AI Studio:
2:31

Watch this demo from Thor 雷神 ⚡️ to see Gemini 3.1 Flash Live in action as a real-time DJ. The model uses function calling (to the Gemini API) to generate custom 30-second clips using Lyria 3️⃣ Start your own studio session in Google AI Studio:

Google AI Developers

11,216 Aufrufe • vor 3 Monaten

Gemini 3.5 Live Translate! We just shipped a real-time babel fish. - 70+ languages, 2,000+ language pairs. - Natural translated speech, works in noisy environments. - Stays in sync with the speaker, no lag, no awkward pauses. - Auto-detects the language being spoken. Available today in Google Translate (Android & iOS), the Gemini API (Public Preview), and Google Meet (Private Preview). I genuinely think this is the beginning of the end of language barriers. Anyone can now speak and understand anyone.
1:33

Gemini 3.5 Live Translate! We just shipped a real-time babel fish. - 70+ languages, 2,000+ language pairs. - Natural translated speech, works in noisy environments. - Stays in sync with the speaker, no lag, no awkward pauses. - Auto-detects the language being spoken. Available today in Google Translate (Android & iOS), the Gemini API (Public Preview), and Google Meet (Private Preview). I genuinely think this is the beginning of the end of language barriers. Anyone can now speak and understand anyone.

Philipp Schmid

17,485 Aufrufe • vor 2 Monaten

Gemini 3 is indeed impressive. In my limited testing with it, it feels like it is able to take on larger, complex prompts with ease and low failure rate. I think one of the biggest strengths that Gemini 3 has is its ability to generate coherent looking 3D voxel or SVG art, which makes it super easy to create game prototypes without having to implement art at all. Made this Heroes of might and magic-inspired overworld prototype with mini map and basic UI in less than 2 hours. Really excited to test this more. Demis Hassabis and Logan Kilpatrick and the Gemini team, you guys really cooked this time!
0:39

Gemini 3 is indeed impressive. In my limited testing with it, it feels like it is able to take on larger, complex prompts with ease and low failure rate. I think one of the biggest strengths that Gemini 3 has is its ability to generate coherent looking 3D voxel or SVG art, which makes it super easy to create game prototypes without having to implement art at all. Made this Heroes of might and magic-inspired overworld prototype with mini map and basic UI in less than 2 hours. Really excited to test this more. Demis Hassabis and Logan Kilpatrick and the Gemini team, you guys really cooked this time!

Danny Limanseta

119,970 Aufrufe • vor 8 Monaten

Super exciting… Google Maps is getting a huge Gemini-powered upgrade! We’re bringing together Maps’s fresh, reliable information about the real world with the conversational intelligence of Gemini. It’s our next big step in building the map of the future. 🧵 (1/3)
0:58

Super exciting… Google Maps is getting a huge Gemini-powered upgrade! We’re bringing together Maps’s fresh, reliable information about the real world with the conversational intelligence of Gemini. It’s our next big step in building the map of the future. 🧵 (1/3)

Nick Fox

203,746 Aufrufe • vor 9 Monaten

Really happy to see the interest around our “Hands-on with Gemini” video. In our developer blog yesterday, we broke down how Gemini was used to create it. We gave Gemini sequences of different modalities — image and text in this case — and had it respond by predicting what might come next. Devs can try similar things when access to Pro opens on 12/13 🚀. The knitting demo used Ultra⚡ All the user prompts and outputs in the video are real, shortened for brevity. The video illustrates what the multimodal user experiences built with Gemini could look like. We made it to inspire developers. When you’re building an app, you can get similar results (there’s always some variability with LLMs) by prompting Gemini with an instruction that allows the user to "configure" the behavior of the model, like inputting “you are an expert in science …” before a user can engage in the same kind of back and forth dialogue. Here’s a clip of what this looks like in AI Studio with Gemini Pro. We’ve come a long way since Flamingo 🦩 & PALI, looking forward to seeing what people build with it!
2:24

Really happy to see the interest around our “Hands-on with Gemini” video. In our developer blog yesterday, we broke down how Gemini was used to create it. We gave Gemini sequences of different modalities — image and text in this case — and had it respond by predicting what might come next. Devs can try similar things when access to Pro opens on 12/13 🚀. The knitting demo used Ultra⚡ All the user prompts and outputs in the video are real, shortened for brevity. The video illustrates what the multimodal user experiences built with Gemini could look like. We made it to inspire developers. When you’re building an app, you can get similar results (there’s always some variability with LLMs) by prompting Gemini with an instruction that allows the user to "configure" the behavior of the model, like inputting “you are an expert in science …” before a user can engage in the same kind of back and forth dialogue. Here’s a clip of what this looks like in AI Studio with Gemini Pro. We’ve come a long way since Flamingo 🦩 & PALI, looking forward to seeing what people build with it!

Oriol Vinyals

180,982 Aufrufe • vor 2 Jahren