Загрузка видео...

Не удалось загрузить видео

На главную

Further tinkering with my little French tutor app. This version is using the Gemini Multimodal Live API. The speech understanding in Gemini is quite something. In this video you can see Gemini correcting my pronunciation. (Very patiently.) The language tutor use case really highlights the strengths of a next-generation...

kwindla's profile picture

kwindla

9,989 subscribers

13,005 просмотров • 1 год назад •via X (Twitter)

ОбразованиеНаука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 12

Фото профиля kwindla
kwindla1 год назад

Code is here: Here are the docs for Pipecat's open source client SDKs for javascript, React, iOS, Android, React Native, and C++: The Pipecat clients all support WebRTC, WebSocket, and HTTP network transports.

Фото профиля Nexus 👾
Nexus 👾1 год назад

Nexus combines CMC’s market insights with Zerion and Debank’s wallet tracking—all in one easy-to-use dashboard. Simplify your crypto journey today, completely free 🚀 Try it now!

Фото профиля The Canaanite
The Canaanite1 год назад

bravo, excellente prononciation monsieur 🤣

Фото профиля عای‌ خان
عای‌ خان1 год назад

Amazing. I need to code up a Maths tutor for my kids for those inevitable times where my patience sadly failed. 😅

Фото профиля Emanuel Perez
Emanuel Perez1 год назад

Whoa

Фото профиля Samuel Ekpe
Samuel Ekpe1 год назад

interesting

Фото профиля Mitja Martini
Mitja Martini1 год назад

Thanks for sharing this very nice use case! I've been using ChatGPT with my daughter to help her learning Latin, not in voice mode, though :)

Фото профиля Nicolas MICHEL
Nicolas MICHEL1 год назад

it is funny, the machine has an African accent. This will lead to some funny situations if you start speaking in public with this accent 😂

Фото профиля Jake
Jake1 год назад

cool any other interesting uncommon use cases? I feel like the voice use cases are going to blow up this year

Фото профиля kwindla
kwindla1 год назад

> any other interesting uncommon use cases? Check out the voice interaction experiments that @trudypainter has been posting!

Фото профиля Dan Goodman 🍊
Dan Goodman 🍊1 год назад

Can you have it take in audio but output text in pipecat?

Фото профиля kwindla
kwindla1 год назад

> Can you have it take in audio but output text in pipecat? Yes!

Похожие видео

Gemini 2.0 drops the beat. Watch the video all the way through — I had four legit "no way it did that" reactions when Jon Taylor sent this to me. "It looks like it's only hitting on the first beat of each bar." This is Jon collaborating with Gemini to create a song in Ableton Live. Jon is using the Multimodal Live API to stream audio and video to Gemini and have a conversation about the song he's creating.
4:15

Gemini 2.0 drops the beat. Watch the video all the way through — I had four legit "no way it did that" reactions when Jon Taylor sent this to me. "It looks like it's only hitting on the first beat of each bar." This is Jon collaborating with Gemini to create a song in Ableton Live. Jon is using the Multimodal Live API to stream audio and video to Gemini and have a conversation about the song he's creating.

kwindla

86,132 просмотров • 1 год назад

Better/faster/cheaper voice AI turn detection with Gemini 2.0 The code that determines when the agent should respond to the user is some of the most important code in your voice AI agent. The technical terms for this job are "turn detection" or "phrase endpointing." If the voice AI responds before the user has finished their thought, the conversation is choppy and unproductive. If the AI waits too long, the conversation is slow and frustrating. There are a number of ways to approach this. You can: 1. Use a fast "voice activity detection" model to detect pauses in speech. Respond when the user pauses. 2. Use a specialized phrase endpointing model that operates on transcribed text, pattern-matching on text semantics. 3. Train a specialized phrase endpointing model that operates directly on audio. 4. Leverage the native audio capabilities of a SOTA LLM like Gemini 2.0. We've benchmarked all four of these, and Gemini 2.0 currently beats other approaches. Using Gemini is also cheaper than transcribing the audio separately using a transcription service or model. Here's a short video showing Gemini phrase endpointing in two scenarios. First, correctly handling pauses in natural conversation. Second, requesting a phone number (which is a common activity in a use case like customer support). You can see the Completeness check lines in the terminal output, printed each time Gemini processes a chunk of audio.
1:23

Better/faster/cheaper voice AI turn detection with Gemini 2.0 The code that determines when the agent should respond to the user is some of the most important code in your voice AI agent. The technical terms for this job are "turn detection" or "phrase endpointing." If the voice AI responds before the user has finished their thought, the conversation is choppy and unproductive. If the AI waits too long, the conversation is slow and frustrating. There are a number of ways to approach this. You can: 1. Use a fast "voice activity detection" model to detect pauses in speech. Respond when the user pauses. 2. Use a specialized phrase endpointing model that operates on transcribed text, pattern-matching on text semantics. 3. Train a specialized phrase endpointing model that operates directly on audio. 4. Leverage the native audio capabilities of a SOTA LLM like Gemini 2.0. We've benchmarked all four of these, and Gemini 2.0 currently beats other approaches. Using Gemini is also cheaper than transcribing the audio separately using a transcription service or model. Here's a short video showing Gemini phrase endpointing in two scenarios. First, correctly handling pauses in natural conversation. Second, requesting a phone number (which is a common activity in a use case like customer support). You can see the Completeness check lines in the terminal output, printed each time Gemini processes a chunk of audio.

kwindla

64,190 просмотров • 1 год назад

Multilingual voice AI tutor using the new Cartesia Sonic-2 voice model and Google DeepMind Gemini 2.0 Flash native audio understanding ...
0:48

Multilingual voice AI tutor using the new Cartesia Sonic-2 voice model and Google DeepMind Gemini 2.0 Flash native audio understanding ...

kwindla

18,793 просмотров • 1 год назад

Just vibe coded with Gemini 3: an AI sparring partner to help you practice pitching skills. general chatbot is "too nice" to do that quite shocked by the low-latency audio stream by Gemini live api Google Gemini , vibe building a audio native app is no longer a dream
5:45

Just vibe coded with Gemini 3: an AI sparring partner to help you practice pitching skills. general chatbot is "too nice" to do that quite shocked by the low-latency audio stream by Gemini live api Google Gemini , vibe building a audio native app is no longer a dream

Jay Ding

12,289 просмотров • 7 месяцев назад

Gemini 3 Pro is the best model in the world for multimodal understanding. One of its most exciting capabilities is document understanding and reasoning. This means you can convert information in any format and into the medium that works best for you. Gemini 3 also has leading multilingual capabilities, enabling it to process, reason and even capture cultural relevance across a variety of languages. For example, here Gemini 3 is translating handwritten recipes in Korean and English to build a digital family cookbook in different languages.
0:37

Gemini 3 Pro is the best model in the world for multimodal understanding. One of its most exciting capabilities is document understanding and reasoning. This means you can convert information in any format and into the medium that works best for you. Gemini 3 also has leading multilingual capabilities, enabling it to process, reason and even capture cultural relevance across a variety of languages. For example, here Gemini 3 is translating handwritten recipes in Korean and English to build a digital family cookbook in different languages.

Google AI

36,696 просмотров • 8 месяцев назад

The new Sonic-3 voice model from launched today. The big additions are increased emotional range, emotion steering, and laughter tags. For example: I wrote some quick demo code, prompting Gemini Flash to know about the emotion tags. You can hear the results in the video, and see the emotion tags in the Pipecat developer console. Code is below.
1:32

The new Sonic-3 voice model from launched today. The big additions are increased emotional range, emotion steering, and laughter tags. For example: I wrote some quick demo code, prompting Gemini Flash to know about the emotion tags. You can hear the results in the video, and see the emotion tags in the Pipecat developer console. Code is below.

kwindla

12,350 просмотров • 9 месяцев назад

Talk to Gemini on your phone ... Deploy your own Gemini Multimodal Live voice agent that you can call on the phone (or that can call you) in 5 minutes.
2:12

Talk to Gemini on your phone ... Deploy your own Gemini Multimodal Live voice agent that you can call on the phone (or that can call you) in 5 minutes.

kwindla

38,680 просмотров • 1 год назад

Today we’re taking a big step on the path toward AGI and releasing Gemini 3— our most intelligent model yet. With Gemini 3, you can bring any idea to life. It is state-of-the-art in reasoning, the best model in the world for multimodal understanding, and our best agentic and vibe coding model.
0:51

Today we’re taking a big step on the path toward AGI and releasing Gemini 3— our most intelligent model yet. With Gemini 3, you can bring any idea to life. It is state-of-the-art in reasoning, the best model in the world for multimodal understanding, and our best agentic and vibe coding model.

Google AI

492,973 просмотров • 8 месяцев назад

(1/5) Gemini 3, our most intelligent model, is landing in Google Search today – starting with AI Mode. Excited that this is the first time we’re shipping a new Gemini model in Search on day one! 🚀 In Search, Gemini 3 with generative layouts will make it easy to get a rich understanding of anything on your mind. It has state-of-the-art reasoning, deep multimodal understanding and advanced agentic capabilities. That allows the model to shine when you ask it to explain advanced concepts or ideas – it reasons and can code interactive visuals in real-time. It can tackle your toughest questions like advanced science.
1:13

(1/5) Gemini 3, our most intelligent model, is landing in Google Search today – starting with AI Mode. Excited that this is the first time we’re shipping a new Gemini model in Search on day one! 🚀 In Search, Gemini 3 with generative layouts will make it easy to get a rich understanding of anything on your mind. It has state-of-the-art reasoning, deep multimodal understanding and advanced agentic capabilities. That allows the model to shine when you ask it to explain advanced concepts or ideas – it reasons and can code interactive visuals in real-time. It can tackle your toughest questions like advanced science.

Robby Stein

94,877 просмотров • 8 месяцев назад

With Gemini 3, you can take an image of anything, like a board game, a floor plan, or a doodle, and make it an interactive experience. This is because of Gemini 3’s deep multimodal understanding. The model is able to analyze the content of the image, distill the most compelling parts, and then translate the image’s details into a functional, interactive experience.
0:47

With Gemini 3, you can take an image of anything, like a board game, a floor plan, or a doodle, and make it an interactive experience. This is because of Gemini 3’s deep multimodal understanding. The model is able to analyze the content of the image, distill the most compelling parts, and then translate the image’s details into a functional, interactive experience.

Google AI

98,340 просмотров • 8 месяцев назад

How does an AI model actually learn to see? 🤖 Learn about the tech behind native multimodality, how models reason over visual data like documents and video, and the future of proactive AI assistants with Logan Kilpatrick and Gemini Model Behavior Product Lead, Ani Baddepudi. ↓ Timestamps: 01:12 Why Gemini is natively multimodal 02:23 The technology behind multimodal models 05:15 Video understanding with Gemini 2.5 09:25 Deciding what to build next 13:23 Building new product experiences with multimodal AI 17:15 The vision for proactive assistants 24:13 Improving video usability with variable FPS and frame tokenization 27:35 What’s next for Gemini’s multimodal development 31:47 Deep dive on Gemini’s document understanding capabilities 37:56 The teamwork and collaboration behind Gemini 40:56 What’s next with model behavior
44:17

How does an AI model actually learn to see? 🤖 Learn about the tech behind native multimodality, how models reason over visual data like documents and video, and the future of proactive AI assistants with Logan Kilpatrick and Gemini Model Behavior Product Lead, Ani Baddepudi. ↓ Timestamps: 01:12 Why Gemini is natively multimodal 02:23 The technology behind multimodal models 05:15 Video understanding with Gemini 2.5 09:25 Deciding what to build next 13:23 Building new product experiences with multimodal AI 17:15 The vision for proactive assistants 24:13 Improving video usability with variable FPS and frame tokenization 27:35 What’s next for Gemini’s multimodal development 31:47 Deep dive on Gemini’s document understanding capabilities 37:56 The teamwork and collaboration behind Gemini 40:56 What’s next with model behavior

Google AI

58,703 просмотров • 1 год назад

Can AI chain tools together to complete complex tasks in real-time? Gemini 2.0 can. Here's a React demo using the new Multimodal Live API to chain code execution + real-time Google Search. I'm amazed at how fast this is.
2:17

Can AI chain tools together to complete complex tasks in real-time? Gemini 2.0 can. Here's a React demo using the new Multimodal Live API to chain code execution + real-time Google Search. I'm amazed at how fast this is.

Addy Osmani

17,804 просмотров • 1 год назад

Gemini 2.0 with Code Execution is actually INSANE🔥 90% don't even know about this feature. - Run Python code in real-time - Handles complex data analysis & makes charts instantly - Works in both API & AI Studio - Use Multimodal Live API and Reasoning model Here's how you can use 👇 1/4 Example 1: Real-time analysis & visualization
2:22

Gemini 2.0 with Code Execution is actually INSANE🔥 90% don't even know about this feature. - Run Python code in real-time - Handles complex data analysis & makes charts instantly - Works in both API & AI Studio - Use Multimodal Live API and Reasoning model Here's how you can use 👇 1/4 Example 1: Real-time analysis & visualization

AshutoshShrivastava

92,641 просмотров • 1 год назад

We’re expanding the Gemini 3 family with the launch of Gemini 3 Flash. This model: — Combines Gemini 3’s Pro-grade reasoning with Flash-level latency, efficiency, and cost — Delivers frontier-level performance on PHD-level reasoning and knowledge benchmarks — Is our most impressive model for agentic workflows Watch Gemini 3 Flash handle hundreds of function calling options at low latency, compiling global recipes from 100 ingredients and 100 kitchen tools.
1:06

We’re expanding the Gemini 3 family with the launch of Gemini 3 Flash. This model: — Combines Gemini 3’s Pro-grade reasoning with Flash-level latency, efficiency, and cost — Delivers frontier-level performance on PHD-level reasoning and knowledge benchmarks — Is our most impressive model for agentic workflows Watch Gemini 3 Flash handle hundreds of function calling options at low latency, compiling global recipes from 100 ingredients and 100 kitchen tools.

Google AI

229,722 просмотров • 7 месяцев назад

Watch this demo from Thor 雷神 ⚡️ to see Gemini 3.1 Flash Live in action as a real-time DJ. The model uses function calling (to the Gemini API) to generate custom 30-second clips using Lyria 3️⃣ Start your own studio session in Google AI Studio:
2:31

Watch this demo from Thor 雷神 ⚡️ to see Gemini 3.1 Flash Live in action as a real-time DJ. The model uses function calling (to the Gemini API) to generate custom 30-second clips using Lyria 3️⃣ Start your own studio session in Google AI Studio:

Google AI Developers

11,216 просмотров • 3 месяцев назад

Gemini 3 is indeed impressive. In my limited testing with it, it feels like it is able to take on larger, complex prompts with ease and low failure rate. I think one of the biggest strengths that Gemini 3 has is its ability to generate coherent looking 3D voxel or SVG art, which makes it super easy to create game prototypes without having to implement art at all. Made this Heroes of might and magic-inspired overworld prototype with mini map and basic UI in less than 2 hours. Really excited to test this more. Demis Hassabis and Logan Kilpatrick and the Gemini team, you guys really cooked this time!
0:39

Gemini 3 is indeed impressive. In my limited testing with it, it feels like it is able to take on larger, complex prompts with ease and low failure rate. I think one of the biggest strengths that Gemini 3 has is its ability to generate coherent looking 3D voxel or SVG art, which makes it super easy to create game prototypes without having to implement art at all. Made this Heroes of might and magic-inspired overworld prototype with mini map and basic UI in less than 2 hours. Really excited to test this more. Demis Hassabis and Logan Kilpatrick and the Gemini team, you guys really cooked this time!

Danny Limanseta

119,970 просмотров • 8 месяцев назад

Gemini 3.5 Live Translate! We just shipped a real-time babel fish. - 70+ languages, 2,000+ language pairs. - Natural translated speech, works in noisy environments. - Stays in sync with the speaker, no lag, no awkward pauses. - Auto-detects the language being spoken. Available today in Google Translate (Android & iOS), the Gemini API (Public Preview), and Google Meet (Private Preview). I genuinely think this is the beginning of the end of language barriers. Anyone can now speak and understand anyone.
1:33

Gemini 3.5 Live Translate! We just shipped a real-time babel fish. - 70+ languages, 2,000+ language pairs. - Natural translated speech, works in noisy environments. - Stays in sync with the speaker, no lag, no awkward pauses. - Auto-detects the language being spoken. Available today in Google Translate (Android & iOS), the Gemini API (Public Preview), and Google Meet (Private Preview). I genuinely think this is the beginning of the end of language barriers. Anyone can now speak and understand anyone.

Philipp Schmid

17,485 просмотров • 2 месяцев назад

Super exciting… Google Maps is getting a huge Gemini-powered upgrade! We’re bringing together Maps’s fresh, reliable information about the real world with the conversational intelligence of Gemini. It’s our next big step in building the map of the future. 🧵 (1/3)
0:58

Super exciting… Google Maps is getting a huge Gemini-powered upgrade! We’re bringing together Maps’s fresh, reliable information about the real world with the conversational intelligence of Gemini. It’s our next big step in building the map of the future. 🧵 (1/3)

Nick Fox

203,746 просмотров • 9 месяцев назад

Really happy to see the interest around our “Hands-on with Gemini” video. In our developer blog yesterday, we broke down how Gemini was used to create it. We gave Gemini sequences of different modalities — image and text in this case — and had it respond by predicting what might come next. Devs can try similar things when access to Pro opens on 12/13 🚀. The knitting demo used Ultra⚡ All the user prompts and outputs in the video are real, shortened for brevity. The video illustrates what the multimodal user experiences built with Gemini could look like. We made it to inspire developers. When you’re building an app, you can get similar results (there’s always some variability with LLMs) by prompting Gemini with an instruction that allows the user to "configure" the behavior of the model, like inputting “you are an expert in science …” before a user can engage in the same kind of back and forth dialogue. Here’s a clip of what this looks like in AI Studio with Gemini Pro. We’ve come a long way since Flamingo 🦩 & PALI, looking forward to seeing what people build with it!
2:24

Really happy to see the interest around our “Hands-on with Gemini” video. In our developer blog yesterday, we broke down how Gemini was used to create it. We gave Gemini sequences of different modalities — image and text in this case — and had it respond by predicting what might come next. Devs can try similar things when access to Pro opens on 12/13 🚀. The knitting demo used Ultra⚡ All the user prompts and outputs in the video are real, shortened for brevity. The video illustrates what the multimodal user experiences built with Gemini could look like. We made it to inspire developers. When you’re building an app, you can get similar results (there’s always some variability with LLMs) by prompting Gemini with an instruction that allows the user to "configure" the behavior of the model, like inputting “you are an expert in science …” before a user can engage in the same kind of back and forth dialogue. Here’s a clip of what this looks like in AI Studio with Gemini Pro. We’ve come a long way since Flamingo 🦩 & PALI, looking forward to seeing what people build with it!

Oriol Vinyals

180,982 просмотров • 2 лет назад