Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Having fun playing with new native audio capabilities in Gemini 1.5 Pro! ♊ Here’s a demo using audio from the #GoogleIO keynote with examples you can try: transcription, word-level timecodes, and searching audio by drawing. (🔊Video has sound)

Alexander Chen

14,280 subscribers

77,097 views • 2 years ago •via X (Twitter)

Science & Technology Education #GoogleIO

Anya Rossi• Live Now

Private livecam show

9 Comments

Alexander Chen2 years ago

Word-level timecodes let you make fun things like this. (🔊Video has sound) My prompt: Find sentences that contain "pickleball" and cite the sentence start and end times. Then give me the exact timecode when the word "pickleball" starts within that sentence. Results will vary and timecodes might not always be correct, so definitely experiment with your own prompts in Google AI Studio. Learn more:

Alexander Chen2 years ago

Nifty video scrubber frontend tool made by @trippedout + @GrantCuster. It's neat to be able to interact directly with the timecodes the model gives us.

Man from Atlantis2 years ago

This is great capability-Can you please share the note book ?

Alexander Chen2 years ago

Thanks! I'll message you and send the prompts I used in AI Studio.

Sina Azizi2 years ago

I wonder how well these would work with 1.5 Flash

Alexander Chen2 years ago

I've tried similar prompts with Gemini 1.5 Flash and gotten generally solid results too.

Victoriano Izquierdo2 years ago

Do they support diarization already?

Alexander Chen2 years ago

I'm not sure, I haven't tried that.

micke2 years ago

amazing

Related Videos

Audio Transcription with Google Gemini 1.5 Flash In this video, Gemini-Flash was able to transcribe 13 minutes of audio in 50-60 seconds. I have tested this with multiple audio files, and the transcription accuracy is close to 99%. If the audio is clear, you get 100% correct transcription. Even if the audio is really bad with a lot of noise, you still get around 95-96% accuracy. The code for this is available on GitHub. If you're interested, you can download and run it. You just need a Google API key.

Audio Transcription with Google Gemini 1.5 Flash In this video, Gemini-Flash was able to transcribe 13 minutes of audio in 50-60 seconds. I have tested this with multiple audio files, and the transcription accuracy is close to 99%. If the audio is clear, you get 100% correct transcription. Even if the audio is really bad with a lot of noise, you still get around 95-96% accuracy. The code for this is available on GitHub. If you're interested, you can download and run it. You just need a Google API key.

AshutoshShrivastava

122,990 views • 1 year ago

New native audio capabilities in Gemini 2.5 enable text-to-speech in over 24 languages. 🔊Voices are more natural and expressive, and you can seamlessly switch between languages.

New native audio capabilities in Gemini 2.5 enable text-to-speech in over 24 languages. 🔊Voices are more natural and expressive, and you can seamlessly switch between languages.

Google

155,360 views • 1 year ago

If you try nothing else today, give the demo at a go - it lets you stream video and audio directly to Gemini 2.0 Flash and get audio back, so you can have a real-time audio conversation about what you can see with the model Feels like science fiction!

If you try nothing else today, give the demo at a go - it lets you stream video and audio directly to Gemini 2.0 Flash and get audio back, so you can have a real-time audio conversation about what you can see with the model Feels like science fiction!

Simon Willison

208,965 views • 1 year ago

Here is something completely new you can do in one shot with Runway. This entire video was generated from a single long audio file. Runway analyzed the file, read the audio transcription, and generated one coherent video to bring it all together visually.

Here is something completely new you can do in one shot with Runway. This entire video was generated from a single long audio file. Runway analyzed the file, read the audio transcription, and generated one coherent video to bring it all together visually.

Cristóbal Valenzuela

21,515 views • 12 days ago

Introducing Seedance 1.5 Pro 🔊 The first Seedance video model to generate audio and film-grade visuals together Create scenes with strong storytelling and emotional expression Plus, built-in sound effects, multilingual dialogue, and native lip-syncing Available now on Freepik

Introducing Seedance 1.5 Pro 🔊 The first Seedance video model to generate audio and film-grade visuals together Create scenes with strong storytelling and emotional expression Plus, built-in sound effects, multilingual dialogue, and native lip-syncing Available now on Freepik

Magnific

32,973 views • 6 months ago

React Native Audio API 0.11.0 is out! 🔊 It’s packed with tons of stability and DX improvements. We've also shipped a fully customizable recording to file and new Web Audio API nodes. Check out the demo 👇

React Native Audio API 0.11.0 is out! 🔊 It’s packed with tons of stability and DX improvements. We've also shipped a fully customizable recording to file and new Web Audio API nodes. Check out the demo 👇

Software Mansion

20,289 views • 6 months ago

Today we’re announcing a set of updates, starting with a new experimental feature for paid subscribers, audio uploads. You can upload an audio clip of your choice, and extend this clip either forward or backward by 32 seconds using up to 2 minutes of context. Audio uploads greatly enrich your prompting vocabulary. You can use audio to set tempo and mood, and explore from there. Maybe you’ve got a great intro but don’t know where to go next, or a full mix that’s missing the perfect bridge–in both cases, Udio can provide inspiration. Check out the video below for some examples (we’ve had a lot of fun with this).

Today we’re announcing a set of updates, starting with a new experimental feature for paid subscribers, audio uploads. You can upload an audio clip of your choice, and extend this clip either forward or backward by 32 seconds using up to 2 minutes of context. Audio uploads greatly enrich your prompting vocabulary. You can use audio to set tempo and mood, and explore from there. Maybe you’ve got a great intro but don’t know where to go next, or a full mix that’s missing the perfect bridge–in both cases, Udio can provide inspiration. Check out the video below for some examples (we’ve had a lot of fun with this).

udio

130,827 views • 2 years ago

Explore the Live API’s new audio capabilities in this episode of Release Notes with Shrestha Basu Mallick and Logan Kilpatrick. From native audio output to proactive dialogue, learn how you can build more natural and engaging multimodal AI applications. Timecodes: 0:00 Intro 01:18 Live API Overview 03:36 Why audio is a special modality 05:07 Speed vs. precision in audio 06:17 Controllable and promptable TTS 08:31 What developers are building with the Live API 11:14 URL context and async calling features 15:02 Proactive audio and affective dialog 16:55 Addressing developer feedback 21:54 Live API roadmap 23:49 The role of long context 24:57 What’s next for the Live API 26:41 State of the AI audio market 30:10 Advice for developers getting started with the Live API 31:16 Live API demo 38:10 Demo wrap up and closing

Explore the Live API’s new audio capabilities in this episode of Release Notes with Shrestha Basu Mallick and Logan Kilpatrick. From native audio output to proactive dialogue, learn how you can build more natural and engaging multimodal AI applications. Timecodes: 0:00 Intro 01:18 Live API Overview 03:36 Why audio is a special modality 05:07 Speed vs. precision in audio 06:17 Controllable and promptable TTS 08:31 What developers are building with the Live API 11:14 URL context and async calling features 15:02 Proactive audio and affective dialog 16:55 Addressing developer feedback 21:54 Live API roadmap 23:49 The role of long context 24:57 What’s next for the Live API 26:41 State of the AI audio market 30:10 Advice for developers getting started with the Live API 31:16 Live API demo 38:10 Demo wrap up and closing

Google AI Developers

24,239 views • 11 months ago

JUST IN: Google releases Gemini 1.5, a powerful MoE model. It's a huge breakthrough. The model has the longest context window ever seen: 1 million tokens. It can process 1 hour of video, 11 hours of audio, 30,000 lines of code, or 700,000 words in a single prompt. When tested on text, code, image, audio and video evaluations, 1.5 Pro outperforms 1.0 Pro on 87% of the benchmarks used for developing LLMs. You can can sign up in AI Studio to try it out.

JUST IN: Google releases Gemini 1.5, a powerful MoE model. It's a huge breakthrough. The model has the longest context window ever seen: 1 million tokens. It can process 1 hour of video, 11 hours of audio, 30,000 lines of code, or 700,000 words in a single prompt. When tested on text, code, image, audio and video evaluations, 1.5 Pro outperforms 1.0 Pro on 87% of the benchmarks used for developing LLMs. You can can sign up in AI Studio to try it out.

Lior Alexander

83,409 views • 2 years ago

Just got early access to Aleph, a new in-context video model from Runway. Here’s a snippet of my first experiment using a toy car. Audio added with the MMAudio Gradio demo on Hugging Face spaces

Just got early access to Aleph, a new in-context video model from Runway. Here’s a snippet of my first experiment using a toy car. Audio added with the MMAudio Gradio demo on Hugging Face spaces

William Lamkin

79,195 views • 11 months ago

Listen up 🔊 We’ve made some updates to our Gemini Audio models and capabilities: — Gemini’s live speech-to-speech translation capability is rolling out in a beta experience to the Google Translate app, bringing you real-time audio translation that captures the nuance of human speech — Gemini 2.5 Flash and 2.5 Pro Text-to-Speech preview models bring improved adherence to style prompts, precision pacing with context-aware speed adjustments, and character voice consistency for multi-speaker scenarios — Gemini 2.5 Flash Native Audio is now updated, with improvements to handle complex workflows, navigate user instructions, and hold natural conversations

Listen up 🔊 We’ve made some updates to our Gemini Audio models and capabilities: — Gemini’s live speech-to-speech translation capability is rolling out in a beta experience to the Google Translate app, bringing you real-time audio translation that captures the nuance of human speech — Gemini 2.5 Flash and 2.5 Pro Text-to-Speech preview models bring improved adherence to style prompts, precision pacing with context-aware speed adjustments, and character voice consistency for multi-speaker scenarios — Gemini 2.5 Flash Native Audio is now updated, with improvements to handle complex workflows, navigate user instructions, and hold natural conversations

Google AI

1,092,358 views • 7 months ago

Beyond just transcription, Saaras V3 excels in three areas. It can automatically detect the language of the audio. It now comes with word-level time-stamps. And it also supports diarisation for multi-speaker audios. Try out your polyglot skills in the blog.

Beyond just transcription, Saaras V3 excels in three areas. It can automatically detect the language of the audio. It now comes with word-level time-stamps. And it also supports diarisation for multi-speaker audios. Try out your polyglot skills in the blog.

Pratyush Kumar

15,445 views • 5 months ago

Along with text, images, video and code, Gemini is able to process raw audio signal end-to-end. 🔊 It can listen to and understand speech, making it not only useful for transcription but a model that has a much more nuanced perception of its environment. ↓

Along with text, images, video and code, Gemini is able to process raw audio signal end-to-end. 🔊 It can listen to and understand speech, making it not only useful for transcription but a model that has a much more nuanced perception of its environment. ↓

Google DeepMind

140,169 views • 2 years ago

Today we’re launching Stable Audio 2.5: The first audio model built for enterprise-grade sound production 🔊 Audio influences brand engagement by 86%, but few enterprises are leveraging audio as an extension of their brand, making customized sound an untapped differentiator. Stable Audio 2.5 is purpose-built for this opportunity to create customizable, high-quality audio at scale, with capabilities that include: ▶️ Improved musical composition: Generate full songs with multi-part structure, meaning a clear intro, middle, and outro. ▶️ Audio inpainting: Input audio, select where the track should start, and the model uses the context to generate the rest of the track. ▶️ Customization: Our team can fine-tune Stable Audio 2.5 to help enterprises create the right sound for their brand. ▶️ Faster inference: The model can generate up to three-minute long tracks in under two seconds on a GPU, outputting in just eight steps (compared to ~50 in the previous model). You can learn more here 👉

Stability AI

67,552 views • 10 months ago

Gemini 2.0 Flash Experimental has the ability to produce native audio in a variety of styles and languages - all from scratch. 🗣️ Here’s how this is different to traditional text-to-speech systems ↓

Gemini 2.0 Flash Experimental has the ability to produce native audio in a variety of styles and languages - all from scratch. 🗣️ Here’s how this is different to traditional text-to-speech systems ↓

Google DeepMind

125,784 views • 1 year ago

I’ve been exploring Gemini 2.0’s new native audio output capability, which is available for early testers. I’m a developer at Google Creative Lab, and wanted to share one of my favorite experiments so far called ✨ VoiceCursor (🔊 sound on for video) Unlike traditional TTS, native audio lets you prompt the model with expressive styles, ie “Say this like a disgruntled pirate…” So I made ✨VoiceCursor… it lets you rapidly try different prompts. Just type, highlight your phrase, then hear it spoken in different ways! My code is open-sourced here: Here’s a thread 🧵

I’ve been exploring Gemini 2.0’s new native audio output capability, which is available for early testers. I’m a developer at Google Creative Lab, and wanted to share one of my favorite experiments so far called ✨ VoiceCursor (🔊 sound on for video) Unlike traditional TTS, native audio lets you prompt the model with expressive styles, ie “Say this like a disgruntled pirate…” So I made ✨VoiceCursor… it lets you rapidly try different prompts. Just type, highlight your phrase, then hear it spoken in different ways! My code is open-sourced here: Here’s a thread 🧵

Trudy Painter

67,567 views • 1 year ago

Meet Veo 3.1 👋 With Veo 3.1, you can generate videos with richer audio, better narrative control, and enhanced realism. This new update also comes with a suite of additional features built for creative control. Including the ability to: — Extend your videos to make them longer (with audio) — Set the first and last frame of your video to control your output (with audio) — Upload multiple ingredient images to craft your scene (with audio) — Add or remove objects directly in your video These features are available in Flow by Google, the Gemini API, and Google Cloud Vertex AI. Veo 3.1 is also available in the Google Gemini. We’re so excited to see what you create!

Meet Veo 3.1 👋 With Veo 3.1, you can generate videos with richer audio, better narrative control, and enhanced realism. This new update also comes with a suite of additional features built for creative control. Including the ability to: — Extend your videos to make them longer (with audio) — Set the first and last frame of your video to control your output (with audio) — Upload multiple ingredient images to craft your scene (with audio) — Add or remove objects directly in your video These features are available in Flow by Google, the Gemini API, and Google Cloud Vertex AI. Veo 3.1 is also available in the Google Gemini. We’re so excited to see what you create!

Google AI

71,238 views • 9 months ago

Introducing OpenVoice 🔊 Instant voice cloning that requires only a short audio clip to replicate voice and generate speech in multiple languages and tones Listen to the examples They are able to replicate voices with only one sentence Demo to try below:

Introducing OpenVoice 🔊 Instant voice cloning that requires only a short audio clip to replicate voice and generate speech in multiple languages and tones Listen to the examples They are able to replicate voices with only one sentence Demo to try below:

Allen T.

32,932 views • 2 years ago

We’re introducing Canvas and welcoming Audio Overview to the Google Gemini today. Turn docs, slides + Deep Research reports into podcasts with Audio Overview. Or create a doc or coding project with Canvas – it spins up prototypes super fast and you can edit with Gemini in real-time.

We’re introducing Canvas and welcoming Audio Overview to the Google Gemini today. Turn docs, slides + Deep Research reports into podcasts with Audio Overview. Or create a doc or coding project with Canvas – it spins up prototypes super fast and you can edit with Gemini in real-time.

Sundar Pichai

384,050 views • 1 year ago

So you can clone any voice 100% locally using this new open source model?! Alibaba has released Qwen3-TTS on Hugging Face. You can easily: - Create custom voices - Clone any voice from a VERY short audio - Generate speech with style instructions Only 0.6B & 1.8B! Sound on🔊

So you can clone any voice 100% locally using this new open source model?! Alibaba has released Qwen3-TTS on Hugging Face. You can easily: - Create custom voices - Clone any voice from a VERY short audio - Generate speech with style instructions Only 0.6B & 1.8B! Sound on🔊

Paul Couvert

148,975 views • 5 months ago