Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement abs: project page:

AK

400,551 subscribers

34,949 views • 3 years ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

6 Comments

Alex Volkov (Thursd/AI)3 years ago

This is so weird haah

RY⚡3 years ago

Wait.. is it..forming the speech just from the visual input? No, right?

GanLike3 years ago

😁🙀

Clément3 years ago

AHAHAHAHHAHAHHAHAHHA

Reyon Gifto Immanuel3 years ago

@scispace_ Summarize this

SciSpace3 years ago

@_akhaliq @ReyonGifto Here you go 👇 It proposes a unified model, ReVISE, for Generalized Speech Enhancement, with the goal of improving intelligibility, quality, and video synchronization. Results show that ReVISE greatly suppresses noise and improves quality, showing its applicability

Related Videos

📢New paper We are announcing ReVISE, the first universal audio-visual speech enhancement model powered by SSL. paper: demo: w/ Yossi Adi @TalRemez BowenShi Jacob Donley

📢New paper We are announcing ReVISE, the first universal audio-visual speech enhancement model powered by SSL. paper: demo: w/ Yossi Adi @TalRemez BowenShi Jacob Donley

Wei-Ning Hsu

46,382 views • 3 years ago

ReBotNet: Fast Real-time Video Enhancement abs: project page:

ReBotNet: Fast Real-time Video Enhancement abs: project page:

AK

111,249 views • 3 years ago

Tesla Full Self-Driving (Supervised) 13.2 exits Universal Studios

Tesla Full Self-Driving (Supervised) 13.2 exits Universal Studios

Whole Mars Catalog

117,562 views • 1 year ago

Word-As-Image for Semantic Typography abs: project page:

Word-As-Image for Semantic Typography abs: project page:

AK

385,996 views • 3 years ago

Mini-Omni 2 understands image, audio and text inputs all via end-to-end voice conversations with users 🔥 > Understands and processes images, speech, and text > Generates real-time speech responses > Supports interruptions during speech Technical Overview: > Concatenates image, audio, and text features for input. > Uses text-guided delayed parallel output for real-time speech > Involves encoder adaptation, modal alignment, and multimodal fine-tuning Best part: MIT licensed ⚡

Mini-Omni 2 understands image, audio and text inputs all via end-to-end voice conversations with users 🔥 > Understands and processes images, speech, and text > Generates real-time speech responses > Supports interruptions during speech Technical Overview: > Concatenates image, audio, and text features for input. > Uses text-guided delayed parallel output for real-time speech > Involves encoder adaptation, modal alignment, and multimodal fine-tuning Best part: MIT licensed ⚡

Vaibhav (VB) Srivastav

45,409 views • 1 year ago

vMAP: Vectorised Object Mapping for Neural Field SLAM abs: project page:

vMAP: Vectorised Object Mapping for Neural Field SLAM abs: project page:

AK

36,137 views • 3 years ago

SceneScape: Text-Driven Consistent Scene Generation abs: project page: text-driven perpetual view generation -- synthesizing long videos of arbitrary scenes solely from an input text describing the scene and camera poses

SceneScape: Text-Driven Consistent Scene Generation abs: project page: text-driven perpetual view generation -- synthesizing long videos of arbitrary scenes solely from an input text describing the scene and camera poses

AK

73,258 views • 3 years ago

Generative Novel View Synthesis with 3D-Aware Diffusion Models abs: project page:

Generative Novel View Synthesis with 3D-Aware Diffusion Models abs: project page:

AK

304,708 views • 3 years ago

My speech today on the Universal Credit and Personal Independence Payment Bill

My speech today on the Universal Credit and Personal Independence Payment Bill

Dr Marie Tidball MP

65,509 views • 11 months ago

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

AK

290,517 views • 3 years ago

GeoCode: Interpretable Shape Programs abs: project page:

GeoCode: Interpretable Shape Programs abs: project page:

AK

1,428,749 views • 3 years ago

Introducing... Amica! 👩‍🦰🤖 Amica is an open source interface for interactive communication with 3D characters with voice synthesis, speech recognition, visual understanding, and an emotion system. 🔗

Introducing... Amica! 👩‍🦰🤖 Amica is an open source interface for interactive communication with 3D characters with voice synthesis, speech recognition, visual understanding, and an emotion system. 🔗

Arbius

41,769 views • 2 years ago

Open-source project Soundstorm (AI generated speech from Google Research) is going to give Elevenlabs a run for it's money: The text-to-speech project specializes in dialogue between multiple parties, and is available on Github:

Open-source project Soundstorm (AI generated speech from Google Research) is going to give Elevenlabs a run for it's money: The text-to-speech project specializes in dialogue between multiple parties, and is available on Github:

AI Breakfast

326,474 views • 3 years ago

Single Motion Diffusion abs: project page:

Single Motion Diffusion abs: project page:

AK

72,767 views • 3 years ago

🔥 Ming-UniAudio: The 「Nano Banana」moment for speech is here! A single model for universal understanding, generation & free-form editing. First Unified Continuous Tokenizer 「MingTok-Audio」and Unified Und & Gen Speech LLM built on it. First Universal Free-form Speech Editing Model Without Timestamp Condition. First FreeForm Audio Editing Benchmark. 💻 GitHub: 🤗 Tokenizer: 🤗 Model: base: edit: 🤗 Benchmark: 🌍 blog: #AI #Speech #SpeechLLM #LLM #GenerativeAI #Audio #ASR #TTS #SpeechEditing

🔥 Ming-UniAudio: The 「Nano Banana」moment for speech is here! A single model for universal understanding, generation & free-form editing. First Unified Continuous Tokenizer 「MingTok-Audio」and Unified Und & Gen Speech LLM built on it. First Universal Free-form Speech Editing Model Without Timestamp Condition. First FreeForm Audio Editing Benchmark. 💻 GitHub: 🤗 Tokenizer: 🤗 Model: base: edit: 🤗 Benchmark: 🌍 blog: #AI #Speech #SpeechLLM #LLM #GenerativeAI #Audio #ASR #TTS #SpeechEditing

Ant Ling

21,365 views • 8 months ago

Supervised learning has held 3D Vision back for too long. Meet RayZer — a self-supervised 3D model trained with zero 3D labels: ❌ No supervision of camera & geometry ✅ Just RGB images And the wild part? RayZer outperforms supervised methods (as 3D labels from COLMAP is noisy) 🌐 Project: (1/4)

Supervised learning has held 3D Vision back for too long. Meet RayZer — a self-supervised 3D model trained with zero 3D labels: ❌ No supervision of camera & geometry ✅ Just RGB images And the wild part? RayZer outperforms supervised methods (as 3D labels from COLMAP is noisy) 🌐 Project: (1/4)

Hanwen Jiang

69,527 views • 1 year ago

Announcing Universal-2, our most advanced speech-to-text model to date. Building on Universal-1's industry-leading performance, we've pushed Speech AI even further. Here's what's new:

Announcing Universal-2, our most advanced speech-to-text model to date. Building on Universal-1's industry-leading performance, we've pushed Speech AI even further. Here's what's new:

AssemblyAI

42,966 views • 1 year ago

MusicLM: Generating Music From Text abs: project page:

MusicLM: Generating Music From Text abs: project page:

AK

434,142 views • 3 years ago

Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion abs: project page:

Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion abs: project page:

AK

262,780 views • 3 years ago