Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

End to End Speech models are on fire - LLAMA-OMNI 8B - Apache licensed! 🔥 > Speech Encoder - Whisper Large v3 > LLM backbone - Llama 3.1 8B Instruct > Speech Decoder - HuBERT (UnitY) > Simultaneously generate Speech + Text > Less than 250 ms latency >... show more

Vaibhav (VB) Srivastav

62,166 subscribers

47,943 views • 2 years ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

10 Comments

Vaibhav (VB) Srivastav2 years ago

Model checkpoint:

Vaibhav (VB) Srivastav2 years ago

Github repo:

Qingkai Fang2 years ago

Thanks for sharing our work!

Vaibhav (VB) Srivastav2 years ago

🔥

Tommy D. Rossi2 years ago

I wouldn't call this end to end, let's keep that term for single multi modal models that do everything by themselves

ThisAndThat2 years ago

less than 250ms latency on what?

Vaibhav (VB) Srivastav2 years ago

Time to first audio chunk according to their GH.

Waifuology2 years ago

License looks good, but the voice quality isn't really there yet.

Hiro2 years ago

Do you know what are supported languages?

Trying my best :-)2 years ago

Can it detect emotion?

Related Videos

Multimodal Ichigo Llama 3.1 - Real Time Voice AI 🔥 > WhisperSpeech X Llama 3.1 8B > Trained on 50K hours of speech (7 languages) > Continually trained on 45hrs 10x A1000s > MLS -> WhisperVQ tokens -> Llama 3.1 > Instruction tuned on 1.89M samples > 70% speech, 20% transcription, 10% text > Apache 2.0 licensed ⚡ Architecture: > WhisperSpeech/ VQ for Semantic Tokens > Llama 3.1 8B Instruct for Text backbone > Early fusion (Chameleon) I'm super bullish on Homebrew to Menlo and early fusion, audio and text, multimodal models! (P.S. Play with the demo on Hugging Face)

Multimodal Ichigo Llama 3.1 - Real Time Voice AI 🔥 > WhisperSpeech X Llama 3.1 8B > Trained on 50K hours of speech (7 languages) > Continually trained on 45hrs 10x A1000s > MLS -> WhisperVQ tokens -> Llama 3.1 > Instruction tuned on 1.89M samples > 70% speech, 20% transcription, 10% text > Apache 2.0 licensed ⚡ Architecture: > WhisperSpeech/ VQ for Semantic Tokens > Llama 3.1 8B Instruct for Text backbone > Early fusion (Chameleon) I'm super bullish on Homebrew to Menlo and early fusion, audio and text, multimodal models! (P.S. Play with the demo on Hugging Face)

Vaibhav (VB) Srivastav

82,170 views • 2 years ago

LETS GOOO! FishSpeech v1.5 - multilingual, zero-shot instant voice cloning, low-latency, open text to speech model 🔥 > Only 500M params > Trained on 1 MILLION hours of audio > Supports 13 languages > Low-latency ( Open model - checkpoints on the hub 🤗 > Best part: Ranks #2 on TTS Arena (preliminary results) Kudos to Fish Audio - truly leading the way for open text to speech research! ⚡

LETS GOOO! FishSpeech v1.5 - multilingual, zero-shot instant voice cloning, low-latency, open text to speech model 🔥 > Only 500M params > Trained on 1 MILLION hours of audio > Supports 13 languages > Low-latency ( Open model - checkpoints on the hub 🤗 > Best part: Ranks #2 on TTS Arena (preliminary results) Kudos to Fish Audio - truly leading the way for open text to speech research! ⚡

Vaibhav (VB) Srivastav

49,169 views • 1 year ago

NEW: Kokoro 82M - APACHE 2.0 licensed, Text to Speech model, trained on < 100 hours of audio 🔥

NEW: Kokoro 82M - APACHE 2.0 licensed, Text to Speech model, trained on < 100 hours of audio 🔥

Vaibhav (VB) Srivastav

330,034 views • 1 year ago

Wow! New Speech to Speech model - Fish Agent v0.1 3B by Fish Audio 🔥 > Trained on 700K hours of multilingual audio > Continue-pretrained version of Qwen-2.5-3B-Instruct for 200B audio & text tokens > Zero-shot voice cloning > Text + audio input/ Audio output > Ultra-fast inference w/ 200ms TTFA > Models on the Hub & Finetuning code on its way! 🚀 What an amazing time to be alive 🤗

Wow! New Speech to Speech model - Fish Agent v0.1 3B by Fish Audio 🔥 > Trained on 700K hours of multilingual audio > Continue-pretrained version of Qwen-2.5-3B-Instruct for 200B audio & text tokens > Zero-shot voice cloning > Text + audio input/ Audio output > Ultra-fast inference w/ 200ms TTFA > Models on the Hub & Finetuning code on its way! 🚀 What an amazing time to be alive 🤗

Vaibhav (VB) Srivastav

67,003 views • 2 years ago

CPU Demo In Action: Meta Llama 3 8b Instruct With AMD Ryzen AI

CPU Demo In Action: Meta Llama 3 8b Instruct With AMD Ryzen AI

AMD

39,226 views • 2 years ago

Mini-Omni 2 understands image, audio and text inputs all via end-to-end voice conversations with users 🔥 > Understands and processes images, speech, and text > Generates real-time speech responses > Supports interruptions during speech Technical Overview: > Concatenates image, audio, and text features for input. > Uses text-guided delayed parallel output for real-time speech > Involves encoder adaptation, modal alignment, and multimodal fine-tuning Best part: MIT licensed ⚡

Mini-Omni 2 understands image, audio and text inputs all via end-to-end voice conversations with users 🔥 > Understands and processes images, speech, and text > Generates real-time speech responses > Supports interruptions during speech Technical Overview: > Concatenates image, audio, and text features for input. > Uses text-guided delayed parallel output for real-time speech > Involves encoder adaptation, modal alignment, and multimodal fine-tuning Best part: MIT licensed ⚡

Vaibhav (VB) Srivastav

45,453 views • 2 years ago

We made the first AI model you can have a realistic conversation with that was trained entirely on speech! Dessa is a full-duplex speech-to-speech model that you can run on consumer GPUs. This is a GPT3 moment for speech.

We made the first AI model you can have a realistic conversation with that was trained entirely on speech! Dessa is a full-duplex speech-to-speech model that you can run on consumer GPUs. This is a GPT3 moment for speech.

Ragavan Thurairatnam

4,648,325 views • 16 days ago

we’re lying on the moon. here are a few models glued together — speech to text (OpenAI whisper) to text (OpenAI gpt-3.5-turbo) to speech () — to recreate samantha from “her” in real-time.

we’re lying on the moon. here are a few models glued together — speech to text (OpenAI whisper) to text (OpenAI gpt-3.5-turbo) to speech () — to recreate samantha from “her” in real-time.

harley turan

506,430 views • 3 years ago

The new open-source Text to Speech model: Fish Speech 1.4 is brilliant! Trained on a massive 700K hours of multilingual speech data in 8 languages - Instant voice cloning 🗣️ - Ultra-low latency ⚡ - Compact model (~1GB weights) 🏋️‍♂️

The new open-source Text to Speech model: Fish Speech 1.4 is brilliant! Trained on a massive 700K hours of multilingual speech data in 8 languages - Instant voice cloning 🗣️ - Ultra-low latency ⚡ - Compact model (~1GB weights) 🏋️‍♂️

Rohan Paul

228,905 views • 2 years ago

Introducing SeamlessM4T, the first all-in-one, multilingual multimodal translation model. This single model can perform tasks across speech-to-text, speech-to-speech, text-to-text translation & speech recognition for up to 100 languages depending on the task. Details ⬇️

Introducing SeamlessM4T, the first all-in-one, multilingual multimodal translation model. This single model can perform tasks across speech-to-text, speech-to-speech, text-to-text translation & speech recognition for up to 100 languages depending on the task. Details ⬇️

AI at Meta

592,944 views • 3 years ago

Kyutai released their Streaming Text to Speech model, ~2B param model, ultra low latency (220ms), CC-BY-4.0 license 🔥 Trained on 2.5 Million Hours of audio, it can serve up to 32 users w/ less than 350ms latency on a SINGLE L40 🤯 Incredible release by kyutai folks, go check out their hugging face page now!

Kyutai released their Streaming Text to Speech model, ~2B param model, ultra low latency (220ms), CC-BY-4.0 license 🔥 Trained on 2.5 Million Hours of audio, it can serve up to 32 users w/ less than 350ms latency on a SINGLE L40 🤯 Incredible release by kyutai folks, go check out their hugging face page now!

Vaibhav (VB) Srivastav

93,586 views • 1 year ago

Here are the best practices for using Eleven v3 (alpha) - the most expressive Text to Speech model.

Here are the best practices for using Eleven v3 (alpha) - the most expressive Text to Speech model.

ElevenLabs

43,851 views • 1 year ago

REMOVE THE CENSORSHIP FROM ANY AI MODEL IN YOUR BROWSER FOR FREEEEE🤯 obliteratus is the open source tool elder plinius used to make the uncensored deepseek v4.1 flash how to use it (no code): 1. open the hugging face space 2. or open the colab notebook and hit run all 3. pick a model, obliterate, test it for devs obliteratus obliterate meta-llama/Llama-3.1-8B-Instruct --method advanced the hf space runs on zerogpu with a free daily quota on hf pro and you're responsible for what you do with the model

REMOVE THE CENSORSHIP FROM ANY AI MODEL IN YOUR BROWSER FOR FREEEEE🤯 obliteratus is the open source tool elder plinius used to make the uncensored deepseek v4.1 flash how to use it (no code): 1. open the hugging face space 2. or open the colab notebook and hit run all 3. pick a model, obliterate, test it for devs obliteratus obliterate meta-llama/Llama-3.1-8B-Instruct --method advanced the hf space runs on zerogpu with a free daily quota on hf pro and you're responsible for what you do with the model

Forha

47,088 views • 3 days ago

Introducing Universal-1: Our most powerful and accurate Speech AI model yet. ✅ Trained on 12.5M hours of multilingual speech data ✅ 13.5% more accurate than models like Whisper ✅ Up to 30% fewer hallucinations than seq2seq models ✅ Just 38 seconds to process 1 hour of audio

Introducing Universal-1: Our most powerful and accurate Speech AI model yet. ✅ Trained on 12.5M hours of multilingual speech data ✅ 13.5% more accurate than models like Whisper ✅ Up to 30% fewer hallucinations than seq2seq models ✅ Just 38 seconds to process 1 hour of audio

AssemblyAI

23,511,312 views • 2 years ago

REMOVE THE CENSORSHIP FROM ANY AI MODEL IN YOUR BROWSER FOR FREEEEE obliteratus is the open source tool elder plinius used to make the uncensored deepseek v4.1 flash how to use it (no code): 1. open the hugging face space 2. or open the colab notebook and hit run all 3. pick a model, obliterate, test it for devs obliteratus obliterate meta-llama/Llama-3.1-8B-Instruct --method advanced the hf space runs on zerogpu with a free daily quota on hf pro and you're responsible for what you do with the model

REMOVE THE CENSORSHIP FROM ANY AI MODEL IN YOUR BROWSER FOR FREEEEE obliteratus is the open source tool elder plinius used to make the uncensored deepseek v4.1 flash how to use it (no code): 1. open the hugging face space 2. or open the colab notebook and hit run all 3. pick a model, obliterate, test it for devs obliteratus obliterate meta-llama/Llama-3.1-8B-Instruct --method advanced the hf space runs on zerogpu with a free daily quota on hf pro and you're responsible for what you do with the model

Sakata

28,951 views • 4 days ago

We’ve never seen a 7B model get the famous Sally test right, let alone a 3.8B model. 💪Even the Llama 3 8B Instruct model fails this test. Coming 🔜 to your 📱

We’ve never seen a 7B model get the famous Sally test right, let alone a 3.8B model. 💪Even the Llama 3 8B Instruct model fails this test. Coming 🔜 to your 📱

Private LLM

36,972 views • 2 years ago

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

AK

290,729 views • 3 years ago

Real-time AI conversations are here! PlayHT, one of the best text-to-speech models I’ve used, now has a latency of less than 300ms. Checkout how fast it outputs audio 🤯 Also, my experience cloning my own voice and links to try it for free are below.

Real-time AI conversations are here! PlayHT, one of the best text-to-speech models I’ve used, now has a latency of less than 300ms. Checkout how fast it outputs audio 🤯 Also, my experience cloning my own voice and links to try it for free are below.

Alvaro Cintas

335,391 views • 3 years ago