Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

End to End Speech models are on fire - LLAMA-OMNI 8B - Apache licensed! 🔥 > Speech Encoder - Whisper Large v3 > LLM backbone - Llama 3.1 8B Instruct > Speech Decoder - HuBERT (UnitY) > Simultaneously generate Speech + Text > Less than 250 ms latency >...

reach_vb's profile picture

Vaibhav (VB) Srivastav

62,166 subscribers

47,943 görüntüleme • 2 yıl önce •via X (Twitter)

Bilim & Teknoloji
LIVE

Anya Rossi• Live Now

Private livecam show

10 Yorum

Vaibhav (VB) Srivastav profil fotoğrafı
Vaibhav (VB) Srivastav2 yıl önce

Model checkpoint:

Vaibhav (VB) Srivastav profil fotoğrafı
Vaibhav (VB) Srivastav2 yıl önce

Github repo:

Qingkai Fang profil fotoğrafı
Qingkai Fang2 yıl önce

Thanks for sharing our work!

Vaibhav (VB) Srivastav profil fotoğrafı
Vaibhav (VB) Srivastav2 yıl önce

🔥

Tommy D. Rossi profil fotoğrafı
Tommy D. Rossi2 yıl önce

I wouldn't call this end to end, let's keep that term for single multi modal models that do everything by themselves

ThisAndThat profil fotoğrafı
ThisAndThat2 yıl önce

less than 250ms latency on what?

Vaibhav (VB) Srivastav profil fotoğrafı
Vaibhav (VB) Srivastav2 yıl önce

Time to first audio chunk according to their GH.

Waifuology profil fotoğrafı
Waifuology2 yıl önce

License looks good, but the voice quality isn't really there yet.

Hiro profil fotoğrafı
Hiro2 yıl önce

Do you know what are supported languages?

Trying my best :-) profil fotoğrafı
Trying my best :-)2 yıl önce

Can it detect emotion?

Benzer Videolar

Multimodal Ichigo Llama 3.1 - Real Time Voice AI 🔥 > WhisperSpeech X Llama 3.1 8B > Trained on 50K hours of speech (7 languages) > Continually trained on 45hrs 10x A1000s > MLS -> WhisperVQ tokens -> Llama 3.1 > Instruction tuned on 1.89M samples > 70% speech, 20% transcription, 10% text > Apache 2.0 licensed ⚡ Architecture: > WhisperSpeech/ VQ for Semantic Tokens > Llama 3.1 8B Instruct for Text backbone > Early fusion (Chameleon) I'm super bullish on Homebrew to Menlo and early fusion, audio and text, multimodal models! (P.S. Play with the demo on Hugging Face)
0:32

Multimodal Ichigo Llama 3.1 - Real Time Voice AI 🔥 > WhisperSpeech X Llama 3.1 8B > Trained on 50K hours of speech (7 languages) > Continually trained on 45hrs 10x A1000s > MLS -> WhisperVQ tokens -> Llama 3.1 > Instruction tuned on 1.89M samples > 70% speech, 20% transcription, 10% text > Apache 2.0 licensed ⚡ Architecture: > WhisperSpeech/ VQ for Semantic Tokens > Llama 3.1 8B Instruct for Text backbone > Early fusion (Chameleon) I'm super bullish on Homebrew to Menlo and early fusion, audio and text, multimodal models! (P.S. Play with the demo on Hugging Face)

Vaibhav (VB) Srivastav

82,170 görüntüleme • 2 yıl önce

LETS GOOO! FishSpeech v1.5 - multilingual, zero-shot instant voice cloning, low-latency, open text to speech model 🔥 > Only 500M params > Trained on 1 MILLION hours of audio > Supports 13 languages > Low-latency ( Open model - checkpoints on the hub 🤗 > Best part: Ranks #2 on TTS Arena (preliminary results) Kudos to Fish Audio - truly leading the way for open text to speech research! ⚡
2:26

LETS GOOO! FishSpeech v1.5 - multilingual, zero-shot instant voice cloning, low-latency, open text to speech model 🔥 > Only 500M params > Trained on 1 MILLION hours of audio > Supports 13 languages > Low-latency ( Open model - checkpoints on the hub 🤗 > Best part: Ranks #2 on TTS Arena (preliminary results) Kudos to Fish Audio - truly leading the way for open text to speech research! ⚡

Vaibhav (VB) Srivastav

49,169 görüntüleme • 1 yıl önce

NEW: Kokoro 82M - APACHE 2.0 licensed, Text to Speech model, trained on < 100 hours of audio 🔥
1:54

NEW: Kokoro 82M - APACHE 2.0 licensed, Text to Speech model, trained on < 100 hours of audio 🔥

Vaibhav (VB) Srivastav

330,034 görüntüleme • 1 yıl önce

Wow! New Speech to Speech model - Fish Agent v0.1 3B by Fish Audio 🔥 > Trained on 700K hours of multilingual audio > Continue-pretrained version of Qwen-2.5-3B-Instruct for 200B audio & text tokens > Zero-shot voice cloning > Text + audio input/ Audio output > Ultra-fast inference w/ 200ms TTFA > Models on the Hub & Finetuning code on its way! 🚀 What an amazing time to be alive 🤗
1:08

Wow! New Speech to Speech model - Fish Agent v0.1 3B by Fish Audio 🔥 > Trained on 700K hours of multilingual audio > Continue-pretrained version of Qwen-2.5-3B-Instruct for 200B audio & text tokens > Zero-shot voice cloning > Text + audio input/ Audio output > Ultra-fast inference w/ 200ms TTFA > Models on the Hub & Finetuning code on its way! 🚀 What an amazing time to be alive 🤗

Vaibhav (VB) Srivastav

67,003 görüntüleme • 2 yıl önce

CPU Demo In Action: Meta Llama 3 8b Instruct With AMD Ryzen AI
1:00

CPU Demo In Action: Meta Llama 3 8b Instruct With AMD Ryzen AI

AMD

39,226 görüntüleme • 2 yıl önce

Mini-Omni 2 understands image, audio and text inputs all via end-to-end voice conversations with users 🔥 > Understands and processes images, speech, and text > Generates real-time speech responses > Supports interruptions during speech Technical Overview: > Concatenates image, audio, and text features for input. > Uses text-guided delayed parallel output for real-time speech > Involves encoder adaptation, modal alignment, and multimodal fine-tuning Best part: MIT licensed ⚡
0:54

Mini-Omni 2 understands image, audio and text inputs all via end-to-end voice conversations with users 🔥 > Understands and processes images, speech, and text > Generates real-time speech responses > Supports interruptions during speech Technical Overview: > Concatenates image, audio, and text features for input. > Uses text-guided delayed parallel output for real-time speech > Involves encoder adaptation, modal alignment, and multimodal fine-tuning Best part: MIT licensed ⚡

Vaibhav (VB) Srivastav

45,453 görüntüleme • 2 yıl önce

We made the first AI model you can have a realistic conversation with that was trained entirely on speech! Dessa is a full-duplex speech-to-speech model that you can run on consumer GPUs. This is a GPT3 moment for speech.
3:25

We made the first AI model you can have a realistic conversation with that was trained entirely on speech! Dessa is a full-duplex speech-to-speech model that you can run on consumer GPUs. This is a GPT3 moment for speech.

Ragavan Thurairatnam

4,648,325 görüntüleme • 16 gün önce

we’re lying on the moon. here are a few models glued together — speech to text (OpenAI whisper) to text (OpenAI gpt-3.5-turbo) to speech () — to recreate samantha from “her” in real-time.
1:18

we’re lying on the moon. here are a few models glued together — speech to text (OpenAI whisper) to text (OpenAI gpt-3.5-turbo) to speech () — to recreate samantha from “her” in real-time.

harley turan

506,430 görüntüleme • 3 yıl önce

The new open-source Text to Speech model: Fish Speech 1.4 is brilliant! Trained on a massive 700K hours of multilingual speech data in 8 languages - Instant voice cloning 🗣️ - Ultra-low latency ⚡ - Compact model (~1GB weights) 🏋️‍♂️
1:23

The new open-source Text to Speech model: Fish Speech 1.4 is brilliant! Trained on a massive 700K hours of multilingual speech data in 8 languages - Instant voice cloning 🗣️ - Ultra-low latency ⚡ - Compact model (~1GB weights) 🏋️‍♂️

Rohan Paul

228,905 görüntüleme • 2 yıl önce

Introducing SeamlessM4T, the first all-in-one, multilingual multimodal translation model. This single model can perform tasks across speech-to-text, speech-to-speech, text-to-text translation & speech recognition for up to 100 languages depending on the task. Details ⬇️
0:56

Introducing SeamlessM4T, the first all-in-one, multilingual multimodal translation model. This single model can perform tasks across speech-to-text, speech-to-speech, text-to-text translation & speech recognition for up to 100 languages depending on the task. Details ⬇️

AI at Meta

592,944 görüntüleme • 3 yıl önce

Kyutai released their Streaming Text to Speech model, ~2B param model, ultra low latency (220ms), CC-BY-4.0 license 🔥 Trained on 2.5 Million Hours of audio, it can serve up to 32 users w/ less than 350ms latency on a SINGLE L40 🤯 Incredible release by kyutai folks, go check out their hugging face page now!
0:41

Kyutai released their Streaming Text to Speech model, ~2B param model, ultra low latency (220ms), CC-BY-4.0 license 🔥 Trained on 2.5 Million Hours of audio, it can serve up to 32 users w/ less than 350ms latency on a SINGLE L40 🤯 Incredible release by kyutai folks, go check out their hugging face page now!

Vaibhav (VB) Srivastav

93,586 görüntüleme • 1 yıl önce

Here are the best practices for using Eleven v3 (alpha) - the most expressive Text to Speech model.
3:55

Here are the best practices for using Eleven v3 (alpha) - the most expressive Text to Speech model.

ElevenLabs

43,851 görüntüleme • 1 yıl önce

REMOVE THE CENSORSHIP FROM ANY AI MODEL IN YOUR BROWSER FOR FREEEEE🤯 obliteratus is the open source tool elder plinius used to make the uncensored deepseek v4.1 flash how to use it (no code): 1. open the hugging face space 2. or open the colab notebook and hit run all 3. pick a model, obliterate, test it for devs obliteratus obliterate meta-llama/Llama-3.1-8B-Instruct --method advanced the hf space runs on zerogpu with a free daily quota on hf pro and you're responsible for what you do with the model
0:36

REMOVE THE CENSORSHIP FROM ANY AI MODEL IN YOUR BROWSER FOR FREEEEE🤯 obliteratus is the open source tool elder plinius used to make the uncensored deepseek v4.1 flash how to use it (no code): 1. open the hugging face space 2. or open the colab notebook and hit run all 3. pick a model, obliterate, test it for devs obliteratus obliterate meta-llama/Llama-3.1-8B-Instruct --method advanced the hf space runs on zerogpu with a free daily quota on hf pro and you're responsible for what you do with the model

Forha

47,088 görüntüleme • 3 gün önce

Introducing Universal-1: Our most powerful and accurate Speech AI model yet. ✅ Trained on 12.5M hours of multilingual speech data ✅ 13.5% more accurate than models like Whisper ✅ Up to 30% fewer hallucinations than seq2seq models ✅ Just 38 seconds to process 1 hour of audio
1:01

Introducing Universal-1: Our most powerful and accurate Speech AI model yet. ✅ Trained on 12.5M hours of multilingual speech data ✅ 13.5% more accurate than models like Whisper ✅ Up to 30% fewer hallucinations than seq2seq models ✅ Just 38 seconds to process 1 hour of audio

AssemblyAI

23,511,312 görüntüleme • 2 yıl önce

REMOVE THE CENSORSHIP FROM ANY AI MODEL IN YOUR BROWSER FOR FREEEEE obliteratus is the open source tool elder plinius used to make the uncensored deepseek v4.1 flash how to use it (no code): 1. open the hugging face space 2. or open the colab notebook and hit run all 3. pick a model, obliterate, test it for devs obliteratus obliterate meta-llama/Llama-3.1-8B-Instruct --method advanced the hf space runs on zerogpu with a free daily quota on hf pro and you're responsible for what you do with the model
0:23

REMOVE THE CENSORSHIP FROM ANY AI MODEL IN YOUR BROWSER FOR FREEEEE obliteratus is the open source tool elder plinius used to make the uncensored deepseek v4.1 flash how to use it (no code): 1. open the hugging face space 2. or open the colab notebook and hit run all 3. pick a model, obliterate, test it for devs obliteratus obliterate meta-llama/Llama-3.1-8B-Instruct --method advanced the hf space runs on zerogpu with a free daily quota on hf pro and you're responsible for what you do with the model

Sakata

28,951 görüntüleme • 4 gün önce

We’ve never seen a 7B model get the famous Sally test right, let alone a 3.8B model. 💪Even the Llama 3 8B Instruct model fails this test. Coming 🔜 to your 📱
0:20

We’ve never seen a 7B model get the famous Sally test right, let alone a 3.8B model. 💪Even the Llama 3 8B Instruct model fails this test. Coming 🔜 to your 📱

Private LLM

36,972 görüntüleme • 2 yıl önce

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.
4:10

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

AK

290,729 görüntüleme • 3 yıl önce

Real-time AI conversations are here! PlayHT, one of the best text-to-speech models I’ve used, now has a latency of less than 300ms. Checkout how fast it outputs audio 🤯 Also, my experience cloning my own voice and links to try it for free are below.
0:30

Real-time AI conversations are here! PlayHT, one of the best text-to-speech models I’ve used, now has a latency of less than 300ms. Checkout how fast it outputs audio 🤯 Also, my experience cloning my own voice and links to try it for free are below.

Alvaro Cintas

335,391 görüntüleme • 3 yıl önce