Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

Announcing 𝐕𝐨𝐢𝐜𝐞𝐂𝐫𝐚𝐟𝐭🪄 SotA for both speech editing and zero-shot text-to-speech, Outperforming VALL-E, XTTS-v2, etc. VoiceCraft works on in-the-wild data such as movies, random videos and podcasts We fully open source it at

Puyuan Peng

1,762 subscribers

160,462 просмотров • 2 лет назад •via X (Twitter)

Наука и технологии

Anya Rossi• Live Now

Private livecam show

Комментарии: 10

Фото профиля Puyuan Peng

Puyuan Peng2 лет назад

𝐕𝐨𝐢𝐜𝐞𝐂𝐫𝐚𝐟𝐭 works well on recordings with diverse accents, emotions, styles, content, background noise, recording conditions. Demo: Paper: code, model, data:

Фото профиля Vaibhav (VB) Srivastav

Vaibhav (VB) Srivastav2 лет назад

@rdesh26 Wow! love itt! I'd love to add this on the TTS Arena: Let's chat in DMs?

Фото профиля Puyuan Peng

Puyuan Peng2 лет назад

@rdesh26 Yea let’s do that!

Фото профиля mrfakename

mrfakename2 лет назад

Thank you for releasing! Any possibility of switching to an open source license?

Фото профиля Puyuan Peng

Puyuan Peng2 лет назад

Thanks! Have been discussing the licensing issue, might change it in the coming days

Фото профиля Kaashika Prajaapat

Kaashika Prajaapat2 лет назад

I tried out. Insane!!! Amazing work @PuyuanPeng Can the license be commercially available? Would love to incorporate this into our product.

Фото профиля Jonathan Fly 👾

Jonathan Fly 👾2 лет назад

Cool practical solutions done in sampling: discarding longest/shortest, special cases for silence tokens, rep penalties and silence tokens interacting. Might be able to even penalize the scratching sound if it's consistent enough (or maybe CFG negative prompt?).

Фото профиля Paulo Vasconcellos

Paulo Vasconcellos2 лет назад

Wish it had a more permissive license. Is it possible to change it to be used commercially?

Фото профиля Puyuan Peng

Puyuan Peng2 лет назад

have been discussing the licensing issue, might change it in the coming days

Фото профиля Siddharth Sharma

Siddharth Sharma2 лет назад

Great work! Would love to connect with you

Похожие видео

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale blog: Large-scale generative models such as GPT and DALL-E have revolutionized natural language processing and computer vision research. These models not only generate high fidelity text or image outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are neither filtered nor enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster.

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale blog: Large-scale generative models such as GPT and DALL-E have revolutionized natural language processing and computer vision research. These models not only generate high fidelity text or image outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are neither filtered nor enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster.

AK

429,355 просмотров • 3 лет назад

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

AK

290,517 просмотров • 3 лет назад

Introducing Scribe — the most accurate Speech to Text model. It has the highest accuracy on benchmarks, outperforming previous state-of-the-art models such as Gemini 2.0 and OpenAI Whisper v3. It’s now the leading model for English, Spanish, Italian, and many more. With support for 99 languages, speaker diarization, character-level timestamps, and non-speech events such as laughing.

Introducing Scribe — the most accurate Speech to Text model. It has the highest accuracy on benchmarks, outperforming previous state-of-the-art models such as Gemini 2.0 and OpenAI Whisper v3. It’s now the leading model for English, Spanish, Italian, and many more. With support for 99 languages, speaker diarization, character-level timestamps, and non-speech events such as laughing.

ElevenLabs

464,967 просмотров • 1 год назад

UPDATE: Four new open models on the Text to Speech Arena! 🔥 *sound on🔉* As the Text-to-Speech ecosystem is heating up, we decided to add more competition. > Parler TTS > VoiceCraft > Vokan > GPT-SOVITS Why is this important? The TTS ecosystem is riddled with opaque metrics and meaningless MOS scores. By crowdsourcing the evals, we test these models in real-life conditions and much more methodically. ⚡ Rank one, rank'em all! 🚀

UPDATE: Four new open models on the Text to Speech Arena! 🔥 sound on🔉 As the Text-to-Speech ecosystem is heating up, we decided to add more competition. > Parler TTS > VoiceCraft > Vokan > GPT-SOVITS Why is this important? The TTS ecosystem is riddled with opaque metrics and meaningless MOS scores. By crowdsourcing the evals, we test these models in real-life conditions and much more methodically. ⚡ Rank one, rank'em all! 🚀

Vaibhav (VB) Srivastav

40,944 просмотров • 2 лет назад

ElevenLabs just solved speech transcription. Their new model, Scribe v2, is SOTA in word error rate. It handles pauses, laughs, complex words with ease. It's so good at segmenting speakers and made ZERO errors on the ultimate test: identical twin voices. How it works 👇

ElevenLabs just solved speech transcription. Their new model, Scribe v2, is SOTA in word error rate. It handles pauses, laughs, complex words with ease. It's so good at segmenting speakers and made ZERO errors on the ultimate test: identical twin voices. How it works 👇

Justine Moore

42,223 просмотров • 8 месяцев назад

Today we're releasing WildDet3D—an open model for monocular 3D object detection in the wild. It works with text, clicks, or 2D boxes, and on zero-shot evals it nearly doubles the best prior scores. 🧵

Today we're releasing WildDet3D—an open model for monocular 3D object detection in the wild. It works with text, clicks, or 2D boxes, and on zero-shot evals it nearly doubles the best prior scores. 🧵

Ai2

87,538 просмотров • 5 месяцев назад

NEW: Higgs Audio V2 from BosonAI open, unified TTS model w/ voice cloning, beats GPT 4o mini tts and ElevenLabs v2 🔥 > Trained on 10M hours (speech, music, events) > Built on top of Llama 3.2 3B > Works real-time and on edge > Beats GPT-4o-mini-tts, ElevenLabs v2 in prosody & emotion Multi-speaker dialog > Zero-shot voice cloning 🤩 > Available on Hugging Face Kudos to folks at Boson AI for releasing such a brilliant work and all the details around the model! 🤗

NEW: Higgs Audio V2 from BosonAI open, unified TTS model w/ voice cloning, beats GPT 4o mini tts and ElevenLabs v2 🔥 > Trained on 10M hours (speech, music, events) > Built on top of Llama 3.2 3B > Works real-time and on edge > Beats GPT-4o-mini-tts, ElevenLabs v2 in prosody & emotion Multi-speaker dialog > Zero-shot voice cloning 🤩 > Available on Hugging Face Kudos to folks at Boson AI for releasing such a brilliant work and all the details around the model! 🤗

Vaibhav (VB) Srivastav

79,645 просмотров • 1 год назад

Today we released Meta Spirit LM — our first open source multimodal language model that freely mixes text and speech. Many existing AI voice experiences today use ASR to techniques to process speech before synthesizing with an LLM to generate text — but these approaches compromise the expressive aspects of speech. Using phonetic, pitch and tone tokens, Spirit LM models can overcome these limitations for both inputs and outputs to generate more natural sounding speech while also learning new tasks across ASR, TTS and speech classification. We hope that sharing this work will enable the research community to further new approaches for text and speech integration.

Today we released Meta Spirit LM — our first open source multimodal language model that freely mixes text and speech. Many existing AI voice experiences today use ASR to techniques to process speech before synthesizing with an LLM to generate text — but these approaches compromise the expressive aspects of speech. Using phonetic, pitch and tone tokens, Spirit LM models can overcome these limitations for both inputs and outputs to generate more natural sounding speech while also learning new tasks across ASR, TTS and speech classification. We hope that sharing this work will enable the research community to further new approaches for text and speech integration.

AI at Meta

351,764 просмотров • 1 год назад

Open-source project Soundstorm (AI generated speech from Google Research) is going to give Elevenlabs a run for it's money: The text-to-speech project specializes in dialogue between multiple parties, and is available on Github:

Open-source project Soundstorm (AI generated speech from Google Research) is going to give Elevenlabs a run for it's money: The text-to-speech project specializes in dialogue between multiple parties, and is available on Github:

AI Breakfast

326,486 просмотров • 3 лет назад

Say It — private, local text-to-speech for macOS Open models 100% on-device, speak text from any app with a hotkey, and even clone your own voice. Infinite voices. Free & open-source. Listen to the video 🔊

Say It — private, local text-to-speech for macOS Open models 100% on-device, speak text from any app with a hotkey, and even clone your own voice. Infinite voices. Free & open-source. Listen to the video 🔊

calle

69,707 просмотров • 1 месяц назад

Voice AI without the wait! ⏱️ Thanks to Hugging Face and Cerebras, developers can now use the Gemma 4 31B model as the brain for voice AI at ultra-fast inference speeds. Add it to a fully open-source, cascaded speech-to-speech stack that can be used to power existing voice apps! 🗣️

Voice AI without the wait! ⏱️ Thanks to Hugging Face and Cerebras, developers can now use the Gemma 4 31B model as the brain for voice AI at ultra-fast inference speeds. Add it to a fully open-source, cascaded speech-to-speech stack that can be used to power existing voice apps! 🗣️

Google Gemma

161,360 просмотров • 2 месяцев назад

Eleven v3 (alpha) is the most expressive Text to Speech model. v3 introduces: • Multi-speaker dialogue with contextual awareness • Support for 70+ languages, up from 33 in v2 • Audio tags such as [excited], [sighs], [laughing], and [whispers]

Eleven v3 (alpha) is the most expressive Text to Speech model. v3 introduces: • Multi-speaker dialogue with contextual awareness • Support for 70+ languages, up from 33 in v2 • Audio tags such as [excited], [sighs], [laughing], and [whispers]

ElevenLabs

39,424 просмотров • 1 год назад

Introducing Indic Parler-TTS: Open-Source Text-to-Speech for Over a Billion Indic Speakers! 🌏 In collaboration with Hugging Face, we are excited to release Indic Parler-TTS, a state-of-the-art open-source text-to-speech system designed to bring accessible and high-quality speech technology to India’s diverse linguistic community. Supporting 20 of the 22 scheduled Indian languages—and English in various accents (US, British, Indian)—it’s built to serve over a billion speakers and empower companies, developers, researchers, and communities. Why Indic Parler-TTS Stands Out: 1. Open and Accessible: Fully open-source with permissive licensing for unrestricted usage. 2. Wide Language Support: Includes a vast range of Indic languages, with rich diversity in voices. 3. High-Quality Audio: Produces natural, clear, and lifelike speech. 4. Adaptable and Fine-Tunable: Customize it to new languages, accents, or specific applications. 5. State-of-the-Art Performance: Proven through rigorous evaluation. 6. Versatile and Inclusive: Indic Parler-TTS offers 69 unique voices across 18 Indian languages, making it a perfect fit for diverse use cases like audiobooks, virtual assistants, and educational tools. Let's democratize speech technology together and make speech technology more inclusive and accessible for everyone. ▶️ Experience it now: Demo: Model page:

Introducing Indic Parler-TTS: Open-Source Text-to-Speech for Over a Billion Indic Speakers! 🌏 In collaboration with Hugging Face, we are excited to release Indic Parler-TTS, a state-of-the-art open-source text-to-speech system designed to bring accessible and high-quality speech technology to India’s diverse linguistic community. Supporting 20 of the 22 scheduled Indian languages—and English in various accents (US, British, Indian)—it’s built to serve over a billion speakers and empower companies, developers, researchers, and communities. Why Indic Parler-TTS Stands Out: 1. Open and Accessible: Fully open-source with permissive licensing for unrestricted usage. 2. Wide Language Support: Includes a vast range of Indic languages, with rich diversity in voices. 3. High-Quality Audio: Produces natural, clear, and lifelike speech. 4. Adaptable and Fine-Tunable: Customize it to new languages, accents, or specific applications. 5. State-of-the-Art Performance: Proven through rigorous evaluation. 6. Versatile and Inclusive: Indic Parler-TTS offers 69 unique voices across 18 Indian languages, making it a perfect fit for diverse use cases like audiobooks, virtual assistants, and educational tools. Let's democratize speech technology together and make speech technology more inclusive and accessible for everyone. ▶️ Experience it now: Demo: Model page:

AI4Bharat

28,681 просмотров • 1 год назад

Huge drop from Tencent! AuK - turns speech gen/editing into a text-prompt playground. You can control literally anything with natural language. - zero-shot voice cloning, no reference transcript - swap an accent, inject anger, fix background noise, or change a single spoken word in a track. - full TTS, vocal separation - studio-grade cleanup The scale behind this is wild: trained on ~2 million hours of data. - Ridiculously low error rates - 2.65% uses Qwen3-Omni/ASR/ForcedAligner. also AuK-Flash for lightning-fast inference. for dubbing, podcasting, game audio... weights are fully open-source!

Huge drop from Tencent! AuK - turns speech gen/editing into a text-prompt playground. You can control literally anything with natural language. - zero-shot voice cloning, no reference transcript - swap an accent, inject anger, fix background noise, or change a single spoken word in a track. - full TTS, vocal separation - studio-grade cleanup The scale behind this is wild: trained on ~2 million hours of data. - Ridiculously low error rates - 2.65% uses Qwen3-Omni/ASR/ForcedAligner. also AuK-Flash for lightning-fast inference. for dubbing, podcasting, game audio... weights are fully open-source!

Wildminder

17,510 просмотров • 10 дней назад

Most people should probably update their priors on the state of open-source speech-to-speech. It's honestly kind of mind-blowing. We teamed up with Cerebras to build a fully open-source realtime voice demo (models + code) to show what's possible today. Demo : Blog: Go test it, fork it, tweak it, and impress your friends. video is raw, no cut, no speed-up, first take

Most people should probably update their priors on the state of open-source speech-to-speech. It's honestly kind of mind-blowing. We teamed up with Cerebras to build a fully open-source realtime voice demo (models + code) to show what's possible today. Demo : Blog: Go test it, fork it, tweak it, and impress your friends. video is raw, no cut, no speed-up, first take

Thomas Wolf

251,786 просмотров • 2 месяцев назад

4 years ago we were on the brink of AI becoming proprietary and centralized, when OpenAI kept GPT3 closed and VCs started dumping money on researchers. From fully open science, to fully closed, in a matter of months. It was scary, and 1,000+ leading researchers and scientists banded together to show the world that it was possible to do the same work in the open, and build an ecosystem that benefits everyone. That was the BigScience Research Workshop BLOOM project, and it put us back on track to open science, starting with forward-thinking organizations like Meta releasing OPT. Look at us now. Open models have not only caught up, they're state of the art now. Not just LLMs, but models for document AI, speech to text, text to speech, generating images and more. We're closing in on 2 million open weight models on Hugging Face. Thanks for the reminder Thomas Wolf .

4 years ago we were on the brink of AI becoming proprietary and centralized, when OpenAI kept GPT3 closed and VCs started dumping money on researchers. From fully open science, to fully closed, in a matter of months. It was scary, and 1,000+ leading researchers and scientists banded together to show the world that it was possible to do the same work in the open, and build an ecosystem that benefits everyone. That was the BigScience Research Workshop BLOOM project, and it put us back on track to open science, starting with forward-thinking organizations like Meta releasing OPT. Look at us now. Open models have not only caught up, they're state of the art now. Not just LLMs, but models for document AI, speech to text, text to speech, generating images and more. We're closing in on 2 million open weight models on Hugging Face. Thanks for the reminder Thomas Wolf .

Jeff Boudier 🤗

21,947 просмотров • 1 год назад

Researchers at Meta recently shared MAGNeT, a single non-autoregressive transformer model for text-to-music & text-to-sound generation capable of generating audio on-par with the quality of SOTA models — at 7x the speed. MAGNeT is open source as part of AudioCraft. Hear audio samples and get more details on this work ➡️

Researchers at Meta recently shared MAGNeT, a single non-autoregressive transformer model for text-to-music & text-to-sound generation capable of generating audio on-par with the quality of SOTA models — at 7x the speed. MAGNeT is open source as part of AudioCraft. Hear audio samples and get more details on this work ➡️

AI at Meta

122,001 просмотров • 2 лет назад

Smart Turn v2: open source, native audio turn detection in 14 languages. New checkpoint of the open source, open data, open training code, semantic VAD model on Hugging Face, fal, and Pipecat AI. - 3x faster inference (12ms on an L40) - 14 languages (13 more than v1, which was english-only) - New synthetic data set `chirp_3_all` with ~163k audio samples - 99% accuracy on held out `human_5_all` test data Good turn detection is critical for voice agents. This model "understands" both semantic and audio patterns, and mitigates the voice AI trade-off between unwanted turn latency vs the agent interrupting people before they are finished speaking. Training scripts for both Modal and local training are in the repo. We want to make it as easy as possible to contribute to or customize this model! Here's a demo running the smart-turn model with default settings, aimed at generally hitting 400ms total turn detection time. You can tune things to be faster, too. You can help by contributing data, doing architecture expermints, or cleaning open source data! Keep reading ...

Smart Turn v2: open source, native audio turn detection in 14 languages. New checkpoint of the open source, open data, open training code, semantic VAD model on Hugging Face, fal, and Pipecat AI. - 3x faster inference (12ms on an L40) - 14 languages (13 more than v1, which was english-only) - New synthetic data set `chirp_3_all` with ~163k audio samples - 99% accuracy on held out `human_5_all` test data Good turn detection is critical for voice agents. This model "understands" both semantic and audio patterns, and mitigates the voice AI trade-off between unwanted turn latency vs the agent interrupting people before they are finished speaking. Training scripts for both Modal and local training are in the repo. We want to make it as easy as possible to contribute to or customize this model! Here's a demo running the smart-turn model with default settings, aimed at generally hitting 400ms total turn detection time. You can tune things to be faster, too. You can help by contributing data, doing architecture expermints, or cleaning open source data! Keep reading ...

kwindla

42,246 просмотров • 1 год назад

This is the best open source text-to-speech I’ve ever heard. It’s free and you can run it locally. It’s also unrestricted as you can give it any voice sample. In this video I have it speak as Peter Griffin, Trump, Jarvis, & Scarjo’s voice from HER. More in thread 👇

This is the best open source text-to-speech I’ve ever heard. It’s free and you can run it locally. It’s also unrestricted as you can give it any voice sample. In this video I have it speak as Peter Griffin, Trump, Jarvis, & Scarjo’s voice from HER. More in thread 👇

Blaine Brown

343,292 просмотров • 1 год назад

I just built an open NotebookLM clone! Here's what it can do for you: - Process multi-modal data - Scrape websites and YouTube videos - Create a unified knowledge base - Lets you do RAG over it - Remember every conversation - Generate a podcast 🎙️ The idea here is not to reinvent the wheel but to understand how one of the most powerful tools for learning and research actually works, by building it step-by-step! So by the end of this video, you'll learn how to: ↳ Process multimodal data (text, audio, video, URLs, and YouTube videos) into a format ready for LLMs ↳ Store everything in a vector database for fast retrieval ↳ Add a memory layer that remembers conversations and preferences for a personalized experience ↳ Chat with your knowledge base or generate podcasts using a fully open-source, locally running text-to-speech model The podcast generation feature is my favorite part! There's something powerful about turning written content into conversational audio that you can listen to while doing something else. The entire code is 100% open-source. I've shared a link in the replies! ____ Don't forget to drop a like if you enjoy my videos. It shows me I should be making more content like this. Cheers! :)

I just built an open NotebookLM clone! Here's what it can do for you: - Process multi-modal data - Scrape websites and YouTube videos - Create a unified knowledge base - Lets you do RAG over it - Remember every conversation - Generate a podcast 🎙️ The idea here is not to reinvent the wheel but to understand how one of the most powerful tools for learning and research actually works, by building it step-by-step! So by the end of this video, you'll learn how to: ↳ Process multimodal data (text, audio, video, URLs, and YouTube videos) into a format ready for LLMs ↳ Store everything in a vector database for fast retrieval ↳ Add a memory layer that remembers conversations and preferences for a personalized experience ↳ Chat with your knowledge base or generate podcasts using a fully open-source, locally running text-to-speech model The podcast generation feature is my favorite part! There's something powerful about turning written content into conversational audio that you can listen to while doing something else. The entire code is 100% open-source. I've shared a link in the replies! ____ Don't forget to drop a like if you enjoy my videos. It shows me I should be making more content like this. Cheers! :)

Akshay 🚀

83,187 просмотров • 11 месяцев назад