Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

MARS5 TTS: Open Source Text to Speech with insane prosodic control! 🔥 > Voice cloning with less than 5 seconds of audio > Two stage Auto-Regressive (750M) + Non-Auto Regressive (450M) model architecture > Used BPE tokenizer to enable control over punctuations, pauses, stops etc. > AR model predicts... show more

Vaibhav (VB) Srivastav

38,088 subscribers

162,180 Aufrufe • vor 2 Jahren •via X (Twitter)

Kunst Bildung Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

10 Kommentare

Profilbild von Vaibhav (VB) Srivastav

Vaibhav (VB) Srivastavvor 2 Jahren

Check out the model here:

Profilbild von Vaibhav (VB) Srivastav

Vaibhav (VB) Srivastavvor 2 Jahren

GitHub for more deets:

Profilbild von Carlos DP

Carlos DPvor 2 Jahren

Wow, these outputs are incredible. Like, is this the new SOTA? The samples sound better than the 11labs ones, at least, but idk what params were used

Profilbild von Vaibhav (VB) Srivastav

Vaibhav (VB) Srivastavvor 2 Jahren

750M + 450M -> pretty lightweight overall, in the GitHub README they promise more updates coming soon :D

Profilbild von Furkan Gözükara

Furkan Gözükaravor 2 Jahren

5 seconds to clone is always a lie but i can't say for sure without testing i asked them for gradio demo app to be shared

Profilbild von marko.

marko.vor 2 Jahren

Released under GNU AGPL 3.0, a very curious choice for a model but I'll take it 🎉

Profilbild von Marouane Belkouri

Marouane Belkourivor 2 Jahren

Finnetunning code ?

Profilbild von adivina_soy3

adivina_soy3vor 2 Jahren

@huggingface Impresionante. Crees que seria posible combinarlo con Hallo?

Profilbild von Thomas Hill

Thomas Hillvor 2 Jahren

Nice share 🔥

Profilbild von STEVE blowJOBS

STEVE blowJOBSvor 2 Jahren

This is racist ask me why

Ähnliche Videos

ElevenLabs just lost the crown - to open source. Chatterbox by Resemble AI (I remember writing about this team back in 2022 as a pioneer of AI audio) just released as an open-source alternative for audio generation and voice cloning. - Zero-shot voice cloning from just 5 seconds of audio - Unique emotion intensity control—from subtle to dramatically expressive - Real-time voice synthesis faster than real-time inference - Built-in watermarking for secure, trusted audio - Consistently preferred over ElevenLabs in blind evaluations Fully open-source. No hidden restrictions. This is the way. Check out the demo:

ElevenLabs just lost the crown - to open source. Chatterbox by Resemble AI (I remember writing about this team back in 2022 as a pioneer of AI audio) just released as an open-source alternative for audio generation and voice cloning. - Zero-shot voice cloning from just 5 seconds of audio - Unique emotion intensity control—from subtle to dramatically expressive - Real-time voice synthesis faster than real-time inference - Built-in watermarking for secure, trusted audio - Consistently preferred over ElevenLabs in blind evaluations Fully open-source. No hidden restrictions. This is the way. Check out the demo:

AI Breakfast

454,132 Aufrufe • vor 1 Jahr

LETS GOO! Parler TTS 🔥 A fully open-source, Apache 2.0 licensed Text-to-speech model focused on providing maximum controllability. Through voice prompts, you can control the pitch, speed, gender, noise levels, emotion characteristics and more! > Trained on 10K hours of permissive data. > Offers control over the generations. > Training + Inference code released. > The processed dataset and tagging scripts were released for further research. > English only for now. Next, we're scaling the training to 50K hours and even better dataset processing! Want to help us out? DMs open! 🤗

LETS GOO! Parler TTS 🔥 A fully open-source, Apache 2.0 licensed Text-to-speech model focused on providing maximum controllability. Through voice prompts, you can control the pitch, speed, gender, noise levels, emotion characteristics and more! > Trained on 10K hours of permissive data. > Offers control over the generations. > Training + Inference code released. > The processed dataset and tagging scripts were released for further research. > English only for now. Next, we're scaling the training to 50K hours and even better dataset processing! Want to help us out? DMs open! 🤗

Vaibhav (VB) Srivastav

156,386 Aufrufe • vor 2 Jahren

NVIDIA just removed one of the biggest friction points in Voice AI. PersonaPlex-7B is an open-source, full-duplex conversational model. Free, open source (MIT), with open model weights on Hugging Face 🤗 Links to repo and weights in 🧵↓ The traditional ASR → LLM → TTS pipeline forces rigid turn-taking. It’s efficient, but it never feels natural. PersonaPlex-7B changes that. This NVIDIA model can listen and speak at the same time. It runs directly on continuous audio tokens with a dual-stream transformer, generating text and audio in parallel instead of passing control between components. That unlocks: → instant back-channel responses → interruptions that feel human → real conversational rhythm Persona control is fully zero-shot! If you’re building low-latency assistants or support agents, this is a big step forward 🔥

NVIDIA just removed one of the biggest friction points in Voice AI. PersonaPlex-7B is an open-source, full-duplex conversational model. Free, open source (MIT), with open model weights on Hugging Face 🤗 Links to repo and weights in 🧵↓ The traditional ASR → LLM → TTS pipeline forces rigid turn-taking. It’s efficient, but it never feels natural. PersonaPlex-7B changes that. This NVIDIA model can listen and speak at the same time. It runs directly on continuous audio tokens with a dual-stream transformer, generating text and audio in parallel instead of passing control between components. That unlocks: → instant back-channel responses → interruptions that feel human → real conversational rhythm Persona control is fully zero-shot! If you’re building low-latency assistants or support agents, this is a big step forward 🔥

Charly Wargnier

564,995 Aufrufe • vor 5 Monaten

🚀 VoxCPM 2 is live! 🎉 Another open-source AI #TTS model from China — and one that stands shoulder to shoulder with Qwen3-TTS, while bringing everything into a single unified model. After rapid iterations from V1 (zero-shot cloning) to V1.5 (long-form + fine-tuning), #VoxCPM has consistently pushed quality and usability forward. Now, VoxCPM 2 takes it further: 🔹30+ languages — truly global, truly local. 🔹Infinite voice design — type it, hear it, control it. From a whisper to a booming cinematic voice. 🔹Studio-grade audio — 48kHz ultra-high fidelity with emotional depth 🔹Diffusion-Autoregressive cloning — preserves more acoustic and emotional detail than token-based models like Qwen3-TTS 💡 Big shoutout to Grok — used your multi-image video magic for our launch demo. It’s scarily good at keeping visuals consistent across shots. Elon Elon Musk, this one’s for you. 😉 Check the demo & start cloning your dream voice: 🌐 Hugging Face Space: 🤗 Hugging Face Model: 🤖 ModelScope Model: 💻 GitHub： #TTS #AI #VoiceCloning #GrokImagine #ElonMusk #OpenBMB #VoxCPM

🚀 VoxCPM 2 is live! 🎉 Another open-source AI #TTS model from China — and one that stands shoulder to shoulder with Qwen3-TTS, while bringing everything into a single unified model. After rapid iterations from V1 (zero-shot cloning) to V1.5 (long-form + fine-tuning), #VoxCPM has consistently pushed quality and usability forward. Now, VoxCPM 2 takes it further: 🔹30+ languages — truly global, truly local. 🔹Infinite voice design — type it, hear it, control it. From a whisper to a booming cinematic voice. 🔹Studio-grade audio — 48kHz ultra-high fidelity with emotional depth 🔹Diffusion-Autoregressive cloning — preserves more acoustic and emotional detail than token-based models like Qwen3-TTS 💡 Big shoutout to Grok — used your multi-image video magic for our launch demo. It’s scarily good at keeping visuals consistent across shots. Elon Elon Musk, this one’s for you. 😉 Check the demo & start cloning your dream voice: 🌐 Hugging Face Space: 🤗 Hugging Face Model: 🤖 ModelScope Model: 💻 GitHub： #TTS #AI #VoiceCloning #GrokImagine #ElonMusk #OpenBMB #VoxCPM

OpenBMB

557,835 Aufrufe • vor 3 Monaten

Today we released Meta Spirit LM — our first open source multimodal language model that freely mixes text and speech. Many existing AI voice experiences today use ASR to techniques to process speech before synthesizing with an LLM to generate text — but these approaches compromise the expressive aspects of speech. Using phonetic, pitch and tone tokens, Spirit LM models can overcome these limitations for both inputs and outputs to generate more natural sounding speech while also learning new tasks across ASR, TTS and speech classification. We hope that sharing this work will enable the research community to further new approaches for text and speech integration.

Today we released Meta Spirit LM — our first open source multimodal language model that freely mixes text and speech. Many existing AI voice experiences today use ASR to techniques to process speech before synthesizing with an LLM to generate text — but these approaches compromise the expressive aspects of speech. Using phonetic, pitch and tone tokens, Spirit LM models can overcome these limitations for both inputs and outputs to generate more natural sounding speech while also learning new tasks across ASR, TTS and speech classification. We hope that sharing this work will enable the research community to further new approaches for text and speech integration.

AI at Meta

351,698 Aufrufe • vor 1 Jahr

Text-to-speech is moving way too fast. Just a few days ago, I tweeted about PersonaPlex-7B, NVIDIA's new open source TTS ( And today, Qwen just open-sourced Qwen3-TTS 🤯 It’s a revolutionary text-to-speech model built for control. Not just about generating speech, but about shaping how it sounds directly from language. You can guide the pace, the tone, and the expressiveness straight from text, without touching audio graphs or hand-tuning parameters. That’s the real shift! What makes Qwen3-TTS stand out is how practical it already is: → voice cloning from just a few seconds of audio → voice creation without any reference sample → support for 10 languages out of the box → end-to-end latency down to ~97ms → works in both streaming and non-streaming setups The models come in two sizes (0.6B and 1.7B), so you can trade off quality and hardware cost depending on your setup. You can work with curated voices, designed voices, or cloned ones, and it integrates cleanly with vLLM for production use. It also ships as a simple Python package you can pip install. If you’re building real-time voice systems, this removes a lot of friction! 100% free and open source. I put the repo in the 🧵↓

Text-to-speech is moving way too fast. Just a few days ago, I tweeted about PersonaPlex-7B, NVIDIA's new open source TTS ( And today, Qwen just open-sourced Qwen3-TTS 🤯 It’s a revolutionary text-to-speech model built for control. Not just about generating speech, but about shaping how it sounds directly from language. You can guide the pace, the tone, and the expressiveness straight from text, without touching audio graphs or hand-tuning parameters. That’s the real shift! What makes Qwen3-TTS stand out is how practical it already is: → voice cloning from just a few seconds of audio → voice creation without any reference sample → support for 10 languages out of the box → end-to-end latency down to ~97ms → works in both streaming and non-streaming setups The models come in two sizes (0.6B and 1.7B), so you can trade off quality and hardware cost depending on your setup. You can work with curated voices, designed voices, or cloned ones, and it integrates cleanly with vLLM for production use. It also ships as a simple Python package you can pip install. If you’re building real-time voice systems, this removes a lot of friction! 100% free and open source. I put the repo in the 🧵↓

Charly Wargnier

59,144 Aufrufe • vor 5 Monaten

Introducing Lightning V3 - it beats every model we tested against. ElevenLabs, Cartesia, OpenAI. Lightning sets a new SOTA with V3 in conversational text-to-speech. → Highest MOS score for conversational TTS at 3.9 → ~76% win rate vs gpt-4o-mini-tts on naturalness → 15 languages with mid-sentence code-switching → Built from scratch for voice agents, not read-aloud Every TTS model sounds clean in a demo. You type a sentence and you get beautiful audio. Voice agents don't work that way. They stream. They're generating audio in real-time chunks with half the context missing. That's where everything breaks. A great reading voice and a great conversational voice are fundamentally different things. A conversational voice has to sound like it's thinking - with the pauses, the rhythm shifts, the reactions. It has to handle the way real people actually talk, including switching languages mid-sentence. That's what V3 does. V3.1 also ships voice cloning. 5 to 15 seconds of audio, no fine-tuning, production-grade clone across 15 languages. Blog link in the comments.

Introducing Lightning V3 - it beats every model we tested against. ElevenLabs, Cartesia, OpenAI. Lightning sets a new SOTA with V3 in conversational text-to-speech. → Highest MOS score for conversational TTS at 3.9 → ~76% win rate vs gpt-4o-mini-tts on naturalness → 15 languages with mid-sentence code-switching → Built from scratch for voice agents, not read-aloud Every TTS model sounds clean in a demo. You type a sentence and you get beautiful audio. Voice agents don't work that way. They stream. They're generating audio in real-time chunks with half the context missing. That's where everything breaks. A great reading voice and a great conversational voice are fundamentally different things. A conversational voice has to sound like it's thinking - with the pauses, the rhythm shifts, the reactions. It has to handle the way real people actually talk, including switching languages mid-sentence. That's what V3 does. V3.1 also ships voice cloning. 5 to 15 seconds of audio, no fine-tuning, production-grade clone across 15 languages. Blog link in the comments.

Sudarshan Kamath

71,298 Aufrufe • vor 3 Monaten

Today we’re releasing Chroma 1.0 → the world first open-source, end-to-end, real-time speech-to-speech model → with personalized voice cloning Trained by FlashLabs. Deployed on FlashAI👉 An open research-grade alternative to the OpenAI Realtime model. Voice Test dubbing Elon Musk and Lex Fridman: 🔥What’s real (evals and benchmarks attached): ⚡ 2× faster than real-time ) 📘 SGLang Cookbook: 📄 Paper + benchmarks: 🤗 Models: 💻 Inference code: 🔁 RT if you believe real-time AI should be open 💬 Reply if you’re building on Conversational Voice AI products

Today we’re releasing Chroma 1.0 → the world first open-source, end-to-end, real-time speech-to-speech model → with personalized voice cloning Trained by FlashLabs. Deployed on FlashAI👉 An open research-grade alternative to the OpenAI Realtime model. Voice Test dubbing Elon Musk and Lex Fridman: 🔥What’s real (evals and benchmarks attached): ⚡ 2× faster than real-time ) 📘 SGLang Cookbook: 📄 Paper + benchmarks: 🤗 Models: 💻 Inference code: 🔁 RT if you believe real-time AI should be open 💬 Reply if you’re building on Conversational Voice AI products

FlashLabs

1,248,697 Aufrufe • vor 5 Monaten

Smart Turn v2: open source, native audio turn detection in 14 languages. New checkpoint of the open source, open data, open training code, semantic VAD model on Hugging Face, fal, and Pipecat AI. - 3x faster inference (12ms on an L40) - 14 languages (13 more than v1, which was english-only) - New synthetic data set `chirp_3_all` with ~163k audio samples - 99% accuracy on held out `human_5_all` test data Good turn detection is critical for voice agents. This model "understands" both semantic and audio patterns, and mitigates the voice AI trade-off between unwanted turn latency vs the agent interrupting people before they are finished speaking. Training scripts for both Modal and local training are in the repo. We want to make it as easy as possible to contribute to or customize this model! Here's a demo running the smart-turn model with default settings, aimed at generally hitting 400ms total turn detection time. You can tune things to be faster, too. You can help by contributing data, doing architecture expermints, or cleaning open source data! Keep reading ...

Smart Turn v2: open source, native audio turn detection in 14 languages. New checkpoint of the open source, open data, open training code, semantic VAD model on Hugging Face, fal, and Pipecat AI. - 3x faster inference (12ms on an L40) - 14 languages (13 more than v1, which was english-only) - New synthetic data set `chirp_3_all` with ~163k audio samples - 99% accuracy on held out `human_5_all` test data Good turn detection is critical for voice agents. This model "understands" both semantic and audio patterns, and mitigates the voice AI trade-off between unwanted turn latency vs the agent interrupting people before they are finished speaking. Training scripts for both Modal and local training are in the repo. We want to make it as easy as possible to contribute to or customize this model! Here's a demo running the smart-turn model with default settings, aimed at generally hitting 400ms total turn detection time. You can tune things to be faster, too. You can help by contributing data, doing architecture expermints, or cleaning open source data! Keep reading ...

kwindla

42,219 Aufrufe • vor 1 Jahr

Mistral just released Voxtral TTS, an open-weight voice model that tries to match ElevenLabs while keeping it free. A 3-billionn param model that fits on a laptop and runs 6X faster than real-time speech. It is split into 3 parts that handle different jobs. - The 3.4B part is the main language brain that understands the text and plans what should be spoken. - The 390M part turns that plan into speech features like rhythm, pronunciation, and tone. - The 300M part converts those speech features into the final compressed audio signal you can play. That split helps because each part only does the job it is best at, which can make the whole system faster and lighter than one giant model doing everything alone. Mistral claims about 90ms to first audio, about 6x faster than real-time speech output, about 3GB RAM after quantization, 9 languages, and voice copying from 5s of audio, even across languages. Its internal tests report 62.8% preference over ElevenLabs Flash on default voices and 69.9% on custom voices, but the bigger point is that companies can keep speech data on their own machines. IMO, enterprise buyers usually care most about control, cost, and latency, So this model should give it great push for voice-based AI apps/products.

Mistral just released Voxtral TTS, an open-weight voice model that tries to match ElevenLabs while keeping it free. A 3-billionn param model that fits on a laptop and runs 6X faster than real-time speech. It is split into 3 parts that handle different jobs. - The 3.4B part is the main language brain that understands the text and plans what should be spoken. - The 390M part turns that plan into speech features like rhythm, pronunciation, and tone. - The 300M part converts those speech features into the final compressed audio signal you can play. That split helps because each part only does the job it is best at, which can make the whole system faster and lighter than one giant model doing everything alone. Mistral claims about 90ms to first audio, about 6x faster than real-time speech output, about 3GB RAM after quantization, 9 languages, and voice copying from 5s of audio, even across languages. Its internal tests report 62.8% preference over ElevenLabs Flash on default voices and 69.9% on custom voices, but the bigger point is that companies can keep speech data on their own machines. IMO, enterprise buyers usually care most about control, cost, and latency, So this model should give it great push for voice-based AI apps/products.

Rohan Paul

15,672 Aufrufe • vor 3 Monaten

Molmo by Ai2 - Open source SoTA Multimodal (Vision) Language model, beating Claude 3.5 Sonnet, GPT4V and comparable to GPT4o 🔥 They release four model checkpoints: 1. MolmoE-1B, a mixture of experts model with 1B (active) 7B (total) 2. Molmo-7B-O, most open 7B model 3. Molmo-7B-D, demo model 4. Molmo-72B, best model System Architecture > Input: Multi-scale, multi-crop images generated from the original image. > Vision Encoder: OpenAI's ViT-L/14 336px CLIP model, a powerful ViT, encodes images into vision tokens. > Connector: MLP projects tokens to LLM input space, followed by pooling for dimensionality reduction. > LLM: Decoder-only Transformer, various options (OLMo, OLMoE, Qwen2, Mistral, Gemma2, Phi) with diverse scales and openness. Model Variants > Vision Encoder: Consistent ViT-L/14 CLIP model across variants. > LLM: OLMo-7B-1024, OLMoE-1B-7B-0924, Qwen2 (7B, 72B), Mistral 7B, Gemma2 9B, Phi 3 Medium, offering different capacities and openness levels. Training Strategy > Stage 1: Multimodal pre-training for caption generation with new captioning data. > Stage 2: Supervised fine-tuning on a dataset mixture, updating all parameters. > No RLHF involved, Learning rates adjusted based on component types and pre-training status. > All the weights are available on Hugging Face Hub 🤗 > Compatible with Transformers (Remote Code) Kudos Ai2 for such a brilliant and open work! 🐐 Video credits: Allen AI YT Channel

Molmo by Ai2 - Open source SoTA Multimodal (Vision) Language model, beating Claude 3.5 Sonnet, GPT4V and comparable to GPT4o 🔥 They release four model checkpoints: 1. MolmoE-1B, a mixture of experts model with 1B (active) 7B (total) 2. Molmo-7B-O, most open 7B model 3. Molmo-7B-D, demo model 4. Molmo-72B, best model System Architecture > Input: Multi-scale, multi-crop images generated from the original image. > Vision Encoder: OpenAI's ViT-L/14 336px CLIP model, a powerful ViT, encodes images into vision tokens. > Connector: MLP projects tokens to LLM input space, followed by pooling for dimensionality reduction. > LLM: Decoder-only Transformer, various options (OLMo, OLMoE, Qwen2, Mistral, Gemma2, Phi) with diverse scales and openness. Model Variants > Vision Encoder: Consistent ViT-L/14 CLIP model across variants. > LLM: OLMo-7B-1024, OLMoE-1B-7B-0924, Qwen2 (7B, 72B), Mistral 7B, Gemma2 9B, Phi 3 Medium, offering different capacities and openness levels. Training Strategy > Stage 1: Multimodal pre-training for caption generation with new captioning data. > Stage 2: Supervised fine-tuning on a dataset mixture, updating all parameters. > No RLHF involved, Learning rates adjusted based on component types and pre-training status. > All the weights are available on Hugging Face Hub 🤗 > Compatible with Transformers (Remote Code) Kudos Ai2 for such a brilliant and open work! 🐐 Video credits: Allen AI YT Channel

Vaibhav (VB) Srivastav

80,471 Aufrufe • vor 1 Jahr

VITA Towards Open-Source Interactive Omni Multimodal LLM discuss: The remarkable multimodal capabilities and interactive experience of GPT-4o underscore their necessity in practical applications, yet open-source models rarely excel in both areas. In this paper, we introduce VITA, the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text, and Audio modalities, and meanwhile has an advanced multimodal interactive experience. Starting from Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary followed by bilingual instruction tuning. We further endow the language model with visual and audio capabilities through two-stage multi-task learning of multimodal alignment and instruction tuning. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. Beyond foundational capabilities, we have made considerable progress in enhancing the natural multimodal human-computer interaction experience. To the best of our knowledge, we are the first to exploit non-awakening interaction and audio interrupt in MLLM. VITA is the first step for the open-source community to explore the seamless integration of multimodal understanding and interaction. While there is still lots of work to be done on VITA to get close to close-source counterparts, we hope that its role as a pioneer can serve as a cornerstone for subsequent research.

VITA Towards Open-Source Interactive Omni Multimodal LLM discuss: The remarkable multimodal capabilities and interactive experience of GPT-4o underscore their necessity in practical applications, yet open-source models rarely excel in both areas. In this paper, we introduce VITA, the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text, and Audio modalities, and meanwhile has an advanced multimodal interactive experience. Starting from Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary followed by bilingual instruction tuning. We further endow the language model with visual and audio capabilities through two-stage multi-task learning of multimodal alignment and instruction tuning. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. Beyond foundational capabilities, we have made considerable progress in enhancing the natural multimodal human-computer interaction experience. To the best of our knowledge, we are the first to exploit non-awakening interaction and audio interrupt in MLLM. VITA is the first step for the open-source community to explore the seamless integration of multimodal understanding and interaction. While there is still lots of work to be done on VITA to get close to close-source counterparts, we hope that its role as a pioneer can serve as a cornerstone for subsequent research.

AK

23,958 Aufrufe • vor 1 Jahr

Microsoft did it again! Speech AI models have a major limitation. They slice long recordings into tiny chunks, lose track of who's speaking, and forget all context halfway through. This is exactly what Microsoft's VibeVoice solves. It's an open-source family of frontier voice AI models for both speech recognition and speech generation. Here's what it can do: > VibeVoice-ASR processes up to 60 minutes of audio in a single pass. No chunking. It outputs structured transcriptions with who spoke, when they spoke, and what they said. > You can feed it custom hotwords like names, technical jargon, or domain-specific terms. The model uses them to significantly improve accuracy on specialized content. > VibeVoice-TTS generates up to 90 minutes of multi-speaker speech with up to 4 distinct speakers. Natural turn-taking, emotional expression, all in one pass. > VibeVoice-Realtime is a 0.5B streaming TTS model with ~300ms first-audio latency. Small enough to deploy practically anywhere. All of this is powered by continuous speech tokenizers running at just 7.5 Hz. This ultra-low frame rate preserves audio quality while making long sequences computationally feasible. I have shared the link to the GitHub repo in the replies!

Microsoft did it again! Speech AI models have a major limitation. They slice long recordings into tiny chunks, lose track of who's speaking, and forget all context halfway through. This is exactly what Microsoft's VibeVoice solves. It's an open-source family of frontier voice AI models for both speech recognition and speech generation. Here's what it can do: > VibeVoice-ASR processes up to 60 minutes of audio in a single pass. No chunking. It outputs structured transcriptions with who spoke, when they spoke, and what they said. > You can feed it custom hotwords like names, technical jargon, or domain-specific terms. The model uses them to significantly improve accuracy on specialized content. > VibeVoice-TTS generates up to 90 minutes of multi-speaker speech with up to 4 distinct speakers. Natural turn-taking, emotional expression, all in one pass. > VibeVoice-Realtime is a 0.5B streaming TTS model with ~300ms first-audio latency. Small enough to deploy practically anywhere. All of this is powered by continuous speech tokenizers running at just 7.5 Hz. This ultra-low frame rate preserves audio quality while making long sequences computationally feasible. I have shared the link to the GitHub repo in the replies!

Akshay 🚀

45,206 Aufrufe • vor 3 Monaten

Tencent presents GameGen-O Open-world Video Game Generation We introduce GameGen-O, the first diffusion transformer model tailored for the generation of open-world video games. This model facilitates high-quality, open-domain generation by simulating a wide array of game engine features, such as innovative characters, dynamic environments, complex actions, and diverse events. Additionally, it provides interactive controllability, thus allowing for the gameplay simulation. The development of GameGen-O involves a comprehensive data collection and processing effort from scratch. We collect and build the first Open-World Video Game Dataset (OGameData), amassed extensive data from over a hundred of next-generation open-world games, employing a proprietary data pipeline for efficient sorting, scoring, filtering, and decoupled captioning. This robust and extensive OGameData forms the foundation of our model's training process. GameGen-O undergoes a two-stage training process, consisting of foundation model pretraining and instruction tuning. In the first phase, the model is pre-trained on the OGameData via the text-to-video and video continuation, endowing GameGen-O with the capability for open-domain video game generation. In the second phase, the pre-trained model is frozen, and we fine-tuned using a trainable InstructNet, which enables the production of subsequent frames based on multimodal structural instructions. This whole training process imparts the model with the ability to generate and interactively control content. In summary, GameGen-O represents a notable initial step forward in the realm of open-world video game generation via generative models. It underscores the potential of generative models to serve as an alternative to rendering techniques, which can efficiently combine creative generation with interactive capabilities.

Tencent presents GameGen-O Open-world Video Game Generation We introduce GameGen-O, the first diffusion transformer model tailored for the generation of open-world video games. This model facilitates high-quality, open-domain generation by simulating a wide array of game engine features, such as innovative characters, dynamic environments, complex actions, and diverse events. Additionally, it provides interactive controllability, thus allowing for the gameplay simulation. The development of GameGen-O involves a comprehensive data collection and processing effort from scratch. We collect and build the first Open-World Video Game Dataset (OGameData), amassed extensive data from over a hundred of next-generation open-world games, employing a proprietary data pipeline for efficient sorting, scoring, filtering, and decoupled captioning. This robust and extensive OGameData forms the foundation of our model's training process. GameGen-O undergoes a two-stage training process, consisting of foundation model pretraining and instruction tuning. In the first phase, the model is pre-trained on the OGameData via the text-to-video and video continuation, endowing GameGen-O with the capability for open-domain video game generation. In the second phase, the pre-trained model is frozen, and we fine-tuned using a trainable InstructNet, which enables the production of subsequent frames based on multimodal structural instructions. This whole training process imparts the model with the ability to generate and interactively control content. In summary, GameGen-O represents a notable initial step forward in the realm of open-world video game generation via generative models. It underscores the potential of generative models to serve as an alternative to rendering techniques, which can efficiently combine creative generation with interactive capabilities.

AK

367,000 Aufrufe • vor 1 Jahr

VoxCPM 2 just dropped by OpenBMB Only 2B-param open-source TTS (Text-to-Speech) model built for production-grade multilingual voice work. Apache-2.0 license, Can run on only 8GB VRAM. • Eliminates the "robotic" feel of traditional TTS, delivering prosody and emotional depth suitable for high-stakes professional environments like filmmaking, gaming, animation, and audiobooks. • 30-language multilingual: no language tag needed, just type in a supported language and generate directly. • Voice design: create a brand-new voice from a text description alone, like age, tone, pace, or emotion. No reference audio required. Describe the desired voice characteristics (gender, age, tone, emotion, pace …) in Control Instruction, and VoxCPM2 will craft a unique voice from your description alone. • Controllable cloning: clone from a short clip, then steer delivery style without losing the speaker’s core voice. • Ultimate cloning: use reference audio + transcript for continuation-style cloning that keeps the tiny vocal details. • 48kHz output: takes 16kHz reference audio and produces studio-quality speech without an external upsampler. • Real-time ready: around 0.3 RTF on RTX 4090, even lower with Nano-VLLM. • Commercial use: Apache-2.0 licensed. Developer-Friendly Infrastructure: - Native Torch Inference: Direct support for PyTorch-based workflows. - Training Flexibility: Supports both full-parameter and LoRA fine-tuning for specific domain adaptation. - Production Readiness: Compatible with voxcpm-nanovllm for large-scale, high-concurrency deployment.

Rohan Paul

13,541 Aufrufe • vor 3 Monaten

AI transcribing 1 hour of audio in less than 30 seconds! Think about that! That's lightning-fast, especially when it can do that with very few mistakes. The team Deepgram released their new Nova 2 model. It's another leap forward: • 36% more accurate than OpenAI Whisper • 5-40x faster than every other alternative • 30% fewer errors than the competitors in real-time transcriptions I like to travel and can't wait for headphones with integrated live transcription and translation. We are getting close! By the way, this is the cheapest transcription service out in the market now (if you discount Whisper, which is open-source.) It costs $0.0043 to transcribe a minute of audio. When you sign up to access their API, you get $200 of credit for free, which will let you transcribe around 45,000 minutes of audio! Here is the link: I wanted to record a quick video showing their live transcription in Spanish and English. You can see it attached here. This post was sponsored by Deepgram.

AI transcribing 1 hour of audio in less than 30 seconds! Think about that! That's lightning-fast, especially when it can do that with very few mistakes. The team Deepgram released their new Nova 2 model. It's another leap forward: • 36% more accurate than OpenAI Whisper • 5-40x faster than every other alternative • 30% fewer errors than the competitors in real-time transcriptions I like to travel and can't wait for headphones with integrated live transcription and translation. We are getting close! By the way, this is the cheapest transcription service out in the market now (if you discount Whisper, which is open-source.) It costs $0.0043 to transcribe a minute of audio. When you sign up to access their API, you get $200 of credit for free, which will let you transcribe around 45,000 minutes of audio! Here is the link: I wanted to record a quick video showing their live transcription in Spanish and English. You can see it attached here. This post was sponsored by Deepgram.

Santiago

354,266 Aufrufe • vor 2 Jahren

Here are 10 AI video editor GitHub repos worth bookmarking: 1. Shotcut Most actively maintained open source video editor in 2026. 14K stars. Cross-platform with AI-assisted features. Just shipped a new release April 30, 2026. 2. Kdenlive The closest open source alternative to Adobe Premiere Pro. Multi-track editing, proxy editing, VST audio, and customizable workspace. Best for professional workflows. 3. OpenShot The easiest entry point for beginners. Drag and drop, 400+ transitions, 3D titles, and AI-assisted trimming. 5,700 stars. 4. Blender Not just 3D. Blender's video sequence editor and compositing pipeline is used in professional film production. 18,300 stars. Unmatched for VFX. 5. Recordly Screen recorder with auto-zoom, cursor polish, webcam overlays, and styled frames built in. Built for demo videos and walkthroughs. 6. Wan2.1 Alibaba's open source text-to-video model. Cinema-grade 1080p generation. Apache 2.0. The gold standard for open source video generation in 2026. 7. HunyuanVideo Tencent's 13B parameter open source video model. 11.9K stars. Handles 720p and 1080p with high temporal coherence. 8. CogVideoX Apache 2.0 licensed. Loads natively via Hugging Face Diffusers. Strong prompt following and smooth frame transitions. Needs 16GB VRAM minimum. 12.5K stars. 9. Open-Sora Most starred open source video generation project at 24K stars. Full training pipeline for $200K. Production-level output quality. 10. Mochi 1 Focused entirely on motion quality. The most natural-looking physics of any open source video model. Water, fabric, and human gestures without AI jitter. Apache 2.0.

Here are 10 AI video editor GitHub repos worth bookmarking: 1. Shotcut Most actively maintained open source video editor in 2026. 14K stars. Cross-platform with AI-assisted features. Just shipped a new release April 30, 2026. 2. Kdenlive The closest open source alternative to Adobe Premiere Pro. Multi-track editing, proxy editing, VST audio, and customizable workspace. Best for professional workflows. 3. OpenShot The easiest entry point for beginners. Drag and drop, 400+ transitions, 3D titles, and AI-assisted trimming. 5,700 stars. 4. Blender Not just 3D. Blender's video sequence editor and compositing pipeline is used in professional film production. 18,300 stars. Unmatched for VFX. 5. Recordly Screen recorder with auto-zoom, cursor polish, webcam overlays, and styled frames built in. Built for demo videos and walkthroughs. 6. Wan2.1 Alibaba's open source text-to-video model. Cinema-grade 1080p generation. Apache 2.0. The gold standard for open source video generation in 2026. 7. HunyuanVideo Tencent's 13B parameter open source video model. 11.9K stars. Handles 720p and 1080p with high temporal coherence. 8. CogVideoX Apache 2.0 licensed. Loads natively via Hugging Face Diffusers. Strong prompt following and smooth frame transitions. Needs 16GB VRAM minimum. 12.5K stars. 9. Open-Sora Most starred open source video generation project at 24K stars. Full training pipeline for $200K. Production-level output quality. 10. Mochi 1 Focused entirely on motion quality. The most natural-looking physics of any open source video model. Water, fabric, and human gestures without AI jitter. Apache 2.0.

Kanika

17,309 Aufrufe • vor 1 Monat

Today we’re celebrating 10 years of the Meta FAIR lab in Paris by sharing a collection of new models, datasets and some exciting milestones in the impacts of open source — all laddering up to our ongoing work to achieve Advanced Machine Intelligence (AMI). 1️⃣ Meta PARTNR is a framework for human-robot collaboration that builds on our existing work in this space with a new dataset and a large planning model enabling robots to accomplish complex tasks alongside humans. 2️⃣ Meta Audiobox Aesthetics enables the automatic evaluation of audio aesthetics, providing a comprehensive assessment of audio quality across speech, music and sound. 3️⃣ Open Source Machine Translation Benchmark is a carefully crafted collection with the aim of building an unprecedented multilingual machine translation benchmark for the community. 4️⃣ Two new breakthrough studies using AI to further our understanding of language in the brain.

Today we’re celebrating 10 years of the Meta FAIR lab in Paris by sharing a collection of new models, datasets and some exciting milestones in the impacts of open source — all laddering up to our ongoing work to achieve Advanced Machine Intelligence (AMI). 1️⃣ Meta PARTNR is a framework for human-robot collaboration that builds on our existing work in this space with a new dataset and a large planning model enabling robots to accomplish complex tasks alongside humans. 2️⃣ Meta Audiobox Aesthetics enables the automatic evaluation of audio aesthetics, providing a comprehensive assessment of audio quality across speech, music and sound. 3️⃣ Open Source Machine Translation Benchmark is a carefully crafted collection with the aim of building an unprecedented multilingual machine translation benchmark for the community. 4️⃣ Two new breakthrough studies using AI to further our understanding of language in the brain.

AI at Meta

85,774 Aufrufe • vor 1 Jahr

🚀 Today Photoroom is announcing the next generation of AI photo-editing features. 🚀 We’re dropping 5 new AI features, including a text-to-image model, generative fill, and our new-and-improved AI backgrounds, plus launching the world’s first foundation model trained specifically for product photography. 🖼️ AI Images: Use AI to generate high-quality images from scratch. Create anything you can imagine, at the click of a button. 💗 AI Expand: Use generative AI to realistically expand the scene around your image. 🎑 AI Backgrounds: Generate stunningly realistic AI backgrounds for product photos. ⏳ AI Fill: Take total control of your AI images, by customizing an item or area of the picture. 🔄 Auto-rotate: Correct your camera angle with AI for more professional results. 🔎 AI Upscale: Get better quality images by enhancing the resolution of your images. If our 6x feature drop wasn’t enough, we also just closed our series B funding round, to the tune of $43 million 🎤 #Photoroom #generativeAI #AI #productphotography #ecommerce

🚀 Today Photoroom is announcing the next generation of AI photo-editing features. 🚀 We’re dropping 5 new AI features, including a text-to-image model, generative fill, and our new-and-improved AI backgrounds, plus launching the world’s first foundation model trained specifically for product photography. 🖼️ AI Images: Use AI to generate high-quality images from scratch. Create anything you can imagine, at the click of a button. 💗 AI Expand: Use generative AI to realistically expand the scene around your image. 🎑 AI Backgrounds: Generate stunningly realistic AI backgrounds for product photos. ⏳ AI Fill: Take total control of your AI images, by customizing an item or area of the picture. 🔄 Auto-rotate: Correct your camera angle with AI for more professional results. 🔎 AI Upscale: Get better quality images by enhancing the resolution of your images. If our 6x feature drop wasn’t enough, we also just closed our series B funding round, to the tune of $43 million 🎤 #Photoroom #generativeAI #AI #productphotography #ecommerce

Photoroom

22,236 Aufrufe • vor 2 Jahren