Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

How to train a model that actually understands both audio and text like Voxtral from Mistral AI? Here is a quick video walkthrough of the paper.

sophiamyang's profile picture

Sophia Yang, Ph.D.

30,074 subscribers

49,801 Aufrufe • vor 1 Jahr •via X (Twitter)

Gesundheit & WellnessWissenschaft & TechnologieBildung
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

How to train a reasoning model, specifically Magistral from Mistral AI? Here is a quick video walkthrough of the paper.
2:10

How to train a reasoning model, specifically Magistral from Mistral AI? Here is a quick video walkthrough of the paper.

Sophia Yang, Ph.D.

31,752 Aufrufe • vor 1 Jahr

Introducing Voxtral WebGPU: Real-time speech transcription entirely in your browser. This demo runs Voxtral-Mini-4B, a powerful streaming ASR model from Mistral AI, locally on WebGPU. The model supports 13 languages and is capable of <500 ms latency. Fully private. Zero cost.
0:39

Introducing Voxtral WebGPU: Real-time speech transcription entirely in your browser. This demo runs Voxtral-Mini-4B, a powerful streaming ASR model from Mistral AI, locally on WebGPU. The model supports 13 languages and is capable of <500 ms latency. Fully private. Zero cost.

Xenova

94,356 Aufrufe • vor 4 Monaten

This is what happens when you plug LLMs into voice assistants, instead of a decade of handwritten rules. This video dissects Voxtral (a family of OSS speech models) and the foundational work behind it (audio tokenization, semantic/acoustic disaggregation, etc). Thank you Mistral AI for your collaboration and for your detailed technical reports in an increasingly opaque industry! 00:00 Intro 01:03 Modular vs end-to-end speech models 03:30 Speech-to-Text 06:07 Delayed Streams Modeling (DSM) 09:41 Whisper Streaming 10:33 Voxtral Realtime 13:07 Voxtral Text-to-Speech 14:28 Throwback: WaveNet 15:24 Audio tokenization 20:39 The Voxtral Codec 21:49 Back to Voxtral TTS 25:30 Outro
26:36

This is what happens when you plug LLMs into voice assistants, instead of a decade of handwritten rules. This video dissects Voxtral (a family of OSS speech models) and the foundational work behind it (audio tokenization, semantic/acoustic disaggregation, etc). Thank you Mistral AI for your collaboration and for your detailed technical reports in an increasingly opaque industry! 00:00 Intro 01:03 Modular vs end-to-end speech models 03:30 Speech-to-Text 06:07 Delayed Streams Modeling (DSM) 09:41 Whisper Streaming 10:33 Voxtral Realtime 13:07 Voxtral Text-to-Speech 14:28 Throwback: WaveNet 15:24 Audio tokenization 20:39 The Voxtral Codec 21:49 Back to Voxtral TTS 25:30 Outro

Julia Turc

296,612 Aufrufe • vor 26 Tagen

What a week for open audio models! 🕺 💃 I demo: 🗣️ Voxtral 4B TTS from Mistral AI 🎙️ Transcribe 2B from Cohere 🏭 and how to run a batch transcribe job in 1 line of CLI using Daniel van Strien uv script links below
4:39

What a week for open audio models! 🕺 💃 I demo: 🗣️ Voxtral 4B TTS from Mistral AI 🎙️ Transcribe 2B from Cohere 🏭 and how to run a batch transcribe job in 1 line of CLI using Daniel van Strien uv script links below

Jeff Boudier 🤗

32,918 Aufrufe • vor 3 Monaten

A quick walkthrough of Terminal Mode on Even G2 to show what it actually looks like in use and how easy it is to setup. #DevTools #BuildInPublic #vibecoding
1:43

A quick walkthrough of Terminal Mode on Even G2 to show what it actually looks like in use and how easy it is to setup. #DevTools #BuildInPublic #vibecoding

Even Realities

14,666 Aufrufe • vor 2 Monaten

Ok this new Indy AI from Contra actually feels like a humongous deal... so I made a quick ~1 min walkthrough 👀
1:14

Ok this new Indy AI from Contra actually feels like a humongous deal... so I made a quick ~1 min walkthrough 👀

Ridd 🤿

16,692 Aufrufe • vor 11 Monaten

1. Meta’s open-sourced multisensory model Meta is back (again!) with yet another exciting open-source project. Introducing ImageBind, a new AI research model that understands and combines text, audio, visual, movement, thermal, AND depth data.
1:04

1. Meta’s open-sourced multisensory model Meta is back (again!) with yet another exciting open-source project. Introducing ImageBind, a new AI research model that understands and combines text, audio, visual, movement, thermal, AND depth data.

Rowan Cheung

173,984 Aufrufe • vor 3 Jahren

Ovi is out on Hugging Face Twin Backbone Cross-Modal Fusion for Audio-Video Generation Ovi is a veo-3 like, video+audio generation model that simultaneously generates both video and audio content from text or text+image inputs. Video+Audio Generation: Generate synchronized video and audio content simultaneously Flexible Input: Supports text-only or text+image conditioning 5-second Videos: Generates 5-second videos at 24 FPS, area of 720×720, at various aspect ratios (9:16, 16:9, 1:1, etc)
2:08

Ovi is out on Hugging Face Twin Backbone Cross-Modal Fusion for Audio-Video Generation Ovi is a veo-3 like, video+audio generation model that simultaneously generates both video and audio content from text or text+image inputs. Video+Audio Generation: Generate synchronized video and audio content simultaneously Flexible Input: Supports text-only or text+image conditioning 5-second Videos: Generates 5-second videos at 24 FPS, area of 720×720, at various aspect ratios (9:16, 16:9, 1:1, etc)

AK

23,082 Aufrufe • vor 9 Monaten

LFM2-Audio just dropped! It's a 1.5B model that understands and generates both text and audio Inference 10x faster + quality on par with models 10x larger Available today on Hugging Face and our playground 🥳
2:00

LFM2-Audio just dropped! It's a 1.5B model that understands and generates both text and audio Inference 10x faster + quality on par with models 10x larger Available today on Hugging Face and our playground 🥳

Maxime Labonne

84,952 Aufrufe • vor 9 Monaten

More Adobe Substance Modeler +AI experiments. Using quick, gestural 3d sketching to drive generative AI with much higher levels of creative control than you could ever get from a text prompt alone. The more detailed your 3d model is, the more you can leave the ai to just finish details like skin texture and individual strands of hair.. my model is obviously just a quick sketch😅 #art #ai
0:54

More Adobe Substance Modeler +AI experiments. Using quick, gestural 3d sketching to drive generative AI with much higher levels of creative control than you could ever get from a text prompt alone. The more detailed your 3d model is, the more you can leave the ai to just finish details like skin texture and individual strands of hair.. my model is obviously just a quick sketch😅 #art #ai

Martin Nebelong

56,540 Aufrufe • vor 2 Jahren

The video is a Llama v1 7B model implemented in MLX and running on an M2 Ultra. More here: * Train a Transformer LM or fine-tune with LoRA * Text generation with Mistral * Image generation with Stable Diffusion * Speech recognition with Whisper
1:35

The video is a Llama v1 7B model implemented in MLX and running on an M2 Ultra. More here: * Train a Transformer LM or fine-tune with LoRA * Text generation with Mistral * Image generation with Stable Diffusion * Speech recognition with Whisper

Awni Hannun

66,565 Aufrufe • vor 2 Jahren

OK, this is insane.. Alibaba just dropped 4 new AI models at Apsara 2025, and they’re wild: → a 1 trillion parameter LLM → a vision model that codes from images → an omni-model for text/audio/video → and a new Wan 2.5 preview for video + audio gen more details below:👇
0:28

OK, this is insane.. Alibaba just dropped 4 new AI models at Apsara 2025, and they’re wild: → a 1 trillion parameter LLM → a vision model that codes from images → an omni-model for text/audio/video → and a new Wan 2.5 preview for video + audio gen more details below:👇

Hamza Khalid

26,707 Aufrufe • vor 9 Monaten

AI Text-to-Music has arrived. MusicLM is a model by Google Research that generates high-fidelity music from text descriptions. Basically just enter some text and it will create the music. 🤯 Attached video shows sample text prompts and AI generated music.
1:24

AI Text-to-Music has arrived. MusicLM is a model by Google Research that generates high-fidelity music from text descriptions. Basically just enter some text and it will create the music. 🤯 Attached video shows sample text prompts and AI generated music.

Dave Lee

72,546 Aufrufe • vor 3 Jahren

Mistral just released Voxtral TTS, an open-weight voice model that tries to match ElevenLabs while keeping it free. A 3-billionn param model that fits on a laptop and runs 6X faster than real-time speech. It is split into 3 parts that handle different jobs. - The 3.4B part is the main language brain that understands the text and plans what should be spoken. - The 390M part turns that plan into speech features like rhythm, pronunciation, and tone. - The 300M part converts those speech features into the final compressed audio signal you can play. That split helps because each part only does the job it is best at, which can make the whole system faster and lighter than one giant model doing everything alone. Mistral claims about 90ms to first audio, about 6x faster than real-time speech output, about 3GB RAM after quantization, 9 languages, and voice copying from 5s of audio, even across languages. Its internal tests report 62.8% preference over ElevenLabs Flash on default voices and 69.9% on custom voices, but the bigger point is that companies can keep speech data on their own machines. IMO, enterprise buyers usually care most about control, cost, and latency, So this model should give it great push for voice-based AI apps/products.
1:21

Mistral just released Voxtral TTS, an open-weight voice model that tries to match ElevenLabs while keeping it free. A 3-billionn param model that fits on a laptop and runs 6X faster than real-time speech. It is split into 3 parts that handle different jobs. - The 3.4B part is the main language brain that understands the text and plans what should be spoken. - The 390M part turns that plan into speech features like rhythm, pronunciation, and tone. - The 300M part converts those speech features into the final compressed audio signal you can play. That split helps because each part only does the job it is best at, which can make the whole system faster and lighter than one giant model doing everything alone. Mistral claims about 90ms to first audio, about 6x faster than real-time speech output, about 3GB RAM after quantization, 9 languages, and voice copying from 5s of audio, even across languages. Its internal tests report 62.8% preference over ElevenLabs Flash on default voices and 69.9% on custom voices, but the bigger point is that companies can keep speech data on their own machines. IMO, enterprise buyers usually care most about control, cost, and latency, So this model should give it great push for voice-based AI apps/products.

Rohan Paul

15,672 Aufrufe • vor 3 Monaten

you might've heard about context engineering, but nobody has explained what it really is... that's why I made a full practical walkthrough on how to use it to make your AI outputs personalized and consistent in this walkthrough, I go over: – what context engineering actually is (without the jargon) – how to train AI on your own data and preferences – how to get outputs that sound like you wrote them (not generic AI) reply "context" and I'll DM it to you (must be following)
0:36

you might've heard about context engineering, but nobody has explained what it really is... that's why I made a full practical walkthrough on how to use it to make your AI outputs personalized and consistent in this walkthrough, I go over: – what context engineering actually is (without the jargon) – how to train AI on your own data and preferences – how to get outputs that sound like you wrote them (not generic AI) reply "context" and I'll DM it to you (must be following)

Tyler

20,012 Aufrufe • vor 10 Monaten

Most conversational AI understands words, not people. Introducing Raven-1, our audio and video perception model that gives AI the ability to understand emotion, intent, and context the way humans do.
4:50

Most conversational AI understands words, not people. Introducing Raven-1, our audio and video perception model that gives AI the ability to understand emotion, intent, and context the way humans do.

Tavus

1,672,503 Aufrufe • vor 5 Monaten

New to AIOZ AI V1? Start Here. Ahead of tomorrow's launch of AIOZ AI Challenges, we have created a new demo video to help you get started smoothly! In this video, you’ll get a complete walkthrough of AIOZ AI V1, from signing up and uploading your AI model to topping up and unlocking AI tools & datasets. Learn how to navigate the Web3 collaborative AI marketplace powered by AIOZ DePIN—and get ready to dive into the first challenge tomorrow. Watch the demo here👇
3:44

New to AIOZ AI V1? Start Here. Ahead of tomorrow's launch of AIOZ AI Challenges, we have created a new demo video to help you get started smoothly! In this video, you’ll get a complete walkthrough of AIOZ AI V1, from signing up and uploading your AI model to topping up and unlocking AI tools & datasets. Learn how to navigate the Web3 collaborative AI marketplace powered by AIOZ DePIN—and get ready to dive into the first challenge tomorrow. Watch the demo here👇

AIOZ Network

465,681 Aufrufe • vor 1 Jahr

Introducing SAM Audio: the first unified AI model that allows you to isolate and edit sound from complex audio mixtures. This could mean isolating the guitar in a video of your band, filtering out traffic noises, or removing the sound of a dog barking in your podcast, all with text, visual, and time span prompts.
0:34

Introducing SAM Audio: the first unified AI model that allows you to isolate and edit sound from complex audio mixtures. This could mean isolating the guitar in a video of your band, filtering out traffic noises, or removing the sound of a dog barking in your podcast, all with text, visual, and time span prompts.

Meta Newsroom

204,498 Aufrufe • vor 7 Monaten

To answer all the questions on how this works, the model is actually an image of a plushie that I split into layers and rigged like normal from there in Live2D. Here's a quick speed paint reference to show how the layers were made in CSP using photo manip strats 🧡
1:01

To answer all the questions on how this works, the model is actually an image of a plushie that I split into layers and rigged like normal from there in Live2D. Here's a quick speed paint reference to show how the layers were made in CSP using photo manip strats 🧡

Clover 🍀🐮

95,121 Aufrufe • vor 1 Jahr

Text to Sound Effects is here. Our newest AI Audio model generates sound effects, short instrumental tracks, soundscapes, and a wide variety of character voices, all from a text prompt. Available now for all users. Everyone from content creators, video game developers, to film and television studios, uses sound effects to create rich and immersive content. Now, in addition to AI voiceovers, you can generate all of the sounds you need with just a prompt. Everything you hear in this video was generated by ElevenLabs sound and voice models. In this thread, we shared some additional clips that help show off the range of this new model.
1:08

Text to Sound Effects is here. Our newest AI Audio model generates sound effects, short instrumental tracks, soundscapes, and a wide variety of character voices, all from a text prompt. Available now for all users. Everyone from content creators, video game developers, to film and television studios, uses sound effects to create rich and immersive content. Now, in addition to AI voiceovers, you can generate all of the sounds you need with just a prompt. Everything you hear in this video was generated by ElevenLabs sound and voice models. In this thread, we shared some additional clips that help show off the range of this new model.

ElevenLabs

450,763 Aufrufe • vor 2 Jahren