Andi Marafioti's banner

Andi Marafioti

@andimarafioti • 9,040 subscribers

leading multimodal research @huggingface (prev @unity)

Shorts

ITS HAPPENING!

ITS HAPPENING!

290,924 görüntüleme

Can a VLM see without a vision encoder? We trained one for $100, inspired by Gemma 4 12B. Latency on an M3 Pro MacBook: 112 ms -> 1.1 ms for the image path 30% lower end-to-end image+LLM The architecture is just: patchify the image -> linear projection with pos embeddings -> LLM Writeup:

Can a VLM see without a vision encoder? We trained one for $100, inspired by Gemma 4 12B. Latency on an M3 Pro MacBook: 112 ms -> 1.1 ms for the image path 30% lower end-to-end image+LLM The architecture is just: patchify the image -> linear projection with pos embeddings -> LLM Writeup:

60,664 görüntüleme

ok who put Hugging Face in Rick and Morty

ok who put Hugging Face in Rick and Morty

47,997 görüntüleme

Microducks will take a few months to arrive, but we have reachy minis in stock! Try him out here:

Microducks will take a few months to arrive, but we have reachy minis in stock! Try him out here:

11,352 görüntüleme

Videos

Anya Rossi

sweetdream.ai

SweetDream.ai•Sponsored•Livecam

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

Exclusive private shows

1.2k viewers online

Private Show

Join now for exclusive access

Free preview available • Premium content

Can you explain to me exactly what you want 4 microducks for??

Can you explain to me exactly what you want 4 microducks for??

288,506 görüntüleme • 1 ay önce

Voice agents still don’t understand who’s speaking to them. That’s a huge gap compared with humans, hidden by all the “phone-call” demos. But that changes today! NVIDIA is open-sourcing Nemotron 3 Diarization: a model that can reliably track speakers in live conversations, under a commercial-friendly license! In my tests, the quality is really good with one-second speech chunks. So we can use it for voice agents! I tested it with Reachy Mini and speech-to-speech running on a DGX Spark. It’s super fun to see the robot notice a new voice, ask for a name, and remember it. The model has day-zero integration with Transformers! Kudos to the NVIDIA team for shipping useful tools for the whole community!

Voice agents still don’t understand who’s speaking to them. That’s a huge gap compared with humans, hidden by all the “phone-call” demos. But that changes today! NVIDIA is open-sourcing Nemotron 3 Diarization: a model that can reliably track speakers in live conversations, under a commercial-friendly license! In my tests, the quality is really good with one-second speech chunks. So we can use it for voice agents! I tested it with Reachy Mini and speech-to-speech running on a DGX Spark. It’s super fun to see the robot notice a new voice, ask for a name, and remember it. The model has day-zero integration with Transformers! Kudos to the NVIDIA team for shipping useful tools for the whole community!

78,540 görüntüleme • 26 gün önce

Speech-to-speech no longer needs speech-to-text! Until now, our stack was VAD -> STT -> LLM -> TTS. Now it can send audio directly to multimodal LLMs: VAD → MLLM → TTS No STT. The model understands your voice. Now go build better voice agents!

Speech-to-speech no longer needs speech-to-text! Until now, our stack was VAD -> STT -> LLM -> TTS. Now it can send audio directly to multimodal LLMs: VAD → MLLM → TTS No STT. The model understands your voice. Now go build better voice agents!

122,326 görüntüleme • 2 ay önce

Introducing local Reachy Mini conversations: free chats forever! So fast that we had to hardcode delays to stop it from interrupting you mid-sentence. We built an open-source Realtime API powered by llama.cpp: Parakeet -> Gemma 4 E4B -> Qwen3TTS Run it anywhere you run local LMs. Video shows DGX Spark and a 36GB M3 Pro MacBook. Blog:

Introducing local Reachy Mini conversations: free chats forever! So fast that we had to hardcode delays to stop it from interrupting you mid-sentence. We built an open-source Realtime API powered by llama.cpp: Parakeet -> Gemma 4 E4B -> Qwen3TTS Run it anywhere you run local LMs. Video shows DGX Spark and a 36GB M3 Pro MacBook. Blog:

77,716 görüntüleme • 4 ay önce

30x real-time speech-to-text in your browser. No installs. No servers. Just open the website.

30x real-time speech-to-text in your browser. No installs. No servers. Just open the website.

105,127 görüntüleme • 8 ay önce

Introducing Faster Qwen3TTS! Realistic voice generation at 4x real time: - Same amazing voice quality from Qwen's model - Streaming support with <200 ms to first audio - 5x faster than the official implementation Just pip install faster-qwen3-tts Try the demo!

Introducing Faster Qwen3TTS! Realistic voice generation at 4x real time: - Same amazing voice quality from Qwen's model - Streaming support with <200 ms to first audio - 5x faster than the official implementation Just pip install faster-qwen3-tts Try the demo!

92,691 görüntüleme • 7 ay önce

Reachy Mini just got a new Brain! We released a fully open-source backend for talking to Reachy Mini. In the last 48 hours, 3,000+ robots have already hit the version deployed on Hugging Face's infra. Until today, if you wanted a real-time voice agent in your robot, you were looking at $20+ a day. Realtime APIs are expensive because every second the connection is up, you are being charged! But with our approach, you can run the audio models locally and use your OpenAI or Claude subscription for the LLM, at no extra costs. The audio pipeline is very efficient; in the video, it's running on the cheapest Mac mini, 16GB of RAM (the one that sold out a couple of months back when people were buying them up for openclaws setups). Tutorials coming in the next few weeks: how to run your robot fully locally, or with your favorite agent. Stay tuned!

Reachy Mini just got a new Brain! We released a fully open-source backend for talking to Reachy Mini. In the last 48 hours, 3,000+ robots have already hit the version deployed on Hugging Face's infra. Until today, if you wanted a real-time voice agent in your robot, you were looking at $20+ a day. Realtime APIs are expensive because every second the connection is up, you are being charged! But with our approach, you can run the audio models locally and use your OpenAI or Claude subscription for the LLM, at no extra costs. The audio pipeline is very efficient; in the video, it's running on the cheapest Mac mini, 16GB of RAM (the one that sold out a couple of months back when people were buying them up for openclaws setups). Tutorials coming in the next few weeks: how to run your robot fully locally, or with your favorite agent. Stay tuned!

56,158 görüntüleme • 5 ay önce

Mixing vision and robotics is incredibly hard, but when it finally works it feels like magic.

Mixing vision and robotics is incredibly hard, but when it finally works it feels like magic.

92,929 görüntüleme • 11 ay önce

I got NVIDIA’s PersonaPlex running on Reachy Mini and it’s uncanny. Ultra-low latency changes everything for voice agents. BTW, NVIDIA is offering a ~$7k prize (including a DGX Spark!) for the best Reachy Mini community app in the next 15 days.

I got NVIDIA’s PersonaPlex running on Reachy Mini and it’s uncanny. Ultra-low latency changes everything for voice agents. BTW, NVIDIA is offering a ~$7k prize (including a DGX Spark!) for the best Reachy Mini community app in the next 15 days.

55,032 görüntüleme • 8 ay önce

🚀 Today, we are introducing SmolTools! 🚀 Last week, at Hugging Face we made a significant leap forward with the release of SmolLM2, a compact 1.7B language model that sets a new benchmark for performance among models of its size. But beyond the impressive stats, SmolLM2 truly shines in practical, real-world applications, unlocking new possibilities for on-device inference. To demonstrate its potential, we're thrilled to introduce Smol-Tools, a suite of simple yet powerful applications that showcase the capabilities of SmolLM2. 🌟 Today, we present you two key tools: Summarize and Rewrite. 🔹 Summarize: Feed SmolLM2 a text up to 20 pages long, and it will provide you with a concise summary. Need to dig deeper? Just ask follow-up questions to clarify any details – it's that simple. 🔹 Rewrite: Draft a response to a message with your main points, and SmolLM2 will transform it into a clear, polished version that's easy for your recipient to read and understand. These tools are designed to make your workflow smoother and more efficient, leveraging the power of SmolLM2 in practical ways. 💡We hope you give these tools a try and see their potential for yourself! Feel free to build your own SmolTools and contribute them to the project. We'd love to see what you build! Check out the code here:

🚀 Today, we are introducing SmolTools! 🚀 Last week, at Hugging Face we made a significant leap forward with the release of SmolLM2, a compact 1.7B language model that sets a new benchmark for performance among models of its size. But beyond the impressive stats, SmolLM2 truly shines in practical, real-world applications, unlocking new possibilities for on-device inference. To demonstrate its potential, we're thrilled to introduce Smol-Tools, a suite of simple yet powerful applications that showcase the capabilities of SmolLM2. 🌟 Today, we present you two key tools: Summarize and Rewrite. 🔹 Summarize: Feed SmolLM2 a text up to 20 pages long, and it will provide you with a concise summary. Need to dig deeper? Just ask follow-up questions to clarify any details – it's that simple. 🔹 Rewrite: Draft a response to a message with your main points, and SmolLM2 will transform it into a clear, polished version that's easy for your recipient to read and understand. These tools are designed to make your workflow smoother and more efficient, leveraging the power of SmolLM2 in practical ways. 💡We hope you give these tools a try and see their potential for yourself! Feel free to build your own SmolTools and contribute them to the project. We'd love to see what you build! Check out the code here:

100,813 görüntüleme • 1 yıl önce

Sneak peek of Reachy Mini's conversation capabilities! He can chat fluidly and change languages naturally. You can interrupt him and he can see and react to his environment. They start shipping soon, I'm excited to see what people start building with him!

Sneak peek of Reachy Mini's conversation capabilities! He can chat fluidly and change languages naturally. You can interrupt him and he can see and react to his environment. They start shipping soon, I'm excited to see what people start building with him!

54,534 görüntüleme • 11 ay önce

You can now train SOTA models without any storage!🌩️ We completely revamped the Hub’s backend to enable streaming at scale. We streamed TBs of data to 100s of H100s to train SOTA VLMs and saw serious speed-ups. But how?

You can now train SOTA models without any storage!🌩️ We completely revamped the Hub’s backend to enable streaming at scale. We streamed TBs of data to 100s of H100s to train SOTA VLMs and saw serious speed-ups. But how?

49,971 görüntüleme • 11 ay önce

🚨 New paper out! “FineVision: Open Data Is All You Need” 🥳 We unified 200+ data sources into 24M samples. That’s 17.3M images and 9.5B answer tokens, the largest open VLM dataset ever released. All fully documented, reproducible, and available for everyone. And there's more! 🎢

🚨 New paper out! “FineVision: Open Data Is All You Need” 🥳 We unified 200+ data sources into 24M samples. That’s 17.3M images and 9.5B answer tokens, the largest open VLM dataset ever released. All fully documented, reproducible, and available for everyone. And there's more! 🎢

46,507 görüntüleme • 1 yıl önce

The Reachy mini desktop app is so aesthetically pleasing. Huge fan.

The Reachy mini desktop app is so aesthetically pleasing. Huge fan.

34,449 görüntüleme • 10 ay önce

This is getting crazy good. Running locally on my laptop with streaming transcriptions without compromising quality.

This is getting crazy good. Running locally on my laptop with streaming transcriptions without compromising quality.

24,568 görüntüleme • 8 ay önce

Daha fazla içerik yok.