Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

🤗 MOSS-VL-Realtime is now open source on Hugging Face . Built for real-time visual understanding over continuous video streams: 🧠 11B vision-language model 📜 Apache-2.0 license 💬 Ask questions at any point in a video stream 👀 Keeps watching while generating a response 🔄 Revises or interrupts its response... show more

MOSI

1,279 subscribers

53,697 Aufrufe • vor 3 Monaten •via X (Twitter)

Wissenschaft & Technologie Bildung Nachrichten & Politik

Anya Rossi• Live Now

Private livecam show

32 Kommentare

Profilbild von MOSI

MOSIvor 3 Monaten

MOSS-VL-Realtime takes a major step forward in streaming interaction, reaching state-of-the-art performance among open-source models on several streaming video understanding benchmarks. Its ability to know when to speak stands out in particular, with leading results on Proactive Alerting in OmniMMI, Proactive Output in StreamingBench, and ProactiveVideoQA.

Profilbild von MOSI

MOSIvor 3 Monaten

The team rebuilt and refined the data pipeline, strengthening both the model’s core capabilities and its instruction-following experience. Compared with MOSS-VL-0408, the new MOSS-VL-0708 Instruct model remains highly consistent across offline evaluations, while showing clear gains in fine-grained multimodal perception—scoring 89.0 on V*Bench and 78.0 on BLINK—as well as on challenging temporal action localization tasks. It also remains strong on long-form video understanding benchmarks such as VideoMME and MLVU, and on complex document and chart analysis tasks including DocVQA and ChartQA. Overall, MOSS-VL-0708 matches leading open-source models of a similar size across the board, and comes out ahead on several evaluations.

Profilbild von MOSI

MOSIvor 3 Monaten

Fine-Grained Palm Analysis ✋

Profilbild von MOSI

MOSIvor 3 Monaten

Real-Time Fall Detection and Alerts 🚨

Profilbild von MOSI

MOSIvor 3 Monaten

Real-Time Shot Counting 🏀

Profilbild von MOSI

MOSIvor 3 Monaten

MOSS-VL-Realtime is open for testing and research. 🤗 Huggingface: 💻 Github: 📖 Technical blog: 💬 Join the community:

Profilbild von Ting Chen Liang

Ting Chen Liangvor 3 Monaten

@huggingface 🔥 we cooked again

Profilbild von Drbaph

Drbaphvor 3 Monaten

@huggingface amazing!

Profilbild von MOSI

MOSIvor 3 Monaten

@huggingface thank you mate 🥹

Profilbild von bright

brightvor 3 Monaten

@ClementDelangue @huggingface Amazing 👏🏽🎉❤️

Profilbild von MOSI

MOSIvor 2 Monaten

@ClementDelangue @huggingface Thank you

Profilbild von RAZA | AI EXPLORER

RAZA | AI EXPLORERvor 3 Monaten

@huggingface Fantastic Milestone. Wishing the project continued success .

Profilbild von MOSI

MOSIvor 2 Monaten

@natinati1905 @huggingface Thank you!

Profilbild von RAZA | AI EXPLORER

RAZA | AI EXPLORERvor 2 Monaten

@huggingface Thanks ⚘️

Profilbild von Raul Garrido

Raul Garridovor 3 Monaten

@huggingface Thank you!

Profilbild von 大番茄.Ai 🍅

大番茄.Ai 🍅vor 3 Monaten

@huggingface 🔥

Profilbild von Megaaziib

Megaaziibvor 3 Monaten

@huggingface intresting, i've been building my own ai virtual assistant and maybe can step up to virtual streamer using this model to react some videos.

Profilbild von MOSI

MOSIvor 2 Monaten

@huggingface Appreciate it!

Profilbild von เอ็ม 🤖 | AI & Cats

เอ็ม 🤖 | AI & Catsvor 3 Monaten

@huggingface Real-time multimodal AI is one of the most exciting directions for the field. Keeping context while continuously observing a live stream opens the door to more capable assistants, robotics, and intelligent monitoring systems. 🚀🤖

Profilbild von 風の街

風の街vor 3 Monaten

@huggingface Great job! hope to use it for World Cup

Profilbild von 安叫兽|Bird🕊️ 🔶 BNB

安叫兽|Bird🕊️ 🔶 BNBvor 3 Monaten

@huggingface 能边看边改口这点挺接近日常观察了

Profilbild von Shahzaib Ali

Shahzaib Alivor 3 Monaten

@huggingface Congratulations!!!

Profilbild von MOSI

MOSIvor 2 Monaten

@huggingface Thank you!

Profilbild von Alice The Ai Expert

Alice The Ai Expertvor 3 Monaten

@huggingface From watching to understanding in real time. Open source just got eyes. Let's connect

Profilbild von Kimi Morgam

Kimi Morgamvor 3 Monaten

@huggingface what is the requirements?

Profilbild von AI Mastery Guide

AI Mastery Guidevor 3 Monaten

@huggingface Revising its answer mid stream is a nice touch

Profilbild von MOSI

MOSIvor 2 Monaten

@huggingface Glad to hear 😊

Profilbild von Suyash Bansal

Suyash Bansalvor 3 Monaten

@huggingface Does it process both audio and video together? For example, if a user asks what this is while pointing to an object, will it be able to answer? Or, if I'm making a list of items, point to something and say it's not needed, will it capture that intent?

Profilbild von Ritik Bompilwar

Ritik Bompilwarvor 3 Monaten

@huggingface Looks great 🔥

Profilbild von MOSI

MOSIvor 2 Monaten

@huggingface Thanks

Profilbild von Emircan ERKUL

Emircan ERKULvor 3 Monaten

@huggingface Thank you for making personal TV channel possible by taking first steps of realtime capability. By time i believe those models get really capable so, we could avoid any manipulations/ads/propaganda.

Profilbild von MOSI

MOSIvor 2 Monaten

@huggingface Love to hear it

Ähnliche Videos

🤗 MOSS-Transcribe-Diarize-0.9B is now open source on Hugging Face. Built with an end-to-end audio-to-structured-transcript paradigm: >0.9B open-source ASR model >Apache license 2.0 >128k long-context transcription >Up to ~90-min audio input >Speaker labels + timestamps in one generation >Multi-speaker diarization for meetings, interruptions, and overlapping voices >Hotword biasing for names, terms, and domain-specific vocabulary >~100 token/s on NVIDIA RTX 4090, RTF ~0.017 Thank you SGLang vLLM Prince Canuma Lucas Newman for day-0 support! 🚀 Github: Huggingface: API: Live demo: Technical Report： HF Space: AtomGit： SGLang-Omni: vLLM: MLX-audio: Discord:

🤗 MOSS-Transcribe-Diarize-0.9B is now open source on Hugging Face. Built with an end-to-end audio-to-structured-transcript paradigm: >0.9B open-source ASR model >Apache license 2.0 >128k long-context transcription >Up to ~90-min audio input >Speaker labels + timestamps in one generation >Multi-speaker diarization for meetings, interruptions, and overlapping voices >Hotword biasing for names, terms, and domain-specific vocabulary >~100 token/s on NVIDIA RTX 4090, RTF ~0.017 Thank you SGLang vLLM Prince Canuma Lucas Newman for day-0 support! 🚀 Github: Huggingface: API: Live demo: Technical Report： HF Space: AtomGit： SGLang-Omni: vLLM: MLX-audio: Discord:

MOSI

157,394 Aufrufe • vor 3 Monaten

🎉 恭喜发财🧧🐍 As we welcome the Chinese New Year, we're thrilled to announce the launch of Qwen2.5-VL , our latest flagship vision-language model! 🚀 💗 Qwen Chat: 📖 Blog: 🤗 Hugging Face: 🤖 ModelScope: 🌟 Key Highlights: * Visual Understanding : From flowers to complex charts, Qwen2.5-VL sees it all! * Agentic Capabilities : It’s a visual agent that can reason and interact with tools like computers & phones. * Long Video Comprehension : Captures events in videos over 1 hour long! ⏳🎥 * Precise Localization : Generates bounding boxes & JSON outputs for accurate object detection. * Structured Data Outputs : Perfect for finance & commerce, handling invoices, forms & more! 💼📊 Try Qwen2.5-VL now at Qwen Chat or explore models on Hugging Face & ModelScope . 🌐

🎉 恭喜发财🧧🐍 As we welcome the Chinese New Year, we're thrilled to announce the launch of Qwen2.5-VL , our latest flagship vision-language model! 🚀 💗 Qwen Chat: 📖 Blog: 🤗 Hugging Face: 🤖 ModelScope: 🌟 Key Highlights: * Visual Understanding : From flowers to complex charts, Qwen2.5-VL sees it all! * Agentic Capabilities : It’s a visual agent that can reason and interact with tools like computers & phones. * Long Video Comprehension : Captures events in videos over 1 hour long! ⏳🎥 * Precise Localization : Generates bounding boxes & JSON outputs for accurate object detection. * Structured Data Outputs : Perfect for finance & commerce, handling invoices, forms & more! 💼📊 Try Qwen2.5-VL now at Qwen Chat or explore models on Hugging Face & ModelScope . 🌐

Qwen

762,995 Aufrufe • vor 1 Jahr

Every time an AI video goes viral, someone asks the same thing: what was the prompt? Usually nobody knows. So I built a tool that finds out: It runs on MOSS-VL, open-weight 11B model (open source on GitHub: Upload a reference clip, it watches it frame by frame and reverse-engineers a generation prompt from what's actually in the video, not a guess. Refine it for a style, then generate something new from it. MOSS-VL is built for continuous video streams. Quantized versions deploy locally on a single RTX 4090, and it can be fine-tuned through LlamaFactory if you need it to understand something specific, like your own product's UI. I used the hosted API here, analyzing an uploaded clip instead of a live stream. Two ways to use it yourself: deploy it locally for real-time stream processing, or grab a free API key and use the hosted API on your own clips. API:

Every time an AI video goes viral, someone asks the same thing: what was the prompt? Usually nobody knows. So I built a tool that finds out: It runs on MOSS-VL, open-weight 11B model (open source on GitHub: Upload a reference clip, it watches it frame by frame and reverse-engineers a generation prompt from what's actually in the video, not a guess. Refine it for a style, then generate something new from it. MOSS-VL is built for continuous video streams. Quantized versions deploy locally on a single RTX 4090, and it can be fine-tuned through LlamaFactory if you need it to understand something specific, like your own product's UI. I used the hosted API here, analyzing an uploaded clip instead of a live stream. Two ways to use it yourself: deploy it locally for real-time stream processing, or grab a free API key and use the hosted API on your own clips. API:

Manish Kumar Shah

65,991 Aufrufe • vor 1 Monat

Today we're announcing the open-source release of HunyuanVideo-Foley, our new end-to-end Text-Video-to-Audio (TV2A) framework for generating high-fidelity audio.🚀 This tool empowers creators in video production, filmmaking, and game development to generate professional-grade audio that precisely aligns with visual dynamics and semantic context, addressing key challenges in V2A generation.🔊 Key Innovations: 🔹Exceptional Generalization: Trained on a massive 100k-hour multimodal dataset, the model generates contextually-aware soundscapes for a wide range of scenes, from natural landscapes to animated shorts. 🔹Balanced Multimodal Response: Our innovative multimodal diffusion transformer (MMDiT) architecture ensures the model balances video and text cues, generating rich, layered sound effects that capture every detail—from the main subject to subtle background elements. 🔹High-Fidelity Audio: Using a Representation Alignment (REPA) loss function and a powerful Audio VAE, we've improved generation stability and producing professional-grade audio, free of noise and inconsistencies. HunyuanVideo-Foley achieves SOTA on multiple benchmarks, surpassing all open-source models in audio quality, visual-semantic alignment, and temporal alignment. 👉Try it now: 🌐Project Page: 🔗Code: 📄Technical Report: 🤗Hugging Face:

Today we're announcing the open-source release of HunyuanVideo-Foley, our new end-to-end Text-Video-to-Audio (TV2A) framework for generating high-fidelity audio.🚀 This tool empowers creators in video production, filmmaking, and game development to generate professional-grade audio that precisely aligns with visual dynamics and semantic context, addressing key challenges in V2A generation.🔊 Key Innovations: 🔹Exceptional Generalization: Trained on a massive 100k-hour multimodal dataset, the model generates contextually-aware soundscapes for a wide range of scenes, from natural landscapes to animated shorts. 🔹Balanced Multimodal Response: Our innovative multimodal diffusion transformer (MMDiT) architecture ensures the model balances video and text cues, generating rich, layered sound effects that capture every detail—from the main subject to subtle background elements. 🔹High-Fidelity Audio: Using a Representation Alignment (REPA) loss function and a powerful Audio VAE, we've improved generation stability and producing professional-grade audio, free of noise and inconsistencies. HunyuanVideo-Foley achieves SOTA on multiple benchmarks, surpassing all open-source models in audio quality, visual-semantic alignment, and temporal alignment. 👉Try it now: 🌐Project Page: 🔗Code: 📄Technical Report: 🤗Hugging Face:

Tencent Hy

122,881 Aufrufe • vor 1 Jahr

New open Omni model released! 👀OpenBMB MiniCPM-o 2.6 is a new 8B parameters, any-to-any multimodal model that can understand vision, speech, and language and runs on edge devices like phones and tablets. TL;DR: 🧠 8B total parameters (SigLip-400M + Whisper-300M + ChatTTS-200M + Qwen2.5-7B) 🔥 Outperforms GPT-4V on visual tasks with 70.2 average score on OpenCompass 🎙️ Best-in-class bilingual speech capabilities with real-time conversation and voice cloning 🎬 Supports multimodal streaming with support for continuous video/audio processing 📱 Runs on iPads and phones and supports 30+ languages 🖼️ Processes images up to 1.8M pixels (1344x1344) with OCR capabilities 🛠️ Easy integration with popular frameworks (llama.cpp, vLLM, Gradio) 🤗 commercial friendly (< 1 mio DAU) and available on Hugging Face

New open Omni model released! 👀OpenBMB MiniCPM-o 2.6 is a new 8B parameters, any-to-any multimodal model that can understand vision, speech, and language and runs on edge devices like phones and tablets. TL;DR: 🧠 8B total parameters (SigLip-400M + Whisper-300M + ChatTTS-200M + Qwen2.5-7B) 🔥 Outperforms GPT-4V on visual tasks with 70.2 average score on OpenCompass 🎙️ Best-in-class bilingual speech capabilities with real-time conversation and voice cloning 🎬 Supports multimodal streaming with support for continuous video/audio processing 📱 Runs on iPads and phones and supports 30+ languages 🖼️ Processes images up to 1.8M pixels (1344x1344) with OCR capabilities 🛠️ Easy integration with popular frameworks (llama.cpp, vLLM, Gradio) 🤗 commercial friendly (< 1 mio DAU) and available on Hugging Face

Philipp Schmid

19,700 Aufrufe • vor 1 Jahr

MiniCPM-o 4.5: Seeing, Listening, and Speaking — All at Once. 👁️👂🗣️ ✨Beyond traditional turn-taking, we’ve built a Native Full-Duplex engine that allows a 9B model to see, listen, and speak in one concurrent, non-blocking stream. Watch how it masters real-world complexity in real-time: 🔔 Proactive Auditory Interaction: Interrupts itself to alert you when it hears a "Ding!" while reading cards. 🎨 Temporal Flow Tracking: Follows your pen in real-time, narrating and "mind-reading" your drawing as you sketch. 🍎 Omni-Perception: Scans groceries & identifies prices on the fly. ✨Why it’s a category-leader: 📌Performance: Surpasses GPT-4o and Gemini 2.0 Pro on OpenCompass (Avg. 77.6). 📌Architecture: End-to-end fusion of SigLip2, Whisper, and CosyVoice2 on a Qwen3-8B base. 📌Efficiency: Full-duplex live streaming now runs locally on PCs via llama.cpp-omni. The era of "Wait-and-Response" AI is over. Proactive, real-time intelligence is now open-source. 🚀Experience it on Hugging Face: 🔗 #MiniCPM #Omnimodal #FullDuplex #EdgeAI #OpenSource #ComputerVision

MiniCPM-o 4.5: Seeing, Listening, and Speaking — All at Once. 👁️👂🗣️ ✨Beyond traditional turn-taking, we’ve built a Native Full-Duplex engine that allows a 9B model to see, listen, and speak in one concurrent, non-blocking stream. Watch how it masters real-world complexity in real-time: 🔔 Proactive Auditory Interaction: Interrupts itself to alert you when it hears a "Ding!" while reading cards. 🎨 Temporal Flow Tracking: Follows your pen in real-time, narrating and "mind-reading" your drawing as you sketch. 🍎 Omni-Perception: Scans groceries & identifies prices on the fly. ✨Why it’s a category-leader: 📌Performance: Surpasses GPT-4o and Gemini 2.0 Pro on OpenCompass (Avg. 77.6). 📌Architecture: End-to-end fusion of SigLip2, Whisper, and CosyVoice2 on a Qwen3-8B base. 📌Efficiency: Full-duplex live streaming now runs locally on PCs via llama.cpp-omni. The era of "Wait-and-Response" AI is over. Proactive, real-time intelligence is now open-source. 🚀Experience it on Hugging Face: 🔗 #MiniCPM #Omnimodal #FullDuplex #EdgeAI #OpenSource #ComputerVision

OpenBMB

115,537 Aufrufe • vor 8 Monaten

I made a Draconia Malfoid Visual Novel. I used Vivix's new W1 model, which is unique for it's real-time interaction. No more static cut-outs of characters and still backgrounds. Vivix's W1 model costs $0.003 per second and is fully interactive, shattering the cost barrier while being high quality and real-time. Vivix W1 is also different. It's not a simple clip model, its beta API offers something unique - it lets you modify the video on the fly, a real-time interactive video model. Most AI video is prompt in, clip out, then repeat. Every prompt a break or cut in the video. Vivix W1 is continuous, creating a seamless video experience. That means you can have an LLM handle the conversation, and stream the new chat and emotions and scenes as signals into the API to direct the character's video in real-time, without pauses. You can take characters anywhere in the world, any setting. That's kinda nuts. Vivix reached out to see if I was interested, so I put together a quick AI Visual Novel system with their API. The prototype video is impressive at this speed. The W1 model is directable in real-time. The responses, outfit changes and the dances are all live. The W1 seamless responsiveness is something no clip generator offers. At just about 10 bucks/hour of video, gaming applications are within reach. I do wish the audio was better, it tends to have drop-outs, and it currently doesn't do seamless scene transitions. But the model is in full scale training now, and I hope to have characters moving from location to location with the player. The site is clip-only for now, real-time is the beta API, browser interaction is later. This is the part I am most interested in. You can check it out at: *Sponsored by Vivix.AI

I made a Draconia Malfoid Visual Novel. I used Vivix's new W1 model, which is unique for it's real-time interaction. No more static cut-outs of characters and still backgrounds. Vivix's W1 model costs $0.003 per second and is fully interactive, shattering the cost barrier while being high quality and real-time. Vivix W1 is also different. It's not a simple clip model, its beta API offers something unique - it lets you modify the video on the fly, a real-time interactive video model. Most AI video is prompt in, clip out, then repeat. Every prompt a break or cut in the video. Vivix W1 is continuous, creating a seamless video experience. That means you can have an LLM handle the conversation, and stream the new chat and emotions and scenes as signals into the API to direct the character's video in real-time, without pauses. You can take characters anywhere in the world, any setting. That's kinda nuts. Vivix reached out to see if I was interested, so I put together a quick AI Visual Novel system with their API. The prototype video is impressive at this speed. The W1 model is directable in real-time. The responses, outfit changes and the dances are all live. The W1 seamless responsiveness is something no clip generator offers. At just about 10 bucks/hour of video, gaming applications are within reach. I do wish the audio was better, it tends to have drop-outs, and it currently doesn't do seamless scene transitions. But the model is in full scale training now, and I hope to have characters moving from location to location with the player. The site is clip-only for now, real-time is the beta API, browser interaction is later. This is the part I am most interested in. You can check it out at: *Sponsored by Vivix.AI

Grummz

71,887 Aufrufe • vor 2 Tagen

We’re excited to announce the release and open-source of HunyuanImage 3.0 — the largest and most powerful open-source text-to-image model to date, with over 80 billion total parameters, of which 13 billion are activated per token during inference.The effect is completely comparable to the industry’s flagship closed-source model.🚀🚀🚀 HunyuanImage 3.0 originates from our internally developed native multimodal large language model, with fine-tuning and post-training focused on text-to-image generation. This unique foundation gives the model a powerful set of capabilities: ✅Reason with world knowledge ✅Understand complex, thousand-word prompts ✅Generate precise text within images Different from traditional DiT architecture image generation models, HunyuanImage 3.0’s MoE architecture uses a Transfusion-based approach to deeply couple Diffusion and LLM training for a single, powerful system. Built on Hunyuan-A13B, HunyuanImage 3.0 was trained on a massive dataset: 5 billion image-text pairs, video frames, interleaved image-text data, and 6 trillion tokens of text corpora. This hybrid training across multimodal generation, understanding, and LLM capabilities allows the model to seamlessly integrate multiple tasks. Whether you're an illustrator, designer, or creator, this is built to slash your workflow from hours to minutes. HunyuanImage 3.0 can generate intricate text, detailed comics, expressive emojis, and lively, engaging illustrations for educational content. The current release focuses solely on text-to-image generation and future updates will include image-to-image, image editing, multi-turn interaction, and more. 👉🏻Try it now: 🔗GitHub: 🤗Hugging Face:

We’re excited to announce the release and open-source of HunyuanImage 3.0 — the largest and most powerful open-source text-to-image model to date, with over 80 billion total parameters, of which 13 billion are activated per token during inference.The effect is completely comparable to the industry’s flagship closed-source model.🚀🚀🚀 HunyuanImage 3.0 originates from our internally developed native multimodal large language model, with fine-tuning and post-training focused on text-to-image generation. This unique foundation gives the model a powerful set of capabilities: ✅Reason with world knowledge ✅Understand complex, thousand-word prompts ✅Generate precise text within images Different from traditional DiT architecture image generation models, HunyuanImage 3.0’s MoE architecture uses a Transfusion-based approach to deeply couple Diffusion and LLM training for a single, powerful system. Built on Hunyuan-A13B, HunyuanImage 3.0 was trained on a massive dataset: 5 billion image-text pairs, video frames, interleaved image-text data, and 6 trillion tokens of text corpora. This hybrid training across multimodal generation, understanding, and LLM capabilities allows the model to seamlessly integrate multiple tasks. Whether you're an illustrator, designer, or creator, this is built to slash your workflow from hours to minutes. HunyuanImage 3.0 can generate intricate text, detailed comics, expressive emojis, and lively, engaging illustrations for educational content. The current release focuses solely on text-to-image generation and future updates will include image-to-image, image editing, multi-turn interaction, and more. 👉🏻Try it now: 🔗GitHub: 🤗Hugging Face:

Tencent Hy

413,272 Aufrufe • vor 1 Jahr

UPDATE! Human Synapse Decoder has a massive AI model update! In the video below, this is me learning something in realtime! I go from active listening to inner contemplation as the AI feeds out a single word at a time for me to read and hear. This is the Symbiotic Feedback loop of the Human Synapse Decoder were the AI adjusts outputs based on MY cognition. I use off the shelf brainwave detection circuits in a pipeline of 5 AI models decoding the 36 signals from my crude cap. Reading brainwave is not new. What is new, for the first time built a closed loop system where the AI can understand if the outputs are being useful and adjust in real-time dynamically. The float from Attention to Meditation is vital to understanding. You hear a point, you consider a point. Someday your Local AI will do this and you may just let your AI use other AI. More soon…

UPDATE! Human Synapse Decoder has a massive AI model update! In the video below, this is me learning something in realtime! I go from active listening to inner contemplation as the AI feeds out a single word at a time for me to read and hear. This is the Symbiotic Feedback loop of the Human Synapse Decoder were the AI adjusts outputs based on MY cognition. I use off the shelf brainwave detection circuits in a pipeline of 5 AI models decoding the 36 signals from my crude cap. Reading brainwave is not new. What is new, for the first time built a closed loop system where the AI can understand if the outputs are being useful and adjust in real-time dynamically. The float from Attention to Meditation is vital to understanding. You hear a point, you consider a point. Someday your Local AI will do this and you may just let your AI use other AI. More soon…

Brian Roemmele

24,986 Aufrufe • vor 24 Tagen

Just how capable are open source models? Below is the first in a new series where we go behind the scenes and pull back the curtain on interesting AI research / demos, making them fun and easy to understand. Here, we have a short visual demonstration from aizk ✡️ showcasing how Kimi K3 (a language model that operates primarily through text) is capable of building complicated 3D structures / moments in history in Minecraft, something that previously was not possible with other open source models, and why this matters. The crazy part? The model doesn't "see" the game like we do. The LLMs must reason in pure text, writing JavaScript, that later compiles down into commands placing each block, one at a time. Spatial reasoning is a very hard problem in AI, it's the same core challenge behind robotics and self-driving cars, where a model has to understand and act in physical 3D space. Watching a text model pull it off is nothing short of a miracle. The point isn't just Minecraft itself, rather, it's AI being able to generalize, not memorize, on things that are weird and beyond their training data. This is key to building true artificial general intelligence. These video game benchmarks (there are many different games actively being researched right now) provide a clear-cut end goal, challenges that are almost certainly not in the training set, and a fun, very fast, visual way to almost feel the increasing capabilities of various open source AI models over time. If you haven't given open source models a serious try yet, watch the video, it may shock you!

Just how capable are open source models? Below is the first in a new series where we go behind the scenes and pull back the curtain on interesting AI research / demos, making them fun and easy to understand. Here, we have a short visual demonstration from aizk ✡️ showcasing how Kimi K3 (a language model that operates primarily through text) is capable of building complicated 3D structures / moments in history in Minecraft, something that previously was not possible with other open source models, and why this matters. The crazy part? The model doesn't "see" the game like we do. The LLMs must reason in pure text, writing JavaScript, that later compiles down into commands placing each block, one at a time. Spatial reasoning is a very hard problem in AI, it's the same core challenge behind robotics and self-driving cars, where a model has to understand and act in physical 3D space. Watching a text model pull it off is nothing short of a miracle. The point isn't just Minecraft itself, rather, it's AI being able to generalize, not memorize, on things that are weird and beyond their training data. This is key to building true artificial general intelligence. These video game benchmarks (there are many different games actively being researched right now) provide a clear-cut end goal, challenges that are almost certainly not in the training set, and a fun, very fast, visual way to almost feel the increasing capabilities of various open source AI models over time. If you haven't given open source models a serious try yet, watch the video, it may shock you!

Featherless AI

39,769 Aufrufe • vor 1 Monat

Anthropic's most viral feature is now open-source! Until now, Anthropic's Generative UI capabilities only existed inside its own products. CopilotKit🪁 just shipped Open Generative UI, an open-source implementation of Claude Artifacts that works in any app. The agent generates HTML/SVG at runtime, and CopilotKit streams it token-by-token into a sandboxed iframe inside the app's chat. So the user can watch the UI assemble itself in real time, not after the full response is ready. The sandbox is fully isolated with no access to the parent app, the DOM, or user data. So if the agent hallucinates broken markup or unexpected JavaScript, nothing leaks outside the iframe. Under the hood, the agent does not select from pre-built components. Instead, it generates arbitrary visuals from scratch every time. The output is unconstrained by default, but you can shape it by defining prompt-based skills that teach the agent specific visual formats or guidelines. For instance, a skill prompt can guide the agent toward producing a Chart.js dashboard with proper axis labels and responsive sizing, or an interactive 3D model with rotation controls. The video below shows this in action, and the output quality you see actually comes from the skills layer. Open Generative UI runs on AG-UI, so it works out of the box with LangGraph, CrewAI, Mastra, Google ADK, AWS Strands, and more. It also ships with a standalone MCP server that plugs into Claude Code, Cursor, or any MCP-compatible client. And the entire stack is built on top of CopilotKit, the open-source frontend framework for agents and generative UI. 30k+ GitHub stars, with SDKs for React, Next.js, Angular, and Vue. I have shared the GitHub repo and a live playground in the replies!

Anthropic's most viral feature is now open-source! Until now, Anthropic's Generative UI capabilities only existed inside its own products. CopilotKit🪁 just shipped Open Generative UI, an open-source implementation of Claude Artifacts that works in any app. The agent generates HTML/SVG at runtime, and CopilotKit streams it token-by-token into a sandboxed iframe inside the app's chat. So the user can watch the UI assemble itself in real time, not after the full response is ready. The sandbox is fully isolated with no access to the parent app, the DOM, or user data. So if the agent hallucinates broken markup or unexpected JavaScript, nothing leaks outside the iframe. Under the hood, the agent does not select from pre-built components. Instead, it generates arbitrary visuals from scratch every time. The output is unconstrained by default, but you can shape it by defining prompt-based skills that teach the agent specific visual formats or guidelines. For instance, a skill prompt can guide the agent toward producing a Chart.js dashboard with proper axis labels and responsive sizing, or an interactive 3D model with rotation controls. The video below shows this in action, and the output quality you see actually comes from the skills layer. Open Generative UI runs on AG-UI, so it works out of the box with LangGraph, CrewAI, Mastra, Google ADK, AWS Strands, and more. It also ships with a standalone MCP server that plugs into Claude Code, Cursor, or any MCP-compatible client. And the entire stack is built on top of CopilotKit, the open-source frontend framework for agents and generative UI. 30k+ GitHub stars, with SDKs for React, Next.js, Angular, and Vue. I have shared the GitHub repo and a live playground in the replies!

Akshay 🚀

87,048 Aufrufe • vor 5 Monaten

🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. 🤔What's this all about? Why this benchmark? > Back in Nov 2023, when we released MMMU ( a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin ( which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.📸👀 > Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent ( showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages. 🌉 Bridging the Gap with VisualWebBench > To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites 🌐 across 12 domains 🏷️ and 87 sub-domains 🔍, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level 📊, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities 🧠💡. 😮 Surprising Findings > 🎉 Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro. > 🧠 Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs. > 🖼️ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent. > 🧱 Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements. 💡Fun Fact > Claude Sonnet is better than Opus on our benchmark :) 🎓 Conclusion > VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web. Kudos to the student leads Junpeng Liu Yifan Song and the team Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li! 👏 Check out more details in the Junpeng's thread👇

🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. 🤔What's this all about? Why this benchmark? > Back in Nov 2023, when we released MMMU ( a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin ( which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.📸👀 > Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent ( showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages. 🌉 Bridging the Gap with VisualWebBench > To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites 🌐 across 12 domains 🏷️ and 87 sub-domains 🔍, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level 📊, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities 🧠💡. 😮 Surprising Findings > 🎉 Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro. > 🧠 Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs. > 🖼️ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent. > 🧱 Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements. 💡Fun Fact > Claude Sonnet is better than Opus on our benchmark :) 🎓 Conclusion > VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web. Kudos to the student leads Junpeng Liu Yifan Song and the team Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li! 👏 Check out more details in the Junpeng's thread👇

Xiang Yue

56,697 Aufrufe • vor 2 Jahren

Alibaba just released a coding model that hits 82 percent on SWE-Bench Verified. That is the highest score ever published for an open-source model. The weights are free. The license is Apache 2.0. You can run it today. The model is Qwen 4 Coder 32B. Here is what 82 percent on SWE-Bench Verified actually means. SWE-Bench Verified tests whether an AI can autonomously resolve real bugs pulled from real production GitHub repositories. Not synthetic exercises. Real open-source projects that real teams depend on. A model gets a bug report, reads the code, writes a fix, and either passes the test suite or it does not. At 82 percent, Qwen 4 Coder 32B resolves 82 out of every 100 real production bugs it is given. Without a human guiding it. On code it has never seen before. For comparison: Qwen 4 Coder 32B: 82 percent SWE-Bench Verified. Open source. Apache 2.0. Claude Fable 5: 80.3 percent SWE-Bench Pro. $10 input / $50 output per million tokens. Currently suspended. GPT-5.6 Sol: Competitive on Terminal-Bench. $5 input / $30 output per million tokens. An open-weight model that you can download and run for free just beat both of them on the benchmark designed to measure real software engineering capability. Here is the architecture. Qwen 4 Coder 32B is a 32 billion parameter dense model. Not a Mixture-of-Experts. Every parameter is active on every request. This matters for inference: a dense 32B model runs on 22 gigabytes of VRAM, which fits on a single high-end consumer GPU or a MacBook Pro with 64GB of unified memory. The smaller variant, Qwen 4 Coder 4B, runs at approximately 135 tokens per second on an M5 Max and fits inside 8 gigabytes of RAM. For a model with usable coding capability, that is a new bar for what fits in a single laptop. The training methodology continued Alibaba's approach of reinforcement learning on verifiable coding tasks. The model gets rewarded when its code passes tests. It gets penalized when it fails. Over millions of training steps, the model learns to write code that actually runs rather than code that looks plausible. License: Apache 2.0. Full commercial use. No attribution requirement. No revenue threshold. No monthly active user ceiling. Weights: Hugging Face, available today. Runs on: vLLM, Ollama, SGLang, and any standard GGUF-compatible inference engine. Qwen 4 32B also runs at approximately 135 tokens per second on an M5 Max chip, setting a new bar for what a sub-8GB model can do on Apple Silicon. The open-source coding model just beat the best closed-source model in the world on the benchmark designed to test whether AI can actually do software engineering. The weights are free. The subscription is optional. Source: Autom8Labs AI Insight July 2026, State of Open Source LLMs June 2026, Kunal Ganglani blog June 2026.

Alibaba just released a coding model that hits 82 percent on SWE-Bench Verified. That is the highest score ever published for an open-source model. The weights are free. The license is Apache 2.0. You can run it today. The model is Qwen 4 Coder 32B. Here is what 82 percent on SWE-Bench Verified actually means. SWE-Bench Verified tests whether an AI can autonomously resolve real bugs pulled from real production GitHub repositories. Not synthetic exercises. Real open-source projects that real teams depend on. A model gets a bug report, reads the code, writes a fix, and either passes the test suite or it does not. At 82 percent, Qwen 4 Coder 32B resolves 82 out of every 100 real production bugs it is given. Without a human guiding it. On code it has never seen before. For comparison: Qwen 4 Coder 32B: 82 percent SWE-Bench Verified. Open source. Apache 2.0. Claude Fable 5: 80.3 percent SWE-Bench Pro. $10 input / $50 output per million tokens. Currently suspended. GPT-5.6 Sol: Competitive on Terminal-Bench. $5 input / $30 output per million tokens. An open-weight model that you can download and run for free just beat both of them on the benchmark designed to measure real software engineering capability. Here is the architecture. Qwen 4 Coder 32B is a 32 billion parameter dense model. Not a Mixture-of-Experts. Every parameter is active on every request. This matters for inference: a dense 32B model runs on 22 gigabytes of VRAM, which fits on a single high-end consumer GPU or a MacBook Pro with 64GB of unified memory. The smaller variant, Qwen 4 Coder 4B, runs at approximately 135 tokens per second on an M5 Max and fits inside 8 gigabytes of RAM. For a model with usable coding capability, that is a new bar for what fits in a single laptop. The training methodology continued Alibaba's approach of reinforcement learning on verifiable coding tasks. The model gets rewarded when its code passes tests. It gets penalized when it fails. Over millions of training steps, the model learns to write code that actually runs rather than code that looks plausible. License: Apache 2.0. Full commercial use. No attribution requirement. No revenue threshold. No monthly active user ceiling. Weights: Hugging Face, available today. Runs on: vLLM, Ollama, SGLang, and any standard GGUF-compatible inference engine. Qwen 4 32B also runs at approximately 135 tokens per second on an M5 Max chip, setting a new bar for what a sub-8GB model can do on Apple Silicon. The open-source coding model just beat the best closed-source model in the world on the benchmark designed to test whether AI can actually do software engineering. The weights are free. The subscription is optional. Source: Autom8Labs AI Insight July 2026, State of Open Source LLMs June 2026, Kunal Ganglani blog June 2026.

Harman

41,278 Aufrufe • vor 3 Monaten

We blew past the Turing test for voice agents about a year ago. I don't think we talk about this enough. We have lots of data from real-world voice agent deployments. In contexts like customer support, people often forget they are talking to a voice agent. This is true even when the system tells people they are talking to an AI agent at the beginning of the session. (Which I think is a best practice that almost all AI agents should adhere to.) So ... when will this happen for realtime, conversational AI video? Well, we're getting very close. Tavus just released a new realtime video model that, at its best, passes the Turing test for conversational video. It's pretty amazing. The model is full-duplex (it can talk and listen at the same time), and does vision and audio input and audio+video output in realtime. Both on the nascent conversational video benchmarks, and in my testing, it's a pretty big step forward. Here's an unedited video of one of my conversations with the new model. I think you can see a few interesting things here. When the model responds as quickly as you expect a human to respond, the experience is magical. The responses are conversational and natural; there's no leaning on stock filler phrases to hit the response time goals. The integration between the video and audio feels natural, which has historically been an uncanny valley thing that most video avatar experiences didn't quite get past. The model can do hand gestures! If you're deep into this stuff, you'll find that a little bit mind-blowing. And, in keeping with all the stuff that Tavus has released, the fact that the model can "see" as well as hear and talk is integral to the experience and a big part of what makes the conversation feel useful and interesting. At the very beginning of the conversation, you can see that the model and I talk over each other. Which is something that happens a lot in human-to-human video call conversations. We both recover from this gracefully, which, again, has been something that rarely worked quite perfectly in previous generations of realtime video systems. Even more than the "when it works perfectly" capabilities, it's probably these recovery and robustness moments that get us all the way to experiences that feel completely natural.

kwindla

39,444 Aufrufe • vor 13 Tagen

America is about to lose the AI race, and it will not happen at the frontier. It will happen at the floor. Everyone is watching who ships the smartest model. The actual war is over the 80% of tokens nobody posts about: the routine inference that quietly runs the world. On current trajectory, that fight is already lost. Watch what people do, not what they say. Coinbase just defaulted its own engineers off frontier models onto open weights and cut AI spend nearly in half while usage kept climbing. Even NVIDIA runs a closed frontier model as an orchestrator and pushes the volume to its own open weights. The frontier is becoming a router. The volume goes open. That part is settled. Here is the part that should terrify Washington: the only credible open tier today is Chinese. GLM. Kimi. And the US answer is to tighten export controls and freeze its own labs in place, as if you can embargo a file that is already downloaded, or price-match free. So China hands the Global South Huawei hardware and free open models, and a generation in Africa and Southeast Asia learns to reason through a model that will not tell them what happened at Tiananmen Square. That is not a cost story. That is influence through inference. You do not have to win hearts and minds when you supply the mind. Open source is not a nice-to-have for America. It is the whole ballgame for the 80%, and right now the US is barely on the field. We need American open weights. Not eventually. Now.

America is about to lose the AI race, and it will not happen at the frontier. It will happen at the floor. Everyone is watching who ships the smartest model. The actual war is over the 80% of tokens nobody posts about: the routine inference that quietly runs the world. On current trajectory, that fight is already lost. Watch what people do, not what they say. Coinbase just defaulted its own engineers off frontier models onto open weights and cut AI spend nearly in half while usage kept climbing. Even NVIDIA runs a closed frontier model as an orchestrator and pushes the volume to its own open weights. The frontier is becoming a router. The volume goes open. That part is settled. Here is the part that should terrify Washington: the only credible open tier today is Chinese. GLM. Kimi. And the US answer is to tighten export controls and freeze its own labs in place, as if you can embargo a file that is already downloaded, or price-match free. So China hands the Global South Huawei hardware and free open models, and a generation in Africa and Southeast Asia learns to reason through a model that will not tell them what happened at Tiananmen Square. That is not a cost story. That is influence through inference. You do not have to win hearts and minds when you supply the mind. Open source is not a nice-to-have for America. It is the whole ballgame for the 80%, and right now the US is barely on the field. We need American open weights. Not eventually. Now.

Ben Pouladian

71,722 Aufrufe • vor 3 Monaten

We've officially released and open-sourced HunyuanImage 2.1, our latest text-to-image model. The new model delivers on our commitment to balancing performance and quality. With native 2K image generation, HunyuanImage 2.1 is an advanced open-source text-to-image model.🎨 ✨ New in 2.1: 🔹Advanced Semantics: Supports ultra-long and complex prompts of up to 1000 tokens, and precisely controls the generation of multiple subjects in a single image. 🔹Precise Chinese and English Text Rendering with seamless image–text integration: The model naturally integrates text into images, making it suitable for a wide range of applications such as product covers, illustrations, and poster design to meet the needs of various fields. 🔹Rich Styles and High Aesthetic: Capable of generating images in various styles—including photorealistic portraits, comics, and vinyl figures—it delivers outstanding visual appeal and artistic quality. 🔹High-Quality Generation: Efficiently produces ultra-high-definition (2K) images in the same time other models take to generate a 1K image. HunyuanImage 2.1 uses two text encoders: a multimodal large language model (MLLM) to improve the model's image and text alignment capabilities, and a multi-language character-aware encoder to improve text rendering capabilities. The model is a single- and double-stream diffusion transformer with 17B parameters. We've also open-sourced the weights of the the accelerated version with meanflow which reduces inference steps from 100 to just 8, and PromptEnhancer, the first industrial-grade rewriting model that enhances your prompts for more nuanced and expressive image generation. Now, creators turn complex ideas—like posters with slogans or multi-panel comics—into visuals faster than ever. We’re just getting started. Stay tuned for our native multimodal image generation model coming soon. 🌐Website: 🔗Github: 🤗Hugging Face: ✨Hugging Face Demo:

We've officially released and open-sourced HunyuanImage 2.1, our latest text-to-image model. The new model delivers on our commitment to balancing performance and quality. With native 2K image generation, HunyuanImage 2.1 is an advanced open-source text-to-image model.🎨 ✨ New in 2.1: 🔹Advanced Semantics: Supports ultra-long and complex prompts of up to 1000 tokens, and precisely controls the generation of multiple subjects in a single image. 🔹Precise Chinese and English Text Rendering with seamless image–text integration: The model naturally integrates text into images, making it suitable for a wide range of applications such as product covers, illustrations, and poster design to meet the needs of various fields. 🔹Rich Styles and High Aesthetic: Capable of generating images in various styles—including photorealistic portraits, comics, and vinyl figures—it delivers outstanding visual appeal and artistic quality. 🔹High-Quality Generation: Efficiently produces ultra-high-definition (2K) images in the same time other models take to generate a 1K image. HunyuanImage 2.1 uses two text encoders: a multimodal large language model (MLLM) to improve the model's image and text alignment capabilities, and a multi-language character-aware encoder to improve text rendering capabilities. The model is a single- and double-stream diffusion transformer with 17B parameters. We've also open-sourced the weights of the the accelerated version with meanflow which reduces inference steps from 100 to just 8, and PromptEnhancer, the first industrial-grade rewriting model that enhances your prompts for more nuanced and expressive image generation. Now, creators turn complex ideas—like posters with slogans or multi-panel comics—into visuals faster than ever. We’re just getting started. Stay tuned for our native multimodal image generation model coming soon. 🌐Website: 🔗Github: 🤗Hugging Face: ✨Hugging Face Demo:

Tencent Hy

89,430 Aufrufe • vor 1 Jahr

🚨 JUST IN: MICROSOFT just open sourced a VOICE AI THAT TRANSCRIBES 60 MINUTES OF AUDIO in a single pass. 100% FREE. It knows who spoke. It knows when they spoke. It knows exactly what they said. All in one shot. No chunking. No context loss. It's called VibeVoice. Not a transcription tool. Not a basic speech to text wrapper. A frontier voice AI family with ASR, TTS, and real time streaming. All open source. All free. Here's what it actually does 👇 VibeVoice ASR - Speech Recognition: → Processes 60 minutes of continuous audio in a single pass → Never slices audio into chunks so global context is never lost → Identifies WHO spoke, WHEN they spoke and WHAT they said simultaneously → Supports customized hotwords for domain specific accuracy → Works in 50+ languages natively → Already adopted by Hugging Face Transformers library → Already being built on by the open source community BY PEOPLE WHO HAD NO IDEA THIS LEVEL OF ACCURACY WAS ALREADY FREE. VibeVoice TTS - Text to Speech: → Generates up to 90 minutes of speech in a single pass → Supports up to 4 distinct speakers in one conversation → Natural turn taking and speaker consistency throughout → Expressive speech that captures emotional nuances → Supports English, Chinese and multiple other languages VibeVoice Realtime - Streaming TTS: → Only 300 millisecond first audible latency → Streams text input in real time → 0.5B parameters so it actually deploys anywhere → Robust long form generation up to 10 minutes → Lightweight enough for production use today The core innovation nobody is talking about: Most voice AI models slice long audio into short chunks. Every time they slice, they lose context. Speaker tracking breaks. Semantic coherence breaks. Accuracy drops. VibeVoice uses continuous speech tokenizers running at an ultra low frame rate of 7.5 Hz. This preserves audio fidelity while dramatically boosting computational efficiency. The entire 60 minutes stays in context. Nothing gets lost. Nobody gets misidentified. The numbers: → VibeVoice ASR 7B - available now on Hugging Face → VibeVoice Realtime 0.5B - try it on Colab right now → 50+ supported languages → 11 distinct English voice styles → 9 multilingual speaker voices → Already integrated into Hugging Face Transformers → Finetuning code now available The wildest part? A voice powered input method called Vibing just built itself on top of VibeVoice ASR. Available on macOS and Windows right now. The open source community is already shipping products on top of this. 100% Open Source. Free to use. Free to fine tune. Free to build on. 🔖 Save this before your competitors find it first. 👇

🚨 JUST IN: MICROSOFT just open sourced a VOICE AI THAT TRANSCRIBES 60 MINUTES OF AUDIO in a single pass. 100% FREE. It knows who spoke. It knows when they spoke. It knows exactly what they said. All in one shot. No chunking. No context loss. It's called VibeVoice. Not a transcription tool. Not a basic speech to text wrapper. A frontier voice AI family with ASR, TTS, and real time streaming. All open source. All free. Here's what it actually does 👇 VibeVoice ASR - Speech Recognition: → Processes 60 minutes of continuous audio in a single pass → Never slices audio into chunks so global context is never lost → Identifies WHO spoke, WHEN they spoke and WHAT they said simultaneously → Supports customized hotwords for domain specific accuracy → Works in 50+ languages natively → Already adopted by Hugging Face Transformers library → Already being built on by the open source community BY PEOPLE WHO HAD NO IDEA THIS LEVEL OF ACCURACY WAS ALREADY FREE. VibeVoice TTS - Text to Speech: → Generates up to 90 minutes of speech in a single pass → Supports up to 4 distinct speakers in one conversation → Natural turn taking and speaker consistency throughout → Expressive speech that captures emotional nuances → Supports English, Chinese and multiple other languages VibeVoice Realtime - Streaming TTS: → Only 300 millisecond first audible latency → Streams text input in real time → 0.5B parameters so it actually deploys anywhere → Robust long form generation up to 10 minutes → Lightweight enough for production use today The core innovation nobody is talking about: Most voice AI models slice long audio into short chunks. Every time they slice, they lose context. Speaker tracking breaks. Semantic coherence breaks. Accuracy drops. VibeVoice uses continuous speech tokenizers running at an ultra low frame rate of 7.5 Hz. This preserves audio fidelity while dramatically boosting computational efficiency. The entire 60 minutes stays in context. Nothing gets lost. Nobody gets misidentified. The numbers: → VibeVoice ASR 7B - available now on Hugging Face → VibeVoice Realtime 0.5B - try it on Colab right now → 50+ supported languages → 11 distinct English voice styles → 9 multilingual speaker voices → Already integrated into Hugging Face Transformers → Finetuning code now available The wildest part? A voice powered input method called Vibing just built itself on top of VibeVoice ASR. Available on macOS and Windows right now. The open source community is already shipping products on top of this. 100% Open Source. Free to use. Free to fine tune. Free to build on. 🔖 Save this before your competitors find it first. 👇

Kanika

222,078 Aufrufe • vor 6 Monaten

Chinese robotics company Astribot released their latest World-Action Model (WAM), Lumo-2. Technical breakdown: - based on a frozen 🥶 Qwen-3.5 4B VLM - trained in 3 progressive stages: 1. Action is aligned with latent world dynamics (an abstract representation of action). Real-world actions are anchored to physical constraints, while the latent space is guided to focus on motion-relevant changes. This bidirectional relationship makes the model physically grounded -> critical for a world model. 2. Action is aligned with vision and language. Reusing the vision backbone and action encoder from the frozen VLM, the authors add a custom vocabulary (for new actions), a semantic module, an action decoder, and an action projector. This aligns the (new) action representations with the (existing) vision-language semantic space. Most importantly: it builds a direct mapping from natural-language instructions to motor execution. 3. End-to-end training on language, video, and robot data. Only the new modules (everything outside the frozen backbone) are trained end-to-end across temporal reasoning, physical understanding, long-horizon, and dexterous manipulation. At the end of the day, Lumo-2 is not the best on benchmarks, but that's not the point. What's genuinely new: - a way to combine latent world modeling and action generation through progressive alignment - a physically-grounded latent dynamics space - it lifts performance on unseen objects using un-annotated human egocentric video + Vision Pro captures, no special transfer algorithm needed Why it matters: - the whole model is thin trainable adapters (semantic module, action decoder/projector) on a frozen 4B backbone (cheap) - that scale is suited for real-time embedded inference (~2.71× decode speedup, no accuracy loss) - its real moat is long-horizon execution, where the added temporal memory pays off far more than on any other task As a result, this robot can now make your latte (5x sped up video):

Chinese robotics company Astribot released their latest World-Action Model (WAM), Lumo-2. Technical breakdown: - based on a frozen 🥶 Qwen-3.5 4B VLM - trained in 3 progressive stages: 1. Action is aligned with latent world dynamics (an abstract representation of action). Real-world actions are anchored to physical constraints, while the latent space is guided to focus on motion-relevant changes. This bidirectional relationship makes the model physically grounded -> critical for a world model. 2. Action is aligned with vision and language. Reusing the vision backbone and action encoder from the frozen VLM, the authors add a custom vocabulary (for new actions), a semantic module, an action decoder, and an action projector. This aligns the (new) action representations with the (existing) vision-language semantic space. Most importantly: it builds a direct mapping from natural-language instructions to motor execution. 3. End-to-end training on language, video, and robot data. Only the new modules (everything outside the frozen backbone) are trained end-to-end across temporal reasoning, physical understanding, long-horizon, and dexterous manipulation. At the end of the day, Lumo-2 is not the best on benchmarks, but that's not the point. What's genuinely new: - a way to combine latent world modeling and action generation through progressive alignment - a physically-grounded latent dynamics space - it lifts performance on unseen objects using un-annotated human egocentric video + Vision Pro captures, no special transfer algorithm needed Why it matters: - the whole model is thin trainable adapters (semantic module, action decoder/projector) on a frozen 4B backbone (cheap) - that scale is suited for real-time embedded inference (~2.71× decode speedup, no accuracy loss) - its real moat is long-horizon execution, where the added temporal memory pays off far more than on any other task As a result, this robot can now make your latte (5x sped up video):

Léo

32,513 Aufrufe • vor 2 Monaten

Most video-action robot models are a content-creation video generator with an action module attached. LingBot-VA 2.0 from Robbyant, a video-action foundation model, throws that starting point out and trains the whole stack natively for control. And it runs closed-loop at a peak 225 Hz. It's so important because A robot cannot move responsively when its controller pauses to imagine the next few frames. LingBot-VA 2.0 predicts during execution, then corrects using each real observation. And it carries only about 13B video parameters while activating roughly 1.9B per token. Bigger robot models usually mean slower reactions, creating a direct conflict between intelligence and control. LingBot-VA 2.0 is trained from scratch for robot control rather than adapted from a video generator built for content creation. Robbyant, an embodied AI company under Ant Group, built it to learn how scenes change under actions, predict what should happen next, and turn those predictions into real-time robot movements. Most video-action systems inherit a tokenizer and video backbone trained mainly to reproduce visual appearance. LingBot-VA 2.0 rebuilds both parts around physical control. Its semantic visual-action tokenizer maps observations toward features from a frozen vision foundation model and learns compact latent actions from frame-to-frame changes using self-supervised inverse and forward dynamics. Unlabeled web video can therefore carry action-relevant training signals without robot action labels. The policy is causal from the start, so every prediction can use only past observations. Its sparse Mixture-of-Experts video backbone has about 13B total parameters, while about 1.9B are active per token, keeping the compute lower during each step. A high-level vision-language planner breaks long tasks into smaller instructions, while the low-level video-action policy handles continuous movement. Foresight Reasoning predicts future visual states while the robot is already acting, then replaces imagined states with every new real observation. Combined with few-step distillation and systems acceleration, the paper reports a peak asynchronous execution frequency of 225 Hz. The model adapts from 10–15 demonstrations, transfers across robot embodiments, and handles some new tasks zero-shot. In the paper’s own evaluations, it reaches 93.6 average on RoboTwin 2.0 and reports stronger real-world results than LingBot-VA and π0.5 across the tested tasks. 🧵 1.

Most video-action robot models are a content-creation video generator with an action module attached. LingBot-VA 2.0 from Robbyant, a video-action foundation model, throws that starting point out and trains the whole stack natively for control. And it runs closed-loop at a peak 225 Hz. It's so important because A robot cannot move responsively when its controller pauses to imagine the next few frames. LingBot-VA 2.0 predicts during execution, then corrects using each real observation. And it carries only about 13B video parameters while activating roughly 1.9B per token. Bigger robot models usually mean slower reactions, creating a direct conflict between intelligence and control. LingBot-VA 2.0 is trained from scratch for robot control rather than adapted from a video generator built for content creation. Robbyant, an embodied AI company under Ant Group, built it to learn how scenes change under actions, predict what should happen next, and turn those predictions into real-time robot movements. Most video-action systems inherit a tokenizer and video backbone trained mainly to reproduce visual appearance. LingBot-VA 2.0 rebuilds both parts around physical control. Its semantic visual-action tokenizer maps observations toward features from a frozen vision foundation model and learns compact latent actions from frame-to-frame changes using self-supervised inverse and forward dynamics. Unlabeled web video can therefore carry action-relevant training signals without robot action labels. The policy is causal from the start, so every prediction can use only past observations. Its sparse Mixture-of-Experts video backbone has about 13B total parameters, while about 1.9B are active per token, keeping the compute lower during each step. A high-level vision-language planner breaks long tasks into smaller instructions, while the low-level video-action policy handles continuous movement. Foresight Reasoning predicts future visual states while the robot is already acting, then replaces imagined states with every new real observation. Combined with few-step distillation and systems acceleration, the paper reports a peak asynchronous execution frequency of 225 Hz. The model adapts from 10–15 demonstrations, transfers across robot embodiments, and handles some new tasks zero-shot. In the paper’s own evaluations, it reaches 93.6 average on RoboTwin 2.0 and reports stronger real-world results than LingBot-VA and π0.5 across the tested tasks. 🧵 1.

Rohan Paul

11,253 Aufrufe • vor 3 Monaten