Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

DiT4DiT is now open source! As the first humanoid-deployable Video-Action Model built on a world model, DiT4DiT continues to surprise us. In our paper last month, we showed its strong data efficiency. Now, with only slight modifications, it enables real-time whole-body autonomous pick-and-place. Paper: Code: Website:

Shuo Yang

5,468 subscribers

15,969 views • 6 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

5 Comments

Alfred Cueva6 months ago

Really interesting work! I am also trying to use VAM for humanoid whole body control

Michael Cho - Rbt/Acc6 months ago

Really cool!

Tommy AutoTech6 months ago

Insert me pitching world models to my VP instead of traditional vision systems. This pick-and-place fluidity is undeniable.

𓅋 𐎫𐎤𐎶𐏀 ‎ﷺ6 months ago

Are you training in sim? How many episodes averagely per task?

Jie Wang6 months ago

very cool! Do you see an advantage of language steer-ability here?

Related Videos

1/ Tencent releases HY World 1.5 (WorldPlay), an open-source real-time world model with 24fps A streaming video diffusion model that enables real-time, interactive world modeling with long-term geometric consistency In direct comparison to Genie 3, it's technically inferior, and there's no information on how long the consistency will last. However, Tencent has released a significant world model as open source for the first time, making it available to everyone. A few more examples, a thread 🧵

1/ Tencent releases HY World 1.5 (WorldPlay), an open-source real-time world model with 24fps A streaming video diffusion model that enables real-time, interactive world modeling with long-term geometric consistency In direct comparison to Genie 3, it's technically inferior, and there's no information on how long the consistency will last. However, Tencent has released a significant world model as open source for the first time, making it available to everyone. A few more examples, a thread 🧵

Chubby♨️

37,776 views • 10 months ago

We’re excited to share DiT4DiT, an end-to-end Video-Action Model for robot learning that unifies a video Diffusion Transformer and an action Diffusion Transformer in a single cascaded framework. By leveraging the rich spatiotemporal and physical dynamics learned through video generation, rather than static image-text priors, DiT4DiT achieves state-of-the-art results on LIBERO (98.6%) and RoboCasa GR1 (50.8%) with far less training data, delivering over 10× better sample efficiency and up to 7× faster convergence. Real-world deployment on a humanoid robot further shows robust generalization. We believe this is a step toward making video generation a powerful backbone for robot policy learning. This work builds upon the brilliant foundations laid by Nvidia's GR00T and Cosmos. Project: Paper: Code: Coming soon. In the meantime, you can ask your coding agent to reproduce the method based on GR00T/Cosmos.

We’re excited to share DiT4DiT, an end-to-end Video-Action Model for robot learning that unifies a video Diffusion Transformer and an action Diffusion Transformer in a single cascaded framework. By leveraging the rich spatiotemporal and physical dynamics learned through video generation, rather than static image-text priors, DiT4DiT achieves state-of-the-art results on LIBERO (98.6%) and RoboCasa GR1 (50.8%) with far less training data, delivering over 10× better sample efficiency and up to 7× faster convergence. Real-world deployment on a humanoid robot further shows robust generalization. We believe this is a step toward making video generation a powerful backbone for robot policy learning. This work builds upon the brilliant foundations laid by Nvidia's GR00T and Cosmos. Project: Paper: Code: Coming soon. In the meantime, you can ask your coding agent to reproduce the method based on GR00T/Cosmos.

Shuo Yang

31,872 views • 7 months ago

🔥Character AI in VR Space🔥 We present #SOLAMI, a social vision-language-action (VLA) model that enables 3D autonomous characters with *speech and body language* interaction - Project: - Paper Hugging Face : . Thanks AK !

🔥Character AI in VR Space🔥 We present #SOLAMI, a social vision-language-action (VLA) model that enables 3D autonomous characters with speech and body language interaction - Project: - Paper Hugging Face : . Thanks AK !

Ziwei Liu

27,957 views • 1 year ago

A humanoid robot policy trained solely on synthetic data generated by a world model. Research Scientist Joel Jang presents NVIDIA's DreamGen pipeline: ⦿ Post-train the world model Cosmos-Predict2 with a small set of real teleoperation demos. ⦿ Prompt the world model to generate synthetic video data with verbs and scenarios not used in the world model’s post-training. ⦿ Auto-label synthetic video data with action sequences. ⦿ Train robot policies using only synthetic data. That's it. Deploy zero-shot to a real humanoid robot.

A humanoid robot policy trained solely on synthetic data generated by a world model. Research Scientist Joel Jang presents NVIDIA's DreamGen pipeline: ⦿ Post-train the world model Cosmos-Predict2 with a small set of real teleoperation demos. ⦿ Prompt the world model to generate synthetic video data with verbs and scenarios not used in the world model’s post-training. ⦿ Auto-label synthetic video data with action sequences. ⦿ Train robot policies using only synthetic data. That's it. Deploy zero-shot to a real humanoid robot.

The Humanoid Hub

20,968 views • 1 year ago

Today, we are releasing Stable Video Diffusion, our first foundation model for generative AI video based on the image model, Stable Diffusion. As part of this research preview, the code, weights, and research paper are now available. Additionally, today you can sign up for our waitlist to access a new upcoming web experience featuring a Text-To-Video interface. To access the model & sign up for our waitlist, visit our website here:

Today, we are releasing Stable Video Diffusion, our first foundation model for generative AI video based on the image model, Stable Diffusion. As part of this research preview, the code, weights, and research paper are now available. Additionally, today you can sign up for our waitlist to access a new upcoming web experience featuring a Text-To-Video interface. To access the model & sign up for our waitlist, visit our website here:

Stability AI

1,024,917 views • 2 years ago

The Waypoint-1.5 technical paper is now live. Waypoint-1.5 is a real-time video diffusion world model designed to run on consumer GPUs, bringing interactive world models closer to practical, accessible deployment.

The Waypoint-1.5 technical paper is now live. Waypoint-1.5 is a real-time video diffusion world model designed to run on consumer GPUs, bringing interactive world models closer to practical, accessible deployment.

Overworld

81,687 views • 4 months ago

The first open-source unified world model for scalable robot manipulation: 5B-parameter open-source unified video-action world model that combines policy and world modeling to generate robot actions, predict future visuals, and evaluate task progress from observations, language, and state. The model is trained on 27.3K hours of heterogeneous data (17.8K real-robot teleop, 6.5K UMI demos, 3K egocentric human videos), enabling it to perform complex manipulation tasks like faucet connecting, bag packing, and toolbox storing as shown in demo videos. The approach supports test-time action refinement and points toward deployment-driven continuous improvement via fleet data. Thanks for sharing, Jianlan Luo (Jianlan Luo)! 📌 Resource links for τ0-WM: • Project page: • GitHub (code): • Hugging Face (model weights): • Paper (PDF): ——- Weekly robotics and AI insights. Subscribe free:

The first open-source unified world model for scalable robot manipulation: 5B-parameter open-source unified video-action world model that combines policy and world modeling to generate robot actions, predict future visuals, and evaluate task progress from observations, language, and state. The model is trained on 27.3K hours of heterogeneous data (17.8K real-robot teleop, 6.5K UMI demos, 3K egocentric human videos), enabling it to perform complex manipulation tasks like faucet connecting, bag packing, and toolbox storing as shown in demo videos. The approach supports test-time action refinement and points toward deployment-driven continuous improvement via fleet data. Thanks for sharing, Jianlan Luo (Jianlan Luo)! 📌 Resource links for τ0-WM: • Project page: • GitHub (code): • Hugging Face (model weights): • Paper (PDF): ——- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

28,276 views • 4 months ago

1/5 🚀 Thrilled to open-source OSCAR 🤖 — an action-conditioned world model for robotics, led by the visiting student in my group Zhuoyuan Wu! It generalizes across different robot embodiments with precise action controllability. All trained on a single GH200 GPU, and outperforms existing open-sourced baselines, which have larger model capacity and need more compute. Everything is public, including training data. 📄 Paper: 🌐 Project: 💻 Code: 🤗 Robot data: 🤗 Human data: 🤗 Weights: #Robotics #WorldModels #AI #OpenSource

1/5 🚀 Thrilled to open-source OSCAR 🤖 — an action-conditioned world model for robotics, led by the visiting student in my group Zhuoyuan Wu! It generalizes across different robot embodiments with precise action controllability. All trained on a single GH200 GPU, and outperforms existing open-sourced baselines, which have larger model capacity and need more compute. Everything is public, including training data. 📄 Paper: 🌐 Project: 💻 Code: 🤗 Robot data: 🤗 Human data: 🤗 Weights: #Robotics #WorldModels #AI #OpenSource

Jun Gao

105,237 views • 4 months ago

Our Puffin-World technical paper is now available. From one image or text prompt, it generates camera-controlled views with appearance, geometry and physical grounding, then reconstructs them into an explorable 3D world—all in one unified model. Paper: Blog: Dataset: Code:

Our Puffin-World technical paper is now available. From one image or text prompt, it generates camera-controlled views with appearance, geometry and physical grounding, then reconstructs them into an explorable 3D world—all in one unified model. Paper: Blog: Dataset: Code:

Chen Change Loy

10,439 views • 1 month ago

Unitree Breakthrough: The World’s First Real-Time World Model-Driven Fully Autonomous Humanoid Robot Combat🥊

Unitree Breakthrough: The World’s First Real-Time World Model-Driven Fully Autonomous Humanoid Robot Combat🥊

Unitree

7,399,519 views • 1 month ago

Google Genie 3 just got its biggest open-source rival 🤯 LingBot-World is an open-source world model that generates playable environments in real-time. → Built on Alibaba’s Wan2.2 → 10 minutes of stable generation → 16fps real-time interaction 100% Open Source.

Google Genie 3 just got its biggest open-source rival 🤯 LingBot-World is an open-source world model that generates playable environments in real-time. → Built on Alibaba’s Wan2.2 → 10 minutes of stable generation → 16fps real-time interaction 100% Open Source.

Alvaro Cintas

1,060,413 views • 9 months ago

Spatial Intelligence just went Open Source. 🌐 Introducing InSpatio-WorldFM — the world’s first open-source real-time interactive 3D world model. Last October, World Labs's RTFM showed the vision. Today, we’re bringing that idea into the open for everyone. ✅Efficient, multi-view consistent spatial reasoning ✅Real-time interactive exploration ✅Open-source weights & code Stop watching demos. Start building now. 👇

Spatial Intelligence just went Open Source. 🌐 Introducing InSpatio-WorldFM — the world’s first open-source real-time interactive 3D world model. Last October, World Labs's RTFM showed the vision. Today, we’re bringing that idea into the open for everyone. ✅Efficient, multi-view consistent spatial reasoning ✅Real-time interactive exploration ✅Open-source weights & code Stop watching demos. Start building now. 👇

InSpatio｜影溯

37,476 views • 7 months ago

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video

Figure

46,041 views • 1 year ago

Open-sourcing the whole package here! The last piece of our SONIC open-source, data collection, gr00t VLA post-training, inference just hit the repo! Train your Autonomous policies on G1 Whole-body with SONIC and gr00t N1.7! 🧑‍💻Code: 📑Docs:

Open-sourcing the whole package here! The last piece of our SONIC open-source, data collection, gr00t VLA post-training, inference just hit the repo! Train your Autonomous policies on G1 Whole-body with SONIC and gr00t N1.7! 🧑‍💻Code: 📑Docs:

Zhengyi “Zen” Luo

49,006 views • 5 months ago

🚀 We open-sourced LongLive — interactive, real-time long-video generation. 👥Generates video in real time as users enter text prompts. ⚡️20.7 FPS on a single H100,⏱️up to 240s per clip. 🎬Fine-tunes SOTA short-video models (e.g., Wan) into long-video generators. 🌍One step closer to World Models. All code for training & inference, model weights, demo page, and videos released! Paper: Code: Model: Demo Page: Introduction Video:

🚀 We open-sourced LongLive — interactive, real-time long-video generation. 👥Generates video in real time as users enter text prompts. ⚡️20.7 FPS on a single H100,⏱️up to 240s per clip. 🎬Fine-tunes SOTA short-video models (e.g., Wan) into long-video generators. 🌍One step closer to World Models. All code for training & inference, model weights, demo page, and videos released! Paper: Code: Model: Demo Page: Introduction Video:

Yukang Chen

11,943 views • 1 year ago

1/4Introducing Qwen3-Coder-Next: Our latest open-weights model designed specifically to power the next generation of autonomous Coding Agents. Built on Qwen3-Next, this model is engineered to handle complex, long-horizon programming tasks with unprecedented efficiency. High-performance agentic intelligence is now in your hands.

1/4Introducing Qwen3-Coder-Next: Our latest open-weights model designed specifically to power the next generation of autonomous Coding Agents. Built on Qwen3-Next, this model is engineered to handle complex, long-horizon programming tasks with unprecedented efficiency. High-performance agentic intelligence is now in your hands.

Tongyi Lab

212,645 views • 8 months ago

We're thrilled to release & open-source Hunyuan3D World Model 1.0! This model enables you to generate immersive, explorable, and interactive 3D worlds from just a sentence or an image. It's the industry's first open-source 3D world generation model, compatible with CG pipelines for full editability & simulation. Set to transform game development, VR, digital content creation and so on. Get started now👇🏻 Project Page： Try it now： Github： Hugging Face：

We're thrilled to release & open-source Hunyuan3D World Model 1.0! This model enables you to generate immersive, explorable, and interactive 3D worlds from just a sentence or an image. It's the industry's first open-source 3D world generation model, compatible with CG pipelines for full editability & simulation. Set to transform game development, VR, digital content creation and so on. Get started now👇🏻 Project Page： Try it now： Github： Hugging Face：

Tencent Hy

1,231,203 views • 1 year ago

This is the best way to understand how ML models actually work! Use Drawdata to draw a 2D dataset in Jupyter. Use it to actively pick data from the widget and update the model as the data is being drawn! Fully interactive, real-time, and open-source!

This is the best way to understand how ML models actually work! Use Drawdata to draw a 2D dataset in Jupyter. Use it to actively pick data from the widget and update the model as the data is being drawn! Fully interactive, real-time, and open-source!

Daily Dose of Data Science

52,132 views • 1 year ago

We don’t generate videos. 🎬 We generate worlds from videos. 🌍 Introducing InSpatio-World — the world's first open-source real-time 4D world model‼️ Your input: a video clip Our output: a dynamic, navigable, persistent world 🕹️ explore freely across viewpoints ⏪ control time forward and backward 🔓 open-source and ready to build on :) Live demo: 🔗 Code & weights: 🔗 Project page: 🔗

We don’t generate videos. 🎬 We generate worlds from videos. 🌍 Introducing InSpatio-World — the world's first open-source real-time 4D world model‼️ Your input: a video clip Our output: a dynamic, navigable, persistent world 🕹️ explore freely across viewpoints ⏪ control time forward and backward 🔓 open-source and ready to build on :) Live demo: 🔗 Code & weights: 🔗 Project page: 🔗

InSpatio｜影溯

110,315 views • 7 months ago