Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

🤖What if a robot could perform a new task just from a natural language command, with zero demonstrations? Our new work, NovaFlow, makes it possible! We use pre-trained video generative model to create a video of the task, then translate it into a plan for real-world robot execution. 1/6... show more

Hongyu Li

1,668 subscribers

105,915 views • 11 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

34 Comments

Hongyu Li11 months ago

We start by giving a video model a text prompt like "hang the mug." It generates a short clip of a human performing the task, acting as a visual instruction manual. This video provides the commonsense physical knowledge needed for the robot to understand the motion. 🎥 2/6

Hongyu Li11 months ago

The generated video is 2D, so how does the robot understand 3D space? We lift it! 📏 A monocular video depth estimation model predicts depth for every frame. Then, we calibrate it with the robot's real-world depth camera for precise spatial awareness. 3/6

Hongyu Li11 months ago

🤔 But how do you bridge the gap between that video and physical control? We introduce "actionable 3D object flow," a novel representation distilled from the video that guides the robot’s motion for complex rigid, articulated, and even deformable objects. 🌟 4/6

Hongyu Li11 months ago

NovaFlow shows incredible generalization. From rigid blocks to deformable ropes, on both a tabletop arm and a mobile Spot! It adapts to new objects and embodiments without any retraining. It achieves SOTA zero-shot results, even outperforming policies trained on 10-30 demos. 🏆 5/6

Hongyu Li11 months ago

By leveraging the commonsense knowledge in video models, NovaFlow helps solve the robot data bottleneck. This is a leap towards more general-purpose robots. 🚀 Dive deeper into our work: 📜 Paper: 🌐 Project Website: 👀 P.S. You can check out the 3D generated videos and planned robot trajectories in a live viewer on our website! 6/6

Hongyu Li11 months ago

This is a collaboration with @ling_feng_sun, @YafeiHuCMU, Duy Ta, Jenny Barry, and @jiahuifu_carol.

Nils11 months ago

Beautiful.

Hongyu Li11 months ago

Thanks, Nils!

Xiatao Sun11 months ago

Using video generation models to create task plans from language is clever! The actionable 3D object flow representation seems like the key to making this work across embodiments.

Hongyu Li11 months ago

Thanks, Xiatao. I believe video models capture the most general form of dynamics, although they can’t be directly understood by robots. Actionable flow enables robots to comprehend the generated videos.

Zhiyang (Frank) Dou11 months ago

Nice work!

Hongyu Li11 months ago

Thanks, Zhiyang!

Bozheng Li11 months ago

great work!

Hongyu Li11 months ago

Thanks, Bozheng!

Balveer Singh11 months ago

Wow, this is such a cool approach.

Hongyu Li11 months ago

Thanks🙏

Ted Xiao11 months ago

Nice work!

Hongyu Li11 months ago

Thanks, Ted! 🙏

Sam Green11 months ago

Laying the groundwork for Soma

Atharva Sawant11 months ago

So cool

Hongyu Li11 months ago

Thanks

Cryptlesh11 months ago

Wow. It’s like imagining whole take on head and then implementing it like human break does. Kind of.

Hongyu Li11 months ago

Exactly, a video model serves as the imagination capabilities of human beings.

Bercan11 months ago

Great work. Can you try on these examples in mcap.

Hongyu Li11 months ago

Thanks for the great suggestions! Egocentric data is more challenging, and we’re still working on it!

Bercan11 months ago

Happy to work together we are also working on real2sim

Stanley Wei11 months ago

That will save a lot of time. Hope you post some cool demos.

BlockChan9 months ago

holy shit i personaly owned name for a while and have no idea what to build

towerofshadow11 months ago

is it more effective to train robots using human actions that are tracked using actionable flow, or to teleoperate robots into direct movement data?

Hongyu Li11 months ago

NovaFlow is object-centric and embodiment-agnostic. So it should work the same for both human and robot data. In practice, we find video generative models work better with generating human data.

Danny Ng11 months ago

I wonder how long it takes to perform one task?

Hongyu Li11 months ago

It depends on the choice of video model. For Veo, it takes around 2-3min for the entire pipeline.

Urs Gehrig11 months ago

Zero shot Robotic AI manipulation.

Mariano Phielipp6 months ago

Interesting work. How you see the work of Florence, Manuelli, Redrake in the context of this work Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation Florence,Manuelli, Russ Tedrake?

Related Videos

S1 learns new tasks like a language model. You prompt it with a video demonstration, and it outputs robot actions to complete the task in any environment and in any embodiment.

S1 learns new tasks like a language model. You prompt it with a video demonstration, and it outputs robot actions to complete the task in any environment and in any embodiment.

Skild AI

308,754 views • 18 days ago

Xiaomi Robotics-1 🤖a new robot foundation model from Xiaomi Tech - 100K+ hours real world robot data - VLA architecture - Cross-embodiment generalization - Fast adaptation to new tasks

Xiaomi Robotics-1 🤖a new robot foundation model from Xiaomi Tech - 100K+ hours real world robot data - VLA architecture - Cross-embodiment generalization - Fast adaptation to new tasks

Adina Yakup

17,348 views • 1 month ago

World Labs co-founder Justin Johnson predicts a real-to-sim-to-real future where Atlas could train a robot for any new environment in minutes using just 5 photos from your phone: "One is this notion of fully in-context learning. Maybe I've got a robotics foundation model, then I can demonstrate the robot once how a task should be performed, and that's enough for the robot to figure out." "Another version is real to sim to real. Maybe I've got my pre-trained robotics foundation model, but I want to adapt it to this particular environment, like this studio, this table, moving this microphone around." "There's a version where you could come in here and take just a couple casual videos with your phone or a couple images with your phone, use Atlas to reconstruct the space, and now stage all kinds of robotics interactions in this studio space." "If you could lower the barrier to doing that, you could do it in five minutes to take five photos, upload it to Atlas, have it generate a simulation, maybe describe in natural language what kind of task you want the robot to do, have an agent go build that simulation for you, RL fine-tune your general purpose robotics foundation model, and now you've got a robot maybe in the span of a couple minutes that could come in and is perfectly adapted to this space." World Labs

World Labs co-founder Justin Johnson predicts a real-to-sim-to-real future where Atlas could train a robot for any new environment in minutes using just 5 photos from your phone: "One is this notion of fully in-context learning. Maybe I've got a robotics foundation model, then I can demonstrate the robot once how a task should be performed, and that's enough for the robot to figure out." "Another version is real to sim to real. Maybe I've got my pre-trained robotics foundation model, but I want to adapt it to this particular environment, like this studio, this table, moving this microphone around." "There's a version where you could come in here and take just a couple casual videos with your phone or a couple images with your phone, use Atlas to reconstruct the space, and now stage all kinds of robotics interactions in this studio space." "If you could lower the barrier to doing that, you could do it in five minutes to take five photos, upload it to Atlas, have it generate a simulation, maybe describe in natural language what kind of task you want the robot to do, have an agent go build that simulation for you, RL fine-tune your general purpose robotics foundation model, and now you've got a robot maybe in the span of a couple minutes that could come in and is perfectly adapted to this space." World Labs

MTS

25,204 views • 11 days ago

World Labs co-founder Justin Johnson on how five photos from your phone could teach a robot to work in a room it has never seen: "I think there's a couple different tech trees that people are working on. One is this notion of fully in-context learning. Maybe I've got a robotics foundation model, then I can demonstrate to the robot once how a task should be performed, and that's enough for the robot to figure out that task." "Another version is what we're calling real-to-sim-to-real. Maybe I've got my pre-trained robotics foundation model, but I want to adapt it to this particular environment." "Like this studio... You could come in here and take just a couple casual videos with your phone or a couple images with your phone, use Atlas to reconstruct the space, and now stage all kinds of robotics interactions in this studio space in particular." "Literally you could do it in five minutes. Take five photos, upload it to Atlas, have it generate a simulation, describe in natural language what kind of task you want the robot to do, have an agent go build that simulation for you, RL fine-tune your general purpose robotics foundation model, and now you've got a robot... that's perfectly adapted to this space." Justin Johnson on MTS

World Labs co-founder Justin Johnson on how five photos from your phone could teach a robot to work in a room it has never seen: "I think there's a couple different tech trees that people are working on. One is this notion of fully in-context learning. Maybe I've got a robotics foundation model, then I can demonstrate to the robot once how a task should be performed, and that's enough for the robot to figure out that task." "Another version is what we're calling real-to-sim-to-real. Maybe I've got my pre-trained robotics foundation model, but I want to adapt it to this particular environment." "Like this studio... You could come in here and take just a couple casual videos with your phone or a couple images with your phone, use Atlas to reconstruct the space, and now stage all kinds of robotics interactions in this studio space in particular." "Literally you could do it in five minutes. Take five photos, upload it to Atlas, have it generate a simulation, describe in natural language what kind of task you want the robot to do, have an agent go build that simulation for you, RL fine-tune your general purpose robotics foundation model, and now you've got a robot... that's perfectly adapted to this space." Justin Johnson on MTS

a16z

35,947 views • 10 days ago

Most robot demos are scripted. Generalist's GEN-1 is not. > GEN-1 was doing a task > Mid-task, an extra object was thrown into the bin > GEN-1 quickly adapted to the new environment > Still completed the task smoothly This is the difference between a scripted policy (seen in viral launch videos) and a real foundation model. GEN-1 was trained on 500,000 hours of proprietary, real-world dexterous manipulation data. It has NEVER seen that exact scenario before. This was the coolest demo at Automate Show Generalist Flexiv Robotics

Most robot demos are scripted. Generalist's GEN-1 is not. > GEN-1 was doing a task > Mid-task, an extra object was thrown into the bin > GEN-1 quickly adapted to the new environment > Still completed the task smoothly This is the difference between a scripted policy (seen in viral launch videos) and a real foundation model. GEN-1 was trained on 500,000 hours of proprietary, real-world dexterous manipulation data. It has NEVER seen that exact scenario before. This was the coolest demo at Automate Show Generalist Flexiv Robotics

mete

11,651 views • 2 months ago

Xiaomi-Robotics-1 just dropped on Hugging Face 🔥 A robot foundation model trained on 100,000 hours of real-world manipulation. They turned it loose in a real apartment: folding laundry, loading the washer, doing the dishes, packing a suitcase. Fully autonomous.

Xiaomi-Robotics-1 just dropped on Hugging Face 🔥 A robot foundation model trained on 100,000 hours of real-world manipulation. They turned it loose in a real apartment: folding laundry, loading the washer, doing the dishes, packing a suitcase. Fully autonomous.

Victor M

131,193 views • 1 month ago

How well do egocentric human videos scale? Apparently, really well. Dyna Robotics has introduced Dyna-2, a world-action model (WAM) pre-trained on more than 1 million hours of real egocentric human video. It learns by jointly predicting the next video frames and the actions, and that video prediction (world modeling) is what carries learning from humans to robots. • More human video predictably improves action prediction. • Task accuracy improves across embodiments (bimanual arms and a humanoid) with zero robot data in pre-training. • Post-trained on just a few hours of robot data, mean task performance rose 20% to 53% as human pre-training scaled 1k to 1M hours. WAM vs VLA, matched apple-to-apple: the WAM hit 1.55x the VLA's success rate and won 65% of head-to-heads. Deployed zero-shot at unseen customer sites, it passed 87% vs the VLA's 46%.

How well do egocentric human videos scale? Apparently, really well. Dyna Robotics has introduced Dyna-2, a world-action model (WAM) pre-trained on more than 1 million hours of real egocentric human video. It learns by jointly predicting the next video frames and the actions, and that video prediction (world modeling) is what carries learning from humans to robots. • More human video predictably improves action prediction. • Task accuracy improves across embodiments (bimanual arms and a humanoid) with zero robot data in pre-training. • Post-trained on just a few hours of robot data, mean task performance rose 20% to 53% as human pre-training scaled 1k to 1M hours. WAM vs VLA, matched apple-to-apple: the WAM hit 1.55x the VLA's success rate and won 65% of head-to-heads. Deployed zero-shot at unseen customer sites, it passed 87% vs the VLA's 46%.

The Humanoid Hub

15,837 views • 1 month ago

The Real-to-Sim part of R2S2R transforms physical robots, sensors, environments, objects, and interactions into simulations that preserve task-relevant observations and dynamics - not only how the world looks, but how it acts when the robot interacts with it. These results set a new bar for sim-real alignment in robot manipulation.

The Real-to-Sim part of R2S2R transforms physical robots, sensors, environments, objects, and interactions into simulations that preserve task-relevant observations and dynamics - not only how the world looks, but how it acts when the robot interacts with it. These results set a new bar for sim-real alignment in robot manipulation.

Fei-Fei Li

25,970 views • 1 month ago

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Jingyun Yang

88,294 views • 2 years ago

The robot is learning several novel tasks instantly, after just ONE demonstration each... Instant Policy makes it possible: no extra training, no weight updates, just pure in-context learning. It just got accepted at ICLR 2025, and it’s changing how robots learn. With just a single demo, a robot can pick up a new task and start performing it right away. Why this is a big deal: ✅ Learns tasks instantly with just one or a few demonstrations ✅ Improves over time as more demonstrations are given ✅ Uses simulation-based training with “pseudo-demonstrations” for scalability ✅ Can transfer skills across different robots and even follow language-defined tasks It brings in-context learning to robotics, opening up new possibilities for flexible, real-world automation. You can try it yourself: code and weights are available at • • • • Thank you to Edward Johns, Director of the Robot Learning Lab at Imperial College for sharing their work! 🙏

The robot is learning several novel tasks instantly, after just ONE demonstration each... Instant Policy makes it possible: no extra training, no weight updates, just pure in-context learning. It just got accepted at ICLR 2025, and it’s changing how robots learn. With just a single demo, a robot can pick up a new task and start performing it right away. Why this is a big deal: ✅ Learns tasks instantly with just one or a few demonstrations ✅ Improves over time as more demonstrations are given ✅ Uses simulation-based training with “pseudo-demonstrations” for scalability ✅ Can transfer skills across different robots and even follow language-defined tasks It brings in-context learning to robotics, opening up new possibilities for flexible, real-world automation. You can try it yourself: code and weights are available at • • • • Thank you to Edward Johns, Director of the Robot Learning Lab at Imperial College for sharing their work! 🙏

Ilir Aliu

46,285 views • 1 year ago

Humanoid robotics is entering a new phase： Xiaomi Robotics 0 (XR-0) explores learning manipulation directly from large-scale human videos — aligning video understanding with robot embodiment to generate executable actions. Watch humans->learn representations ->map to robot control. Robot data is scarce. Internet video is not. If video pretraining transfers reliably into embodied control, the scaling law of robotics changes. Deployment speed becomes a function of model alignment, not just hardware iteration. We’re moving from engineered skills to learned behavior. That’s a structural shift. project:

Humanoid robotics is entering a new phase： Xiaomi Robotics 0 (XR-0) explores learning manipulation directly from large-scale human videos — aligning video understanding with robot embodiment to generate executable actions. Watch humans->learn representations ->map to robot control. Robot data is scarce. Internet video is not. If video pretraining transfers reliably into embodied control, the scaling law of robotics changes. Deployment speed becomes a function of model alignment, not just hardware iteration. We’re moving from engineered skills to learned behavior. That’s a structural shift. project:

CyberRobo

18,356 views • 6 months ago

How do you know your robot actually succeeded? In v0.6.0, you ask a VLM 🤖 LeRobot ships a new unified reward models API with two zero-shot additions. Robometer is a general-purpose reward model trained on 1M+ trajectories across 21 robot embodiments: point it at any LeRobot dataset and it scores task progress and success straight from video and a language instruction, no task-specific training needed. TOPReward goes even further: no reward weights at all. It wraps an off-the-shelf VLM and reads the log-probability of the token "True" given the video and the instruction. Both come with labeling scripts that write per-frame progress curves into your dataset, ready for reward-aware behavior cloning. More on how they work:

How do you know your robot actually succeeded? In v0.6.0, you ask a VLM 🤖 LeRobot ships a new unified reward models API with two zero-shot additions. Robometer is a general-purpose reward model trained on 1M+ trajectories across 21 robot embodiments: point it at any LeRobot dataset and it scores task progress and success straight from video and a language instruction, no task-specific training needed. TOPReward goes even further: no reward weights at all. It wraps an off-the-shelf VLM and reads the log-probability of the token "True" given the video and the instruction. Both come with labeling scripts that write per-frame progress curves into your dataset, ready for reward-aware behavior cloning. More on how they work:

LeRobot

22,513 views • 2 months ago

Show it a single human demonstration of a long-horizon task and the robot performs it right away. No fine-tuning, no retraining. Skild has released S1, a robot foundation model where you specify the task with a video demonstration instead of a language instruction. Why video, not language Language works for "hand me the mug." It falls apart for anything delicate or long-horizon. You don't learn to fold a fitted sheet from a sentence. So S1 takes a video demo into its context window and translates it to its own body and scene. What it does - Tasks up to 10 minutes, dozens of sequential steps - Unseen tasks: potting plants, pour-over coffee, kit assembly, pancakes. Skild says even the underlying primitives, like flipping a pancake, weren't in pre-training - Recovers from its own mistakes without being trained to - Improves on flawed demos. Demonstrator drops an egg, S1 doesn't make that mistake. It reads the demo as a goal, not a trajectory to copy.

Show it a single human demonstration of a long-horizon task and the robot performs it right away. No fine-tuning, no retraining. Skild has released S1, a robot foundation model where you specify the task with a video demonstration instead of a language instruction. Why video, not language Language works for "hand me the mug." It falls apart for anything delicate or long-horizon. You don't learn to fold a fitted sheet from a sentence. So S1 takes a video demo into its context window and translates it to its own body and scene. What it does - Tasks up to 10 minutes, dozens of sequential steps - Unseen tasks: potting plants, pour-over coffee, kit assembly, pancakes. Skild says even the underlying primitives, like flipping a pancake, weren't in pre-training - Recovers from its own mistakes without being trained to - Improves on flawed demos. Demonstrator drops an egg, S1 doesn't make that mistake. It reads the demo as a goal, not a trajectory to copy.

The Humanoid Hub

18,901 views • 18 days ago

🚀 We’re excited to announce LingBot-VA, a new state-of-the-art robot policy model from Robbyant ! LingBot-VA is built on a causal, autoregressive video-action world model for generalist robot control. Highlights: (1) First unified autoregressive video-action world model for robot control (2) Low-latency inference with a new asynchronous execution pipeline (3) SOTA on RoboTwin (92.9%, firstever > 90%) and LIBERO (98.5%) (4) +20% over π0.5 on challenging real-world long-horizon & high-precision tasks

🚀 We’re excited to announce LingBot-VA, a new state-of-the-art robot policy model from Robbyant ! LingBot-VA is built on a causal, autoregressive video-action world model for generalist robot control. Highlights: (1) First unified autoregressive video-action world model for robot control (2) Low-latency inference with a new asynchronous execution pipeline (3) SOTA on RoboTwin (92.9%, firstever > 90%) and LIBERO (98.5%) (4) +20% over π0.5 on challenging real-world long-horizon & high-precision tasks

Yinghao Xu

47,566 views • 7 months ago

Introducing FLUX-mimic, a next-generation Video-Action Model for general purpose dexterity, developed in partnership with Black Forest Labs. Late last year we published mimic-video and introduced Video-Action Models (VAM): a new family of robotics foundation models built on top of video generation models. We showed that robot control reduces to visual prediction, and that robot capability is downstream of improvements in video modeling accuracy. The obvious implication was that advances in the video modeling frontier would directly translate to increased capabilities in end-to-end robot learning. FLUX-mimic is that thesis at frontier scale: We've applied our VAM architecture to the strongest video backbone available today, FLUX 3 from Black Forest Labs, and trained it on data from our own robots and wearables. General-purpose dexterity, running on a single GPU on premises. Because the model already understands world dynamics, it needs far fewer demonstrations to learn a new task. This is game-changing for our mission to deploy robots to factory floors, where industrial robot data is scarce and expensive to collect. We're now testing and deploying FLUX-mimic with manufacturing leaders like Audi USA, on complex, multi-step manipulation long considered impossible for conventional automation.

Introducing FLUX-mimic, a next-generation Video-Action Model for general purpose dexterity, developed in partnership with Black Forest Labs. Late last year we published mimic-video and introduced Video-Action Models (VAM): a new family of robotics foundation models built on top of video generation models. We showed that robot control reduces to visual prediction, and that robot capability is downstream of improvements in video modeling accuracy. The obvious implication was that advances in the video modeling frontier would directly translate to increased capabilities in end-to-end robot learning. FLUX-mimic is that thesis at frontier scale: We've applied our VAM architecture to the strongest video backbone available today, FLUX 3 from Black Forest Labs, and trained it on data from our own robots and wearables. General-purpose dexterity, running on a single GPU on premises. Because the model already understands world dynamics, it needs far fewer demonstrations to learn a new task. This is game-changing for our mission to deploy robots to factory floors, where industrial robot data is scarce and expensive to collect. We're now testing and deploying FLUX-mimic with manufacturing leaders like Audi USA, on complex, multi-step manipulation long considered impossible for conventional automation.

mimic

118,128 views • 1 month ago

Excited to release τ0-WM: an open-source unified video-action world model for robotic manipulation. It's a 5B-parameter robotic foundation model trained on 27.3K hours of real-robot teleoperation, UMI-style demonstrations, and egocentric interaction videos.

Excited to release τ0-WM: an open-source unified video-action world model for robotic manipulation. It's a 5B-parameter robotic foundation model trained on 27.3K hours of real-robot teleoperation, UMI-style demonstrations, and egocentric interaction videos.

Jianlan Luo

63,831 views • 3 months ago

In my first week at Generalist, I trained a robot to pour liquids using GEN-1 🤖💧 I wanted to challenge the robot with a non-rigid manipulation task, so liquid felt like the perfect choice. The task involved: - unscrewing the bottle cap - pouring liquid into espresso glasses - rebalancing uneven pours Best of all, the robot was able to complete the task fully autonomously 3 times in a row (out of 3)! Pour-fect 😉 Excited for the journey ahead and grateful to be building alongside such an incredible team!

In my first week at Generalist, I trained a robot to pour liquids using GEN-1 🤖💧 I wanted to challenge the robot with a non-rigid manipulation task, so liquid felt like the perfect choice. The task involved: - unscrewing the bottle cap - pouring liquid into espresso glasses - rebalancing uneven pours Best of all, the robot was able to complete the task fully autonomously 3 times in a row (out of 3)! Pour-fect 😉 Excited for the journey ahead and grateful to be building alongside such an incredible team!

Kylie Ying

40,386 views • 3 months ago

Robots might learn better from video than from language! 📼 Most Vision-Language-Action (VLA) models learn what to do from text, but still struggle with how things move in the real world. That makes them data-hungry and slow to train. mimic video takes a different route. Instead of grounding robot control in text, it grounds it in video, using large pre-trained video models that already capture physical motion and dynamics. The idea is straightforward: let the video model handle “what will happen next,” and let a smaller control model focus only on turning that visual plan into robot actions. The result is big gains in practice. Robots trained this way need 10× less data, converge twice as fast, and perform better on both simulated benchmarks and real bimanual manipulation tasks. If robots can “imagine” motion using video, control becomes a much simpler problem. Shoutout to Jonas Pai, Liam Achenbach, Oier Mees, Elvis Nava and the rest of the team! Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Robots might learn better from video than from language! 📼 Most Vision-Language-Action (VLA) models learn what to do from text, but still struggle with how things move in the real world. That makes them data-hungry and slow to train. mimic video takes a different route. Instead of grounding robot control in text, it grounds it in video, using large pre-trained video models that already capture physical motion and dynamics. The idea is straightforward: let the video model handle “what will happen next,” and let a smaller control model focus only on turning that visual plan into robot actions. The result is big gains in practice. Robots trained this way need 10× less data, converge twice as fast, and perform better on both simulated benchmarks and real bimanual manipulation tasks. If robots can “imagine” motion using video, control becomes a much simpler problem. Shoutout to Jonas Pai, Liam Achenbach, Oier Mees, Elvis Nava and the rest of the team! Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

50,090 views • 8 months ago

A humanoid robot policy trained solely on synthetic data generated by a world model. Research Scientist Joel Jang presents NVIDIA's DreamGen pipeline: ⦿ Post-train the world model Cosmos-Predict2 with a small set of real teleoperation demos. ⦿ Prompt the world model to generate synthetic video data with verbs and scenarios not used in the world model’s post-training. ⦿ Auto-label synthetic video data with action sequences. ⦿ Train robot policies using only synthetic data. That's it. Deploy zero-shot to a real humanoid robot.

A humanoid robot policy trained solely on synthetic data generated by a world model. Research Scientist Joel Jang presents NVIDIA's DreamGen pipeline: ⦿ Post-train the world model Cosmos-Predict2 with a small set of real teleoperation demos. ⦿ Prompt the world model to generate synthetic video data with verbs and scenarios not used in the world model’s post-training. ⦿ Auto-label synthetic video data with action sequences. ⦿ Train robot policies using only synthetic data. That's it. Deploy zero-shot to a real humanoid robot.

The Humanoid Hub

20,968 views • 1 year ago