A humanoid robot policy trained solely on synthetic data generated by a world model. Research Scientist Joel Jang presents NVIDIA's DreamGen pipeline: ⦿ Post-train the world model Cosmos-Predict2 with a small set of real teleoperation demos. ⦿ Prompt the world model to generate synthetic video data with verbs and... scenarios not used in the world model’s post-training. ⦿ Auto-label synthetic video data with action sequences. ⦿ Train robot policies using only synthetic data. That's it. Deploy zero-shot to a real humanoid robot.show more

Mhm

Go ad-free on X with Premium+ It's the highest return on investment you can make.

Companies like 1x that develop their own models will cumulate benefits from in-house efforts plus everything Nvidia comes up with.

Synthetic sim data training is good but it will always fail in real world due to randomness of world will overwhelm them. This deterministic approach will fail in real world , same reason tesla ain,t confident enough to create real self driving cars yet despite so much training

the approach sounds interesting, especially using real data for fine-tuning. balancing synthetic with real-world scenarios is tricky though. curious to see how it performs in practice.

And it directly worked on the real robot. The sim2real pipeline was just magically good at that time. Unfortunately it took us too much time to make it a publication.

XLeRobot 0.2.3 (XLeVR) out! • VR (Quest 3) Whole-body Control • Reads everything: head and hand poses, joysticks, and all the buttons • Minimal dependencies, web-based • Modular, can use it on other robots • Open-source, easy install, play in 20min

What if your robot could catch things mid-air... without breaking or snapping a joint? [📍Bookmark for later] IMA-Catcher is a new framework that lets robots catch nonprehensile, free-falling objects while reducing impact forces and avoiding hardware damage. Here’s what they did: ✅ Matched robot velocity to the object’s for low-impact catching ✅ Used human demos to teach smooth post-catch movements ✅ Minimized joint stress with a smart controller ✅ Generalized from 1D to full 3D catching Without velocity matching? The robot would fail the task completely. Catch the full paper here: 📍 ------------------------------ 🧠 Repost to help someone find the paper they’ve been searching for.

Exactly 2 years ago during #Arche, my thesis supervisor @FJenelten51897 and I deployed this mpc+rl hybrid controller on very complicated collapsed buildings. Two years later its performance is still at the top tier! 😎 Paper: