Announcing DreamDojo: our open-source, interactive world model that takes robot motor controls and generates the future in pixels. No engine, no meshes, no hand-authored dynamics. It's Simulation 2.0. Time for robotics to take the bitter lesson pill. Real-world robot learning is bottlenecked by time, wear, safety, and resets. If... we want Physical AI to move at pretraining speed, we need a simulator that adapts to pretraining scale with as little human engineering as possible. Our key insights: (1) human egocentric videos are a scalable source of first-person physics; (2) latent actions make them "robot-readable" across different hardware; (3) real-time inference unlocks live teleop, policy eval, and test-time planning *inside* a dream. We pre-train on 44K hours of human videos: cheap, abundant, and collected with zero robot-in-the-loop. Humans have already explored the combinatorics: we grasp, pour, fold, assemble, fail, retry—across cluttered scenes, shifting viewpoints, changing light, and hour-long task chains—at a scale no robot fleet could match. The missing piece: these videos have no action labels. So we introduce latent actions: a unified representation inferred directly from videos that captures "what changed between world states" without knowing the underlying hardware. This lets us train on any first-person video as if it came with motor commands attached. As a result, DreamDojo generalizes zero-shot to objects and environments never seen in any robot training set, because humans saw them first. Next, we post-train onto each robot to fit its specific hardware. Think of it as separating "how the world looks and behaves" from "how this particular robot actuates." The base model follows the general physical rules, then "snaps onto" the robot's unique mechanics. It's kind of like loading a new character and scene assets into Unreal Engine, but done through gradient descent and generalizes far beyond the post-training dataset. A world simulator is only useful if it runs fast enough to close the loop. We train a real-time version of DreamDojo that runs at 10 FPS, stable for over a minute of continuous rollout. This unlocks exciting possibilities: - Live teleoperation *inside* a dream. Connect a VR controller, stream actions into DreamDojo, and teleop a virtual robot in real time. We demo this on Unitree G1 with a PICO headset and one RTX 5090. - Policy evaluation. You can benchmark a policy checkpoint in DreamDojo instead of the real world. The simulated success rates strongly correlate with real-world results - accurate enough to rank checkpoints without burning a single motor. - Model-based planning. Sample multiple action proposals → simulate them all in parallel → pick the best future. Gains +17% real-world success out of the box on a fruit packing task. We open-source everything!! Weights, code, post-training dataset, eval set, and whitepaper with tons of details to reproduce. DreamDojo is based on NVIDIA Cosmos, which is open-weight too. 2026 is the year of World Models for physical AI. We want you to build with us. Happy scaling! Links in thread:show more

- Project website: - Paper: - Code repo and model ckpts: This is a huge team work at NVIDIA. All credits go to the wonderful teams who poured their hearts into it!

Check out @ShenyuanGao's technical deep dive:

We are working on bringing DreamDojo workflow to the Worlds Hack! @DrJimFan Checkout @GetSoloTech at the upcoming GDC X GTC Hackathon in San Francisco!

were actually going to have a dream dojo workflow in our upcoming hackathon :)

Nice work! Congratulations!

Thanks Shane!!

Super excited to build with this!

this is exactly what robotics needed - learning from human videos at scale instead of expensive robot data is genius, can't wait to see what people build with this

This is the ImageNet moment for robotics. Separating world dynamics from robot actuation & learning from egocentric human video is exactly how Physical AI scales. Real time rollouts, teleop in dreams, a& checkpoint ranking without hardware wear… Simulation 2.0 is real.

Really nice.

🙏🙏

DreamDojo’s shift from hand-authored dynamics to learning world models straight from human egocentric videos is a really exciting move for simulation. The idea of latent actions is a clever bridge.

Wow this is so cool!

Very nice work. Can DreamDojo be used as a egocentric human hand world model? In other words, is a human hand a possible robot post-training setting?

this is what's missing from web agents too. current browser agents parse the DOM, pick an action, get surprised by the result. no internal model of what the page looks like after clicking. world models for UI would change everything.

Congratulations!

Nice!

OSS ftw!!

Super cool!! Congrats!

No more sim2real gap, this is just real2real!

This is a huge step for robotics simulation. Learning world dynamics directly from data instead of hand-crafting physics engines feels like the same paradigm shift we saw with LLMs. Excited to dig into the paper.

no engine, no meshes, no hand-authored dynamics. if this actually generalizes to new scenarios robotics simulation is about to get way cheaper

woow

check dms rq doctor

World models are becoming robotics’ compiler. 44k hours of human POV video, latent actions, then you snap it to a robot. The killer question is calibration. How do you spot when the dream is confidently wrong in contact-rich edge cases?

how does it handle the latency between prediction and actual motor feedback? that's where most sim-to-real breaks down imo

The real test is how well these world models transfer when the robot meets the real world.

World models are the bridge between AI understanding physics and AI acting in physics. DreamDojo generating futures from motor controls without a physics engine is wild. This is how we get to robots that actually generalize instead of memorizing.

Generative world models replacing physics engines is the bitter lesson landing in sim-to-real. If the model learns dynamics from pixels instead of hand-crafted rules, the sim-to-real gap becomes a data problem, not a domain problem. That’s the unlock.

World models are promising. The next challenge is proving they transfer reliably to the real world.

Asking for a friend, when diaper change for babies by robots?

Impressive , congrats !

This could really shake up how robots learn. Curious to see it in action.

DreamDojo looks insane, Jim — turning human videos into a real-time physics simulator with zero engine? This is exactly the kind of world model embodied AI has been waiting for. The teleop inside the dream is next level!

DreamDojo is a massive validation of the human-data thesis. 44k hours of egocentric video → robot transfer without physics engines. The logical next step: make that 44k → 4.4M. That's the crowdsourcing unlock. What bottleneck do you see on the contributor side at that scale?

Congrats Jim and team! Super curious to see how well it generalizes to failures/near failures - the most important thing for policy improvement

6DyYEaKni12VMwdKAAttJ3hSc2dVinu9FcyMHyZ1pump