What representation enables open-world robot manipulation from generated videos? Introducing Dream2Flow, our recent work that bridges video generation and robot control with 3D object flow. Stanford University #ICRA2026 1/N

Robot manipulation is about inducing changes in an environment through actions. We observe that video models (e.g., Veo) excel at producing plausible object motions from an in-the-wild image and language instructions. Intriguingly, these motions are more physically realistic when the actor is human rather than robot, likely because the internet contains far more human interaction data than robot data.

But how do we turn those generated videos into low-level robot actions? This is a nuanced question beyond simple retargeting, because strategies taken by a human may not work on a robot (consider picking up and using scissors).

We propose Dream2Flow, which uses 3D object flow to separate what should happen in the scene from how a robot should realize it. We extract this flow from generated videos using off-the-shelf vision models (SAM2, SpatialTrackerV2, CoTrackerV3), then use it as a shared objective for both trajectory optimization and reinforcement learning.

Dream2Flow can perform a range of in-the-wild tasks zero-shot with trajectory optimization, including rigid, articulated, and deformable manipulation. The robot plans by asking a counterfactual question using a dynamics model: if I take this action, will the scene evolve toward the desired 3D object flow? Dream2Flow supports different dynamics models, from rigid-grasp dynamics to learned models (e.g., PointWorld All results here use no expert demonstrations.

Using as rewards for RL, Dream2Flow enables different embodiments to discover emergent behaviors that achieve the same effect. To open a door, notice how: - the quadruped leverages base motions to extend its reach - the humanoid pushes the handle with its palm instead of fingers. Dream2Flow unifies these behaviors through a shared task interface and unifies model-free and model-based methods around a shared tracking goal.

By leveraging purely off-the-shelf video models, Dream2Flow allows generalization to different object instances, backgrounds, and camera viewpoints. It is also surprisingly steerable: different language instructions in the same scene can induce different desired behaviors.

There are many exciting works in this space, including RIGVID @shivanshpatel35 ( and NovaFlow @Hongyu_Lii ( Make sure to check them out if you haven’t! World modeling encodes rich priors about not only environment dynamics but also behaviors within it. It is immensely useful for robotics, yet we are only scratching the surface of understanding it.

The project was led by @KDharmarajan123 and has been a year in the making, together with @jiajunwu_cs , @drfeifei , and @RuohanZhang76 . Also, @KDharmarajan123 will be joining UC Berkeley as a PhD student this fall! Website: Paper: Code:

@Stanford Nice work @wenlong_huang !

@Stanford Thanks Jesse, though all credits go to @KDharmarajan123 ! Looking forward to more future works from you!

@Stanford Do you guys have any plans for haptic feedback, force control, or sub-millimeter precision actions?

@Stanford 3D object flow as the bridge between video gen and robot control is a clever framing. Will check the paper!

@Stanford This is really awesome

@Stanford Really cool work - congrats!

@Stanford 3D object flow feels like the right glue between video priors and control.

@Stanford robots finally joining the bull run? wagmi.

@Stanford Looks cool!!

@Stanford Why are you still waving fridges at robots like they're ice-skating partners?! 🌨️ Switch to Dream2Flow for smooth, automated flings!

@Stanford Video is the map, but flow is the territory. Most will miss this distinction.