Загрузка видео...

Не удалось загрузить видео

На главную

What representation enables open-world robot manipulation from generated videos? Introducing Dream2Flow, our recent work that bridges video generation and robot control with 3D object flow. Stanford University #ICRA2026 1/N

109,366 просмотров • 8 месяцев назад •via X (Twitter)

Комментарии: 19

Фото профиля Wenlong Huang
Wenlong Huang8 месяцев назад

Robot manipulation is about inducing changes in an environment through actions. We observe that video models (e.g., Veo) excel at producing plausible object motions from an in-the-wild image and language instructions. Intriguingly, these motions are more physically realistic when the actor is human rather than robot, likely because the internet contains far more human interaction data than robot data.

Фото профиля Wenlong Huang
Wenlong Huang8 месяцев назад

But how do we turn those generated videos into low-level robot actions? This is a nuanced question beyond simple retargeting, because strategies taken by a human may not work on a robot (consider picking up and using scissors).

Фото профиля Wenlong Huang
Wenlong Huang8 месяцев назад

We propose Dream2Flow, which uses 3D object flow to separate what should happen in the scene from how a robot should realize it. We extract this flow from generated videos using off-the-shelf vision models (SAM2, SpatialTrackerV2, CoTrackerV3), then use it as a shared objective for both trajectory optimization and reinforcement learning.

Фото профиля Wenlong Huang
Wenlong Huang8 месяцев назад

Dream2Flow can perform a range of in-the-wild tasks zero-shot with trajectory optimization, including rigid, articulated, and deformable manipulation. The robot plans by asking a counterfactual question using a dynamics model: if I take this action, will the scene evolve toward the desired 3D object flow? Dream2Flow supports different dynamics models, from rigid-grasp dynamics to learned models (e.g., PointWorld All results here use no expert demonstrations.

Фото профиля Wenlong Huang
Wenlong Huang8 месяцев назад

Using as rewards for RL, Dream2Flow enables different embodiments to discover emergent behaviors that achieve the same effect. To open a door, notice how: - the quadruped leverages base motions to extend its reach - the humanoid pushes the handle with its palm instead of fingers. Dream2Flow unifies these behaviors through a shared task interface and unifies model-free and model-based methods around a shared tracking goal.

Фото профиля Wenlong Huang
Wenlong Huang8 месяцев назад

By leveraging purely off-the-shelf video models, Dream2Flow allows generalization to different object instances, backgrounds, and camera viewpoints. It is also surprisingly steerable: different language instructions in the same scene can induce different desired behaviors.

Фото профиля Wenlong Huang
Wenlong Huang8 месяцев назад

There are many exciting works in this space, including RIGVID @shivanshpatel35 ( and NovaFlow @Hongyu_Lii ( Make sure to check them out if you haven’t! World modeling encodes rich priors about not only environment dynamics but also behaviors within it. It is immensely useful for robotics, yet we are only scratching the surface of understanding it.

Фото профиля Wenlong Huang
Wenlong Huang8 месяцев назад

The project was led by @KDharmarajan123 and has been a year in the making, together with @jiajunwu_cs , @drfeifei , and @RuohanZhang76 . Also, @KDharmarajan123 will be joining UC Berkeley as a PhD student this fall! Website: Paper: Code:

Фото профиля Jesse Zhang
Jesse Zhang8 месяцев назад

@Stanford Nice work @wenlong_huang !

Фото профиля Wenlong Huang
Wenlong Huang8 месяцев назад

@Stanford Thanks Jesse, though all credits go to @KDharmarajan123 ! Looking forward to more future works from you!

Фото профиля Chukwudalu Dumebi-Kachikwu
Chukwudalu Dumebi-Kachikwu8 месяцев назад

@Stanford Do you guys have any plans for haptic feedback, force control, or sub-millimeter precision actions?

Фото профиля Louis-François Bouchard 🎥🤖
Louis-François Bouchard 🎥🤖8 месяцев назад

@Stanford 3D object flow as the bridge between video gen and robot control is a clever framing. Will check the paper!

Фото профиля Chukwudalu Dumebi-Kachikwu
Chukwudalu Dumebi-Kachikwu8 месяцев назад

@Stanford This is really awesome

Фото профиля Kanchana Ranasinghe
Kanchana Ranasinghe8 месяцев назад

@Stanford Really cool work - congrats!

Фото профиля Thomas Tao
Thomas Tao8 месяцев назад

@Stanford 3D object flow feels like the right glue between video priors and control.

Фото профиля Chain Alpha
Chain Alpha8 месяцев назад

@Stanford robots finally joining the bull run? wagmi.

Фото профиля Chris
Chris8 месяцев назад

@Stanford Looks cool!!

Фото профиля LightShift Studio
LightShift Studio8 месяцев назад

@Stanford Why are you still waving fridges at robots like they're ice-skating partners?! 🌨️ Switch to Dream2Flow for smooth, automated flings!

Фото профиля Bika.ai
Bika.ai8 месяцев назад

@Stanford Video is the map, but flow is the territory. Most will miss this distinction.

Похожие видео