Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

🤖What if a robot could perform a new task just from a natural language command, with zero demonstrations? Our new work, NovaFlow, makes it possible! We use pre-trained video generative model to create a video of the task, then translate it into a plan for real-world robot execution. 1/6...

106,422 Aufrufe • vor 1 Jahr •via X (Twitter)

34 Kommentare

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

We start by giving a video model a text prompt like "hang the mug." It generates a short clip of a human performing the task, acting as a visual instruction manual. This video provides the commonsense physical knowledge needed for the robot to understand the motion. 🎥 2/6

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

The generated video is 2D, so how does the robot understand 3D space? We lift it! 📏 A monocular video depth estimation model predicts depth for every frame. Then, we calibrate it with the robot's real-world depth camera for precise spatial awareness. 3/6

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

🤔 But how do you bridge the gap between that video and physical control? We introduce "actionable 3D object flow," a novel representation distilled from the video that guides the robot’s motion for complex rigid, articulated, and even deformable objects. 🌟 4/6

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

NovaFlow shows incredible generalization. From rigid blocks to deformable ropes, on both a tabletop arm and a mobile Spot! It adapts to new objects and embodiments without any retraining. It achieves SOTA zero-shot results, even outperforming policies trained on 10-30 demos. 🏆 5/6

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

By leveraging the commonsense knowledge in video models, NovaFlow helps solve the robot data bottleneck. This is a leap towards more general-purpose robots. 🚀 Dive deeper into our work: 📜 Paper: 🌐 Project Website: 👀 P.S. You can check out the 3D generated videos and planned robot trajectories in a live viewer on our website! 6/6

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

This is a collaboration with @ling_feng_sun, @YafeiHuCMU, Duy Ta, Jenny Barry, and @jiahuifu_carol.

Profilbild von Nils
Nilsvor 1 Jahr

Beautiful.

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

Thanks, Nils!

Profilbild von Xiatao Sun
Xiatao Sunvor 1 Jahr

Using video generation models to create task plans from language is clever! The actionable 3D object flow representation seems like the key to making this work across embodiments.

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

Thanks, Xiatao. I believe video models capture the most general form of dynamics, although they can’t be directly understood by robots. Actionable flow enables robots to comprehend the generated videos.

Profilbild von Zhiyang (Frank) Dou
Zhiyang (Frank) Douvor 1 Jahr

Nice work!

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

Thanks, Zhiyang!

Profilbild von Bozheng Li
Bozheng Livor 1 Jahr

great work!

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

Thanks, Bozheng!

Profilbild von Balveer Singh
Balveer Singhvor 1 Jahr

Wow, this is such a cool approach.

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

Thanks🙏

Profilbild von Ted Xiao
Ted Xiaovor 1 Jahr

Nice work!

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

Thanks, Ted! 🙏

Profilbild von Sam Green
Sam Greenvor 1 Jahr

Laying the groundwork for Soma

Profilbild von Atharva Sawant
Atharva Sawantvor 1 Jahr

So cool

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

Thanks

Profilbild von Cryptlesh
Cryptleshvor 1 Jahr

Wow. It’s like imagining whole take on head and then implementing it like human break does. Kind of.

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

Exactly, a video model serves as the imagination capabilities of human beings.

Profilbild von Bercan
Bercanvor 1 Jahr

Great work. Can you try on these examples in mcap.

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

Thanks for the great suggestions! Egocentric data is more challenging, and we’re still working on it!

Profilbild von Bercan
Bercanvor 1 Jahr

Happy to work together we are also working on real2sim

Profilbild von Stanley Wei
Stanley Weivor 1 Jahr

That will save a lot of time. Hope you post some cool demos.

Profilbild von BlockChan
BlockChanvor 11 Monaten

holy shit i personaly owned name for a while and have no idea what to build

Profilbild von towerofshadow
towerofshadowvor 1 Jahr

is it more effective to train robots using human actions that are tracked using actionable flow, or to teleoperate robots into direct movement data?

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

NovaFlow is object-centric and embodiment-agnostic. So it should work the same for both human and robot data. In practice, we find video generative models work better with generating human data.

Profilbild von Danny Ng
Danny Ngvor 1 Jahr

I wonder how long it takes to perform one task?

Profilbild von Hongyu Li
Hongyu Livor 1 Jahr

It depends on the choice of video model. For Veo, it takes around 2-3min for the entire pipeline.

Profilbild von Urs Gehrig
Urs Gehrigvor 1 Jahr

Zero shot Robotic AI manipulation.

Profilbild von Mariano Phielipp
Mariano Phielippvor 8 Monaten

Interesting work. How you see the work of Florence, Manuelli, Redrake in the context of this work Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation Florence,Manuelli, Russ Tedrake?

Ähnliche Videos

The robot flipped a pancake nobody taught it! 🥞 Skild AI team assumed pancake flipping had to be somewhere in the training data. So they searched. Millions of hours of pre-training data. Nothing. S1 inferred the whole task from a single human demonstration. That's their new general robot model, built as an in-context learner from the ground up. Every new robot task today starts with days of teleoperation and a fine-tuning run on a specialist policy. S1 skips all of it. Much like a language model, it never updates its weights to learn a new task. The demonstration enters the context window, and the policy uses it to decide what to do next. → Ten-minute tasks it was never trained on, composed from primitives learned in pre-training: a new style of coffee, potting a plant, frying pancakes. → Soil and pots arrived at their office at 8:54 PM. The robot was running the task autonomously by 9:27 PM. → Slide objects away mid-reach, swap them, change the lighting, it still finishes. → The prompt waters a plant with a watering can, but only a cup is available. It uses the cup. It doesn't rigidly replay what it saw, but it recovers from its own errors, and sometimes executes with more precision than the demonstrator, when the human fumbles an egg and makes a mess, S1 performs the same step cleanly. The demonstration is a specification of the goal, and not a trajectory to copy. On unseen tasks after 100K hours of pre-training: language-prompted VLAs reach 9%. Their new model reaches 66%. It's already deploying with industrial partners, with a wider rollout over the coming months. Congrats Deepak Pathak and team behind this! 😮‍💨 🔗 Link to their latest blog: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

11,406 Aufrufe • vor 2 Monaten