🤖What if a robot could perform a new task just from a natural language command, with zero…
106.422 görüntüleme
34 Yorum
Yorumlar X (Twitter) üzerindeki orijinal gönderiden alınmıştır

@Hongyu_Lii
We start by giving a video model a text prompt like "hang the mug." It generates a short clip of a human performing the task, acting as a visual instruction manual. This video provides the commonsense physical knowledge needed for the robot to understand the motion. 🎥 2/6
18 beğeni

@Hongyu_Lii
The generated video is 2D, so how does the robot understand 3D space? We lift it! 📏 A monocular video depth estimation model predicts depth for every frame. Then, we calibrate it with the robot's real-world depth camera for precise spatial awareness. 3/6
15 beğeni

@Hongyu_Lii
🤔 But how do you bridge the gap between that video and physical control? We introduce "actionable 3D object flow," a novel representation distilled from the video that guides the robot’s motion for complex rigid, articulated, and even deformable objects. 🌟 4/6
18 beğeni

@Hongyu_Lii
NovaFlow shows incredible generalization. From rigid blocks to deformable ropes, on both a tabletop arm and a mobile Spot! It adapts to new objects and embodiments without any retraining. It achieves SOTA zero-shot results, even outperforming policies trained on 10-30 demos. 🏆 5/6
15 beğeni

@Hongyu_Lii
By leveraging the commonsense knowledge in video models, NovaFlow helps solve the robot data bottleneck. This is a leap towards more general-purpose robots. 🚀 Dive deeper into our work: 📜 Paper: 🌐 Project Website: 👀 P.S. You can check out the 3D generated videos and planned robot trajectories in a live viewer on our website! 6/6
28 beğeni

@Hongyu_Lii
This is a collaboration with @ling_feng_sun, @YafeiHuCMU, Duy Ta, Jenny Barry, and @jiahuifu_carol.
11 beğeni

@broodsugar
Beautiful.
4 beğeni

@Hongyu_Lii
Thanks, Nils!

@xiatao_sun
Using video generation models to create task plans from language is clever! The actionable 3D object flow representation seems like the key to making this work across embodiments.
2 beğeni

@Hongyu_Lii
Thanks, Xiatao. I believe video models capture the most general form of dynamics, although they can’t be directly understood by robots. Actionable flow enables robots to comprehend the generated videos.
1 beğeni

@frankzydou
Nice work!
1 beğeni

@Hongyu_Lii
Thanks, Zhiyang!
1 beğeni

@PLB4704
great work!
1 beğeni

@Hongyu_Lii
Thanks, Bozheng!

@balveersinghyt
Wow, this is such a cool approach.
1 beğeni

@Hongyu_Lii
Thanks🙏
1 beğeni

@xiao_ted
Nice work!
1 beğeni

@Hongyu_Lii
Thanks, Ted! 🙏
2 beğeni

@0xsamgreen
Laying the groundwork for Soma
1 beğeni

@AtharvaSaw41799
So cool
1 beğeni







