Gezinmeyi atla
🤖What if a robot could perform a new task just from a natural language command, with zero…
0:00 / 0:27

🤖What if a robot could perform a new task just from a natural language command, with zero…

Hongyu Li

@Hongyu_Lii1,9 B takipçi

X profili
556 beğeniİndirOrijinal gönderi

106.422 görüntüleme

🤖What if a robot could perform a new task just from a natural language command, with zero demonstrations? Our new work, NovaFlow, makes it possible! We use pre-trained video generative model to create a video of the task, then translate it into a plan for real-world robot execution. 1/6 #Robotics #AI #ZeroShot #Manipulation

34 Yorum

Yorumlar X (Twitter) üzerindeki orijinal gönderiden alınmıştır

  • Hongyu Li profil fotoğrafı

    @Hongyu_Lii

    We start by giving a video model a text prompt like "hang the mug." It generates a short clip of a human performing the task, acting as a visual instruction manual. This video provides the commonsense physical knowledge needed for the robot to understand the motion. 🎥 2/6

    18 beğeni

  • Hongyu Li profil fotoğrafı

    @Hongyu_Lii

    The generated video is 2D, so how does the robot understand 3D space? We lift it! 📏 A monocular video depth estimation model predicts depth for every frame. Then, we calibrate it with the robot's real-world depth camera for precise spatial awareness. 3/6

    15 beğeni

  • Hongyu Li profil fotoğrafı

    @Hongyu_Lii

    🤔 But how do you bridge the gap between that video and physical control? We introduce "actionable 3D object flow," a novel representation distilled from the video that guides the robot’s motion for complex rigid, articulated, and even deformable objects. 🌟 4/6

    18 beğeni

  • Hongyu Li profil fotoğrafı

    @Hongyu_Lii

    NovaFlow shows incredible generalization. From rigid blocks to deformable ropes, on both a tabletop arm and a mobile Spot! It adapts to new objects and embodiments without any retraining. It achieves SOTA zero-shot results, even outperforming policies trained on 10-30 demos. 🏆 5/6

    15 beğeni

  • Hongyu Li profil fotoğrafı

    @Hongyu_Lii

    By leveraging the commonsense knowledge in video models, NovaFlow helps solve the robot data bottleneck. This is a leap towards more general-purpose robots. 🚀 Dive deeper into our work: 📜 Paper: 🌐 Project Website: 👀 P.S. You can check out the 3D generated videos and planned robot trajectories in a live viewer on our website! 6/6

    28 beğeni

  • Hongyu Li profil fotoğrafı

    @Hongyu_Lii

    This is a collaboration with @ling_feng_sun, @YafeiHuCMU, Duy Ta, Jenny Barry, and @jiahuifu_carol.

    11 beğeni

  • Nils profil fotoğrafı

    @broodsugar

    Beautiful.

    4 beğeni

  • Hongyu Li profil fotoğrafı

    @Hongyu_Lii

    Thanks, Nils!

  • Xiatao Sun profil fotoğrafı

    @xiatao_sun

    Using video generation models to create task plans from language is clever! The actionable 3D object flow representation seems like the key to making this work across embodiments.

    2 beğeni

  • Hongyu Li profil fotoğrafı

    @Hongyu_Lii

    Thanks, Xiatao. I believe video models capture the most general form of dynamics, although they can’t be directly understood by robots. Actionable flow enables robots to comprehend the generated videos.

    1 beğeni

  • Zhiyang (Frank) Dou profil fotoğrafı

    @frankzydou

    Nice work!

    1 beğeni

  • Hongyu Li profil fotoğrafı

    @Hongyu_Lii

    Thanks, Zhiyang!

    1 beğeni

  • Bozheng Li profil fotoğrafı

    @PLB4704

    great work!

    1 beğeni

  • Hongyu Li profil fotoğrafı

    @Hongyu_Lii

    Thanks, Bozheng!

  • Balveer Singh profil fotoğrafı

    @balveersinghyt

    Wow, this is such a cool approach.

    1 beğeni

  • Hongyu Li profil fotoğrafı

    @Hongyu_Lii

    Thanks🙏

    1 beğeni

  • Ted Xiao profil fotoğrafı

    @xiao_ted

    Nice work!

    1 beğeni

  • Hongyu Li profil fotoğrafı

    @Hongyu_Lii

    Thanks, Ted! 🙏

    2 beğeni

  • Sam Green profil fotoğrafı

    @0xsamgreen

    Laying the groundwork for Soma

    1 beğeni

  • Atharva Sawant profil fotoğrafı

    @AtharvaSaw41799

    So cool

    1 beğeni