Introducing CoPa, a novel framework that can autonomously complete various complex robotic manipulation tasks w/o any training: - Make pour-over coffee☕️ - Set up a romantic table💘 - Hammer a nail🔨 🧵👇

CoPa incorporates common sense knowledge embedded within VLMs into robotic manipulation tasks. We demonstrate the complete execution flow of CoPa through *Put flowers into vase* task. (1/N)

Most manipulation tasks can be decomposed into two phases: initial grasp of the object and subsequent motion required to complete the task. Motivated by this observation, we structure our approach into two modules: *Task-Oriented Grasping* and *Task-Aware Motion Planning*. (2/N)

We observe that most manipulation tasks require a part-level, fine-grained physical understanding of objects within the scene. Hence, we design a coarse-to-fine grounding module to identify task-relevant parts. (3/N)

We design *Task-Oriented Grasping* to generate grasp pose. The grasp candidates are generated from the scene point cloud using GraspNet. Concurrently, a grounding module is used to identify the grasping part, allowing us to filter candidates to obtain the final grasp pose. (4/N)

To leverage VLMs for aiding robotic low-level control, it's necessary to design an interface that not only allows VLMs to reason in language but also facilitates robot’s object manipulation. Thus, we propose utilizing spatial constraints as a bridge between VLMs and robots. (5/N)

Boasting a fine-grained physical understanding of scenes, CoPa can generalize to open-world scenarios, handling open-set instructions and objects with minimal prompt engineering and without any training. (6/N)

CoPa can be seamlessly integrated with high-level planning methods (e.g. ViLa) to accomplish complex, long-horizon tasks, such as *Make pour-over coffee*☕️ and *Set up romantic table*💘 (7/N)

For more，check out 👇 Project site: Paper: Amazing work done with @haoxu_huang @lfqirrrrr @yingdong_hu99 @ShengjieWa34067 (8/N)

🧠 Unified Search. Smarter Meetings. Effortless CRM. MightyBot is your AI agent platform for seamless workflows—record meetings, automate CRM updates, and find answers across apps in seconds. 🌟 Focus on what matters. We'll handle the grind.