What structural task representation enables multi-stage, in-the-wild, bimanual, reactive manipulation? Introducing ReKep: LVM to label keypoints & VLM to write keypoint-based constraints, solve w/ optimization for diverse tasks, w/o task-specific training or env models. 🧵👇

Relational Keypoint Constraints (ReKep) represent tasks as seq of keypoint relations. Eg in pouring task: pull together gripper kp & handle kp -> keep handle & spout kp at same height (avoid spillage) -> align spout & cup kp -> handle & spout kp form a tilting angle to pour. 2/N

ReKeps are Python functions mapping kp to costs w/ NumPy operations, specifying relations b/w robot, obj, and obj parts. While each kp has only (x,y,z), multiple kp can specify SO(3) rotations, vectors, surfaces, volumes to capture rich geometric structures in manipulation. 3/N

By sequencing multiple ReKeps, our framework organically integrates high-level task planning with dense low-level actions as a unified continuous optimization problem. With tracked keypoints, this enables rapid backtracking & replanning behaviors both within/across stages. 4/N

ReKep can also be fully automated w/ foundation models for in-the-wild task execution. We use large vision models (SAM+DINOv2) to identify keypoints, overlay on image, and prompt VLM (GPT-4o) to write a seq of ReKep constraints based on the task instruction. 5/N

We test ReKep on two setups – in-the-wild and bimanual. It can perform diverse 6-12 DoF tasks w/ a perception-action loop at 10 Hz. It can also fold diff. clothes with diff. (human-like) strategies. And entire pipeline does not require task-specific training or env models. 6/N

ReKep is implemented upon common packages like SciPy, and the code has also been open-sourced! The code runs on BEHAVIOR ( a large-scale benchmark with diverse scenes and objects, so you can easily try ReKep without setting up a real robot. 7/N

Project website: Walkthrough video: Paper: Code: Work done together with @chenwang_j, @YunzhuLiYZ, @RuohanZhang76, and @drfeifei. (N/N)

Congrats, @wenlong_huang ! This is impressive!

Thanks Huazhe!!

Cool work @wenlong_huang 👏

Thanks Karol!!