Can we learn whole-body mobile manipulation directly from human demonstrations? Introducing Whole-Body Mobile Manipulation Interface (HoMMI) Egocentric + UMI, 0 teleop -> bimanual & whole-body manipulation, long-horizon navigation, active perception

We extend UMI with 𝗲𝗴𝗼𝗰𝗲𝗻𝘁𝗿𝗶𝗰 𝘀𝗲𝗻𝘀𝗶𝗻𝗴 to enable scalable 𝙢𝙤𝙗𝙞𝙡𝙚 manipulation with 𝙖𝙘𝙩𝙞𝙫𝙚 𝙥𝙚𝙧𝙘𝙚𝙥𝙩𝙞𝙤𝙣 – capabilities that cannot be achieved with the original UMI with only wrist sensing. (1/n)

However, the new egocentric view creates a substantial embodiment gap in both observation and action space. Visual gap: Different egocentric viewpoints & arms and torso appearances. Kinematic gap: The robot head is lower than the human, and the robot neck only has 2-DoF. (2/n)

To bridge the observation gap, we use a 3D visual representation for egocentric observations. This allows us to use embodiment-agnostic coordinate frames (i.e., end-effector frame), and remove embodiment-specific observations (e.g., demonstrator’s arms and body). (3/n)

To bridge the kinematic gap, we relax the head action constraint by representing the robot gaze as a “3D look-at point”. This allows effective active perception for gathering task-relevant information without over-constraining the robot to mimic human head motions exactly. (4/n)

We design a whole-body controller that can coordinate whole-body motions to precisely track end-effector trajectories for accurate manipulation, while respecting the unique constraints in a bimanual mobile robot system for stable and safe motions. (5/n)

Finding 1: Wrist-only sensing under-observes global task context and bimanual coordination. On the contrary, HoMMI augments UMI with egocentric sensing, providing global context and active perception behaviors crucial for mobile manipulation. (6/n)

Finding 2: Head-mounted camera alone is insufficient for manipulation. HoMMI combines head camera views with wrist camera views, which provide essential local visual servoing and contact cues for fine-grained manipulation. (7/n)

Finding 3: Naively adding egocentric RGB and regressing 6-DoF head motion hurts policy performance under embodiment mismatch. (8/n)

Finding 4: Active head control effectively gathers task-relevant information and maintains policy observability. (9/n)

Paper: Website: Code : Joint work with my amazing collaborators at @StanfordAILab and @ToyotaResearch : @gsanpark @DoubleHan07 Eric Cousineau, Aditya Bhat, Jose Barreiros, @Dian_Wang_ @SongShuran (10/n)

Congrats!

Great Work!

Thank you!

The long-horizon navigation piece is especially interesting!

amazing work!

Thank you!

@StanfordAILab This is seriously impressive learning full-body manipulation from human demos is a huge step forward 🤯 The combination of perception, navigation, and coordination here feels like a big leap toward real-world robotics applications. Excited to see where this goes 🚀

Cool work!

@JinghanSun3 Thank you!

HoMMI looks impressive — learning whole-body mobile manipulation directly from egocentric human demos is the right direction! 🔥🤖 Tradeye is doing the same in the real world: narrated egocentric POV footage from actual HVAC, plumbing, and electrical installs in tight, messy, occupied homes. This gives robots the hard, unstructured bimanual + whole-body coordination they need beyond lab tables. Would love to explore collaboration or data sharing for more realistic home/service tasks. Happy to connect! ⚡ @danfei_xu @elonmusk @Tesla_AI @NVIDIARobotics @Figure_robot #RoboticsData #PhysicalAI #HoMMI

Sounds great!

Thanks so much — really glad it resonates! 😊 Name is Justin. I've been capturing narrated egocentric clips from actual HVAC, plumbing, and electrical installs in tight, messy homes. Haha I'd love to explore collaboration or data sharing with the HoMMI team! Happy to send over some sample clips or hop on a quick call whenever it works for you. Thanks again — looking forward to hearing from you! ⚡🤖

Using a mobile base instead of two legs is a great alternative. This type of base allows for easier mechanical navigation on a flat surface. Did you use omnidirectional wheels for movement?

Yes! The robot’s base is omnidirectional

The idea of robots learning directly from human demonstrations reminds me of how parents teach their children, not through instructions but through example. It feels so surprisingly "human." Curious to learn about how you came up with the idea!