Excited to finally share Pixel: a full-stack robot learning project Tony Tao @ CoRL and I worked on earlier this year! We’re releasing two papers: 1. Pixel: A low-cost mobile manipulator, co-designed with its UMI system and learning stack. 2. Pixel-UMI: Improved UMI pipeline for learning environment-aware, whole-body manipulation (1/N)

In this thread, I will mainly cover the Pixel-UMI First, check out @_tonytao_'s thread for more on the robot and UMI hardware we co-designed! (2/N)

In Pixel-UMI, we first introduce Environment-Aware Retargeting (EAR). We reconstruct the scene offline from each demonstration, then use the reconstruction to convert UMI human data into collision-aware whole-body robot trajectories. (3/N)

We also introduce several tricks that help with learning whole-body policies from UMI: from assigning actions that better account for real robot dynamics to training-time techniques that reduce the effects of human–robot kinematic mismatches. Check out these policies trained from ~200 UMI demos per task! (4/N)

Finally, we use DAgger to improve long-horizon, contact-rich manipulation. With two rounds of 60 corrected rollouts per task, we substantially improve performance on challenging bimanual tasks like folding a box and hanging a jacket. (5/N)

This work was done @CMU_Robotics with co-lead @_tonytao_ as well as @deepakpathak and @rsalakhu. Website: Pixel Paper: Pixel-UMI Paper: (6/N)

@_tonytao_ I love how community’s idea come together around the same time. Super impressive work! We just released our open source bimanual robot MABEL too lol Pixel should meet MABEL

@_tonytao_ 🫡

@_tonytao_ very dextrous! this is great

@_tonytao_ Awesome work Jason!

@_tonytao_ Let’s go!!!!

@_tonytao_ Dude. Awesome.

@_tonytao_ Very cool!

@_tonytao_ Thank you for sharing both papers — co-designing the UMI stack with the low-cost arm seems like the key decision. I assume the environment-aware whole-body results in the Pixel-UMI paper are where the gain shows up most.

@_tonytao_ this is great mann! quick question does pixel umi capture base motion alongside arm trajectories??

@_tonytao_ The VR headset provides head tracking wrt the world, and we derive base motion from it via retargeting

@_tonytao_ Reproducible hardware costs would help labs compare setups.

@_tonytao_ the DoF ablation is the most useful part for anyone building one: dropping the single torso-pitch joint cuts task progress to 47%, worse than losing the whole 430mm lift (71%). one pitch joint buying more than a lift is a great takeaway for low-cost mobile manipulators

@_tonytao_ incredible

@_tonytao_ Wish I could hire this @JasonJZLiu guy 😀 , so cracked.

@_tonytao_ Ben my goat!!

@_tonytao_ Teaching a robot by wearing its grippers and just doing the chore yourself is the real unlock. Data was always the bottleneck, not the hardware. Under $15k plus open source means the next breakthrough can come out of a garage, not just a megalab.

@_tonytao_ Great job! Love the full-stack approach and also the design language 😎

@_tonytao_ very nice!

@_tonytao_ Now I can really be like Tony Stark, having a robot start my vinyl when I’m working

@_tonytao_ Low-cost arm plus a UMI learning stack is the right way in. How many demos per task before it works reliably?

@_tonytao_ Super cool work!!

@_tonytao_ Can the robot learn to tickle me

@_tonytao_ hi what lifting column are you guys using?

@_tonytao_ but we swapped out the motor and transmission to actuate it to achieve higher speed. Check the system paper for more details

@_tonytao_ Les gooo

@_tonytao_ The policy still chains the tasks

@_tonytao_ A low-cost mobile manipulator co-designed with its data pipeline is a useful combo. In real facilities the gap usually shows up in recovery: what does Pixel do after a failed grasp or a blocked path, retry on its own or hand off to a person?

@_tonytao_ What's the actual dollar cost of the low-cost platform?

Low-cost + open research is how robotics actually scales. When a mobile manipulator stops costing as much as a car, the bottleneck shifts from hardware to data. Whole-body manipulation from a UMI-style pipeline feels like the right bet. Congrats on the release! 🤖 What's the hardest task Pixel still fails at?

@_tonytao_ Using scene geometry in the retargeting is the part I'd want to try. How much UMI data got thrown out for collisions before that step existed?

@_tonytao_ if pixel needs daily recalibration, low hardware cost won't translate to low operating cost

@_tonytao_ Bro why do you guys call this "pixel"?