Excited to finally share Pixel: a full-stack robot learning project Tony Tao and I worked on earlier…
@JasonJZLiu2.7K followers
73,113 views
37 Comments
Comments from the original post on X (Twitter)

@JasonJZLiu
In this thread, I will mainly cover the Pixel-UMI First, check out @_tonytao_'s thread for more on the robot and UMI hardware we co-designed! (2/N)
16 likes

@JasonJZLiu
In Pixel-UMI, we first introduce Environment-Aware Retargeting (EAR). We reconstruct the scene offline from each demonstration, then use the reconstruction to convert UMI human data into collision-aware whole-body robot trajectories. (3/N)
13 likes

@JasonJZLiu
We also introduce several tricks that help with learning whole-body policies from UMI: from assigning actions that better account for real robot dynamics to training-time techniques that reduce the effects of human–robot kinematic mismatches. Check out these policies trained from ~200 UMI demos per task! (4/N)
14 likes

@JasonJZLiu
Finally, we use DAgger to improve long-horizon, contact-rich manipulation. With two rounds of 60 corrected rollouts per task, we substantially improve performance on challenging bimanual tasks like folding a box and hanging a jacket. (5/N)
15 likes

@JasonJZLiu
This work was done @CMU_Robotics with co-lead @_tonytao_ as well as @deepakpathak and @rsalakhu. Website: Pixel Paper: Pixel-UMI Paper: (6/N)
14 likes

@thejerrycheng
@_tonytao_ I love how community’s idea come together around the same time. Super impressive work! We just released our open source bimanual robot MABEL too lol Pixel should meet MABEL
4 likes

@JasonJZLiu
@_tonytao_ 🫡

@khushii_w
@_tonytao_ very dextrous! this is great
3 likes

@alfie14x
@_tonytao_ Awesome work Jason!
3 likes

@davidrmcall
@_tonytao_ Let’s go!!!!
3 likes

@lumbogg
@_tonytao_ Dude. Awesome.
2 likes

@praneethsamine
@_tonytao_ Very cool!
2 likes

@Imfulao
@_tonytao_ Thank you for sharing both papers — co-designing the UMI stack with the low-cost arm seems like the key decision. I assume the environment-aware whole-body results in the Pixel-UMI paper are where the gain shows up most.
2 likes

@Van0SS
@_tonytao_ this is great mann! quick question does pixel umi capture base motion alongside arm trajectories??
2 likes

@JasonJZLiu
@_tonytao_ The VR headset provides head tracking wrt the world, and we derive base motion from it via retargeting
2 likes

@Alanogrok
@_tonytao_ Reproducible hardware costs would help labs compare setups.
2 likes

@21J3phy
@_tonytao_ the DoF ablation is the most useful part for anyone building one: dropping the single torso-pitch joint cuts task progress to 47%, worse than losing the whole 430mm lift (71%). one pitch joint buying more than a lift is a great takeaway for low-cost mobile manipulators
2 likes

@k7agar
@_tonytao_ incredible
2 likes

@BenAgro4
@_tonytao_ Wish I could hire this @JasonJZLiu guy 😀 , so cracked.
2 likes

@JasonJZLiu
@_tonytao_ Ben my goat!!
1 like














