Skip navigation
Excited to finally share Pixel: a full-stack robot learning project Tony Tao and I worked on earlier…
0:00 / 2:22

Excited to finally share Pixel: a full-stack robot learning project Tony Tao and I worked on earlier…

Jason Liu

@JasonJZLiu2.7K followers

Visit on X
456 likesDownloadOriginal post

73,113 views

Excited to finally share Pixel: a full-stack robot learning project Tony Tao and I worked on earlier this year! We’re releasing two papers: 1. Pixel: A low-cost mobile manipulator, co-designed with its UMI system and learning stack. 2. Pixel-UMI: Improved UMI pipeline for learning environment-aware, whole-body manipulation (1/N)

37 Comments

Comments from the original post on X (Twitter)

  • Jason Liu ✈️ CoRL's profile picture

    @JasonJZLiu

    In this thread, I will mainly cover the Pixel-UMI First, check out @_tonytao_'s thread for more on the robot and UMI hardware we co-designed! (2/N)

    16 likes

  • Jason Liu ✈️ CoRL's profile picture

    @JasonJZLiu

    In Pixel-UMI, we first introduce Environment-Aware Retargeting (EAR). We reconstruct the scene offline from each demonstration, then use the reconstruction to convert UMI human data into collision-aware whole-body robot trajectories. (3/N)

    13 likes

  • Jason Liu ✈️ CoRL's profile picture

    @JasonJZLiu

    We also introduce several tricks that help with learning whole-body policies from UMI: from assigning actions that better account for real robot dynamics to training-time techniques that reduce the effects of human–robot kinematic mismatches. Check out these policies trained from ~200 UMI demos per task! (4/N)

    14 likes

  • Jason Liu ✈️ CoRL's profile picture

    @JasonJZLiu

    Finally, we use DAgger to improve long-horizon, contact-rich manipulation. With two rounds of 60 corrected rollouts per task, we substantially improve performance on challenging bimanual tasks like folding a box and hanging a jacket. (5/N)

    15 likes

  • Jason Liu ✈️ CoRL's profile picture

    @JasonJZLiu

    This work was done @CMU_Robotics with co-lead @_tonytao_ as well as @deepakpathak and @rsalakhu. Website: Pixel Paper: Pixel-UMI Paper: (6/N)

    14 likes

  • Jerry Chéng's profile picture

    @thejerrycheng

    @_tonytao_ I love how community’s idea come together around the same time. Super impressive work! We just released our open source bimanual robot MABEL too lol Pixel should meet MABEL

    4 likes

  • Jason Liu ✈️ CoRL's profile picture

    @JasonJZLiu

    @_tonytao_ 🫡

  • khushi's profile picture

    @khushii_w

    @_tonytao_ very dextrous! this is great

    3 likes

  • Alfred Cueva ✈️ CoRL's profile picture

    @alfie14x

    @_tonytao_ Awesome work Jason!

    3 likes

  • David McAllister's profile picture

    @davidrmcall

    @_tonytao_ Let’s go!!!!

    3 likes

  • Lumbogg's profile picture

    @lumbogg

    @_tonytao_ Dude. Awesome.

    2 likes

  • Praneeth Samineni's profile picture

    @praneethsamine

    @_tonytao_ Very cool!

    2 likes

  • Stephen's profile picture

    @Imfulao

    @_tonytao_ Thank you for sharing both papers — co-designing the UMI stack with the low-cost arm seems like the key decision. I assume the environment-aware whole-body results in the Pixel-UMI paper are where the gain shows up most.

    2 likes

  • Van0SS's profile picture

    @Van0SS

    @_tonytao_ this is great mann! quick question does pixel umi capture base motion alongside arm trajectories??

    2 likes

  • Jason Liu ✈️ CoRL's profile picture

    @JasonJZLiu

    @_tonytao_ The VR headset provides head tracking wrt the world, and we derive base motion from it via retargeting

    2 likes

  • Alano's profile picture

    @Alanogrok

    @_tonytao_ Reproducible hardware costs would help labs compare setups.

    2 likes

  • Nirav's profile picture

    @21J3phy

    @_tonytao_ the DoF ablation is the most useful part for anyone building one: dropping the single torso-pitch joint cuts task progress to 47%, worse than losing the whole 430mm lift (71%). one pitch joint buying more than a lift is a great takeaway for low-cost mobile manipulators

    2 likes

  • atharva ☆'s profile picture

    @k7agar

    @_tonytao_ incredible

    2 likes

  • Ben Agro's profile picture

    @BenAgro4

    @_tonytao_ Wish I could hire this @JasonJZLiu guy 😀 , so cracked.

    2 likes

  • Jason Liu ✈️ CoRL's profile picture

    @JasonJZLiu

    @_tonytao_ Ben my goat!!

    1 like