Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data? Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun... insights below: 🧵show more

The idea is simple: throw in any reasonable human video and some compute, and out comes robot actions doing the same thing the human is doing. But to make it work, we had to tackle the breadth of human data, hand-object interaction, and dexterous retargeting. 2/n

Most internet-grade “human data” has terrible quality: often they’re blurry, with no meaningful interaction happening, and hand/objects are constantly going out of frame. Want to know if your human data is robot ready? Pass them through our Do as I Do. For example: we start with 2000 100DOH clips that are already filtered for hand/object interactions, and find that only 107 of them actually have the necessary info. Do as I Do can reconstruct 83 of those. 3/n

For hand-object tracking from monocular RGB, we used many of the nice perks of modern CV, and made stuff on our own when that wasn’t enough. For example, did you know you can get SAM3D to do 4D object tracking? Under the hood, it’s a diffusion decoder; so tracking ≈ biasing the distribution with current object priors! The technical details matter, so check out the paper. 4/n

Sometimes there is genuine ambiguity in the data, and vision can’t resolve it. What can you do in this case? You use physics! The physics-grounding step that transfers motion to robot actions also validates that any motion we propose is achievable by a robot. Thus, even when the object wants to float away, the physics staples it to the hand. 5/n

Lots of gems like this in the full work, led by @bhawna_paliwal_, @HarithejaE, & @willjhliang. Also special thanks to @pabbeel & @JitendraMalikCV for advice, and @kyutai_labs for the compute support. Try our code today: 📝: 🌐: ⚙️:

Very cool man

Great stuff!! I got lots of qns (like how u do the physics grounding)...would love to have u guys come on robopapers to share more 🙏

incredible achievement! congrats to the team!

"Do what I do" is a foolish approach... It's full of unnecessary and useless data which comes from ring and pinky fingers. People use only 3 fingers in 99.9% of their daily tasks. “Move object as i move” is the reliable model to teach AI..

Lab data is too fake Just pay people to cook dinner in smart glasses for the real chaotic data

Lab data is too fake Paying people to cook in smart glasses gives robots the real chaos they need

I have some hands on experience with a similar approach and what we should do on the IK for Mujoco is to use additional constraints to favour natural human positions and avoid singularity points. Example:

This is the right kind of messy. Human videos won’t look like clean teleop data, but they contain the stuff robots actually struggle with: contact, recovery, small corrections, changing context. Making that usable is the unlock.

Exactly. Teleoperation is boutique. I suspect open-source robotics scales on uncurated internet video.

Very interesting! And thank you for the contribution to the public through a paper. I find the potential to use AI-generated videos to train physical AI robot incredibly interesting. Another example of AI training AI, how great.