Introducing Neural Jacobian Fields, robot 3D kinematic models learned only from vision! They can model & control robots from just a single RGB camera, even those w/ intractable kinematics & no embedded sensors such as soft, 3D-printed pneumatic hands! 1/n

This project was led by the outstanding @sizhe_lester_li, in a multi-disciplinary collaboration with @annan__zhang, @BoyuanChen0, Hanna Matusik, Chao Liu, and Daniela Rus! Amazing team - truly grateful for the soft robot chops hands-on contributed by @annan__zhang & co! 2/n

While we humans constantly use vision for closed-loop control of our hands and limbs, today’s robots instead rely completely on embedded sensors, precision manufacturing and expert-designed forward kinematic models to understand where they are in 3D: 3/n

Conventional robots are “blind” when it comes to state estimation! This constrains their design to be “simple” such that an expert can design a model & requires expensive manufacturing & sensors. But bio-inspired, multi-material, and cheap robots cannot be modeled easily! 4/n

Our approach enables this pneumatic bio-inspired hand to perform physical tasks and allows the $220 poppy arm to draw “MIT” in the air, all with using a single RGB camera as the only sensor! None of these motion trajectories are prescribed as part of our training data. 5/n

Neural Jacobian Fields enable us to equip any robot with vision-based control, irrespective of its sensors, fabrication, material, or actuation. Each point is mapped to its “system Jacobian”, which maps a change in motor commands to the 3D motion of that point! 6/n

The Jacobian Field can be directly used for inverse dynamics control. Given desired motions, our model solves for the corresponding control command at interactive speeds. 7/n

Neural Jacobian Fields are trained completely self-supervised, from multi-view videos of a robot executing random commands - no human labels or intervention. At test time, a *single* image suffices to reconstruct them, for closed-loop control from a single RGB camera. 8/n

The learned Jacobian Field is interpretable: we can visualize it by color-coding sensitivity to the different control channels. In this way, we can see that the Jacobian field correctly discovers the 3D kinematics of the robot. 9/n

We’re not advocating for *only* using vision. In this paper, that’s what we do, to show how *surprisingly powerful* it can be! But: the more sensors, the better. It is straightforward to add more sensors - just add them to the input to reconstruct the jacobian field! 10/n

Neural Jacobian Fields could make robotic automation more affordable by enabling control of cheap robots, as they allow us to skip expensive sensors, enable new fabrication techniques such as 3D printing, and are robust to imprecise manufacturing! 11/n

There are still ways to go - but we think that this direction is super promising and just the beginning of an exciting arc of building representations of robotic embodiments that can bridge vision, proprioception, touch, and more! 12/n

Again, shoutout to the amazing team at MIT - make sure to check them out! @sizhe_lester_li, @annan__zhang, @BoyuanChen0, Hanna Matusik, Chao Liu, and Daniela Rus! Code & everything is already released: 13/n

Also, I should add we are aware of another paper that is named "Neural Jacobian Fields": It's a completely different idea and problem and not directly related, but still really, really cool (check it out!)...

Our paper is actually called "Unifying 3D Representation and Control of Diverse Robots with a Single Camera". We thought about not naming the core method "Neural Jacobian Fields", but it makes the most sense, b/c we are really modeling the "System Jacobian", a staple in robotics.