💥Introducing FACTR 2, learning external force sensing on commodity robot arms without needing dedicated sensors. We show that learned force signals enable force-feedback teleop on low-cost arms and improve BC policies. FACTR 2 consists of: 1. Neural External Torque (NEXT): learns external forces without needing dedicated force sensors. 2.... Force-Informed Re-Sampling Training (FIRST): uses the learned force signal to identify task-critical regions and upsample them during training. w/ Steven Oh Tony Tao 🧵(1/N)show more

We first introduce NEXT: Neural External Torque, which learns external joint torque without dedicated force sensors. With just 10 min of free-space data and 1 min of training, NEXT learns to predict the torque needed for contact-free motion. At runtime, subtracting this prediction from measured motor torque gives external torque. 🧵(2/N)

We validate NEXT on the Franka, an arm with dedicated force sensing. Despite using no force sensors, NEXT closely matches Franka’s factory external torque estimates. 🧵(3/N)

We further introduce FIRST: Force-Informed Resampling Training. Using the learned force signal, we can automatically segment demonstrations into free-space, pre-contact, and contact regions. 🧵(4/N)

We then up-sample pre-contact and contact data during training to improve policy performance. This is intuitive: most failures do not happen in free space. They happen near contact, where precise alignment, small error recovery, and force-sensitive interaction matter most. 🧵(5/N)

This work was done @CMU_Robotics with co-lead @StevenOh_ and @_tonytao_ as well as @yangphiliphan, @kenny__shaw, @funabashihand, @rsalakhu, @deepakpathak. Website: Paper: 🧵(6/N)

Special shoutout to @StevenOh_. Steven has been visiting us at CMU from Japan for the past few months and worked incredibly hard on this project. I’m very proud of what he has accomplished, and excited for him to start his PhD at UChicago. Please check out his thread for more details.

@StevenOh_ @_tonytao_ @yangphiliphan @kenny__shaw @funabashihand @rsalakhu @deepakpathak Also check out @_tonytao_’s thread as well!

Please get back on we need you for pantheon

😅

This is super cool work! Can’t wait to see the controller code😄 Would your method work on Feetech motors by any chance?

Feetech motors on the follower arms?

both actually, e.g. could you use the method with the so-101 leader/follower setup?

On the follower side, we don’t have these arms so we haven’t tried it ourselves, but I suspect it can work for these. On the leader side for force feedback, this is definitely possible.

Congrats on the release!

@ritvik_singh9 What arms are you using

@ritvik_singh9 Most of the policy videos are with the AgileX Pipers

@ritvik_singh9 Awesome!

Cool work Jason! I’ve been exploring using tactile feedback on policy training lately

This is super cool! Expect much hyper param tuning for new embodiments? Also what’s the intuition for what makes a good training set? Must it also include contact interactions?

Thanks! Hyperparameter tuning hasn’t been too bad since each NEXT model trains in ~1 min, so sweeps are cheap. The training data should be free-space only, without contact. The model learns free-space dynamics, so contacts appear as residual external forces. Appendix A.3 has more data collection guidelines.

Every technology eventually becomes cheaper. The real advantage shifts from hardware to decision-making. That's usually where the biggest companies are built.

Congrats!

Great work. For future I suggest using one of those for groundtruth

Oh, clever approach! Canada's manufacturing sector needs to be adopting stuff like this. But I'm not holding my breath.

Great work @JasonJZLiu!!

cool paper! did you consider evaluating end-effector force directly rather than joint torques? seems like for contact tasks like insertion that's would actually matter a lot

Do you mean evaluating the accuracy of the learned joint torques transformed to end effector forces? Or do you mean feeding end effector forces to the policy?

Transformed to end effector forces

We haven’t done these experiments, but from the perspective of training policies, we didn’t find much difference between using end effector wrench vs joint torque. But considering our joint torque predictions are accurate, I presume the transformed end effector wrenches will be as well.

I see, thanks :) I assume the same but I guess it might not always be true that smaller join torque errors always mean smaller EE errors? Either way, nice work!

This is amazing. Did you use Dynamixels for your leader arms?

Yes

The XL330-M288-Ts or the Dynamixel XL430-W250-Ts?

We use XC330 T288-T

Really impressive work！🤩 We build open-source robot arms and joint modules with torque control and current feedback. If you’re interested in testing FACTR 2 on another low-cost platform, we’d be happy to provide hardware and collaborate.

Love this, how well does NEXT handle vibration on cheap arms?

You mean arms such as the SO-101?

感觉也是另一种触摸世界

How much does this hardware set cost?

The piper arms cost ~$2.5k. The leader arms are ~$600

Thanks