Excited to introduce TWIST2, our next-generation humanoid data collection system. TWIST2 is portable (use anywhere, no MoCap), scalable (100+ demos in 15 mins), and holistic (unlock major whole-body human skills). Fully open-sourced:

2/n Why to build TWIST2? I still remember 1 year ago, painfully teleoperating a full-size humanoid for simple pick & place to collect a few demonstrations. Even getting 10 episodes was tough, and we had to lock the lower body to keep the robot stable. That is a well-known bottleneck of teleop: "Not scalable". This problem gets much harder in humanoids: the state and action space are much larger than tabletop arms, and dynamics are more complex. video from iDP3 (

3/n TWIST2 system overview We finally have such a teleop system that has all the features we need for general-purpose robot data collection: - Scalable - Portable - Holistic

4/n Portable You can just put on a VR headset (and 2 motion trackers) and start teleoperating anywhere. Rapid setup, no motion capture required. From walk in with the VR device to finish setup, it takes only 1 min.

5/n Holistic We find that egocentric active vision is essential to achieve long-horizon complex tasks. Therefore, we build a 2-DoF low-cost neck, which can be easily attached/detached to Unitree G1 without need to remove its original head.

6/n Holistic This is what it looks like in VR when doing teleoperation. The robot view is floating in the center of the VR view.

7/n Teleop skills Our system achieves human-like whole-body skills in a unified manner. Case 1: folding 3 towels consecutively, a very long-horizon and dexterous whole-body manipulation task.

8/n Teleop skills Case 2: walk around like humans

9/n Teleop skills Case 3: Pick an object from the ground

10/n Scalable data collection TWIST2 enables data collection at an unprecedented scale: ~50 mobile manipulation demos in 15 minutes

11/n Scalable data collection TWIST2 enables data collection at an unprecedented scale: ~100 manipulation demos in 15 minutes

12/n Visuomotor policy learning With TWIST2 data, we further build a hierarchical visuomotor policy learning framework, that controls the full body of a humanoid with imitation learning.

13/n Visuomotor policy learning That means, our visuomotor policy can not only pick & place objects,

14/n Visuomotor policy learning ...but can also use feet to kick a T-shaped box to the target region, i.e., Humanoid Kick-T.

15/n Visuomotor policy learning Our visuomotor policy predicts future whole-body joint positions. Across *Space & Time*.

16/n Humanoid data We also believe humanoid datasets should be universally shareable. So we built a platform to host and visualize humanoid data, and welcome contributions from the community:

n/n That's all. Hope you enjoy our work. This work was done while interning at Amazon FAR, with incredible collaborators @sihengzhao @KenWangWeizhuo, and guidance from amazing advisors @akanazawa @rocky_duan @pabbeel @GuanyaShi @jiajunwu_cs Karen. Website: arXiv: YouTube:

Glad to see humanoid teleop getting to useable state for manipulation! Congrats Yanjie

@Scobleizer @zeyanjie, would love to chat about collaborating (@Contact_CI)

Wow this is nice! I didn't know Pico had motion trackers like that, those look very interesting! How is the tracking and SLAM compared to Quest 3?

I am curious why you choose ZED2 rather than RealSense D435i as the head camera?

@chichengcc Nice work!

This is so cool!! Congrats @ZeYanjie!

super cool!! can it do my laundry for me?? (^_~)

So futuristic, good way to train, good work.

@ZeYanjie I remember that you mentioned the human motions in videos can also be retarget to robots instantly in the RoboPaper talk, please remind me which papers are you referring to? Thank you,

牛逼！

The paper mentions a Discord server, is it still active? The link on the paper website is no longer working

Hello! I don’t have a Pico 4, so I tried deploying TWIST2 with the provided motion lib. Something still feels quite weird, the walking isn’t fluid (the video below is from the 001 example, which walks forward, turns, and walks back). Any hint about a misconfig I might have made?

@Scobleizer This is soo good

TWIST2 is like the Swiss Army knife for robotics-portable, scalable, and open-source. Seriously game-changing stuff! Which whole-body skill do you want to see demoed next?

The neck looks great, thanks for sharing with the community!

TWIST2's open-source approach is brilliant! Portable, scalable AND holistic? This will supercharge humanoid tech development. Can't wait to try the 2-DoF neck for active perception!

game changer for data collection