Wanted to get into manipulation. Decided to first build a full stack that scales. Plan: copy Generalist's gripper. Fun weekend project. It was not. So -T.K.- and I documented every design choice and lesson along the way. Meet Specialist: fully open-source hardware, data pipeline, training, and deployment code.

The whole collection system is two handheld grippers and a few printed tags. No computer, headset, or tracking system. $320 per gripper, and under 30 s to set up a new location.

Each gripper camera records two lenses and an IMU. Audio syncs the grippers, with no extra hardware. A factor graph, anchored by the printed tags, turns the footage into metric 6-DoF trajectories. Median error: 3.0 mm against motion capture.

The robot gripper has the same geometry and camera position as the handheld one. So policies trained only on handheld demos run on the robot with no retargeting. Build your own:

Dataset and more demos coming soon. Built at UC Berkeley with Borivoje Nikolić, @Sophia_Shao_, @pabbeel, and @KoushilSreenath. Thanks to @chichengcc and @SongShuran for UMI, and to Generalist for the inspiration.

@_tonytao_ @GeneralistAI @t_k_233 So cool!!

@GeneralistAI @t_k_233 exciting，I tried this for a month,but I failed

@GeneralistAI @t_k_233 I can't imagine this being a weekend project....

@GeneralistAI @t_k_233 320 dollars a gripper plus under 30 seconds setup is wild, did you go with the parallel or adaptive jaw for Specialist

@GeneralistAI @t_k_233 It’s pretty easy.

@GeneralistAI @t_k_233 So cool

@GeneralistAI @t_k_233 nice

@kevin_zakka @GeneralistAI @t_k_233 Wonder how robust is it to occlusion by large objects?

@GeneralistAI @t_k_233 Hi — if my arm uses different actuators than the one in your demos, can the Specialist trajectories still serve as a direct training reference (with me handling the IK), or does the policy expect the same arm dynamics?

@GeneralistAI @t_k_233 Qiayuan on a roll! inspired by (not copying) 😂

@GeneralistAI @t_k_233 Amazing

@GeneralistAI @t_k_233 goat

@GeneralistAI @t_k_233 🐐

I’m curious whether you guys tried to reverse engineer their finger design / thought about their hexagonal lattice design or how they optimized the finger design for manipulation. Also why any thoughts on why a servo instead of offsetting a Daimao motor behind the 2 main gear sets for easier arm integration (more torque, and current feedback, and reliability)

@kevin_zakka @GeneralistAI @t_k_233 This is awesome

@GeneralistAI @t_k_233 "fun weekend project. it was not" is how every real robotics project starts lol. open sourcing the gripper, data pipeline and lessons is the best possible outcome though

@GeneralistAI @t_k_233 so fascinating! could be one of the most cost-effective way to capture data for robots training. great works!

@GeneralistAI @t_k_233 Congrats on the release! Curious how the data pipeline handles teleop vs autonomy, will dig through the repo this weekend. A real one this time, unlike yours apparently.

@GeneralistAI @t_k_233 This exact project has been on my to-do list for a while. Thanks for open sourcing, we’ll try it out!