We (Pantheon) have been working on a robotics data quality pipeline that uncovered a series of major problems in public robotics datasets, especially for world modeling. To improve the quality of data available to open-source robotics, we're publishing annotations for four of the most popular datasets. Some examples of... issues, and our report 🧵show more

1. Camera disconnection One of the wrist cameras in this episode froze, then disconnected. This would show up as pure noise to a world model.

2. Missing gripper actions Despite the left gripper opening and closing, no actions are recorded. The world model is penalized for failing to predict the gripper's actions.

3. Spiky actions In some datasets, action data contains sharp spikes. These are hard for a world action model with broad smoothness priors to fit.

4. Shifted actions Action data is misaligned by a few frames for the grasp, but not the movement. This tricks the world model into predicting state changes too early or late.

Read our full report: Credits to @sambhavgupta159 for uncovering and publishing this analysis. We're hiring for robotics foundation model research, software, data, and hardware engineering. We're also open to data collaborations.

@PantheonInc Such important work! Data is king

@PantheonInc bullish for looking at the data you’d be surprised how many don’t

@PantheonInc Crazy how many issues we found in just two weeks...

@PantheonInc I’ve been waiting for this

@PantheonInc Dream team!!

@hamostaf04 @PantheonInc Great work on the data!

@PantheonInc @Jannikschg @arminsommer_ the european robotics mafia!

@PantheonInc bot see good data, bot happy :)

@PantheonInc let’s goooooo @calixo888 @nettedjay !!!!!! 🚀

@PantheonInc Interesting read! Congrats to the team @_varunnair great to see you in action :)

@PantheonInc @sambhavgupta159 is a beast! very nice work!

@PantheonInc cool!!

@PantheonInc 🙏

@PantheonInc amazing read! cong to the team :D @Jannikschg

@PantheonInc this is super cool!

@PantheonInc KALIX!!

@PantheonInc The beginning

@PantheonInc 🔥🔥

@PantheonInc 🚀🚀🚀

@PantheonInc so amazing calix

@PantheonInc quietly cooking!!

@PantheonInc finally some real data rigor

@PantheonInc Data Interpretability is huge. These are fixable issues and the human data community should know the aspects that must be improved. Overall great for the community as a whole!

@PantheonInc Crushing it!

@PantheonInc Well done Mr Gupta @sambhavgupta159

@PantheonInc very cool work!!

@PantheonInc letsgoo

@PantheonInc Awesome results!!!

@PantheonInc awesome work! congrats pantheon

@PantheonInc long time coming 💪

@PantheonInc thank you!!

@PantheonInc Thank you for looking at the data