Very excited to release the Open X-Embodiment Dataset today — the largest robot dataset to date with 1M+ trajectories! Robotics needs more data & this is a big step! There’s lots to unpack here, so let’s do a deep dive into the dataset! 🧵1/15

We assembled the dataset by pooling *existing* robot datasets from our collaborators @ Google and many many academic labs (34!). In total we included 60 individual datasets with 22 different robot embodiments — many robot arms, bi-manual robots, quadrupeds, wheeled robots etc. 2/

We analyzed the properties of the combined dataset! First, the number of datasets per robot embodiment: many academic labs use Franka robot arms, so we have many (smaller) Franka datasets and a long-tail of other robot embodiments! 3/

There’s a few large datasets with 10s/100s of thousands of trajectories, e.g. from Google, and many comparatively smaller datasets with <1000 trajectories from the academic labs. As a result, a few robot embodiments have more individual trajectories than others. 4/

👆We also estimated the # of visually distinct scenes per dataset & find that this metric is well distributed across robots, w/ many embodiments contributing a significant fraction of the scene diversity. Ultimately, scene diversity may be more important than trajectory count. 5/

Finally, we analyzed the distribution of skills & objects in the dataset based on the language annotations (~60% of the datasets have language instructions). While common pick-place tasks are most frequent, there is a long tail of interesting skills like wiping, assembling etc 6/

The distribution of objects is diverse & reflective of objects a robot would encounter "in the wild”, like common furniture pieces, food items, appliances etc. There is still a long way towards real world diversity, but we hope that this dataset can build a good foundation! 7/

Using the data is easy! All data is stored in tfrecords & we made a colab for visualizing & downloading the data (w/ examples for efficient data loaders)! Each dataset stores observations/actions in its “native” format & resolution, but it's easy to align&mix them on-the-fly! 8/

The full dataset download is ~4.5 TB. We also provide a sheet that allows you to filter the data along many attributes, e.g. if you only want to download Franka robot data or only data with wrist cams, natural language instructions etc! Tailor the data to your use case! 9/

Here are the dataset resource links: ✅Colab (vis / download / data loaders):  ✅Overview Sheet (filtering):  All data is fully open-source under a commercially usable CC-BY 4.0 license! 10/

To show that the data is useful for learning, we trained a series of large-scale policies (RT-1-X, RT-2-X) & found co-training with our data to improve performance substantially! We’re releasing model checkpoints too, check Quan’s tweets for details! 11/

Creating this dataset was a huge community effort (look at that author list 😀)! I led the dataset construction and had calls with countless labs & everybody was very excited to contribute data — there is a lot of momentum in the community towards sharing & reusing data 🙂 12/

We’re hoping to continue this momentum and keep growing the dataset 🚀! We’re still figuring out the details, but if you or your lab have data you’d like to contribute feel free to shoot an email to [email protected] and we will get back to you! :) 13/

Many authors were involved in this project! Special thanks to @QuanVng for leading the overall project and managing everything masterfully! Tagging a few more co-authors @pannag_ @hausman_k @chelseabfinn @svlevine 14/

I’m very excited to see how the community will use this dataset! Let me know if you have any questions! 🙂 💻Project Website: 15/15