Introducing ABC: open data, training, and infrastructure for robotics. We release the largest teleop dataset to date, and extensively investigate design decisions, pretraining, and post-training techniques. Arthur Allshire Himanshu Gaurav Singh Adam Rashid Hongsuk Benjamin Choi David McAllister

Check out our paper, more videos, and our more than 3.5K hour dataset here

In addition to real data, we have simulation data. Using this data, we show sim/real correlation.

One interesting observation we find is across different model classes, training flow loss and validation action error correlate with real world success, while we did not observe any statistically significant correlation between validation flow loss and real world success.

We develop lightweight robot infra, enabling things such as DAgger and intervention data to augment our large scale dataset. This allows us to experiment with post-training our checkpoints to enable dexterous tasks such as airpod insertion

This would not have been possible without the support from Amazon FAR or our amazing mentors @akanazawa, @GuanyaShi, @pabbeel, @peterxichen, @rocky_duan, @JitendraMalikCV, @phillip_isola

Shout out to our collaborators @xdofai -- see their tweets here and also congrats on their company announcement:

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall So happy to be able to help contribute to this amazing work!

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall So great to collaborate with you all on this work!

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Congrats @ritvik_singh9 !!

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Let's gooo congrats y'all

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall congrats!!! very polished amazing work!!!

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall let's goooo

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Congrats!!

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Very cool paper, so much useful data in there. Do you have any theories as to why adaln modulation worked better for VLAs but cross attention worked better for DiT? That was definitely not an intuitive result for me.

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Am I the only one curious about Arthur's exit? Where is he going? 😂

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Congrats to the team!

Circling back on this - why did you guys not compute CI-MSE on the val set? On a (tiny) amount of rollouts we’re actually seeing that it can meaningfully surface segment level failures completely offline , totally in-sync with actual deployments. Hope this holds up with more rollouts

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Great work! Any instructions for reproducing the exact work cell? I know the arm / hand brand is in the paper, but the camera - arm extrinsics etc.

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall 3.5K hours is massive – but it’s all teleop in controlled setups. Does scale in controlled environments actually close the deployment gap, or do we hit a ceiling without real-world diversity? Curious if you saw diminishing returns on generalization as the dataset grew.

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Fantastic work, lots of great insights

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Looks promising. How do I buy/build the right hardware for this to try in my factory for sorting veggies ?

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Congratulations!

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Amazing to see even more great work open sourced! Well done to the team

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall #266 9/30/2019

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Y’all need any data for picking up rubber ducks? Cuz I have wayyyyyy too many episodes in my local hard drive 😅

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall What’s the arm called or link for it

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall This is all free?

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Thanks 👍

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall wen g1 dataset 👀

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall This is very cool, how easy is it to fine tune?

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Great work done, btw are u going to release the rest of the sim data as stated on GitHub? Having it together with the sim itself is very useful

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Open robotics data is a major accelerator.Large teleoperation datasets, shared training methods, and open infrastructure can help the field move faster from isolated demos to more generalizable robot capabilities. 🤖📊

Hey Ritvik! Your ABC work caught our eye, especially the push to make dexterous robot learning more reproducible. We’d love your @Opinion for an upcoming piece. DM me if interested.

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Cool, is it possible to 3d print required accessories?

Large-scale open datasets could be a major accelerator for robotics research. The interesting part is not only the data volume, but understanding which collection and training choices actually translate into better robot capabilities. How much will these lessons shape future data pipelines?

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall In robotics, the long-term advantage may come from the ability to accumulate, share and learn from operational experience at scale. Models can be replicated. Accumulated experience is harder to copy.