Excited to introduce 𝐋𝐀𝐏𝐀: the first unsupervised pretraining method for Vision-Language-Action models. Outperforms SOTA models trained with ground-truth actions 30x more efficient than conventional VLA pretraining 📝: 🧵 1/9

Vision-Language-Action (VLA) models, LLMs aligned with vision encoders, show strong generalization when pretrained on robot datasets, enabling strong generalization capabilities. However, they remain limited by the scope of existing robot datasets. 2/9

This limits scalability—robot companies spend $$$ hiring workers to collect data, requiring 8+ hour shifts 💀 We propose an alternative: training VLA models directly on internet videos. But how can we train VLA on videos without robot actions? 3/9

Introducing 𝐋atent 𝐀ction 𝐏retraining for general 𝐀ction model (𝐋𝐀𝐏𝐀)! LAPA first trains a VQ-VAE to extract latent action tokens, acting as a tokenizer to capture atomic actions from video frames, preserving semantics (see ⬇️). 4/9

Next, we pretrain a VLM to generate latent action tokens from input observations and task descriptions. Finally, the VLA model learns to map latent actions to robot actions by fine-tuning on a small set of action-labeled robot trajectories. 5/9

We experiment on diverse sim & real environments, varying different combinations of pretraining and fine-tuning datasets to test different transfer capabilities (in-domain, cross-task, cross-environment, cross-embodiment, and multi-embodiment). 6/9

LAPA pretrained on Open-X outperforms OpenVLA (SOTA VLA) in average success rate across three tabletop manipulation tasks requiring generalization; we attribute this to positive transfer from multi-embodiment training in a unified latent action space. 7/9

Surprisingly, LAPA pretrained only on human videos outperforms OpenVLA pretrained on Bridgev2, despite the human ↔ robot embodiment gap, highlighting its potential as a general-purpose solution for creating robotic foundation model from internet videos. 8/9

Huge thanks to my co-lead @SeonghyeonYe , and to all of the advisors & collaborators from @kaist_ai , @uwcse , @MicrosoftAI , @NVIDIAAI , and @allen_ai for making this project possible 🙌 For more info, visit our website! 9/9

@kaist_ai @uwcse @MicrosoftAI @NVIDIAAI @allen_ai + I have recently joined @NVIDIAAI GEAR Lab as a research intern, working on scaling LAPA and Project GR00T—a moon-shot to create generalist humanoids w/ @scott_e_reed , @DrJimFan , and @yukez . Stay tuned for exciting updates!

@RemiCadene so bsically google genie?