🚀 First step to unlocking Generalist Robots!
@SeonghyeonYe2 тыс. подписчиков
33 385 просмотров
9 комментариев
Комментарии из оригинального поста в X (Twitter)

@SeonghyeonYe
LAPA consists of a 1) Latent Action Quantization and 2) Latent Pretraining stage. The first stage learns quantized actions through visual deltas. For the second stage, a pretrained VLM (LWM) is trained to predict the quantized latent actions. During finetuning, we map the latent actions to real actions.
4 отметки «Нравится»

@SeonghyeonYe
LAPA beats OpenVLA across cross- and multi-embodiment tasks, all without using action labels during pretraining! 🚀 A step forward in robust, generalizable robot learning.
2 отметки «Нравится»

@SeonghyeonYe
We can built LAPA from 220K human videos where the action labels does not exist and the embodiment gap is huge. Still, LAPA (Human Videos) outperform OpenVLA (Bridge) 😯
3 отметки «Нравится»

@SeonghyeonYe
What do latent actions mean? Latent actions correspond to 🌟semantic 🌟actions across different robot embodiments. Interestingly, despite different robot embodiments, the same latent action maps to similar movements. This suggests latent actions form a ‘shared’ representation space, much like images or language.
3 отметки «Нравится»

@SeonghyeonYe
We also do closed-loop rollout of LAPA for analysis (rollout vs ground truth). Given ‘pick up the broccoli from pot,’ it successfully picks up the object, which then disappears. This highlights LAPA’s potential as an emerging 'world model' with impressive predictive abilities! 🌍
3 отметки «Нравится»

@SeonghyeonYe
All of the things are open-sourced! Code 💻: Huggingface 🤗: Website🌐:
6 отметок «Нравится»

@SeonghyeonYe
Co-led with @jang_yoel With wonderful collaborators: Byeongguk Jeon, @joocjun ,@jw2yang4ai Baolin Peng, @AjayMandlekar, Reuben Tan, Yu-Wei Chao, @billyuchenlin, Lars Liden And advisors: @kimin_le2, @JianfengGao0217, @LukeZettlemoyer, Dieter Fox, @seo_minjoon from @kaist_ai, @UW, @Microsoft, @nvidia, @allen_ai
6 отметок «Нравится»

@EthanNg51931527
Firm step towards scaling up VLA model!Excellent job!
2 отметки «Нравится»

@jiazhi_yang2024
Incredible job! I view it as the first practical evidence that unsupervised latent action could work well at such a large scale.
1 отметка «Нравится»