1/N Most Vision-Language-Action models need tons of data for finetuning, and still fail for new objects and instructions. Introducing OTTER, a lightweight, easy-to-train model that uses text-aware visual features to nail unseen tasks out of the box! Here's how it works 👇

2/N Most VLA models process vision and language separately, making the policy responsible for both understanding tasks and performing control. OTTER simplifies this by directly leveraging a pre-trained VLM that grounds tasks visually—letting the policy focus solely on actions.

3/N Most people know CLIP as a strong model for image-text alignment. But CLIP also understands fine-grained patch-level details! Recent work ClearCLIP found that the intermediate features (after attention layers) actually have stronger language alignment than the final outputs.

4/N We tested ClearCLIP on both the DROID and OXE datasets. It accurately identifies the target objects by highlighting the relevant patches. The ability to precisely ground language visually makes it an ideal backbone for OTTER, enhancing robotic control and task understanding.

5/N Building on these observations, we propose a simple yet effective approach: we measure the similarity between each language token and visual patch. Specifically, we use these similarity scores to “retrieve” the image patches that closely match the task instruction.

6/N OTTER combines ClearCLIP’s fine-grained visual features together with (1) robot proprioception for precise control and (2) language instructions for flexible handling of the same object—whether picking, pushing, or poking.

7/N We collected demonstrations using the DROID setup under challenging generalization settings. For every task, we randomized object locations and added 2-3 distractors. During evaluation, we use different object locations and distractors.

7/N (Cont) To further assess Otter’s zero-shot generalization capabilities for unseen tasks, we introduced new objects and out-of-distribution language instructions that were not present in the training dataset.

8/N Otter performs well on physical robots for pick-and-place tasks (10 out of 19 tasks). It successfully generalizes to both training tasks with unseen configurations and distractors and held-out tasks. Notably, Otter outperforms Octo, OpenVLA, and the recently introduced Pi0.

9/N For more manipulation primitives beyond pick-and-place (covering our full set of tasks), Otter demonstrates superior performance. It outperforms fine-tuned Octo, OpenVLA, and even Pi0, showcasing its strong generalization ability and robustness across diverse robotic tasks.

10/N We open-sourced the code, models, and datasets! Please refer to the paper for more technical details. More pre-trained models are also coming, please stay tuned! Jax Code: Pytorch Code: Paper:

11/N It’s been an exciting collaboration between @berkeley_ai and @Meta! I had a fantastic time working with co-leaders Raven (@RavenHuang4) and Max (@letian_fu). Grateful for the invaluable insights from Tingfan Wu, @mukadammh, @JitendraMalikCV, @Ken_Goldberg, and @pabbeel!

⏰ Countdown alert! 🚀 Get free access to our new AI Writing feature before Dec 1st! Don't miss your chance to explore its power. Try it now—no cost, no catch! 🔗