What happens when we train the largest vision-language model and add in robot experiences? The result is PaLM-E 🌴🤖, a 562-billion parameter, general-purpose, embodied visual-language generalist - across robotics, vision, and language. Website:

PaLM-E enables robot planning directly from pixels – all in a single model, trained end-to-end. Here the model is guiding a robot to get a chip bag from a kitchen. Being integrated into the control loop, PaLM-E is robust to disturbances happening during the robot’s journey.

In a different domain, here the **same** exact PaLM-E model is controlling a robot in real-time. This robot recently required human assistance to guide it through very long-horizon tasks ( but now PaLM-E can learn these tasks autonomously.

PaLM-E is the largest VLM reported to date. We observe emergent capabilities like multimodal chain of thought reasoning, and multi-image inference, despite being trained on only single-image prompts. Though not the focus of our work, PaLM-E sets a new SOTA on OK-VQA benchmark.

The inputs to PaLM-E are multimodal sentences that interleave text, images, states, or other continuous encodings. These multimodal sentences are passed as inputs to an LLM for next token prediction, trained end-to-end. For PaLM-E-562B, it combines PaLM-540B and ViT-22B.

Perhaps most exciting about PaLM-E is **positive transfer**: simultaneously training PaLM-E across several domains, including internet-scale general vision-language tasks, leads to significantly higher performance compared to single-task robot models.

We observe a notable trend with model scale: the larger the language model, the more it maintains its language capabilities when training on visual-language and robotics tasks – quantitatively, the 562B PaLM-E model nearly retains all of its language capabilities.

And that’s PaLM-E: a one-model generalist across robotics, language, and vision-language.

This was an incredibly fun project. Thanks to my collaborators @xf1280, Mehdi S. M. Sajjadi, @coreylynch, @achowdhery, @brian_ichter, @ayzwah, @JonathanTompson, @QuanVng, @TianheYu, @wenlong_huang, @YevgenChebotar, @psermanet, @duck

, Vincent Vanhoucke, @hausman_k, @Marc__Toussaint, Klaus Greff, @andyzengtweets, @IMordatch, @peteflorence

Many people have asked me what is next for PaLM-E. Check out RT-2-PaLM-E where we enable PaLM-E to directly output robot actions!