New work with Yide Shentu! We present Latent Code as Bridges🌉 or LCB. We leverage LLMs for robotics by finetuning the LLM to output latent embeddings to control the lower level policy. This hierarchical approach enables end2end training while maintaining the advantages of LLMs!

Some prior works leverage pretrained skills with a function calling or code interface. Another common approach uses LLMs to output intermediate language goals, which a language conditioned policy ingests. In contrast, LCB offers the flexibility for the LLM to output latent goals.

LCB can result in a better holistic model through end2end training than the sum of its parts. We finetune a pretrained VLM and policy with a language annotated dataset robotics dataset. The model learns to predict an <ACT> token, its respective latent code, and the control action

This procedure enables us to learn a model capable of various long horizon and reasoning tasks. On the Language Table benchmark, we out perform baselines, including using GPT4-V to reason and predict language goals.

In addition, we test on the CALVIN long horizon benchmark, and find our approach is competitive to others when comparing against other recent methods.

I think there's lots of potential for this approach as a policy architecture for robotics! Thanks to my co-lead @YideShentu. Also could not have done it without @aravindr93 and @pabbeel. Checkout the paper on our website 🌟: Arxiv coming soon!