Excited to share RoCo: Dialectic Multi-Robot Collaboration with Large Language Models. We propose a novel approach to multi-robot collaboration that leverages LLMs for both high-level communication and low-level path planning. w/ Shreeya Jain, Shuran Song

Much of the recent work in LLM+robotics has been task planning for a single robot - in this work, we venture into multi-robot settings, and introduce a new benchmark, RoCoBench: 6 multi-robot collaboration tasks that span a variety of scenarios and robot capabilities.

These tasks require efficient high-level allocation, coordinated low-level execution, and generalization to variations. Our key idea: use LLMs to simulate a multi-agent dialog that facilitates collaborative reasoning, then propose sub-task plans to guide motion planning.

We show our system 1) benefits from the generality of LLMs in understanding varying task semantics, and 2) the multi-agent dialog offers flexibility to easily incorporate one human participation - see this human-robot collaboration task in our real-world experiments:

We also curate a text-based dataset that focuses on evaluating LLMs' agent representation and task reasoning ability, without requiring environment interactions. It contains open-ended questions about the RoCoBench tasks, going beyond simply finding the best action plan.

For more details, check out: arxiv: website: code & RoCoBench:

Announcing: Our most advanced speech-to-text model goes beyond accuracy to capture the real-world complexity of human conversation and deliver reliable, source-of-truth audio data. Explore Universal-2 updates 👇

@shreeyaajain @SongShuran Nice work! The field's been searching for years for effective "robot languages" (usually in some learned latent space) to enable emergent communication in MARL settings. ...maybe the best "robot language" is just English?

@shreeyaajain @SongShuran Thanks Ted! I suspect even if there is some kind of optimal latent communication, it just might be more appealing to have robots talk in language that human can also understand & supervise

@shreeyaajain @SongShuran Exciting stuff!

@shreeyaajain @SongShuran RoCo's basilisk...

@AlperCanberk1 @shreeyaajain @SongShuran I did Not think of that lol