With the recent progress in large-scale multi-task robot training, how can we advance the real-world deployment of multi-task robot fleets? Introducing Sirius-Fleet✨, a multi-task interactive robot fleet learning framework with 𝗩𝗶𝘀𝘂𝗮𝗹 𝗪𝗼𝗿𝗹𝗱 𝗠𝗼𝗱𝗲𝗹𝘀! 🌍 #CoRL2024

Our recipe: Stage 1: Pre-train a visual world model on diverse datasets to 𝙥𝙧𝙚𝙙𝙞𝙘𝙩 𝙛𝙪𝙩𝙪𝙧𝙚 𝙤𝙪𝙩𝙘𝙤𝙢𝙚𝙨 across many tasks. Stage 2: Deploy a multi-task policy on a robot fleet, with anomaly predictors monitoring the fleet deployment via the visual world model.

The visual world model predicts future embeddings. It learns a latent space using image reconstruction and predicts future embeddings with cVAE. We train anomaly predictors on the frozen future embeddings, including failure and Out-of-Distribution (OOD) predictors.

The anomaly predictors and the multi-task policy are fine-tuned over time. Over deployment, Sirius-Fleet improves on 1️⃣Combined Policy Performance for human-robot teaming, 2️⃣Autonomous Policy Performance for policy success, and 3️⃣Return of Human Effort for human use efficiency.

Check out our paper and the project website for more info! 📃 🌐 A huge thank you to the team @YZ_Franklin, Vaarij Betala, Evan Zhang, James Liu, Crystal Ding, and @yukez ! @texas_robotics

Impressive! Visual World Models could truly revolutionize multi-task robot fleets.

Cool

Cool work. Is the world model trained on the observations in the same environment (i.e. trained for open door / turn on microwave separately)?

Thanks! The world model is trained a wide range of tasks, not specific to one particular environment.

Cool work! When will you present this work?

Thanks! It will be Thursday afternoon, Session 3 at CoRL :)