MOSAIC A Modular System for Assistive and Interactive Cooking We present MOSAIC, a modular architecture for home robots to perform complex collaborative tasks, such as cooking with everyday users. MOSAIC tightly collaborates with humans, interacts with users using natural

language, coordinates multiple robots, and manages an open vocabulary of everyday objects. At its core, MOSAIC employs modularity: it leverages multiple large-scale pre-trained models for general tasks like language and image recognition, while using streamlined modules designed

for task-specific control. We extensively evaluate MOSAIC on 60 end-to-end trials where two robots collaborate with a human user to cook a combination of 6 recipes. We also extensively test individual modules with 180 episodes of visuomotor picking, 60 episodes of human motion

forecasting, and 46 online user evaluations of the task planner. We show that MOSAIC is able to efficiently collaborate with humans by running the overall system end-to-end with a real human user, completing 68.3% (41/60) collaborative cooking trials of 6 different recipes with a

subtask completion rate of 91.6%. Finally, we discuss the limitations of the current system and exciting open challenges in this domain.

paper page:

It would be helpful if you could write the abstract of a paper in one tweet. If I am not an English speaker, I will use the translation feature, but if you write it in a reply thread, the English will be interrupted in the middle, making translation difficult.

El futuro, eso sí que es Emocionante!!