Cooking in kitchens is fun. BUT doing it collaboratively with two robots is even more satisfying! We introduce MOSAIC, a modular framework that coordinates multiple robots to closely collaborate and cook with humans via natural language interaction and a repository of skills.

To collaboratively cook, the robots must: (1) Interact with users through language (2) Master a wide array of skills (3) Collaborate safely by understanding the user's intent A single end-to-end model would require a lot of demonstrations and be challenging to debug.

MOSAIC's key insight is modularity: decomposing the challenging problem into more manageable modules. MOSAIC uses: * large-scale models for tasks that require broad generalization (e.g. language and image recognition) * specialized modules for task-specific control

Using LLMs for task planning enables fluid natural language interaction, but relying on them alone is prone to hallucinations and constraint violations. Instead, MOSAIC embeds LLM calls within a behavior tree to partition the reasoning process, enhancing system reliability.

MOSAIC enables flexibility! The task planner can flexibly modify or add tasks based on users' requests. Given an instruction, * the LLMs infers users' intention to update the robot's plan * the behavior tree structure ensures the plan is updated without violating constraints.

To handle the wide range of household objects, MOSAIC uses internet-scale VLMs conditioned on the task to extract object position. Using this information, an RL policy trained in simulation can robustly solve the task without needing any demonstrations!

MOSAIC enables recoverability! When the robot is interrupted by users mid-execution, its visuomotor skill can adapt to new tasks because: * the VLM identifies the target object given a new prompt * the RL policy recovers from its current position and completes the new task.

To enable safe and seamless interactions in real-time, MOSAIC predicts future human motion based on past observations. The forecasting module is pre-trained on large-scale human motion data and fine-tuned on human collaboration data, ensuring high accuracy during interaction.

MOSAIC enables safety! When working closely with users, predicting future human motion is critical for seamless collaboration. The human motion forecasting module * extracts users' joint position from an RGB image * predicts future human pose that is used for motion planning

We evaluate MOSAIC on 60 end-to-end trails, where 2 robots collaborate with a human user to cook one of 6 recipes. Each trial involves different robot skills and types of user interaction. We report the number of successes, subtasks completion rate, and failure cases.

This project is the culmination of a massive team endeavor at Cornell's PoRTaL lab! Check out our paper and project

Blown away by what 2 second-year grads (@YukiWang_hw, @kushalk_ ) and a team of Cornell undergrads achieved here! Getting 2 different robots, to cook with humans not once but over multiple hours is robotics at its best. Seen a lot over the years, but this was very special!

Congrats on the great work and the massive systems undertaking!

Really interesting work on collaborative cooking!

What a impressing job. I'm greatly interested in your Interactive Task Planner componnet. So will the project be open sourced in the future?