Meet Casper👻, a friendly robot sidekick who shadows your day, decodes your intents on the fly, and lends a hand while you stay in control! Instead of passively receiving commands, what if a robot actively sense what you need in the background, and step in when confident? (1/n)

📄paper: 🌐website: (2/n)

Given user teleoperation input, Casper predicts user intent in real time. Upon user confirmation, it fulfills the intent with autonomous execution. Casper's background reasoning runs in parallel with foreground human control to minimize user disruption. (3/n)

Casper's key idea #1: Use VLM commonsense reasoning to infer diverse human intents. Casper generates task candidates from observations and infers intent from user inputs among the task candidates, repeating until predictions are self-consistent. (4/n)

Casper's key idea #2: Use a parameterized skill library to fulfill intents. Once confirmed by the user, Casper executes the corresponding skill with estimated parameters. (5/n)

🙋🏻‍♀️We conduct extensive user studies on multi-step mobile manipulation tasks. At each step, the robot disambiguates user intent among multiple plausible goals, selecting the correct one based on user inputs and visual context. (6/n)

🔑Key insight from the user studies: VLM-based commonsense reasoning is crucial for diverse intent inference in real-world assistive tasks. Casper consistently outperforms the baselines on user workload and user satisfaction, as well as task performance metrics. (7/n)

Checkout our paper and website for more details. A huge thank you to the team @rutavms @dafeijing Jack Pittenger @kiwi_sherbet @YuchenCui1 @ybisk @RobobertoMM @yukez ! @texas_robotics @UCLAComSci @CSDatCMU

cool work!!!

Thanks Mina :)

Human Robot Interaction! I like the way combining human command and lower task execution. The user study is helpful, maybe we will first have Doraemon being teleoped before full autonomy