LLM-generated plans in abstract spaces lack physical grounding. Human demonstrations in high-dimensional continuous spaces lack labels for motion constraints. MIT CSAIL-led research grounds language plans in demonstrations as “manipulation modes” to build robust policies: A thread 🧵

The CSAIL researchers train a fully differentiable pipeline by prompting the LLM to describe demonstrations in a mode sequence & augmenting these demonstrations w/counterfactual perturbations. This pipeline predicts trajectory successes and failures, and as a by-product, it learns classifiers that achieve grounding.

Consequently, a mode-agnostic Behavior Cloning that imitates motion demonstrations cannot replan at the task level when all marbles are dropped due to perturbations. In contrast, CSAIL’s mode-conditioned policy enabled by the learned grounding can leverage LLMs to replan.

Paper: Website: Authors: Yanwei Wang (@felixwyw), Tsun-Hsuan Wang (@johnsonwang0810), Jiayuan Mao (@maojiayuan), Michael Hagenow (@hagenowrobotics), and Julie Shah (@julie_a_shah)

Expand the possibilities of your metabolic research. Resipher tracks real-time cellular oxygen consumption in standard 96-well plates, delivering continuous real-time data directly from your incubator. Request a free virtual demo or quote today >>