[NeurIPS D&B Oral] Embodied Agent Interface: Benchmarking LLMs for Embodied Agents A single line of code to evaluate your model! 🌟Standardize Goal Specifications: LTL 🌟Standardize Modules and Interfaces: 4 modules, 438 tasks, 1475 goals 🌟Standardize Fine-grained Metrics: 18 models, 42 metrics, 100+ page analysis Website: Dataset: Code: Docker: PyPI: Doc:

A big shoutout and thank you for our wonderful team @jiajunwu_cs @maojiayuan @drfeifei @percyliang @shiyuzhao @Inevitablevalor @James_KKW @yu_bryan_zhou @RuohanZhang76 @Weiyu_Liu_ @tonyh_lee @cgokmenAI @sanjana__z @erranli !

Ability Module 1: Goal Interpretation

Ability Module 2: Subgoal Decomposition

Ability Module 3: Action Sequencing

Ability Module 4: Transition Modeling

📢We also released 100+ page detailed analysis on 18 LLMs for embodied decision making.

🚀 Key Findings on 18 LLMs for Embodied Decision Making: 🤖 Insight #1 Large Reasoning Models (o1) vs LLMs: -- o1 performs better than all other 16 models in action sequencing (o1 81%, others 60%) and subgoal decomposition (o1 62%, others 48%) -- But NOT in goal interpretation (o1 78%, others 87%) -- Neither in transition modeling (o1 71%, others 68%). -- o1 cost is even higher than all 16 other models in total. It is a trade off. 🏆 Insight #2 Using Models Selectively (​​Specialized Strengths): -- Claude-3.5: Excels in spatial goals (83% F1), Strong transition modeling (68%, o1 is 71% but much slower), Consistently ranks second across tasks on BEHAVIOR (while o1 ranks first but with much higher cost and longer time) -- Gemini 1.5 Pro: Strongest in state goals (87% F1), Performance varies by sequence length: performs much better on VirtualHome (short, avg. length 8.7) than BEHAVIOR (long, avg. length 14.7) -- GPT 4o: Notable in subgoal decomposition (49% Executable Rate, 41% Task Success Rate) -- Mistral & Llama3 (open-weight models) generally perform worse 📊 Insight #3 Comparison of 18 models on 4 Core Abilities: -- Surprisingly, subgoal decomposition is relatively hard, as it is more about declaratively strategizing goal breaking down, requiring precondition understanding of complex scenes and tracking physical locations (e.g., attempting to fetch things from closed containers). -- There is a gap between executable rate and task success rate, around 10% -- Goal interpretation: struggles with complex scenes, common errors include misinterpreting final states (objects, object states, and relations), confusing intermediate subgoals with the final goals, e.g., predicting open(freezer) as a goal for “drinking water”. -- Planning ability is improved a lot by o1. Generally, trajectory feasibility errors are common (45.2%), with a large portion of missing step (19.5%) and additional step (14.2%) errors, often due to overlooking preconditions. For instance, LLMs may ignore the agent’s sitting or lying state before executing other actions. Additional step errors frequently occur when LLMs output actions for previously achieved goals. 📈 Insight #4: How to design better LLMs that can understand the physical world? -- Standardize embodied decision-making using MDP framework -- Balance training between reasoning and transition modeling -- Enhance instruction tuning for embodied decision-making under the MDP framework, with different abilities required. 🔗See more details at

🛠️ Particularly, the evaluation also runs on BEHAVIOR (@drfeifei @jiajunwu which is the first to feature complicated goal annotations (with quantifiers for alternative goal options) and long-sequence trajectory (avg. length 14.6), making it the most challenging embodied decision-making benchmark for LLMs to date. 🔧 We build a symbolic simulator on BEHAVIOR iGibson to enable LLM operating 30 actions to interact with objects through a evolving graph, as well as annotating 100 task trajectories extensively. 🎉 Totally open-sourced! Codebase: Documentation:

@jiajunwu_cs @percyliang @tonyh_lee @maojiayuan @RuohanZhang76 @Weiyu_Liu_ Impressive! Standardizing embodied agent evaluation is a big step forward. Leveraging LTL for goal specs and unifying modules/interfaces across 438 tasks and 1475 goals creates consistency. Fine-grained metrics across 18 models with 100+ pages of analysis highlight rigor. 🚀

@jiajunwu_cs @percyliang @tonyh_lee @maojiayuan @RuohanZhang76 @Weiyu_Liu_ Pattern recognition is not cognition.