Introducing Embodied First Exam: One line of code to run coding agents in 1,090 embodied tasks across 31 simulation suites and 44 robots. Can your agent go far beyond pick-and-place: Puzzles with spatial constraints; Olympic sports with multi-robot collab; Real-time eval, where the world won't pause; Dynamic, agile, whole-body... control; AutoResearch with sim2sim Try your own agent: harbor run -d embodied-first-exam -a -mshow more

Why do we need We built an agent-native infra, to evaluate the coding-agent capabilities that matter for robots: reasoning & planning, contact-rich manipulation, dynamics & latency, safety, whole-body athletic control, competition & collaboration, sim2sim 1,090 tasks, 2,128 task variants, 31 suites, 44 robots, 8 robot types: - With 382 brand-new tasks in 8 original suites: RoboARC, RoboOlympics (Fun + Match), RoboPaint, DexHandExam, RoboDynamicExam, RoboEngineer and SafetyExam. - On top of that, we include 23 suites from community benchmarks and pick hard tasks from them: RoboDojo, BEHAVIOR-1K, RoboCasa, LIBERO, MyoSuite and more. An infra to run coding agents for robotics in one line of code: - Standard: as close to the real world as it gets. Camera only, partial observation, no resets, real time for dynamic tasks. - Privileged: the upper bound. The agent gets raw simulator access and can generate trajectories for training. Allows coding agent to act in three ways: - Direct control: agent as policy - Code to control: code as policy - Train a policy to control: autoresearch

Where we are: 33% on the core leaderboard Leaderboard: Report: Lessons: 1. Privileged ≠ Standard Most tasks agents solve with full simulator access fail when rerun under real-robot rules (camera only, no resets). Standard mode is far from solved, and no single agent covers it. GPT-6 Astra, the strongest agent, solves only about a third of standard and standard-realtime tests. Claude Opus 5.5 and Gemini 4 Argon each solve dozens of standard tests that Astra fails. 2. Where today's agents break Dexterity and tool use, dynamic control and balance, and long-horizon chores. Even with privileged access, GPT-6 Astra took 40 minutes to drive one bolt on IKEA Stegesund and ran out of time on the second. 3. Where today's agents shine They do best when motion can be planned in advance, resets allow free retries, or the rules are spelled out. Opus worked out a toss-and-recatch with the Jacobian before moving. Astra wrote its own C++ brush renderer to paint calligraphy and passed on its first execution. 4. Agents change strategy with the setting. They write programs in privileged mode, steer step by step in standard mode, and run closed-loop controllers in realtime. In a badminton match, Opus's background program tracked the shuttlecock with a drag model and came back from 7–10 down to win 12–10.

RoboARC: Reasoning under Spatial Constraints

EmbodiedOlympics: Competition and Collaboration

RoboDynamics: Real-Time Tasks

RoboEngineer: AutoResearch on Sim2Sim

EmbodiedSafetyExam: Situation Awareness & Safety

Real-world Experiments: the same interface drives a real robot. Here is a real buzz-wire run!

Welcome feedback and contribution: Try your agent. Try your suite. Contributors invited! Website: Report: Code: Discord:

The real time tasks are especially interesting. A coding agent can generate a good plan, but a physical system still needs continuous feedback when balance, contact, or timing changes. Are you measuring how much performance depends on the agent handling that feedback itself versus delegating to a lower level controller? That distinction could tell us a lot about where the actual bottleneck is.

Keeping the world running while the agent thinks is an important constraint. I'd want results broken out by action latency as well as task success.

in the real-time tasks, does a slower model just lose because the sim keeps running while it thinks?

How much of the privileged to standard drop is just state estimation? Wonder how many would pass with the object poses handed back but still no resets

Congrats. I particularly love the diversity of tasks here, especially the dynamic ones

Real-time eval where the world won't pause is the part everyone else dodges, that's the actual test. What's the success spread across all 44 robots?