Robotics: coding agents’ next frontier. So how good are they? We introduce CaP-X: an open-source framework and benchmark for coding agents, where they write code for robot perception and control, execute it on sim and real robots, observe the outcomes, and iteratively improve code reliability. From NVIDIA Berkeley AI... Research CMU Robotics Institute Stanford AI Lab 🧵show more

At the core of CaP-X is CaP-Gym: an open-source simulation playground for robot agents spanning 187 community standard tasks: robosuite, LIBERO-PRO, and BEHAVIOR. It is designed to be easy to extend with new environments and skills, while a web-based chat UI and robot visualizer powered by Viser make it easy to inspect and interact with agents.

Built on top of the CaP-Gym infrastructure, CaP-Bench can measure how far coding models can push robot control, and what is still missing. The best model we evaluated reaches 32.3% average zero-shot success, while human experts reach 88.5%.

To close this gap, we built CaP-Agent0, a training-free agentic framework built on three scaffolds: text-based visual differencing described by a multimodal model, skill libraries, and parallel reasoning. Scene changes are turned into text, reusable functions are distilled from successful runs, and multiple candidate solutions are explored in parallel. Together, these ingredients can significantly improve success rates and bring coding agents closer to human-level performance on several manipulation tasks.

We also present CaP-RL, where we use GRPO to train Qwen2.5-Coder-7B directly from environment rewards obtained by executing robot programs in simulation. Because CaP-X uses code as the interface for robot control, the resulting model can be deployed in both simulation and on real robots. We find that post-training significantly improves agentic coding performance, bringing the model closer to human experts.

We believe the path to future robotic intelligence includes both data-driven methods and coding agents. To help push that future forward, CaP-X is fully open-sourced under the MIT license. We welcome the community to extend it with new environments, skills, models, and agent scaffolding. This was a massive team effort, and I am deeply thankful to the incredible collaborators who made it possible: @uynitsuj @k_elrefai @ethankou24 @HaoruXue @_wenlixiao @RavenHuang4 @guanzhi_wang @drfeifei @GuanyaShi @jiajunwu_cs Shankar Sastry @yukez @Ken_Goldberg @DrJimFan Paper: Code: Project page:

We encourage the community to take our open source code and integrate their own simulators and deploy on real robots as we have done with Robosuite, LIBERO, Behavior, and on the Agibot and Franka Panda!

Congratulations Max and Haoru!

32.3% zero-shot vs 88.5% human expert — that gap is massive but the approach is right. Code as the interface for robot control is fundamentally more debuggable, interpretable, and transferable than end-to-end VLA models. The GRPO training on Qwen2.5-Coder-7B with environment rewards is the same technique that's dominating agentic RL papers this week (Gen-Searcher, etc). This is becoming the default training recipe for tool-using agents. The fact that simulation-trained models transfer to real robots through code is the key insight most people will miss in this paper. Follow @drawais_ai for daily AI paper breakdowns and resources.

I learned this the hard way. The sim-to-real gap is rarely just a software problem.

The sim-to-real loop is the interesting part. Pure code evals miss where agents usually break.

Congrats bro, really hot stuff!

Hi Max, great work. Do you think it might be worth citing our work on very similar settings?

@berkeley_ai Great work Max!

実は3/13、GTC前に発表してました。巨人と方向性が合ってて良かった。 頑張るぞー。

CaP-X's iterative control mirrors UAV stabilization algorithms I work with daily. German robotics engineers: our banks reject this innovation, while UAE venture capital funds it.

I’d love to train a robot, but I have none. Is it possible to provide a platform where people around the world can train and monitor a robot remotely?

I suppose there are a lot of niches in robotics where code rather than AI will remain the way to go so AI agents specializing in coding for such robots makes sense. Hope it yields positive results.

Super FUN paper to materialize. We found some interesting things happening using @MiniMax_AI 2.7 as model. Great job Max

Interesting to see coding agents evaluated through executable robot behavior instead of just code quality. Does this start shifting robotics progress toward better software abstractions, rather than only better models or hardware?

How fast can it adapt to be a world model ?

Congrats on the release. The 'execute, observe, iterate' loop is the right structure. We've been thinking about skipping the policy entirely, using a semantic scene description and a deterministic solver.

CaP-X is a smart benchmark approach, letting coding agents iterate on real robot outcomes. Curious how you think about the perception gap when the visual domain shifts dramatically. If a robot trained on household scenes encounters an industrial workspace or a surgical environment, how far does the pretrained visual encoder get before it needs domain-specific data?