Loading video...
Video Failed to Load
Robotics has spent years scaling demonstration data and model size. What happens when we scale EXPLORATION for robot agents? We're incredibly excited to share RAX, an agentic robotics system that learns by scaling online exploration. A coding agent connects directly to the robot’s perception and control APIs, learns from... show more
76,924 views • 1 month ago •via X (Twitter)
21 Comments

We have open-sourced our code, harness, skill library, and full real robot logs containing Claude sessions! - Github harness code: - Github skill library where anyone can reuse and contribute their own robot skills on their own platforms! - Huggingface dataset with full robot logs and sessions!

scaling online exploration via agentic code generation bypasses the traditional demonstration bottleneck and shifts robotic learning toward autonomous discovery

🚀🚀🚀

Interesting that the frozen model plus a persistent skill repo carries the gains. How do you stop the skill library from drifting when a new embodiment exposes an API quirk that silently breaks an older skill? Regression replays?

👏👏

What limits exploration on the real hardware? Curious how the agent can try something genuinely new without risking the robot or its surroundings.

frozen model + a growing library of programs is the same thing that's working in coding agents. the learning lives in code you can read and diff instead of in weights, which feels like a much bigger deal for robots where you need to know why it did something

What exactly are we scaling? The agent’s budget of online exploration episodes. It executes a strategy, observes what happened, revises its perception and control code, and executes again. Each round builds on the programs, measurements, and discoveries from earlier attempts. The underlying model stays frozen. Learning happens in the agent’s growing workspace of code, notes, and verified skills. As exploration scales, performance on unseen scenes climbs. In our five-benchmark scaling curve: • 4 episodes: 24.9% • 16 episodes: 62.3% • 128 episodes: 78.6% The previous SOTA models average 41.2%. Scaling exploration takes RAX well beyond that, reaching a +37.4-point lead. More exploration lets the agent discover better strategies, turn them into fully autonomous programs, and carry those improvements into new scenes. (2/N)

RAX’s cognitive architecture turns exploration into persistent robot skills. The agent uses RGB-D and force observations to build task-relevant 3D models, then searches for actions through the robot’s control and kinematics tools. It develops the perception pipelines, manipulation strategies, and execution checks itself. As exploration scales, successful decisions become autonomous programs containing the full perception–planning–control loop. These programs run directly; when they encounter an unfamiliar situation, they return control to the agent for further exploration and revision. Memory carries this experience forward: notes preserve discoveries within a session, while a versioned skill repository stores working code, preconditions, and execution evidence across sessions. We also parallelize exploration across agents and environments, cutting the time to reach target performance by more than 2×. (3/N)

RAX separates the agent’s workspace from the robot’s physical stack. The frontend contains the coding agent, programs, notes, and skill repository. The backend provides sensing, low-level controllers, hardware drivers or simulation, and motion checks. They communicate through a tool interface. The agent and its generated programs call exactly the same tools, so code developed interactively can run unchanged as an autonomous program. Every action is recorded. The agent receives compact observations and retrieves detailed execution evidence when needed. Verified skills are committed to a versioned repository for later sessions. This makes RAX multi-embodiment by construction: bring a new robot’s backend and interface documentation, and the same frontend can start exploring. (4/N)

How does one agent control such different bodies? RAX exposes expressive control interfaces alongside model-based tools. The agent can work with joint targets, end-effector poses, coordinated torso–arm motions, or whole-body references, depending on the robot. Low-level controllers handle high-rate tracking. The agent develops the task’s perception, planning, and control logic above them. On the humanoid, it constructs locomotion from whole-body references, measures its gait, and composes walking, turning, and obstacle crossing. On dexterous hands, it plans arm motion and 16 finger-joint targets per hand. On RB-Y1, it coordinates the base, torso, arms, and head. The architecture stays the same. Exploration discovers how to use each body. (5/N)

Scaling exploration produces huge gains on RoboDojo and BEHAVIOR-1K. RoboDojo: 78.3% vs. the prior SOTA’s 24.9%—a +53.4-point gain. Plugging in a charger improves from 26% to 94% with just 4 → 8 exploration episodes. Swapping blocks rises from 34% to 100% with 4 → 16 episodes. On pouring by language, every prior policy baseline scores 0%. RAX reaches 100%. BEHAVIOR-1K: 46.5% vs. 8.2%—a +38.3-point gain on long-horizon household tasks. For loading a car, RAX reaches 100%, compared with the previous best of 20%. Its programs combine perception, navigation, object manipulation, and progress tracking over extended episodes. More exploration builds programs that solve tasks that existing policies barely make progress on. (6/N)

Can scaling exploration handle complex embodiments? On HumanoidArena, RAX reaches 89.8% average success, surpassing WB-WAM’s 81.9% with just 3 task demonstrations instead of 100. WB-WAM is pretrained on 1,880 hours of video and motion data. RAX develops executable programs through exploration, reaching 100% on opening doors and boxing. On DexJoCo, RAX uses ZERO action demonstrations, while prior policies train on 100 per task. With 32 exploration episodes, it reaches 94.3% on the harder bimanual tasks, compared with ME-Dex’s 48.8%. The agent even calibrates its own fingertips from camera observations. Its estimated thumb-tip position is within 1.1 mm of the official hand model it was never given. Whole-body coordination and dexterous manipulation both improve through exploration. (7/N)

On real robots, we collected ZERO teleoperation or action demonstrations. RAX learns precision insertion, air-fryer manipulation, stain wiping, hidden-object tracking, Bandu tower assembly, and long-horizon mobile whole-body control. It builds an articulated 3D model of the air fryer and plans on it. For 0.1 mm-clearance insertion, it uses contact and force feedback to measure surfaces when depth sensing struggles. On RB-Y1, previously learned chair, cup, basket, and box skills combine into a six-stage task. When conditions change, the agent updates the relevant code while retaining working skills. Once exploration produces an autonomous program, execution becomes much more efficient: air-fryer inference overhead falls from ~72 minutes to 18 seconds—about 241× lower. The experience stays with the robot and becomes the starting point for its next task. (8/N)

We’re open-sourcing the full RAX platform: • Agent architecture, instructions, and exploration workflow • Robot interfaces and benchmark integrations • Evaluation protocols • Executable real-robot skills • ALL experiment sessions, including failures, revisions, and successful runs We’re also releasing the RAX Skill Library. Each skill includes its code, preconditions, checks, failure modes, and execution evidence. Later sessions can retrieve, compose, improve, and contribute skills back to the library. Bring a new robot through its perception and control APIs, or contribute what your agent learns. We’re excited to see researchers and industry push scaling exploration further—with new embodiments, harder tasks, and more experience. Framework: Skill library: Project + sessions:

Really cool that every intervention is logged. In the RB-Y1 tidying run the operator gave chat tips like "lift first", how much do you think those tips sped things up compared to the agent figuring it out alone

frozen model plus a growing skill library is a neat setup. nice that the real robot logs are out too

Exploration scales until reset cost on hardware shows up.

Moving away from massive teleoperation datasets toward online exploration is such a fascinating direction. We actually went deeper on this here:

Two unattended runs are a useful floor. I'd tag what changed between them: object pose, clutter, lighting. Otherwise 'verified' can mean 'worked twice before anyone moved the mug.'

Keeping the model frozen while scaling exploration is an interesting approach. How transferable are the discovered skills between robots with different sensors and actuators, versus needing to explore from scratch again?
