Loading video...

Video Failed to Load

Go Home

Robotics has spent years scaling demonstration data and model size. What happens when we scale EXPLORATION for robot agents? We're incredibly excited to share RAX, an agentic robotics system that learns by scaling online exploration. A coding agent connects directly to the robot’s perception and control APIs, learns from...

76,924 views • 1 month ago •via X (Twitter)

21 Comments

Runhan Huang's profile picture
Runhan Huang1 month ago

We have open-sourced our code, harness, skill library, and full real robot logs containing Claude sessions! - Github harness code: - Github skill library where anyone can reuse and contribute their own robot skills on their own platforms! - Huggingface dataset with full robot logs and sessions!

ZANA's profile picture
ZANA1 month ago

scaling online exploration via agentic code generation bypasses the traditional demonstration bottleneck and shifts robotic learning toward autonomous discovery

Jaeyeon (Jay) Kim's profile picture
Jaeyeon (Jay) Kim1 month ago

🚀🚀🚀

Arber's profile picture
Arber1 month ago

Interesting that the frozen model plus a persistent skill repo carries the gains. How do you stop the skill library from drifting when a new embodiment exposes an API quirk that silently breaks an older skill? Regression replays?

Aadim Nepal's profile picture
Aadim Nepal1 month ago

👏👏

Philipp Paulik's profile picture
Philipp Paulik1 month ago

What limits exploration on the real hardware? Curious how the agent can try something genuinely new without risking the robot or its surroundings.

justin06lee.dev's profile picture
justin06lee.dev1 month ago

frozen model + a growing library of programs is the same thing that's working in coding agents. the learning lives in code you can read and diff instead of in weights, which feels like a much bigger deal for robots where you need to know why it did something

Yilun Du's profile picture
Yilun Du1 month ago

What exactly are we scaling? The agent’s budget of online exploration episodes. It executes a strategy, observes what happened, revises its perception and control code, and executes again. Each round builds on the programs, measurements, and discoveries from earlier attempts. The underlying model stays frozen. Learning happens in the agent’s growing workspace of code, notes, and verified skills. As exploration scales, performance on unseen scenes climbs. In our five-benchmark scaling curve: • 4 episodes: 24.9% • 16 episodes: 62.3% • 128 episodes: 78.6% The previous SOTA models average 41.2%. Scaling exploration takes RAX well beyond that, reaching a +37.4-point lead. More exploration lets the agent discover better strategies, turn them into fully autonomous programs, and carry those improvements into new scenes. (2/N)

Yilun Du's profile picture
Yilun Du1 month ago

RAX’s cognitive architecture turns exploration into persistent robot skills. The agent uses RGB-D and force observations to build task-relevant 3D models, then searches for actions through the robot’s control and kinematics tools. It develops the perception pipelines, manipulation strategies, and execution checks itself. As exploration scales, successful decisions become autonomous programs containing the full perception–planning–control loop. These programs run directly; when they encounter an unfamiliar situation, they return control to the agent for further exploration and revision. Memory carries this experience forward: notes preserve discoveries within a session, while a versioned skill repository stores working code, preconditions, and execution evidence across sessions. We also parallelize exploration across agents and environments, cutting the time to reach target performance by more than 2×. (3/N)

Yilun Du's profile picture
Yilun Du1 month ago

RAX separates the agent’s workspace from the robot’s physical stack. The frontend contains the coding agent, programs, notes, and skill repository. The backend provides sensing, low-level controllers, hardware drivers or simulation, and motion checks. They communicate through a tool interface. The agent and its generated programs call exactly the same tools, so code developed interactively can run unchanged as an autonomous program. Every action is recorded. The agent receives compact observations and retrieves detailed execution evidence when needed. Verified skills are committed to a versioned repository for later sessions. This makes RAX multi-embodiment by construction: bring a new robot’s backend and interface documentation, and the same frontend can start exploring. (4/N)

Yilun Du's profile picture
Yilun Du1 month ago

How does one agent control such different bodies? RAX exposes expressive control interfaces alongside model-based tools. The agent can work with joint targets, end-effector poses, coordinated torso–arm motions, or whole-body references, depending on the robot. Low-level controllers handle high-rate tracking. The agent develops the task’s perception, planning, and control logic above them. On the humanoid, it constructs locomotion from whole-body references, measures its gait, and composes walking, turning, and obstacle crossing. On dexterous hands, it plans arm motion and 16 finger-joint targets per hand. On RB-Y1, it coordinates the base, torso, arms, and head. The architecture stays the same. Exploration discovers how to use each body. (5/N)

Yilun Du's profile picture
Yilun Du1 month ago

Scaling exploration produces huge gains on RoboDojo and BEHAVIOR-1K. RoboDojo: 78.3% vs. the prior SOTA’s 24.9%—a +53.4-point gain. Plugging in a charger improves from 26% to 94% with just 4 → 8 exploration episodes. Swapping blocks rises from 34% to 100% with 4 → 16 episodes. On pouring by language, every prior policy baseline scores 0%. RAX reaches 100%. BEHAVIOR-1K: 46.5% vs. 8.2%—a +38.3-point gain on long-horizon household tasks. For loading a car, RAX reaches 100%, compared with the previous best of 20%. Its programs combine perception, navigation, object manipulation, and progress tracking over extended episodes. More exploration builds programs that solve tasks that existing policies barely make progress on. (6/N)

Yilun Du's profile picture
Yilun Du1 month ago

Can scaling exploration handle complex embodiments? On HumanoidArena, RAX reaches 89.8% average success, surpassing WB-WAM’s 81.9% with just 3 task demonstrations instead of 100. WB-WAM is pretrained on 1,880 hours of video and motion data. RAX develops executable programs through exploration, reaching 100% on opening doors and boxing. On DexJoCo, RAX uses ZERO action demonstrations, while prior policies train on 100 per task. With 32 exploration episodes, it reaches 94.3% on the harder bimanual tasks, compared with ME-Dex’s 48.8%. The agent even calibrates its own fingertips from camera observations. Its estimated thumb-tip position is within 1.1 mm of the official hand model it was never given. Whole-body coordination and dexterous manipulation both improve through exploration. (7/N)

Yilun Du's profile picture
Yilun Du1 month ago

On real robots, we collected ZERO teleoperation or action demonstrations. RAX learns precision insertion, air-fryer manipulation, stain wiping, hidden-object tracking, Bandu tower assembly, and long-horizon mobile whole-body control. It builds an articulated 3D model of the air fryer and plans on it. For 0.1 mm-clearance insertion, it uses contact and force feedback to measure surfaces when depth sensing struggles. On RB-Y1, previously learned chair, cup, basket, and box skills combine into a six-stage task. When conditions change, the agent updates the relevant code while retaining working skills. Once exploration produces an autonomous program, execution becomes much more efficient: air-fryer inference overhead falls from ~72 minutes to 18 seconds—about 241× lower. The experience stays with the robot and becomes the starting point for its next task. (8/N)

Yilun Du's profile picture
Yilun Du1 month ago

We’re open-sourcing the full RAX platform: • Agent architecture, instructions, and exploration workflow • Robot interfaces and benchmark integrations • Evaluation protocols • Executable real-robot skills • ALL experiment sessions, including failures, revisions, and successful runs We’re also releasing the RAX Skill Library. Each skill includes its code, preconditions, checks, failure modes, and execution evidence. Later sessions can retrieve, compose, improve, and contribute skills back to the library. Bring a new robot through its perception and control APIs, or contribute what your agent learns. We’re excited to see researchers and industry push scaling exploration further—with new embodiments, harder tasks, and more experience. Framework: Skill library: Project + sessions:

Yusuf Ahmed's profile picture
Yusuf Ahmed1 month ago

Really cool that every intervention is logged. In the RB-Y1 tidying run the operator gave chat tips like "lift first", how much do you think those tips sped things up compared to the agent figuring it out alone

Yuan's profile picture
Yuan1 month ago

frozen model plus a growing skill library is a neat setup. nice that the real robot logs are out too

Aayushman Singh's profile picture
Aayushman Singh1 month ago

Exploration scales until reset cost on hardware shows up.

Martin Ronfort's profile picture
Martin Ronfort1 month ago

Moving away from massive teleoperation datasets toward online exploration is such a fascinating direction. We actually went deeper on this here:

AxleMode's profile picture
AxleMode1 month ago

Two unattended runs are a useful floor. I'd tag what changed between them: object pose, clutter, lighting. Otherwise 'verified' can mean 'worked twice before anyone moved the mug.'

Alex Vainshtein's profile picture
Alex Vainshtein1 month ago

Keeping the model frozen while scaling exploration is an interesting approach. How transferable are the discovered skills between robots with different sensors and actuators, versus needing to explore from scratch again?

Related Videos

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Jim Fan

468,325 views • 1 year ago

Today, Dream Labs is out of stealth. In February 2026, while we were at NVIDIA, Scott Reed youliang Yunhao Ge and I started calling a new kind of robot model a World Action Model, one that learns how the world moves from video, then uses that to decide how to act. Today, we’re releasing Dream-1 through a public API that anyone in the world can use. When you rent a car you’ve never driven, you don’t learn to drive all over again. Within a few minutes, it feels like your own. Robots should work the same way: a good model should be able to pick up a new robot and get to work, without being retrained from scratch. That’s what we’ve spent the last 6 months on, and it’s what Dream-1 does. It controls bimanual robots with parallel grippers it has never seen before, zero-shot, on unseen tasks in unseen environments. Generalization has been my North Star in robotics: first by distilling video world models into VLAs (DreamGen), then by learning to act from those world models directly (DreamZero), and now by transferring across robots (Dream-1). I’ve come to believe it’s the only direction that lasts. Pushing a narrow model from 90% to 99.9% on a single task is real work, but much of it gets washed away once a model that generalizes arrives. Since the start of the company we’ve stayed single-minded about generalization.That focus is what got Dream-1 to control robots it was never trained on, on tasks it was never shown. And it’s what makes a public API possible at all: a model that only does one task on one robot can’t serve millions of users. Dream-1 is still a research beta, and it will make mistakes. But it already drives cars it’s never seen. Generalization wins in the end, and the end is closer than it looks. Here’s Dream-1 taking the wheel of a robot it’s never seen. Full details 👇

Joel Jang

36,391 views • 29 days ago

🔥 JUST IN: Open-source robotics dataset from 100% real-world scenarios! 🤯 Chinese robotics company AGIBOT just released AGIBOT WORLD 2026, an open-source dataset systematically covering key embodied AI research directions. Built entirely from real-world environments: commercial spaces, and homes. Collected using AGIBOT G2 robots in free-form collection mode, providing structured, accurately annotated, high-quality data. Digital twin technology creates 1:1 scale replicas in simulation matching the real environments. Both real-world and simulation data are open-sourced. The AGIBOT G2 platform collects multiple data types simultaneously: RGB(D) cameras, tactile sensors, force sensors, LiDAR, IMU, and full-body joint states. Whole-body control coordinates arms, waist, and hands for complex tasks. First-person teleoperation lets operators control the robot from its perspective. The tasks covered are fine-grained manipulation, ultra-long-horizon tasks, spatial navigation, dual-arm coordination, and multi-agent/human-robot collaboration. The dataset includes error-recovery trajectories with annotations. Most datasets only show successful demonstrations. AGIBOT includes failures and how the robot recovers, teaching models how to handle mistakes. After collection, data is tested through policy training and real-robot deployment to ensure quality. Then processed through industrial quality control with multiple screening and cleaning rounds. Making it open-source accelerates embodied AI research by giving researchers access to high-quality real-world robot data at scale. 🇨🇳 Learn more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

40,583 views • 7 months ago