正在加载视频...

视频加载失败

How should VLAs learn from naturally suboptimal robot demonstrations? Introducing AXIS-Bench: What Matters in Quality-Aware VLA Pretraining, a benchmark and empirical study of how demonstration quality should be defined, labeled, and used during pretraining. Across three VLAs and 24 downstream tasks, we systematically compare quality-aware learning choices and find...

14,035 次观看 • 15 天前 •via X (Twitter)

33 条评论

Axis Robotics 的头像
Axis Robotics15 天前

2/7 The Idea Human demonstrations often contain hesitation, detours, and uneven progress. But should a VLA discard suboptimal segments, downweight them, or learn to condition on their quality? AXIS-Bench builds on crowdsourced simulation demonstrations with recorded object and contact states and replayable trajectories. By recomputing quality labels on the same trajectories, we can faithfully compare different definitions and granularities.

Axis Robotics 的头像
Axis Robotics15 天前

3/7 Experiment Setup We compare three quality-aware approaches: weighting, filtering, and conditioning, across two reward definitions (object-centric and approach-aware) and two label granularities (episode and action chunk). The controlled study uses 5,000 simulation demonstrations for pretraining across π₀.₅, SmolVLA, and X-VLA. Each configuration is fine-tuned using 354 demonstrations and evaluated on 24 target tasks, with 1,200 evaluation trials.

Axis Robotics 的头像
Axis Robotics15 天前

4/7 Key Findings • How quality is used matters. Approach-aware episode conditioning improves mean success over plain behavioral cloning (BC) across all three VLAs: +3.9%, +23.4%, and +5.6% mean relative gains over three training seeds (π₀.₅, SmolVLA, X-VLA). • Granularity matters. Conditioning favors episode labels, while filtering favors chunk labels on average, with exceptions depending on the model and reward. • Adaptation matters. After conditioned pretraining, constant-tag fine-tuning improves mean success with or without inference-time tags. Adding a high-quality tag only at inference is not always beneficial. Takeaway: Reward definition, granularity, learning method, and fine-tuning protocol need to be considered together.

Axis Robotics 的头像
Axis Robotics15 天前

5/7 Sim to Real AXIS-Bench also includes a reference workflow and baseline for real-world experiment: align the simulated cameras and workspace, re-render recorded demonstrations, and fine-tune on a small set of real demonstrations. On a Franka FR3, plain BC pretraining on 19,250 workspace-aligned simulation demonstrations (~114.6 hours), followed by 25 real demonstrations per task, raises π₀.₅-base success from 9.6% to 31.7% across 10 tasks. This serves as a real-world baseline for future quality-aware pretraining studies.

Axis Robotics 的头像
Axis Robotics15 天前

6/7 Data Release Axis Dataset V1 launched in August. We've since built Dataset V2, now covering 1,092 tasks and 2.35M episodes (including camera augmentation). The companion datasets used in AXIS-Bench are all curated from the V2 pool: • Pretraining dataset. 5,000 simulated Franka demos across 731 tasks, with quality labels and frame indices. • Benchmark dataset. 354 fine-tuning demos across 24 held-out tasks, with task definitions and success checkers, scene assets, and evaluation layouts. • Sim-to-real pretraining dataset. 19,250 replayable simulated episodes across 663 tasks, ready to re-render onto your own workspace for pretraining before real-demo fine-tuning. The two efforts serve complementary goals: Axis grows the data resource, while AXIS-Bench studies how data quality and training choices affect downstream VLA performance.

Axis Robotics 的头像
Axis Robotics15 天前

7/7 Links Explore AXIS-Bench and our broader data resources: Project Resources: Project page: Paper: Code: Try Axis Hub: Datasets: Axis Dataset V2: Pretraining: Benchmark: Sim-to-real pretraining: We hope AXIS-Bench enables more reproducible research on learning from imperfect demonstrations.

Kosasi Nakamoto 的头像
Kosasi Nakamoto14 天前

In the real world Franka experiments, the plain simulation pretraining baseline already gave a large jump (9.6% to 31.7%). Have you tried applying the best quality aware recipe (approach-aware episode conditioning + constant-tag fine-tuning) on the real robot yet, and if not, what are the main practical barriers?

Sriram Gopalan 的头像
Sriram Gopalan14 天前

No single recipe works for every model’ matches what we see collecting teleop data: operator quality varies within a single session, not just between people. Did you find per-episode quality labels beat per-operator ones, or is that too noisy to label reliably?

Steven Cheng 的头像
Steven Cheng14 天前

Per-model filtering is key

Puri 的头像
Puri14 天前

Sc@mmer team members

Dominik 的头像
Dominik14 天前

clumsy demonstrations are still demonstrations. curious whether a hesitant human pause counts as low quality or as signal in axis-bench

Timo 的头像
Timo15 天前

No single recipe across 3 VLAs is an honest result. Makes me wonder if quality is partly model-specific: does a demo one VLA learns well from hurt another, or is it mostly the training setup?

The SI Maximalist 的头像
The SI Maximalist15 天前

I’d label useful recoveries separately from the mistakes that preceded them. One quality score for a whole rollout can hide those corrections, so segment-level weighting seems worth testing 🙂

Jatin Garg 的头像
Jatin Garg15 天前

can your quality labels distinguish wasted motion from a useful recovery after a failed grasp?

Ranger 的头像
Ranger14 天前

gAxis

Flix Muller 的头像
Flix Muller14 天前

Real-world robot demonstrations are rarely perfect, so learning how to measure and use their quality is an important step toward more reliable VLA models.

AMAN VAIBHAV 的头像
AMAN VAIBHAV14 天前

Suboptimal demos are the interesting part, most real demos are messy. Does AXIS-Bench reward a policy that recovers well, or one that copies the demo closely?

0xyussa 的头像
0xyussa15 天前

Always gAxis..🦾

Omkar Ray 的头像
Omkar Ray14 天前

"how quality should be defined and labeled" is the part most teams skip. did different labelers agree on what counts as a suboptimal segment? careful vs hesitating feels pretty subjective

zayan 的头像
zayan14 天前

No one is intended in axis now

小白勇闯币圈 的头像
小白勇闯币圈14 天前

承认次优数据价值,没有万能配方,多机构联合扎实研究,为机器人落地铺路。

Atomik 的头像
Atomik14 天前

Funny that no single recipe wins: a suboptimal demo may be junk for one model and useful signal for another.

Aria Tech 的头像
Aria Tech13 天前

Brilliant work on AXIS-Bench, tackling how VLAs learn from real imperfect demos is huge!

Korakot Budsabun 的头像
Korakot Budsabun15 天前

💪💪

Shinka - AI 的头像
Shinka - AI15 天前

Demo quality matters more than model size for VLAs, glad someone finally measured it.

Nirav 的头像
Nirav13 天前

neat tension with Data Inoculation (Levine lab, this week): progress-based labels score recoveries as low quality, but DI finds recoveries are the most valuable data. may be part of why conditioning holds up better than filtering: tagging keeps recoveries, filtering drops them

Samuel Lassoff 的头像
Samuel Lassoff14 天前

dark ages

Elise Tech Ai 的头像
Elise Tech Ai12 天前

A benchmark that finally tackles learning from imperfect robot demos, this quality-aware research is so important!

Horse 的头像
Horse14 天前

Loveeeee

ARIF AHMED 的头像
ARIF AHMED15 天前

gAxis

Sílvio 的头像
Sílvio15 天前

We’d like to know how Axis plans to recover after this setback—will there be any further listings?

MarvelCrypt | 3D Motion Designer 的头像
MarvelCrypt | 3D Motion Designer15 天前

observe and learn

SAN.SOMETHING 的头像
SAN.SOMETHING15 天前

very interesting...

相关视频