正在加载视频...
视频加载失败
How should VLAs learn from naturally suboptimal robot demonstrations? Introducing AXIS-Bench: What Matters in Quality-Aware VLA Pretraining, a benchmark and empirical study of how demonstration quality should be defined, labeled, and used during pretraining. Across three VLAs and 24 downstream tasks, we systematically compare quality-aware learning choices and find... show more
33 条评论

2/7 The Idea Human demonstrations often contain hesitation, detours, and uneven progress. But should a VLA discard suboptimal segments, downweight them, or learn to condition on their quality? AXIS-Bench builds on crowdsourced simulation demonstrations with recorded object and contact states and replayable trajectories. By recomputing quality labels on the same trajectories, we can faithfully compare different definitions and granularities.

3/7 Experiment Setup We compare three quality-aware approaches: weighting, filtering, and conditioning, across two reward definitions (object-centric and approach-aware) and two label granularities (episode and action chunk). The controlled study uses 5,000 simulation demonstrations for pretraining across π₀.₅, SmolVLA, and X-VLA. Each configuration is fine-tuned using 354 demonstrations and evaluated on 24 target tasks, with 1,200 evaluation trials.

4/7 Key Findings • How quality is used matters. Approach-aware episode conditioning improves mean success over plain behavioral cloning (BC) across all three VLAs: +3.9%, +23.4%, and +5.6% mean relative gains over three training seeds (π₀.₅, SmolVLA, X-VLA). • Granularity matters. Conditioning favors episode labels, while filtering favors chunk labels on average, with exceptions depending on the model and reward. • Adaptation matters. After conditioned pretraining, constant-tag fine-tuning improves mean success with or without inference-time tags. Adding a high-quality tag only at inference is not always beneficial. Takeaway: Reward definition, granularity, learning method, and fine-tuning protocol need to be considered together.

5/7 Sim to Real AXIS-Bench also includes a reference workflow and baseline for real-world experiment: align the simulated cameras and workspace, re-render recorded demonstrations, and fine-tune on a small set of real demonstrations. On a Franka FR3, plain BC pretraining on 19,250 workspace-aligned simulation demonstrations (~114.6 hours), followed by 25 real demonstrations per task, raises π₀.₅-base success from 9.6% to 31.7% across 10 tasks. This serves as a real-world baseline for future quality-aware pretraining studies.

6/7 Data Release Axis Dataset V1 launched in August. We've since built Dataset V2, now covering 1,092 tasks and 2.35M episodes (including camera augmentation). The companion datasets used in AXIS-Bench are all curated from the V2 pool: • Pretraining dataset. 5,000 simulated Franka demos across 731 tasks, with quality labels and frame indices. • Benchmark dataset. 354 fine-tuning demos across 24 held-out tasks, with task definitions and success checkers, scene assets, and evaluation layouts. • Sim-to-real pretraining dataset. 19,250 replayable simulated episodes across 663 tasks, ready to re-render onto your own workspace for pretraining before real-demo fine-tuning. The two efforts serve complementary goals: Axis grows the data resource, while AXIS-Bench studies how data quality and training choices affect downstream VLA performance.

7/7 Links Explore AXIS-Bench and our broader data resources: Project Resources: Project page: Paper: Code: Try Axis Hub: Datasets: Axis Dataset V2: Pretraining: Benchmark: Sim-to-real pretraining: We hope AXIS-Bench enables more reproducible research on learning from imperfect demonstrations.

In the real world Franka experiments, the plain simulation pretraining baseline already gave a large jump (9.6% to 31.7%). Have you tried applying the best quality aware recipe (approach-aware episode conditioning + constant-tag fine-tuning) on the real robot yet, and if not, what are the main practical barriers?

No single recipe works for every model’ matches what we see collecting teleop data: operator quality varies within a single session, not just between people. Did you find per-episode quality labels beat per-operator ones, or is that too noisy to label reliably?

Per-model filtering is key

Sc@mmer team members

clumsy demonstrations are still demonstrations. curious whether a hesitant human pause counts as low quality or as signal in axis-bench

No single recipe across 3 VLAs is an honest result. Makes me wonder if quality is partly model-specific: does a demo one VLA learns well from hurt another, or is it mostly the training setup?

I’d label useful recoveries separately from the mistakes that preceded them. One quality score for a whole rollout can hide those corrections, so segment-level weighting seems worth testing 🙂

can your quality labels distinguish wasted motion from a useful recovery after a failed grasp?

gAxis

Real-world robot demonstrations are rarely perfect, so learning how to measure and use their quality is an important step toward more reliable VLA models.

Suboptimal demos are the interesting part, most real demos are messy. Does AXIS-Bench reward a policy that recovers well, or one that copies the demo closely?

Always gAxis..🦾

"how quality should be defined and labeled" is the part most teams skip. did different labelers agree on what counts as a suboptimal segment? careful vs hesitating feels pretty subjective

No one is intended in axis now

承认次优数据价值,没有万能配方,多机构联合扎实研究,为机器人落地铺路。

Funny that no single recipe wins: a suboptimal demo may be junk for one model and useful signal for another.

Brilliant work on AXIS-Bench, tackling how VLAs learn from real imperfect demos is huge!

💪💪

Demo quality matters more than model size for VLAs, glad someone finally measured it.

neat tension with Data Inoculation (Levine lab, this week): progress-based labels score recoveries as low quality, but DI finds recoveries are the most valuable data. may be part of why conditioning holds up better than filtering: tagging keeps recoveries, filtering drops them

dark ages

A benchmark that finally tackles learning from imperfect robot demos, this quality-aware research is so important!

Loveeeee

gAxis

We’d like to know how Axis plans to recover after this setback—will there be any further listings?

observe and learn

very interesting...

