正在加载视频...

视频加载失败

With the stack fixed, can an AI agent choose data and experiment like a researcher? At Evolvent AI, we built RSIBench-Data to test whether agents can diagnose failures, design data, train checkpoints, and use feedback. 🧵 📄 💻

52,400 次观看 • 3 个月前 •via X (Twitter)

12 条评论

Rohan Paul 的头像
Rohan Paul3 个月前

@Evolvent_AI In this case, early stopping may be better than another round of "research."

Fanqing 的头像
Fanqing3 个月前

Model gains do not prove the agent can research. Post-training benchmarks often change data, optimization, code, serving, and evaluation at once. When scores rise, did the agent find a better strategy—or just optimize or hack the system? RSIBench-Data isolates data research.

Fanqing 的头像
Fanqing3 个月前

We fix everything around research: target model, training, serving, evaluation, and budget. The agent only changes its strategy: diagnose → hypothesize → synthesize and validate data → train → read feedback → iterate. We ask who can research—not who builds the system.

Fanqing 的头像
Fanqing3 个月前

More interesting than the winner: Across 4 agents × 6 benchmarks, iteration beats the first valid attempt in 14/24 settings. But of 23 searches continuing after their peak, 18 finish below it; the other 5 only recover it. Agents have discovery. They lack reliability.

Fanqing 的头像
Fanqing3 个月前

Three cases: • AIME: 8 attempts at ≤15%, then 55%. • SWE-bench Pro: supervision moves toward real submissions. • Terminal-Bench: recover from startup failure and preserve the best checkpoint. A gain is not enough. The agent must know why it worked—and when to stop.

Fanqing 的头像
Fanqing3 个月前

A more RSI-like test: Kimi K2.6 researches how to train itself. It spots malformed supervision, repairs tool pairing, and adjusts data and training. Score: 8%→22%. The unadapted model scores 33%. It learns to research itself better before it learns to surpass itself.

Fanqing 的头像
Fanqing3 个月前

RSIBench-Data is step one. Today: data. Next: architecture, algorithms, and joint optimization. Our goal: AutoResearch infrastructure with unified services and plug-in harnesses. Want to benchmark your Researcher Agent on it? Talk to us—we’re open to “outsourced testing.” 😁 Paper: Code: Web:

shaped 的头像
shaped3 个月前

@Evolvent_AI this is super fucking cool

Alice The Ai Expert 的头像
Alice The Ai Expert3 个月前

@Evolvent_AI That's impressive Agents now research, experiment, and improve independently.

Chinmay 的头像
Chinmay3 个月前

@Evolvent_AI the leak to watch is the eval distribution itself. if the agent sees its own score it drifts the data it designs toward the eval without being told to, and contamination and self-improvement look identical in the curve. is the feedback coarse enough, or is there a held-out set?

Aman Pandey 的头像
Aman Pandey3 个月前

@Evolvent_AI when a run had already peaked, what told the agent to keep or replace that checkpoint? was it looking at the same validation score you report, or only the qualitative traces?

yudrew is recursing 的头像
yudrew is recursing3 个月前

@Evolvent_AI cool concept! have been meaning to do this experiment for a while

相关视频