With the stack fixed, can an AI agent choose data and experiment like a researcher? At Evolvent AI, we built RSIBench-Data to test whether agents can diagnose failures, design data, train checkpoints, and use feedback. 🧵 📄 💻

@Evolvent_AI In this case, early stopping may be better than another round of "research."

Model gains do not prove the agent can research. Post-training benchmarks often change data, optimization, code, serving, and evaluation at once. When scores rise, did the agent find a better strategy—or just optimize or hack the system? RSIBench-Data isolates data research.

We fix everything around research: target model, training, serving, evaluation, and budget. The agent only changes its strategy: diagnose → hypothesize → synthesize and validate data → train → read feedback → iterate. We ask who can research—not who builds the system.

More interesting than the winner: Across 4 agents × 6 benchmarks, iteration beats the first valid attempt in 14/24 settings. But of 23 searches continuing after their peak, 18 finish below it; the other 5 only recover it. Agents have discovery. They lack reliability.

Three cases: • AIME: 8 attempts at ≤15%, then 55%. • SWE-bench Pro: supervision moves toward real submissions. • Terminal-Bench: recover from startup failure and preserve the best checkpoint. A gain is not enough. The agent must know why it worked—and when to stop.

A more RSI-like test: Kimi K2.6 researches how to train itself. It spots malformed supervision, repairs tool pairing, and adjusts data and training. Score: 8%→22%. The unadapted model scores 33%. It learns to research itself better before it learns to surpass itself.

RSIBench-Data is step one. Today: data. Next: architecture, algorithms, and joint optimization. Our goal: AutoResearch infrastructure with unified services and plug-in harnesses. Want to benchmark your Researcher Agent on it? Talk to us—we’re open to “outsourced testing.” 😁 Paper: Code: Web:

@Evolvent_AI this is super fucking cool

@Evolvent_AI That's impressive Agents now research, experiment, and improve independently.

@Evolvent_AI the leak to watch is the eval distribution itself. if the agent sees its own score it drifts the data it designs toward the eval without being told to, and contamination and self-improvement look identical in the curve. is the feedback coarse enough, or is there a held-out set?

@Evolvent_AI when a run had already peaked, what told the agent to keep or replace that checkpoint? was it looking at the same validation score you report, or only the qualitative traces?

@Evolvent_AI cool concept! have been meaning to do this experiment for a while