Skip navigation
With the stack fixed, can an AI agent choose data and experiment like a researcher?
0:00 / 1:30

With the stack fixed, can an AI agent choose data and experiment like a researcher?

Fanqing

@FanqingMengAI2.6K followers

Visit on X
115 likesDownloadOriginal post

52,400 views

With the stack fixed, can an AI agent choose data and experiment like a researcher? At Evolvent AI, we built RSIBench-Data to test whether agents can diagnose failures, design data, train checkpoints, and use feedback. 🧵 📄 💻

12 Comments

Comments from the original post on X (Twitter)

  • Rohan Paul's profile picture

    @rohanpaul_ai

    @Evolvent_AI In this case, early stopping may be better than another round of "research."

    3 likes

  • Fanqing's profile picture

    @FanqingMengAI

    Model gains do not prove the agent can research. Post-training benchmarks often change data, optimization, code, serving, and evaluation at once. When scores rise, did the agent find a better strategy—or just optimize or hack the system? RSIBench-Data isolates data research.

    2 likes

  • Fanqing's profile picture

    @FanqingMengAI

    We fix everything around research: target model, training, serving, evaluation, and budget. The agent only changes its strategy: diagnose → hypothesize → synthesize and validate data → train → read feedback → iterate. We ask who can research—not who builds the system.

    2 likes

  • Fanqing's profile picture

    @FanqingMengAI

    More interesting than the winner: Across 4 agents × 6 benchmarks, iteration beats the first valid attempt in 14/24 settings. But of 23 searches continuing after their peak, 18 finish below it; the other 5 only recover it. Agents have discovery. They lack reliability.

    3 likes

  • Fanqing's profile picture

    @FanqingMengAI

    Three cases: • AIME: 8 attempts at ≤15%, then 55%. • SWE-bench Pro: supervision moves toward real submissions. • Terminal-Bench: recover from startup failure and preserve the best checkpoint. A gain is not enough. The agent must know why it worked—and when to stop.

    1 like

  • Fanqing's profile picture

    @FanqingMengAI

    A more RSI-like test: Kimi K2.6 researches how to train itself. It spots malformed supervision, repairs tool pairing, and adjusts data and training. Score: 8%→22%. The unadapted model scores 33%. It learns to research itself better before it learns to surpass itself.

    1 like

  • Fanqing's profile picture

    @FanqingMengAI

    RSIBench-Data is step one. Today: data. Next: architecture, algorithms, and joint optimization. Our goal: AutoResearch infrastructure with unified services and plug-in harnesses. Want to benchmark your Researcher Agent on it? Talk to us—we’re open to “outsourced testing.” 😁 Paper: Code: Web:

    1 like

  • shaped's profile picture

    @shaped

    @Evolvent_AI this is super fucking cool

    1 like

  • Alice The Ai Expert's profile picture

    @AliceInfoAi

    @Evolvent_AI That's impressive Agents now research, experiment, and improve independently.

  • Chinmay's profile picture

    @Chinmay4o

    @Evolvent_AI the leak to watch is the eval distribution itself. if the agent sees its own score it drifts the data it designs toward the eval without being told to, and contamination and self-improvement look identical in the curve. is the feedback coarse enough, or is there a held-out set?

  • Aman Pandey's profile picture

    @adidshaft

    @Evolvent_AI when a run had already peaked, what told the agent to keep or replace that checkpoint? was it looking at the same validation score you report, or only the qualitative traces?

  • yudrew is recursing's profile picture

    @AndrewDYu

    @Evolvent_AI cool concept! have been meaning to do this experiment for a while