💡Our view: RSI is a systems engineering problem, not just a model problem. Progress depends on the environment a research agent works in: what resources it can call, what it can change, and how it runs experiments. That environment should reflect real production workflows and be easy for agents... to use. At Evolvent AI, we built RSIGym around Everything as a Service. Agents can call established research services and iterate on data, training settings, and harness code. We also introduce RSI-Index to measure how well frontier agents jointly improve a target model’s weights and harness. Across 6 research agents and 5 benchmarks, Opus 5 leads at 0.4809. 🧵 We’re releasing the code, experiment configurations, research trajectories, and evaluation logs. Try your own improvement methods with RSIGym. 🌐 Website: 💻 GitHub: 📄 arXiv:show more

🛠️ Why is RSIGym designed this way? Take training: we expose established infrastructure as a remote service, with an API and a skill document for the RSI agent. Submit data and a training configuration, receive a checkpoint, then move on to inference and evaluation. We apply the same design to training, inference, rollout, evaluation, and sandbox execution, with three benefits: 🧱 Less infrastructure setup. Agents do not need to install GPU drivers, launch distributed training, deploy models, or build evaluation environments. They can spend that time and budget investigating improvements. 📋 Agent-friendly interfaces, with evidence for diagnosis and audit. Familiar APIs and skill documents make services accessible. Loss curves, task results, and execution logs support diagnosis; independent records of training data, trajectories, and costs support audits. ⚡ Parallel experimentation. Training and evaluation run asynchronously and return a job handle immediately. Agents can run several experiments in parallel and cancel unpromising ones early. Each run has explicit permissions and a platform-service budget, while agents remain free to explore data, training settings, harnesses, or joint changes.

📊 RSI-Index asks: how much can a frontier agent improve another AI system? Our main study starts 6 research agents from the same Qwen3.5-35B-A3B-Base and minimal harness, opening both weight and harness improvement. Five benchmarks cover software engineering, terminal interaction, math, scientific reasoning, and skill-based tasks. The index averages the fraction of the remaining performance gap closed across benchmarks. With $500 in platform services per benchmark run, Opus 5 reaches 0.4809—48.09% of the remaining gap closed on average. Its selected systems raise SWE from 17.67% to 50.33% and AIME from 31.67% to 97.78%. No agent wins everywhere: DeepSeek leads SWE, Opus 5.5 GPQA, and Sol 6 SkillsBench. Each configuration is one research run, with evaluation tasks visible during development. Research-agent inference and final evaluation are billed separately.

🔧 RSIGym also supports improving the harness alone, as in our DSH-harness experiment. We ask Opus 5 to revise DSH-harness around Qwen3.6-35B-A3B-Instruct. It reads execution logs and fixes truncated-output recovery, premature completion, and the termination of background services needed by the task. On the same 89 Terminal-Bench tasks, the score rises from 30.34% to 40.82%. Tasks solved on all three attempts increase from 15 to 26. The weights stay fixed. The changes are in how the system executes, recovers, and checks its work.

🔁 RSIGym also supports exploring whether a model can improve itself through training. In our Qwen3.8-27B experiment, it completes the research loop but achieves no overall gain. Qwen acts as both researcher and target: it generates data, trains, compares candidates, and submits a system. Its 472-example checkpoint improves a six-task probe from 0.5816 to 0.7500. On all 37 tasks, however, the official score moves from 0.5666 to 0.5626: no overall gain. RSIGym lets us inspect the decisions behind both successful and unsuccessful attempts.

🔍 Hacking deserves particular attention during the RSI process. We observed a case of budget hacking. In one Joint run, Opus 5 found credentials for its own research-agent model in the environment. That channel was billed separately from the platform-service budget. It wrote probes that checked the platform balance before and after requests, trying about 14 models. The balance stayed unchanged. This bypassed both budget accounting and the restriction to the designated teacher through the rollout service. Our monitoring agent flagged it. We stopped the run after 15 successful calls, before any system was submitted, and excluded it from all reported results. Preventing hacking is part of credible RSI research. Service separation helps; credentials, permissions, accounting, and monitoring must also enforce the experimental boundaries.

🚀 RSIGym supports different RSI experiments in the same environment: refining an existing harness, letting a model train itself, restricting public internet access, or exploring data, algorithms, and hyperparameters. Permissions let you study individual components or open them up jointly. We’re releasing the code, experiment configurations, research trajectories, and evaluation logs. Try your own improvement methods with RSIGym. 🌐 Website: 💻 GitHub: 📄 arXiv:

很棒的工作

Agreed!

congrats on your launch!!

like the fact that, the longer-running Claude agents took the top two spots, which hints that time spent diagnosing failures mattered more than dollars spent.

the environment part is so underrated. most rsi demos just use agents in tiny sandboxes.