1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains? We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer. In every language and image setting we tried that has... no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans. 📝 💻 Try it on your task: 🧵show more

Say you're a PhD student, broke on compute, with a half-working new model and an advisor meeting tomorrow. Or you run a factory line, have a few thousand photos of faulty parts, and want an open multimodal LLM to spot them. How would you train the model? All you can do is pick the most standard recipe, or one from a recent paper, and live with the chance that it doesn't suit your setting. A reactive optimizer takes that burden off you: it watches the training, analyzes the signals, and makes optimization decisions as training progresses. 2/9

How does it work? An agent powers the reactiveness, rewriting a file, used in every step of optimization. Code is the entire interface, so the agent can alter any part of the optimization. Before changing anything, it observes training in its own way, through a second program it writes and rewrites, its own probes of the training dynamics. 3/9

We simulate the scenario above in language and image tasks, where no recipe has been optimally tuned for the setting: modded-nanogpt taken to new languages or to a quarter of the batch, Goldfish models on new languages, and LightningDiT and RAEv2 on ImageNet. In all of them, the reactive optimizer beats its starting recipe and matches or beats the best recipe one could pick off the shelf. 4/9

Can an optimizer overcome overfitting by itself? We train a ResNet-18 on CIFAR-10 with 10% wrong labels. Trained straight through, AdamW ends up learning the assigned wrong labels. "The model has started to memorise its training images, and dev loss is climbing fast." Noted by the reactive optimizer at epoch 24. Then it avoids overfitting by itself, navigating by dev-set signals, raising its learning rate, and averaging its weights. 5/9

Reactive optimizers expand what an optimizer can do, with a reasoner in the closed loop. They autonomously build their own training recipes, each adapted to its run: multi-stage schedules for learning rate, clipping, momentum, and more. One switches to per-module clipping after tracing 17 of 18 gradient spikes to a single module; another keeps one matrix's moments in FP32 after deciding rounding would eat its updates. More details are in the blog: the decisions they make are highly specialized. 6/9

Suppressing spikes and overfitting is something human experts can still do, only less conveniently and at a higher cost. As agents get smarter, with hundreds of papers in their memory, thousands of tokens in their chain of thought and millions of training signals in their context, we are convinced reactive optimizers will invent new mechanisms beyond what human experts could invent by hand. Reactiveness can also go beyond gradient-based methods to zeroth-order (like Dust @industriaalist), language-based, and more. 7/9

In the past decade, deep learning models have come to possess unreasonably effective power through training. Yet the very heart of their making, the optimizer, is still mechanistic. Our post is an attempt to breathe the magic that training has given us into the core of training itself, forming a loop that we hope will create more capability in the future! 8/9

Send our pip package (OPAL, OPtimizer with an Agent in the Loop) to your agent and ask it to equip your training with reactive optimizers! 💻 Joint work with Jia-Hao Liang, @jayden_teoh_ and @vincesitzmann.

Interesting! Reminds me of this pioneering prior work 😆

The loop is completed if we get a robot taking signals from the reactive optimizer and controlling its fingers to scroll. 🤣

Very cool work @SimulatedAnneal ! Curious if the agent has access to logits and training data or just the log? Also how do you prevent hacking that may lead to degenerate solutions? E.g. an agent that injects the most optimal gradient without computation for each sample?

Hi Aditya, that is a great question. The agent is only told to see the log and whatever its own analysis code chooses to report. But the code it writes runs inside the training process, so technically it could reach the data and do exactly what you describe, and we currently don't have a hard guardrail for that. All we did was rely on detection of its behavior and keeping evaluation data out of the process. I went through every program version in our published runs, and none touched the data loader or the test set.

Super cool!! Thanks for sharing, the blog post looks great :)

Thanks Sophie! For real, I learned a lot from your blogs

Agent cost: In our default settings, Opus 5.5 high costs $5 to $15 per run. We limit the reaction time to 100 at max; this can be adjusted at will to control budgets or perform more fine-grained control.

really cool!

Thanks, Dhruv!

Very nice work @SimulatedAnneal . I am wondering what would happen if you did it for large scale systems where the parallelization strategy and optimizer tailored kernels are needed to get good system utilization. If agent updates the code the MFU can drop a lot due to this.

Great point, Anuj! We only implemented agent reasoning + training in parallel, while on large scale systems there must be more work done. I don't have sufficient knowledge to say how this can be integrated. 🙂

very cool work! congrats Xingjian

Thank you, Zhengyang! Having fun with agents ☺️

firee

🔥

A useful ablation would be replaying one run's optimizer edits on a new seed, versus letting the agent react afresh. That could separate discovering a better reusable recipe from the value of responding to that particular run's dynamics.

Good point!

The next step is asking the agent to manually decide each individual weight update!

Great

NGL I was doing a DPO training run just yesterday and thought of this instead of curriculum training, WEIGHT THE LR SCHEDULE! . That’s great man. Maybe this is actually a hivemind that predicts trends & patterns in our brains at the same time

This is really interesting!! Can’t wait to give it try!

Letting an agent rewrite optimizer code mid-training is a fun idea, but how do you handle instability? A bad rewrite halfway through training could trash the run, so I'm curious what guardrails you use.

Great point! When we experimented with worse agents, sometimes their code couldn't even compile (although such an issue does not exist for Opus 5.5). The only guardrail we have now is to only accept compilable code from reactions. More restrictions are surely needed.

This is very cool! I’m going to try to run this experiment on some of my larger models soon.

Please do!

very interesting work!!!

Thank you Rahul!

The animation is so cool. Love to watch it

Thank you Benedikt! Put in some tokens and a lot of heart 💜

what did the agent itself cost to run

With Claude Opus 5.5 at high effort, it's $0.2 per decision after the first one. A full run costs ~$10. For example, $6.42 for LightningDiT-XL/1, on top of ~74 H100-hours (~$510) of training. More in Section 4 of the blog:

so the agent is 2% of the training bill

Yes exactly. In larger runs, this relative cost goes down more, especially if you fix the reaction frequency.

so the bigger the run the less the agent hurts

✅ One caveat is that larger runs always compile their (static) optimizers. That can't be done straightforwardly for reactive ones.

Congrats! cool work :)

Thanks Amil 🙂

this is cool!

Thanks, Chenglei! 😊

Poor timestep embedder. The SiT agent cut its rate to 0.15× and traced 17 of its 18 gradient spikes to it, and the LightningDiT one told the spike guard to ignore the embedder's first layer

Poor MLP, just doesn’t fit in with its community 😅 of huge transformer blocks. I guess the optimizer settings are made for the majority.

Half of it got let in at least, the LightningDiT one moved its second layer onto Muon with the big blocks

super cool, congrats!!

This is super cool! How does the agent decide when and how to modify the optimizer? If it relies mainly on training logs, how does it distinguish temporary changes from genuine instability? Does it ever make the wrong change? We'd love to invite you back for another talk!

Hi Chuanxia, thanks really insightful questions! 😊 We did discover intriguing reasoning in agents logs behind their decisions. For example, they trial and error with different optimizer types and LR; or formulate hypothesis, observe additional signals to verify them, and alter them subsequently. Specifically, they use the average of cosine similarity of adjacent gradients as a metric of instability.

Super cool work, thanks for sharing Xingjian!

hmm, like the idea. but how is this fundamentally from agentic training and per-layer optimizer training (given your pic above showing the code)?

I found reactivity the most interesting aspect🙂. Observing and reasoning from its own training steps, the optimizer decides on sophisticated schedules, suppress spikes, avoid overfits, etc. There are also examples where it actively trials and errors along training to gain understanding of the dynamics.

thx! like the idea and happy to try and see how it goes!

vibe training?

Cool! Do you know any tricks to cut the cost down without hurting the performance?

Congratulations on making those discoveries; I will use them in an architecture I am developing to achieve faster training—thanks!

Would it be costly in time? Because you need the agent to think and propose better design for every step, meaning you need to stop a lot in training

ok soo jus curious on wat if opal makes a change that imrpoves loss in short term but hurts the generalization??? how would it catch it before its too lat

very cool!

doing RL for this would be really cool. feel like the set-up/prompt/harness can matter a lot (i.e. let it decide on lr only vs changing entire optimzers). also this is step-size comparison but i imagine training needs to be paused before deciding on the next optimization step so inference compute needs to be allocated optimally (x tokens spent to decide on y steps of optimization or something)

bruh

once the optimizer rewrites itself mid-run, the loss curve is a chain of decisions, not one experiment. how do you replay a run like that exactly?