What if an optimizer could think & react while a neural network trains?
@MIT_CSAIL353K followers
56,311 views
9 Comments
Comments from the original post on X (Twitter)

@rotaruzaurul
Imo, by Goodhart's law, this leads to overfitting. Or at least it assumes something it shouldn't: that the data you have is 100% representative of the task you want to model.
13 likes

@OmkarRay9
wild idea. how many seeds did you compare against? with an agent rewriting the schedule mid run, a lucky run and a smart run could look the same
3 likes

@thesimaximalist
the optimizer’s code becomes part of the search space too, and that’s a fascinating extra lever 💜
2 likes

@aapakari
Fun detail: fifty decisions on the Amharic run, not continuous watching.
1 like

@shuizhuyu
Self-rewriting optimizer? Wild. How’s debugging?

@Cryptnate
Crazy, I thought about something like this recently

@fishjar
The real question is the eval budget: an agent rewriting its own optimizer means every candidate update needs a trustworthy signal, and that search cost rarely shows up in the headline gains. Curious how OPAL amortizes that across tasks versus a tuned AdamW baseline.

@sprabs12
What’s the best way for an academic to learn more about your research without fully committing to a PhD?

@faisalnazir
An optimizer that rewrites its own code mid-run is a strong result and an audit headache. If the winning schedule only exists as a trail of agent edits, reproducing the run means versioning those code diffs alongside the weights. Does OPAL log them?
