Evolution happens to populations.
Useful genes become common in a group over generations, then spread to other groups that test them in their own environments.
The speed depends on generation time, 20 minutes for a bacterium but about 25 years for an elephant.
Using LLMs to evolve better models often looks like an isolated herd of elephants. Each search runs alone on one dataset, and for large models every generation costs a long training run on the full data.
Our new paper, "Discovered, Not Designed," introduces Population Evolution. It has two goals, to discover better models and to discover them faster.
To discover better model designs, the searches share what they find. Several searches run at the same time, each on its own dataset. When one finds a code change that helps, the same change is retrained on the others.
While nature keeps no record of the genes that failed to help, Population Evolution does. All peer results, including weak or unsuccessful transfers, are added to a shared ledger, and the LLM reads that ledger before it writes the next change.
To discover them faster, the searches do not train on the full data, because that is what makes each generation slow.
We found that a search can run on as little as 2% or 5% of the data and still find changes that hold up in full training, as long as the other searches check each change first. In Population Evolution, a change that works on one slice must also work on at least two other slices, then on two larger 10% sets. Only then does it get a full training run.
We tested Population Evolution on RMD-Bench, a benchmark we also introduce for evaluating model discovery.
It covers five task families (LLM pretraining, VLM pretraining, RL algorithm discovery, ranking, and watch-time prediction) with 34 instances in total. Each one comes with a reference implementation and a fixed training and evaluation protocol, so a change found on one dataset or data scale can be retrained and scored on another, and other discovery methods can be compared under the same conditions.
The numbers below are a sneak peek at the two most expensive families of experiments.
The LLM pretraining task starts from a 185M-parameter Gemma 3-style model, and the VLM pretraining task starts from a Qwen3.5-style model with about 646M trainable parameters. The baseline runs the same kind of search directly on the full data, at the same total GPU compute.
Population Evolution finds better models. The best relative accuracy gain for LLM pretraining goes from 0.92% to 2.48%. The best loss reduction for VLM pretraining goes from 5.05% to 8.78%.
Population Evolution finds them faster. In LLM pretraining, each search gets 100 generations where a full-data search gets 14. In VLM pretraining, 57 of 61 full training runs met the success threshold, against 12 of 100.
Evolution works because many groups pass along what helps and get many generations to do it, and Population Evolution gives LLM-driven model search these prerequisites.
Read our paper at for full implementation details and results. The benchmark data and code will be made publicly available shortly.