Can autoresearch agents find better training data, not just better training code? Here’s our new EMNLP paper: AutoData: Agentic Search for Pre-training Data Selection [1/6]

Having an agent read every document would be too expensive. AutoData instead writes code (heuristics) to select documents using features like length and perplexity. It trains a small model on the selected data, then uses validation feedback to improve the heuristics. [2/6]

On an already highly curated dataset (ClimbMix), AutoData finds a recipe that further improves NanoChat scores, outperforming existing data-selection methods. The gains also carry over to larger models without re-tuning the recipe. [3/6]

The two best recipes both keep a random sample of the corpus, then use quality signals to select the rest. One recipe makes the reward for low perplexity depend on n-gram diversity, so repetitive text gets less credit. [4/6]

Paper has been up on arxiv and code are opensourced: Paper: GitHub: Blog: [5/6]

Work led by @vivian_yanmy during her internship at Weco. We'd also like to thanks to @NVIDIA for the compute support! [6/6]

better data selection feels more useful than another coding benchmark.

This is a useful reframing: data selection is itself an agentic search problem. The strongest eval would measure not only validation gain, but search cost, diversity, and whether the agent can explain why each sample earned its place.

moving autoresearch upstream to data selection is a bigger lever than tuning code. did the agent's picks transfer across model scales or is the gain tied to the size it was searched on?

“this is a much harder search space than hyperparams imo. bad code usually fails visibly; bad data selection can train perfectly and only reveal itself later as a capability gap”

Autoresearch that improves data selection could be as important as improving the optimizer itself. The evaluation challenge is proving transfer: a dataset chosen for one benchmark should improve real downstream tasks without overfitting to the curator's proxy or amplifying hidden bias in the source pool.

🤔🤔