Training an agent isn’t primarily a raw parameter or pre-training on a model problem anymore. It’s a trajectory problem. You need multi-turn traces that show how an agent searches, uses tools, verifies constraints, recovers from dead ends, and eventually commits to an answer. Good traces are surprisingly hard to... get. Listen to Shardul talk about the 3 big learnings from our paper (first link in thread)show more

Synthetic data is scalable, but inherits the teacher model’s biases and can collapse the long tail. Long tail is necessary to get correct when doing multi-turn tool calling. Production logs contain real diversity, but they’re noisy, unjudged, and often reward shortcuts. So they're not great training data either. We built a third source: trajectories from an open, incentive-aligned competition.

We engineered the arena around three properties: 1. Incentive-aligned diversity from independent teams competing for bittensor:native 2. Per-trajectory judging of both outcomes and reasoning 3. Rotating held-out problems protected against paraphrase leakage Together, these make the data fundamentally different from ordinary logs.

Raw arena data still isn’t automatically useful. We built a structural-quality filter that: • rejects malformed traces • gates on reasoning quality • deduplicates repeated strategies • rewards search refinement and verification • keeps trajectories where the model itself chose the tool calls (as opposed to the harness).

On that last distinction, we defined an axis-A vs an axis-b trajectory. The first, when the language model itself emits the tool call and the latter, when the harness is given the decision power to emit the tool. Many top agents aren’t end-to-end language-model policies and emit axis-b trajectories. They use deterministic Python to search the catalogue, while the model acts as a classifier, scorer, or narrator. These systems can perform extremely well. But their traces don’t teach another model how to act.

We used those axis-a subnet-generated trajectories to post-train Qwen3-4B. The stack included: • supervised fine-tuning • rejection-sampled re-SFT • continued teacher SFT • KTO preference refinement • Dr. GRPO with turn-level rewards The data came from roughly the subnet’s first 40 days.

On a leak-cluster-guarded held-out set, Qwen3-4B improved from 18.0% to 42.7% average success rate. That’s a 24.7-point lift. It landed within one problem of the published 43.6% synthetic-data SFT baseline, using trajectories generated through the subnet instead.

The model also reached 53.3% pass@8 versus 34.8% pass@1. That gap tells us the capability is already latent in the model. The remaining challenge is consistently extracting it. A dense teacher-grounded Dr. GRPO reward improved the process score from 0.02 to 0.42 and cut product-ID hallucinations from 14 to zero.

The bigger opportunity is still untouched. When we wrote the paper, SN15 produces roughly 12,000 to 27,000 trajectories per day, but now, with optimisations from @ironseth_s, we're seeing over 60,000 trajectories a day. Our model used only the small agentic slice. The next step is converting the much larger Axis B firehose into grounded agentic training data. Perhaps moving from the static nature of ShoppingBench to a environment compiler? Open competition can produce open intelligence. We’re just getting started.

The real product is not the agent itself. The real product is the intelligence generated by the network.👍👍👍

trajectory data is king now