introducing AlphaLab -an autonomous system that runs quantitative research end-to-end give it a dataset + goal, it explores, reads papers, builds evals, and runs parallel experiments across a GPU cluster, learning&evolving from the results all details, code, and paper below

Why build this? as many have noted, something changed with these models in december 2025, the agentic coding ability (to me) had a phase change. they can genuinely operate autonomously for hours - writing code, debugging, iterating. "AI scientist" efforts have been around for a while, but it feels like the time is finally right for domains with measurable outcomes. @karpathy 's auto-research is a great example of this moment - brilliantly simple and effective. We were actually building AlphaLab in parallel (embarrassingly similar timing), but where we ended up diverging is in three areas we think matter a lot: 1. A real research phase - before touching a GPU, the system reads SOTA papers on arXiv, surveys existing approaches, and deeply analyzes the dataset. Our ablations show skipping this alone costs 12.5%. 2. Self-adaptation - the system writes its own evaluation framework, its own prompts, its own metrics. This is what makes it domain-agnostic: you hand it a new problem and it figures out how to measure success, catches its own bugs (lookahead bias, leaky features), and adapts - no human engineering per task. 3. Massive parallel experimentation with synthesis - instead of sequential propose-and-revert, it runs dozens of experiments in parallel across a GPU cluster. But the key part isn't just throughput: after each batch, a Strategist agent synthesizes all the results, updates a persistent knowledge base, and proposes new experiments informed by everything it's learned. The search evolves.

at a high level, AlphaLab is just an agentic harness - tools + a structured environment. the power largely comes from the underlying model. we experiment with opus 4.6 and gpt-5.2 the tools are simple: shell access (~50% of all tool calls), web search, file reading, sub-agent spawning, and a few coordination tools (propose experiment, update playbook, read leaderboard) the environment is the four phases the system moves through - that's where the structure lives. Each phase produces artifacts the next one consumes

Phase 0: the system adapts itself to whatever problem you throw at it. It generates all its own prompts, metrics, and domain knowledge by examining your actual data and searching the web. Same pipeline, zero code changes between domains. We work in financial time series forecasting a lot - but to show this is general, we evaluate on three very different domains: - CUDA kernel optimization - LLM pretraining (inspired by @karpathy's NanoGPT speedrun, using our favorite pretraining dataset, PleIAs SYNTH, from @Dorialexander ) - Traffic forecasting Plus a financial forecasting experiment in the appendix where the adapter was generated entirely from scratch.

Phase 1: the system research the data and the domain it writes a detailed plan, then works through it autonomously - writing Python scripts, generating plots, searching the web for relevant papers and best practices, and updating its notes after each finding. On traffic forecasting it discovered dual seasonality (daily + weekly), cross-sensor correlations, and that you need >= 192h context to capture weekly patterns - all before a single experiment ran. On LLM pretraining it sampled the dataset, discovered byte-level tokenization enables direct BPB computation, and found that including the query prefix materially improves loss. This phase runs 1-2 hours and produces a research report + machine-readable learnings that feed everything downstream.

Phase 2 - the system adversarially build a evaluation harness: A Builder writes the full harness (data loading, splitting, metrics). A Critic - a fresh agent with zero shared context - audits for data leakage, lookahead bias, and metric bugs. A Tester writes and runs automated tests. This loop caught real issues in practice: a context window that could include the target period, a MASE denominator using the wrong seasonal period, a torch.allclose correctness check with swapped arguments. If your eval is wrong, you're optimizing the wrong thing and you'll never know. This is the most important phase.

Phase 3: the system runs experiments - continuously - at scale A Strategist agent proposes experiments, Worker agents implement and submit them to a GPU cluster via Slurm, and after each run the system analyzes what happened and why. Each experiment moves through a kanban board: queued -> implement -> execute -> analyze -> done (with a fix state for failures). It learns from every result - what works, what doesn't, what to try next - and the strategy evolves over the course of the campaign In practice what we love about this is - it scales to whatever cluster you have and it just lives there. Whenever there's an idle GPU, AlphaLab can fill it with something interesting. If someone needs the GPU back, just kill the job, doesn't matter, it adapts and moves on. you can basically just leave it running in the background and it's always making progress. We cap campaigns at 50 experiments in the paper for fair comparison, but in practice it just keeps going as long as you let it

Results: CUDA kernel optimization. We evaluate on KernelBench (extended by @SakanaAILabs 's CUDA Engineer benchmark) -- 200 tasks where you write a CUDA kernel to match PyTorch's output but faster. The system writes custom GPU kernels and benchmarks against torch.compile. Across 119 tasks: - 4.4x mean speedup over torch.compile - Best kernel: 91x speedup (fused LayerNorm + residual add) - Warp-shuffle reductions: 73-75x - Algebraic rewrites (diag matmul -> elementwise): 10-68x Against torch.native (for comparison with prior work), we get 3.47x mean speedup vs Sakana's decontaminated 1.49x. It also learned what NOT to do -- handwritten convolution kernels consistently lost to cuDNN (0.05-0.73x). The playbook explicitly warns future experiments away.

Results: LLM pretraining speedrun. Inspired by NanoGPT speedrun - train a <100M param LM from scratch on the @pleiasfr SYNTH dataset in 20 minutes on one H100, minimize val bits-per-byte. We ran with three models -- they all found different solutions: - Opus 4.6: 0.758 bpb (10L x 752d, custom BPE tokenizer) - Sonnet 4.6: 0.869 bpb (11L x 768d, Muon+AdamW hybrid) - GPT-5.2: 0.970 bpb (8L x 512d, byte tokenizer, GQA) That's a 22% gap between Opus and GPT-5.2 - from the same system, same data, same budget. Different models explore different regions of the search space. A simple greedy loop baseline gets 1.020. Single-shot gets 1.248.

Results: traffic forecasting. 862 Bay Area freeway sensors, predict 24h ahead. Seasonal naive baseline: 0.0287 RMSE. - Opus: 0.0214 (-25%) via TFT -- locked in on this architecture early and refined relentlessly - GPT-5.2: 0.0220 (-23%) via iTransformer -- explored 5+ architecture families The most interesting finding across ALL domains: different models discover qualitatively different solutions. Neither dominates. GPT-5.2 leads on CUDA kernels, Opus leads on LLM pretraining and traffic. This is the strongest argument for multi-model campaigns - you're sampling different regions of a huge search space

Some honest caveats: This is a big, involved system with a huge number of design choices. It's very early. There are a lot of ways to poke holes in it - single-run comparisons, no sandboxing, premature playbook convergence, environmental fragility But it is really working. Messy, sometimes frustrating, but genuinely producing results that would take a human researcher days or weeks. And the thing is - every component gets better as models improve. Better code generation = fewer failures. Better reasoning = smarter strategy. Better context = richer playbooks. This is a system that rides the scaling curve. Each campaign for the paper costs ~$150-200 in API calls and runs 12-48 hours on 4xH100s.

Full paper, code, prompt templates, playbook excerpts, and additional results all available here: If you're interested in autonomous research systems, multi-agent experimentation, or just want to see what frontier LLMs can do when you give them a GPU cluster and tell them to go work - check it out. Would love feedback, especially from people working on similar systems.

that is a fantastic idea to build such pipeline. I also build a similar one but in a more limited scope, ONLY in alpha feature construction. Hope you could give me some guidance:

looks cool! I think giving it web search and encouraging it to do look at related works is very valuable, and also having it to parallel search to try out a lot of ideas helps too

wow, thanks for your valueable suggestions, will add!

the eval loop is what makes this viable. built something similar for a client where the model chose its next experiment based on failure signals. the hand-coded plan got abandoned within 3 iterations every time.

sounds interesting~

seriously cool - thanks for sharing!!

@llmgram

Is this based on auto research ?

was building it before he released, but they are similar for sure - I think main advantage here is the research phase, and the parallel experiments

Cool. You think I could use it for any strategies ? I wanna publish a paper on harmonic patterns, by far I could not find any researched backed paper that shows whip pattern types are effective with which indicator filters.

yes definitely! I think its very generic what it can be used for