Skip navigation
Experts doubted a small neural net could beat Craftax without an LLM's knowledge.
0:00 / 0:30

Experts doubted a small neural net could beat Craftax without an LLM's knowledge.

Dan Kondratyuk

@hyperparticle3.1K followers

Visit on X
734 likesDownloadOriginal post

273,315 views

Experts doubted a small neural net could beat Craftax without an LLM's knowledge. ​ Our agent meta-harness built one in 3 weeks. The 15M-param agent beats the final boss 44% of the time, costing a cent, and 10 seconds. ​ GPT-6 Astra won 1%, taking 24 hours and $83. 🧵

16 Comments

Comments from the original post on X (Twitter)

  • Dan Kondratyuk's profile picture

    @hyperparticle

    Our meta-harness is built on top of trackinizer, our OSS graph DB, self-improving its research over time. It launched 4,805 experiments, each training in a few hours on 1 GPU. We recently released a new UI update, which is free to use:

    20 likes

  • Dan Kondratyuk's profile picture

    @hyperparticle

    The final 15M parameter model demonstrates expert-level play, such as creating a little jail for enemies, or pre-firing when the enemies round corners. It even discovered that ice attacks beat fire, and fire attacks beat ice. And it learned this on its own, from scratch, a first

    11 likes

  • Dan Kondratyuk's profile picture

    @hyperparticle

    rekursiv at every level: agents run the research, the learner restarts from its own best moments, and each generation's games train the model that teaches the next. 📖 Full story: 💬 Join our Discord:

    15 likes

  • Dan Kondratyuk's profile picture

    @hyperparticle

    The "3 weeks" here is also a bit conservative. The campaign actually beat the game in a few days, the rest of the time was spent performing ablations to improve the score, and get the meta-harness to discover their ideas faster.

    4 likes

  • Max Informatiker's profile picture

    @maxm_ideas

    15M params cooking for a cent while GPT-6 burns $83 for 1% 💀

    3 likes

  • Silverarrow's profile picture

    @ta_eis_eauton

    Is meta harness OSS?

    2 likes

  • Dan Kondratyuk's profile picture

    @hyperparticle

    We OSS the engine powering it, trackinizer:

    8 likes

  • David Tao's profile picture

    @Taodav

    it feels like we’ve lost the plot here? craftax is an RL benchmark designed to test general aspects of RL algorithms, not LLM designed task-specific policies

    1 like

  • Dan Kondratyuk's profile picture

    @hyperparticle

    The improvements proposed by the agents were fairly general, not task specific. E.g., representing the input board as a 2D grid and using CNNs. The agent mentioned it was inspired was inspired by works like VARC

    1 like

  • Michael Zhang's profile picture

    @Michaelz1914

    sick

    1 like

  • Andrew Prifer (is hiring)'s profile picture

    @AndrewPrifer

    Congrats on the release, I’ll have to play with this

    1 like

  • Alberto Fuentes (e/acc)'s profile picture

    @AlberFuen

    Starcraft skirmish bot dev by agents seemed promising; but if i get It, this is autoresearch from @karpathy training a neural networks on the Game env: proposing new experiments, checking the loss... Seems promising and a great way to form factor universal recursive intelligences

    1 like

  • Albert Buchard 🇪🇺's profile picture

    @AlbertBuchard

    RSI babay

  • Joshua V. Dillon's profile picture

    @dfacto82

    We should clarify this: the craftax meta-harness and results were achieved in a few days. The three weeks was spent improving the trackinizer system.

  • Mooncast Productions's profile picture

    @MooncastOnline

    Are you using a JEPA-like predictive coding world model + action controller model? I did something very basic like this a few months ago, my model was evolutionary and only like 1000 params though.

  • ✨mintha✨'s profile picture

    @kugelbl81659324

    i was waiting for the wave of persistant agents just to see people using robotics ai primitives for the digital work (small high frequency models that don't use words) inside a bigger asyncronous harness