OpenClaw, Hermes Agent and Prime Agent don't meaningfully learn from experience. On Continual-ARC, our new benchmark at unify, none of them were able to memorize anything of value and none were able to generalize to new tasks. Training a small 9B model's weights did both.

The paper, the code, and replays of every run: Paper: Code: Site:

Notes against weights, on one open model: Qwen3.5-9B, taught by GLM-5.3 Flash. Notes saved nothing measurable on new puzzles. Trained on GLM's moves (DAgger), Qwen spent 57% less than untrained on puzzles it had met, 29% less on 100 new ones. Section 4 ·

How it works: 25 ARC puzzles keep returning over a stream of 200 visits, each time with a new grid, so caching answers is useless. After the stream, two tests: new grids of the puzzles it has seen, and 100 puzzles it has never seen. Section 2 ·

We ran all three on GPT-6 Luna. They solve 93 to 95% of puzzles at high effort, but never get cheaper as puzzles return. Remove their memory, and nothing changes. A baseline that saves a program per puzzle drops straight away and stays down. Appendix I ·

Told exactly what to keep, a checked program per puzzle, all three memorized, saving 17 to 62% of what perfect memory would. But none got better at puzzles they'd never seen. The memory was there all along. The judgment wasn't. Appendix F ·

Why? Prime Agent never stored anything for later (its Python skills also failed to import for us). Hermes Agent opened a skill on 524 of 525 returns, but mostly general tips. OpenClaw kept every conversation, but rarely went back to look. Appendix E ·

Big caveat: one domain and a few models, none the strongest, and stronger ones may well keep more. But I think real learning on the job will need the weights to change, with the harness becoming the thing that trains the model, not a growing pile of markdown files. Section 6

Think your harness or training method learns? Run it on the public stream, then get it verified: we rerun your code on 50 new puzzles nobody outside unify has seen. Feedback of all kinds welcome, especially on where the benchmark falls short 🙏

@letsunifyai Noise matters more than people think.

@letsunifyai Still waiting for an AI that remembers last week

@letsunifyai Hopefully someone builds it soon! My 2 cents is that it will involve evolving weights 👀

@letsunifyai Imagine building an AI that remembers everything, only to forget what you built last week. 😂

@letsunifyai Show the raw number every time.

@letsunifyai Sorry not sure I follow, you mean the absolute performance? The full set of logs are available here if you want to dive deeper!

@letsunifyai Well , I’m impressed 👏.

@letsunifyai Thanks 😁 Easy to poke holes in things, now we just need new research to actually show efficient weight-based learning without a teacher aha

@letsunifyai An agent that can't learn from yesterday isn't really an agent yet. Following this one closely..

@letsunifyai Agreed! We don't have an elegant solution for continual learning yet, but if/when they arrive we'll be sure to measure them! 📏

@letsunifyai Letting people open up agent memory would build a lot of trust.

@letsunifyai How long does improvement usually take to show?

@letsunifyai Great initiative! Current harnesses are still surprisingly ill-equipped to handle evolving tasks. We certainly need more benchmarks that evaluate the dynamism and adaptability of agents if we want to build systems that are adapted to the messiness of the real world

@letsunifyai continuous learning all the way! congrats Dan

@letsunifyai Thanks @Karmedge 🫶

@letsunifyai The baseline that saves a program per puzzle dropping straight away while the frontier models stayed flat is the cleanest demonstration I've seen that context isn't learning.

@letsunifyai starting from zero every time I switch AI tools is rough

@letsunifyai It really depends on the task to be honest, harnesses can deal with some domains pretty well but they just suprisingly suck here!

@letsunifyai Wow, I really thought these agents were already learning from experience, so this is surprising

I think "learning" is a loaded term. They 100% are good at taking notes, and remembering things you told them, but that doesn't always translate into true "learning" imo, as in gaining new understading/intuitive that the base model doesn't already have. In-context learning is limited compared to full training ofc.

@letsunifyai That's a good point! Remembering what happened and actually learning from it are two very different things

@letsunifyai New work is where the claims get tested.

@letsunifyai Yep! I think new work which isn't just a simple memory of past work is the most important test, and this is defo where weights disproportionately help

@letsunifyai The real challenge is building agents that continuously learn, adapt, and improve through experience.

@letsunifyai Yep! There are definitely a lot of open research questions in this regard. I think the next few years will be very exciting.

@letsunifyai Memory is my word of the year.

@letsunifyai ahaha same, or more lackthereof when it comes to agents. Maybe "grep" is my word of the year.

@letsunifyai Release day shouldn't be the only day a model gets better.

@letsunifyai Agreed 😅 Harnesses do help in basic memory for sure, and the products built around them are awesome, but intuitive, associative understanding 100% needs continual weight updates imo

@letsunifyai Continuity between shots is half the work in AI video.

@letsunifyai Sorry not sure what you mean by "in AI video"? 😅 You mean like the fact that this explainer was opus-generated?

@letsunifyai Notes alone feel like a stopgap.

@letsunifyai Agreed 💯

@letsunifyai So it’s editable memory

@letsunifyai Yeah well all harnesses have editable memory, but it has limitations. It acts more like a notepad than true "learning" in the sense that humans continually learn throughout their lifetime.

@letsunifyai Btw sent you dm at @letsunifyai

@letsunifyai Interesting results highlighting the importance of continual learning

@letsunifyai Thanks! To be honest the field is very early, and these are somewhat deliberately provocative. I think there is still a lot to uncover in the coming months/years as weight updates continue to enter the picture.

@letsunifyai Can I run this on my own agent without a GPU?

@letsunifyai Yep! You can see it all here:

@letsunifyai Great work! Looking forward to more updates ❤️

@letsunifyai Thanks! :)

@letsunifyai Waste that most teams never measure. Appreciated

@letsunifyai Agreed!

@letsunifyai AI that learns and remembers is the real breakthrough! 🔥

@letsunifyai Definitely! We're not there yet as a field I don't think, but lots of very cool research out there which is pushing is closer towards this direction 👀

@letsunifyai Notes only help if someone actually reads them.

@letsunifyai lol amen. I also think paradoxically they're less likely to get read as they pile up higher over time.

@letsunifyai Fascinating result small focused training beats big agents that forget everything

@letsunifyai Just to clarify, it's actually the *learning* which is being compared in this work. The bigger finding is that weight-based training results in a bigger delta than in-context learning shows. Big agents can still zero-shot a lot of hard tasks etc. if that makes sense :)

The experimental design is clean, the ablations are informative, and the public stream plus verification path is a generous way to invite the community to test their own systems. Work like this is exactly what the field needs to move beyond narrative claims about “agents that learn on the job.” Looking forward to seeing how others perform on the held out puzzles.

@letsunifyai Context is my bottleneck every single session.

@letsunifyai Real learning makes the difference.

@letsunifyai Yep! It's still not super obvious how we can do continual test-time weight-based learning effectively, but lots of cool research is looking into it.

@letsunifyai Does any agent tool I can buy today actually train on my work?

@letsunifyai a new grid every visit so caching is useless, replays of every run, a hidden set of 50 puzzles for verification, and the big caveat said upfront. this is how a benchmark should ship. great work 👏

@letsunifyai Thanks! 😁🙏

@letsunifyai So far mine mostly remember my name.

@letsunifyai ahahah and even that takes ~10 greps to find out with me

@letsunifyai Impressed honestly

@letsunifyai Thanks! Easy to identify flaws, the real impressive work will be when someone works out how to do true test-time weight-based continual learning 👀

@letsunifyai A teacher's second year is always better than the first.

@letsunifyai I'd hope it would take less than a year 😅