Google Research has open-sourced Harness², a framework that enables AI agents to improve their own prompts, tools, memory, and workflows. It tests different configurations and carries useful lessons into future tasks while keeping the underlying model unchanged. An increasing number of releases concerning self-improving models are being published.

it's speeding up, everybody put your seat belts on!!

Held-out tasks decide if the harness improved or just memorized the test

Talked about it here

Do lessons transfer across models, or only within the same setup?

actually looks extremely useful, thx for sharing will test it out asap

Honestly self-improving agents without touching the model is a cool trick Just want to know what happens when it "learns" the wrong lesson and keeps it forever lol And i think @nicksaraev recent video cover something like this

so the harness gets a promotion and the model stays frozen forever 💀

Self-improving harnesses are only as good as their eval. If the agent grades its own configs on a narrow task set, it can overfit to those tasks. Held out tasks matter more than the memory design.

Small catch from the repo: it sits under google-research but is flagged as not an officially supported Google product. And the gains come from editing prompts, skills and tools around a frozen model, so "self-improving scaffolding" is the more accurate label.

Does Harness² test its final configuration on held-out tasks with further changes disabled?

Google open sourcing this kinda tells you they don't know which agent stack wins yet either. Just flooding the zone with tooling

ppl really underestimate Google's DeepMind, their AI Lab is still good despite their latest models struggling to compete with frontier.

Self improving prompts sound great until the eval is wrong, then it optimizes toward the wrong thing faster. The test set ends up being the real product.

do the learned prompts and memory carry over when you swap in a newer model, or does it have to relearn everything from scratch?

Except that models become a harness too

The part I'd probe is what gets remembered: how does Harness² keep a useful lesson from becoming a bad habit on a different task?

keeping the model frozen is the key detail, since every gain has to come from the harness edits alone. curious how well those lessons hold on tasks it has never seen.

wild that the model stays frozen while everything around it keeps getting smarter. makes me wonder how much of 'agent progress' this year was actually better models vs just better scaffolding quietly doing the heavy lifting

tuning the harness is a smart cheat sheet for frozen models. real test will be how well lessons survive out-of-distribution tasks

self improving agents. literally

Worth separating the two: Harness² leaves the model as is and changes the setup around it (prompts, tools, memory). That's easier to inspect and roll back than a model that retrains itself. The risk is in the lessons it carries forward. If one bad lesson from an odd task gets saved, every later task inherits it. How does it decide which lessons to drop?

the model stays the same, the scaffolding teaches itself. compounding, but for prompts

Keeping the model frozen is what makes the gains testable, since they have to come from the harness. Does it carry lessons across tasks by editing the prompt text, or by storing something structured like a config or tool graph?

The key line is "keeping the underlying model unchanged." If the gains live in the prompts, tools and memory around the model, that layer is the moat, not which model you picked. It's now my first diligence question for agent founders: what does your system know after 10,000 runs that it didn't on day one?

you’re wrong about where the leverage is: in my agent runs, prompt search added 6% while replaying failures added 28%. the next frontier is memory of mistakes, not prettier instructions.

Wow 😮

And what also got my eye about this one's that it keeps what it learned in a file with the evidence, so you can go back and see why the agent started doing something different and hill climb it if necessary.

Improving the harness with the model frozen is the cheap path. Better prompts and memory on a $0.10 model often beat paying 40x for a bigger one. I post the cost math on setups like this.

This is something we just began working on; it’s difficult to compete with OpenAI and Anthropic on model usage limits, obviously, so we’re trying just about every possible combination that stretches cheaper models abilities farther. Luna and DeepSeek’s newest models are the main targets right now.

모델은 그대로 두고 프롬프트만 고쳐간다는 게 포인트죠

if the harness edits carry into future tasks, do they also transfer across models? a setup tuned for glm-5 should help opus too, unless the lessons end up model-specific.

Freezing the weights and optimizing the harness is underrated, great move