Morning Bathrobe Rant: Rethinking Harnesses.

alternative title: uncle bob vs the bitter lesson

I told it several months ago, do not develop orchestrators, save your time. You trying to compete with corporations throwing tons of money in their gpus doing the same you are doing . Just relax. Wait till they come with a large carton box and security. Enjoy your last moments in corporate environment which you used to hate most of the time..

I can absolutely relate! For the past 2 weeks, I’ve also experimented with a super basic setup: one agent writes the plan, one writes the code (product and tests), one reviews it. Almost no rules. Worked surprisingly well 👍 Except: the architecture was ... basically not existing 🤷‍♂️😉

What does it mean not considering agents as component of the software design? Great video for me it opens the perspective

Can you please say something more (or have an agent write a list you can share) about what things your harness was orchestrating and what constraints it imposed? That would help us apply what you’ve learned to our harnesses.

my “harness” is simple when claude finishes a task, a hook invokes codex to review and claude gets the output except for the most simple task - codex always comes back with 2-3 valid findings idk how people gain this one shot confidence

Great topic for a Saturday morning! This is something I've been thinking about a lot too. I wonder how far the single-agent advantage holds, especially on longer tasks. How does context rot factor into that comparison? An architect and a developer, each with a narrow role and focused context, could bring complementary perspectives without one conversation carrying the entire history. Does coordination overhead outweigh those benefits, especially on longer tasks? I'll test it myself. My workflow uses multiple coding agents and models, so I suspect the way responsibilities and context are split matters a lot. I'm also obsessed with autonomous programming. I've been building an IDE for multi-agent coding: define agents through Role.md files, organize them into teams, and run as many team instances as you need, each agent using your preferred coding agent.

But I already created the Jira tickets for my team to setup the harnesses 😭

This is simplistic. There are many factors and considerations. Take token efficiency, well by shifting work to an external software harness, you can reduce your token consumption. Part of the code review of the LLM output can be checked by the software harness using abstract syntax trees. An external software harness allows you to have an LLM agnostic usage of AI. Lots of benefits.

All behaviors must be composable. All behaviors must be serializable. Assembling an agent should resemble assembling a webpage from ready-made components that you can also rewrite from scratch where needed. I think we're getting there.

i don't want to promote my harness so I'm not going to paste a link, but I adapted it from zoocode and it works very well especially in orchestrator mode, and allows for swarms of agents. I definitely couldn't achieve the same with a single agent.

Yes! ONE agent holding the entire session context is better. I've been finding that my old protocols, guardrails, and rigid instructions are becoming something I now need to gradually strip away, rather than continue hardening them.

It happens again and again, that it turns out to be unwise to invest a lot of effort into building an infrastructure over components that are themselves at the moment improving in unexpected ways. The best tip is just don't touch that stuff now even with a 9 meter GPU rack.

Agents start fighting the harness correctness?? and the the rules we try to impose/don't do this that. Collectively pulluting context??I don't know..

Even though it sounds weird to say it, I still think "habitat" is the better mental framing Agents work in the habitat you provide them. You don't put a harness on them to control them

4:20 long Nice

Unbelievable pace. Thanks for sharing. How can we best not be absorbed in one idea and missing what will be possible?

Moving constraints from deterministic tools to agents worked, but delegating architecture-drift prevention and mutation testing to agents didn't. Agent decides it's unnecessary and skips it, while deterministic tools enforcement gets the result we want.

For my design, I don't steer the agent's thinking. I gate its transitions. Inside a phase it works freely; to leave, it must write distilled notes a schema demands. Every run ends in a retrospective that proposes config changes. The harness learns, so friction points are fixed.

Grok @elonmusk

q q

makes me think that orchestration is inherently hard the way multi-threading is and maybe some problems are best done whole by a single agent the way that a single mind can produce a more coherent solution. i suppose a large enough problem that simply is too large for a single mind/agent could be sliced along natural seams (which might be hard to determine a priori) and then those parts done whole by different agents. in other words, slice vertically by feature not horizontally by skill

Engineers engineer. Our humble task now is to get out of our own way.

I think breaking down a plan into sub-agents make sense if the task is completely independent. But I agree with your assessment. LLMs are getting better in handling multi task segments of the overall assignment without the need to break it down into smaller subgents.

With plurnk, I have WORK (new log), FORK (forked log), and BARE (no log or tools, pure inference) as primitives the model can reach for to devise its own arbitrary graphs and topologies as it sees fit. Seems to answer for this question.

yep… did a very complex harness w langgraph before for my use case - thin harness , fat skills idea from Garry of Y combinator worked very well w my use case

Maybe the review framework is the problem and CRAP and the rest of the stuff are actually bad

@unclebobmartin What do you mean by components in software design?

free the agents! ha it depends on the agent but yeah if the task is too small they can't see the bigger picture. they still can't do too big of a task that takes too much context as they'll do stupid things also. so long as the prompt and the task size are in the sweet spot they seem to do well

I keep thinking about Django and Ruby on Rails. Their architectures and design patterns are burned into my brain. It's easy to whiteboard parts of applications and to review sensitive code

What about harnesses for other engineering disciplines: civil-structural-harnesses? Why only for developers?

Do you thing your harness would still be useful with pre-SOTA or smaller open weight models that are not that smart?

I felt the same! I used to decompose tasks very granular, then, it start "feeling" slow, Then i tried the same: "just let the main thread do everything" And then.... WTF?! Anyways... Anybody else is thinking/working on the WHY is this?

Be nice to know the prompt/task so we can evaluate too? Got to wonder where this leaves us

Question: Have you noticed the harness getting better overtime especially with Grok Bot?