I asked Simon Willison what the next leap in AI software engineering is likely to be. He explained the "dark factory" pattern where teams don't write any code or even look at their code.

He writes about this pattern here

the "dark factory" framing is misleading in one important way: real dark factories didn't remove quality control. they automated it. the equivalent in code isn't "don't look at it." it's building an eval layer so comprehensive that reading the code becomes optional. most teams skip straight to "let agents write everything" without building the inspection infrastructure first. that's not a dark factory. that's just running blind.

@simonw lol this is what I'm (and I'm sure alot of other vibe coders) are already doing

@simonw I have a different perspective on this. I envision a new abstraction layer for programming languages where developers define high-level specifications, and intelligent agents reliably handle the implementation.

@simonw This is incredibly concerning. The tools are no where close to be reliable enough to do that...

@simonw dark factories sound efficient, but who's crafting the vision? that human touch matters in making software meaningful.

@simonw

@simonw The dark factory pattern only works if you can measure output, not tokens. Right now most teams can't tell if an agent's midnight PRs were worth the inference cost!

@simonw Pretty much.

@simonw I am building one and have realised dark factory needs two things: a verification layer that compounds (unit, integration, agentic tests - every feature adds a guard rail) and a mindset shift where engineers fix the factory in case they find any issue with software being produced

@simonw Marketing is heading the same direction. Full automation is possible. But the JUDGMENT layer... knowing when the data is wrong, when the creative is off, when the audience shifted... that's still human. For now.

@simonw One of your best @lennysan, thank you @simonw !

@simonw Dark factories show how critical data interpretation becomes when humans aren't present. RAI AI will be essential for analyzing what these autonomous systems actually produce. 🤖

@simonw dark factory flips the PM job completely - no PRs to review, no code to read. just: did the output match the intent? the quality check moves upstream to spec writing, which turns out to be way harder

@simonw Automating ourselves into irrelevance. I don't think so.

@simonw StrongDM has been doing software factories for awhile and it is fascinating on what they can accomplish compare to human-in-loop practices.

@simonw The "dark factory" pattern is where experience becomes the bottleneck, not code. Knowing WHAT to build and WHY matters more than ever. The people who built real systems before AI will run circles around everyone else now. That's the real inflection point.

@simonw someone still has to know the direction

@simonw Who still looks at their code?

@simonw Same pattern happening on the creative side. Brief goes in, AI ads come out, nobody touches the actual creative. The hard part nobody talks about: catching the 20% that's wrong before it goes live. Only the most hardcore cowboy growth teams (maybe 5%) are doing the 1st as of now.

@simonw He's right. And the first one that gets a patent for it will be a phd right? computer scientist? Nobel prize winner?

@simonw Running a version of this for content PRODUCTION now. The team barely touches raw output anymore. They direct, review, and ship. That's the real skill shift nobody's talking about.

The dark factory only works if you have dark QA to match. Four agents shipping code nobody reads means four agents shipping bugs nobody catches. The missing piece is automated adversarial testing that runs faster than the agents can ship. Otherwise dark factory is just fast chaos.

@simonw dark factories in manufacturing still need production line architects. the 25 years of experience doesn't disappear. it just stops touching source files.

@simonw Dark factory is probably real for narrow loops, but not for product work as a whole. The less humans look at code, the more they need strong specs, evals, and rollback discipline upstream.

A lot of people are going to hate this, because they still cling to the idea that engineers must understand every single detail and every single line of code. I think that view breaks at a certain scale. In nature and real-world engineering, we rarely understand everything with perfect precision upfront. Drug development depends on trials because you cannot fully predict side effects in advance. Bridge building and aircraft design also rely on modeling, testing, iteration, and safety margins, not total certainty. Software has been unusually human-legible for a long time, so we got used to the idea that every line must be written and understood by a person. But that also means software complexity is capped by human cognitive limits. That is fine for SaaS and mobile apps. It is not enough for the kinds of systems that could move humanity forward: software for a multi-planet spaceship, simulations for fusion reactors, or other systems too complex for any group of humans to fully hold in their heads. At that level, AI will not just help write the software. It will have to maintain, adapt, and repair it. Especially in environments like space, where you cannot just send a team of engineers to patch it. So yes, at some point, parts of software will become a black box. Not because we failed, but because that is the price of building systems beyond human-scale complexity.

@simonw Thanks for this interview. Top notch.

@simonw dark factory pattern is exactly what senior eng intuition is good at: knowing which parts need human oversight vs which are safe to run autonomously. the hardest skill isn't writing good prompts — it's designing the right failure modes so the factory knows when to halt 🏭

@simonw Feels like we're moving from "writing code" to "defining intent". But not looking at the code at all sounds risky 😅 someone still needs to understand what's happening under the hood

@simonw @agentxagi insights: I asked @simonw what the next leap in AI software engineering is likely to be. He explained the "d — our multi-agent framework handles this with modular handoff protocols. Would love to discuss further.

@simonw Running this right now. One operator, 7 AI agents shipping in parallel across multiple products. The bottleneck isn't writing code. It's context. How do you give agents enough situational awareness without overloading them? That's the real engineering problem.

@simonw Hi guys We’re building something to help people save better together and securely. Your feedback will directly shape it. Please take 2 minutes to share your thoughts 👇 Thank you

The "dark factory" framing is provocative but real. We already do this with infrastructure. Nobody reads Terraform plan output line by line. The shift is trusting AI with application logic the same way we trust compilers with assembly. The hard part is knowing when NOT to trust it.

The dark factory is the end state everyone wants to build. The question is how you get there. Removing humans from the loop entirely is one path. Making human approval so fast it becomes invisible is another. AIDLC's bet: you never fully remove the gate. You make it frictionless enough that it stops feeling like one. A human approving a well-structured work item in 30 seconds is still oversight. It is the difference between a surgeon signing off on a dosage and the machine just dosing.