The software factory is already here. We're seeing bots write code, bots review it, and humans reduced to dispatching the next tool in the chain. Using 500k+ PRs from Code Review Bench, we looked at one question: can the human leave the loop yet?

Everyone talks about the software factory in terms of generation. But factories don't fail at generation. They fail at quality control. In over half of bot-reviewed PRs, nobody acts on the review afterward. We're already starting to take the human out of the loop before establishing that AI review reliably matches human judgment.

A PR with no human in the loop can mean two very different things: - In a healthy factory, the inspector is trustworthy and doesn't need oversight. - In an unhealthy one, bad output is passing through unchecked. Raw scores don't distinguish between the two. So we built filters to isolate the PRs that actually tell you something about trust.

The data shows something important: not all AI code review tools are used the same way. Some tools are review-centric, where humans read the comments, respond, and iterate. Others function as agent side-channels, where the review exists but the main interaction is with the generated code itself.

The #1 tool on our online leaderboard, @cubic_dev_, is new to the benchmark. 52.9% of their reviewed PRs see human follow-up after the review, and their score holds whether or not a human is in the loop.

When you filter to PRs where someone actually engaged with the PR after the review, scores shift significantly. @OpenAI’s Codex sees the largest swing at +17.2pp. @augmentcode climbs to #2 in F1 with +12.3pp. Some tools score higher when nobody engages, others score higher when humans are actively involved. The delta reflects how each tool is being used, not which one is better.

That doesn't mean one tool is "real" and another isn't. It means they're supporting different kinds of factories: → one where the review itself is the product → one where the review is a side effect of a broader agent workflow Both are valid, but they can be measured differently.

The software factory question is really a trust question: when can the human leave the loop without quality collapsing? We're not there yet. But some tools are closing the gap fast: @QodoAI +22pp in 4 months, @coderabbitai +14pp. Our new online filters let us track that threshold more directly.

Take a look at our blog post for more on what we’re seeing in the data: Leaderboard: Repo / methodology: Blog:

52.8% of AI code reviews get zero human follow-up. we've accidentally built RLHF with no HF!

This is the software factory we’ve been waiting for — and you just built the perfect inspection-line benchmark to prove it.

This is the real concern, removing humans before proving the system is actually reliable is a risky move.

This is a good update btw:

Super interesting perspective on the future of dev work.

No. Next question.

People are comparing feedback tools vs autonomous agents like they’re the same thing.

Great work @withmartian team! Love how the filters and deltas reveal that different tools are optimizing for very different workflows. The progress from tools like QodoAI and coderabbitai is impressive. Excited to see how this benchmark evolves as we get closer to trustworthy autonomous review.

🪢🪢

Interesting

they make everything look simple

Big question, not if humans leave the loop, but when