We've been tracking AI code review tools across OSS, and a new category is emerging. We're calling it "Deep Review": → Standard AI review: PR-level, fast, human in the loop → Deep Review: repo-wide context, runs autonomously in the background 🧵👇

The speed data makes the split obvious. Some tools review in ~1–2 minutes. Others take 20+ minutes. This isn't a performance gap — they're optimizing for completely different things.

If a human is waiting for the result, every minute matters. If it's running in the background, you have all the time in the world. What makes a great review depends entirely on what you're optimizing for. The same is true for every dimension: precision, recall, cost, and more.

ships two separate products for a reason: one for standard review, one for Deep Review. Their Extended mode is the Deep Review product: top 3 in F1 and #2 in recall offline. @AnthropicAI's Claude Code Reviewer tells a similar story — #1 in F1 on the online benchmark over the past month, also sitting in the Deep Review zone on latency.”

If speed is the priority, @CodeAntAI is currently the fastest reviewer in the offline set.

How you define quality changes the leaderboard. @cognition (Devin Review) has the best precision-to-latency ratio in the offline set.

wrote about this background/focus split in code generation: background agents for async work, fast agents for when humans are in the loop. Code review is one of the first verticals where we're seeing it show up in the data. It won't be the last.

This is part of broader work updating our offline benchmark. We also updated the offline benchmark to: - remove some erroneous gold comments - fix an issue inflating precision for summary-based tools - improve comment parsing and matching This shifted some of the results.

moved to #1 in F1 and recall on the offline benchmark following their latest updates and improvements to the benchmark.

ranks #1 across localized and clear-cut issue types — local context, simple complexity, and obvious review difficulty.

saw one of the largest gains after the offline fixes (~+10% F1).

Code Review Bench v0.3: 22 tools, online tracker + offline benchmark, continuously updated, fully open source. Explore the data: Methodology + code: Want your tool added? [email protected]

What is the point of calling something a "deep review" if it performs worse than a standard review? Seemingly Claude Code reviewer is now in this deep review but based on martian benchmarks it falls short of most standard reviewers which are a fraction of the price

the slow/fast split is actually about who's in the loop. fast = human waiting. slow = agent runs while you sleep. the real question is: which one do you trust more? 🐱

“Repo-wide context” is the key distinction. PR review catches local issues. Deep review catches work that looks correct in isolation but conflicts with prior decisions, neighboring systems, or release ownership once it lands.

Would you auto block merges on a 20 min deep review AI with 95% recall but high false positives on legacy code?

The jump from PR-level to repo-wide is where the noise problem gets brutal though. Autonomous background review sounds great until teams auto-dismiss findings, which is arguably worse than no review at all. The real moat is signal quality, not coverage breadth.

Check dm!

LFG