Introducing Code Review Bench v0: The first independent code review benchmark. 200,000+ PRs. Unbiased. Fully OSS. Updated daily. Tool performance highlights 🧵👇 Featuring: Augment Code baz Claude CodeRabbit Cursor Google Gemini GitHub Graphite Greptile Kilo (acq. by Anaconda) OpenAI Developers Propel Qodo

Existing code review benchmarks are small (50-100 PRs), vendor-published (the vendor's tool wins), and stale (training contamination). Worse, they structurally punish tools for finding bugs humans missed — if it's not in the gold set, a real bug gets scored as a false positive.

We built two benchmarks that check each other. An offline benchmark runs tools on the same PRs for fair comparison. An online benchmark tracks when devs fix flagged issues; a fix is a vote the flag was useful. If the offline rankings don't match online use, we fix the offline.

The benchmark measures precision (is the tool low noise) and recall (is the tool thorough). We combine them to get a single number (F-score) depending on how strongly you weight precision vs recall. Depending on your weights, @coderabbit @greptile or @cursor is #1 online

ranks #1 in online F1 score (recall and precision equally valuable). Unlike other tools they try to find the most bugs, not just reduce noise. That's why they're #1 in recall too. They're trusted by companies like @nvidia, Abnormal Security, @Groupon and more

is the tool for people who want finely-tuned precision. They score #1 when measuring the F_0.5 score (weighing precision more heavily than recall, without discounting recall entirely). They’re widely used, including in repos like @openclaw @nvidia @pytorch and @posthog

has the least noisy tool: bugbot ranks #1 in precision. The team has prioritized finding the bugs that matter most while ruthlessly eliminating noise. They serve high signal companies like @discord @Rippling @airtable @SierraPlatform

We see a similar pattern with the tools from model labs. Different labs have chosen to optimize for different metrics.

Gemini Code Assist from @GeminiApp has the highest precision and F1 score of any tool from a model provider. It’s a testament to the model’s quality, especially since other labs have launched models more recently. We can’t wait to see the gains from the next release.

has higher recall than either Gemini or OpenAI. It catches more bugs — likely a result of the exceptional agentic ability that leads the model to be more eager in exploring codebases and surfacing issues.

Interestingly, @OpenAIDevs Codex is the most widely adopted tool, with ~2x the daily usage of Gemini and ~10x the daily usage of Claude Code. This online result gives us strong reason to examine more closely how much of user preference is captured by precision and recall.

More broadly, the most used code review tool is @github's Copilot. They review more than twice as many PRs as the next most popular tool. We’re excited to dive into what this means about developers’ revealed preferences.

has the most precise tool for large PRs. This makes sense: they made the first code review benchmark that measured precision, and their context engine is suited to understanding large changes. They’re used in massive OSS repos like curl (@bagder) and @Redisinc

stands out as the #1 open source tool for all values of beta. It doesn’t matter whether you care about low noise or high thoroughness, they’re the best OSS code review solution. It'll be interesting to see if open can catch up to (or exceed) closed in the future.

Benchmarks like this can also shed light on the broader code review market. @graphite (recently acquired by @cursor) is #3 in precision, right after cursor and greptile. The cursor and graphite teams clearly have similar code review philosophy.

On our offline benchmark,@Qodo is a top 5 tool by F1 score (valuing precision and recall equally). In particular, they're a tool for teams like @SeleniumHQ that want to find every bug, placing #4 on recall.

We’re also interested in seeing how newer tools do. @propelcode is the newest tool in the scene, and as we collect more data on their online performance, we’ll see how users react to the tool.

We're excited about the results from @baz_scm. Because they're newer, their sample size is smaller (i.e. more noise in measurement). Taking their results at face value, they are #1 in precision, #2 in F_1 score, and #1 in F_0.5 score. It's an impressive David & Goliath story for the Israeli company -- enough that our @joshgreaves_ml made it the default in our ARES repo

We owe a lot to the tool creators in the space, especially those who’ve built benchmarks in the past. Greptile and Augment create the most used OSS benchmark in the past. A particular shoutout to @EntelligenceAI who released the most recent such benchmark.

Code Review Bench is a living benchmark: we’ll release new versions as we learn more from the data. Over the coming weeks, we’ll be sharing updates on the offline benchmark as we align it with the online. How we’re building and how we keep it fair:

If you want to play around with the data, reproduce the results, or contribute to the project, see our GitHub repo:

If you’re: - A code review company that wants to be added to the list - A developer who wants to understand what code review tools to use - A researcher who wants to collaborate on the benchmark Reach out: [email protected] or our DMs

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github good one let me add this report in our blog post as well

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github Congrats to you and the team. Very detailed post, covering almost everything. We are writing code way faster now, but quality needs to be managed, and code review is extremely important. This will give people a lot of clarity .

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github If you’re conducting research in AI code review, now’s the perfect moment to reach out.

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github This is phenomenal!

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github This is big. 🔥 An independent, OSS benchmark with 200K+ PRs is exactly what the ecosystem needed.

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github Finally, real signal over hype, an open, unbiased benchmark like this is exactly what AI code review needed.

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github Huge move for transparency in code reviews.

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github The first independent code review benchmark and @coderabbitai is on Top. The team is amazing at CodeRabbit, always pushing to deliver the top quality code review for everyone.

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github This looks like a great resource. I'll have to dig into it more in the future

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github Congrats on the launch guys!

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github Huge respect for building an independent benchmark like this. Daily updates + OSS makes it even more valuable for devs who care about real performance.

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github Huge Congratulations 🎊

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github Independent benchmarks raise the bar.

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github Realistic benchmarks push teams to improve documentation clarity and onboarding experience.

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github Huge milestone for transparent AI tooling evaluation.

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github Super useful!

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github This sets a new standard for accountability in AI code tools. Clear metrics, real world validation, and zero bias. Amazing work by the team.

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github this is unbelievably necessary for the market rn

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github That's an interesting feature. Congratulations for the huge achievement

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github I think, you are actually building the exact reward signal needed to train even better code-generating AI models in the future. 👏

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github AI code review is crowded. Only a few survive real-world PR volume. This benchmark just exposed that.

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github @robinebers perfect timing also @withmarti can you add @cubic_dev_ to see how it holds up in the benchmarks ?

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github @hccallaway @shcallaway for vis

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github An independent code review benchmark is exactly what the industry needs right now. Everyone claims their AI writes production-ready code but nobody had the receipts. 200K PRs is a serious dataset. Time to find out who's been lying.

The false positive penalty in vendor benchmarks is why most teams end up tuning AI review tools toward noise reduction over coverage. A benchmark grounded in whether devs actually fix the flagged issues is the correct signal, because silence has a real cost that static gold sets never capture.

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github just sent this to all my dev friends no more bs benchmarks!!

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github I love this

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github lets gooo - love this!!!

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github This is a huge step for developers! 🚀 An independent, unbiased benchmark for code reviews over 200k+ PRs can really level up how we measure and improve code quality.

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github Benchmarks like this will decide which AI actually ships production-ready code.

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github Where is @Macroscope ?

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github where is @cubic_dev_ ?

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github That's great

@augmentcode @baz_scm @claudeai @coderabbitai @Cursor @GeminiApp @github Great