Existing coding benchmarks stop at the first working version. We are releasing Vibe Code Bench 1-100 today, to measure what comes next. This benchmark asks if models can handle a large number of modifications to a product, without breaking what already works.

The model receives a fully working web app. It must handle up to ten “PM-style” requests (in plain English) like adding a new feature, fixing a bug, or performing a DB migration. Each iteration has to both add the new behavior and preserve everything that came before it. The first failed iteration ends the run.

No model has been able to pass 30%. The top three models are Opus 5, Fable 5.1, and GPT 6-Astra. However, they are 8x to 30x more expensive than the fourth model, GPT Luna, which is 6pts behind the #1. Fable 5.1 comes in at a whopping $150 average cost / test ($7500 for a single run).

When a model failed a request, we looked at what broke down. In the majority of cases (61%), the new feature was not fully working. But 39% of failures broke something that already worked, either alone (14%) or alongside the new feature (25%).

A big difference in model behavior is how much they self-test their own work. One proxy for this is how much they use the browser. About a third of Opus 5 and Fable 5.1 tool calls are browser use. In contrast, Grok 4.6 almost never opens the browser and lands in the bottom third. Astra also doesn’t use the browser, but instead checks its work via HTTP requests through the shell.

Full results and methodology:

This benchmark is the real deal

How do you score "without breaking what already works"? Is it pass/fail on existing tests, or do you check for subtle degradation in behavior?

Each version includes regression tests targeting the changes from the previous iteration.

If possible, please try to accommodate all reasoning levels

That last clause is the real benchmark. Most AI coding demos stop at “it runs”; production work is preserving behavior while changing ten things at once. Add regression tests, inspect the diff, and make the model explain deleted code—or the benchmark is just a prettier demo.

Finally, a benchmark Gary understands: fix one thing without breaking three others. 😂🔧

getting v1 right matters a lot less if change 37 quietly breaks change 4

a score is much more useful when the failure cases are inspectable enough to engineer against what matters next is whether that holds across repeated runs

A Useful Addition Is A Failure Taxonomy, Not Just A Pass Rate: Track Which Change Broke Existing Behavior, The First Failing Step, And Whether The Model Recovered. That Turns A 1–100 Score Into A Debuggable Release Gate.

ten edits later is the real bench.

Measuring 1 to 100 modifications is a much better stress test than a single greenfield demo. I’d love to see failure modes split into regressions, spec drift, and tests that miss behavior. That breakdown would make the score actionable.

Finally a benchmark that doesn’t pretend code lives in a vacuum. 🤝 just dropped a follow