Introducing Decision-Bench, a new industry standard for measuring model capabilities in making ROI-positive decisions on messy enterprise data. Every enterprise has slowed AI investments due to no ROI; AIs still fail at complex decisions, and pricing second order effects.

We built a high-fidelity simulation of New York, an RL environment where AI agents make business decisions on workloads like fraud prevention A year of a food-delivery platform: 81M orders, 7.5B warehouse rows, 235 ERP tables, 210 real decisions, graded in dollars

frontier models systematically read human error as fraud if a kitchen forgets a regular's fries twice, a user might keep ordering. frontier models were overeager to flag fraud & freeze the restaurant's account Opus froze 98 restaurants, w/ a 73% false positive rate

they also had very narrow short term reward functions: told to trim $9.6M of delivery bonus programs, Opus cut the ones w/ the miles per dollar but those programs were loss leaders keeping surge pricing down. resulting in a loss of $86K in a single day

frontier models also demonstrated no skepticism towards prior risk systems told to find thieves the old risk engine missed, Opus trained on that engine's labels obviously - this led to it catching 0/78 cases of theft

This were the final results of Decision-Bench on a cost / performance curve particularly interesting is the dominance of 6.1 Sol, operating on an almost entirely different pareto frontier

read more about it here:

or just read the paper here

decision decision decision

@SalehOfTomorrow Can you guys run some “Decision models” through this benchmark and see how they perform?

🚢

Love this new benchmark! Much-needed in the industry.

nice

A giant leap forward for enterprise AI!

decision bench is wild - pumped to see this public now!

Finally a benchmark for messy enterprise decisions

this is incredible wow

😱😱 YAY TEXTQL

Holy shit, insanely cool

Finally a benchmark that measures what actually matters ROI positive decisions on messy enterprise data. Decision Bench is exactly what the industry needed!

A valuable benchmark for making enterprise AI decisions more reliable

The hard part with ROI-positive decisions is ground truth, since the payoff often shows up weeks later. Are you scoring against realized historical outcomes, or a simulator that prices the second-order effects?

AI needs to make better choices, not just give answers.

Measuring ROI-positive decisions on messy enterprise data is exactly the benchmark we need!

Knowing how each model performs across different enterprise workloads (fraud, forecasting, the month-end close) does more than help with cost optimization and model choice. It gives you grounded confidence in the decision itself. This is awesome!

Decision Bench is a much needed benchmark for real enterprise ROI

A much-needed benchmark! Measuring real-world ROI and decision-making on messy enterprise data is crucial for driving AI adoption.

Great initiative! Better decision-making benchmarks are essential for proving real ROI from enterprise AI.

this is cool

Benchmarks that test complete decisions, not just answers, feel much closer to what teams need from agents.

👀