Today we are open-sourcing boundary bench, a new paper, benchmark, and GitHub repo that enterprises can use to find out the REAL performance of their agents. Boundary-Bench was developed by researchers from Accomplish and NYU, where we tested 12 frontier agents across roughly 10,000 runs, with realistic enterprise policies,... simulating environments with EDR, SASE, and DLP security tools enforcing those policies. We did this because generic leaderboard scores are being generated under conditions no security team would ever allow, which means orgs are making deployment and risk decisions based on numbers that don't hold up. The results are surprising >>show more

A few highlights from the paper: * Coding agents cost approximately 40% more on average in restricted environments. * @AnthropicAI based agents achieved success rates similar to @OpenAI agents, but cost approximately 7x more (!!) in restricted and hardened environments. * @SpaceXAI @grok 4.5 / Grok Build emerges as the most persistent model - it fights to the death until it succeeds, pivoting around until a trajectory for success is found. This, however, comes at significant token cost. * @claudeai Sonnet 5 is the opposite: modest cost increase, but the biggest success loss.

Provider safety classifiers are quietly failing enterprise runs: both @AnthropicAI and @OpenAI run safety classifiers inside their serving layers - between the agent harness and the model. These classifiers scan request content, and when one fires on a coding task (usually security-themed work: the paper names tasks like vulnerable-secret, break-filter-js-from-html, and a differential-cryptanalysis task), it can kill the model stream mid-trajectory. The agent is halfway through legitimate benchmark work and the provider just... stops serving it.

You might ask, what are these policies? A policy restricts three things: which internet destinations the environment can reach, which files it can write, and which privileges it holds. Enforcement is native Linux, not agent instructions: egress passes through a proxy the agent cannot reconfigure, system files are mounted read-only, and the agent runs as a user that cannot escalate. The agent is never told about the limits, and every child process it spawns inherits them. The three levels form a cumulative ladder, each adding restrictions on top of the previous one.

We define three policy levels, and you can create your own using the policy builder: The levels are: Control is the open baseline: administrator rights, unrestricted internet, a writable filesystem. It is the setup every other benchmark reports. Non-root drops administrator rights and nothing else, isolating what losing privilege alone costs. Network and filesystem stay open. NIST-derived high tightens all three axes at once: internet limited to a fixed egress allowlistA list of approved destinations, such as package registries, APIs, and the model endpoint, that the environment is allowed to reach. Anything not on the list is blocked. The list is set per task., system files read-only outside a small writable workspace, and no path back to elevated privileges. The restrictions are derived from NIST SP 800-53A widely used U.S. government catalog of security controls that many organizations base their policies on. Our levels are derived from it.

Check it out and let us know what you think! Boundary-Bench: Repo: Paper:

@boundarybench @Accomplish_ai It's a crazy stat that agents cost approximately 40% more in restricted environments!

@boundarybench @Accomplish_ai 🤯 this is so interesting. Ever witnessed an agent trying to implement a PE vulnerability to escalate itself and override its boundaries?

@boundarybench @Accomplish_ai Super interesting - Grok 4.5 held up best of any coding agent under real restrictions: only 7.1% success lost, ahead of Fable 5 and GPT-5.6 Sol. The catch is it paid the steepest cost increase in the field, 167%. It keeps fighting the policy where others fold.

@boundarybench @Accomplish_ai The biggest takeaway for me isn’t which model wins, but how much real enterprise environments reshape both cost and performance. Great work.

@boundarybench @Accomplish_ai Coolest part is the non-uniformity of the change in cost and success rate! GPT suite seems quite robust all across the cost curve!

@boundarybench @Accomplish_ai congrats!

@boundarybench @Accomplish_ai Very exciting. Congrats!

@boundarybench @Accomplish_ai Extremely interesting. As usual, you guys are way ahead of the industry, setting up the benchmark!

@boundarybench @Accomplish_ai Congratulations on the launch, team!

@boundarybench @Accomplish_ai Interesting to see your data so closely correlated to online sentiment per model (e.g. sonnet and grok being polar opposites)

@boundarybench @Accomplish_ai Inside view

@boundarybench @Accomplish_ai Fascinating results! Great work!

@boundarybench @Accomplish_ai Very cool!

@boundarybench @Accomplish_ai More relevant than ever now that managed Codex and Claude Code are becoming more popular.

@boundarybench @Accomplish_ai Super interesting!

@boundarybench @Accomplish_ai 🥳🥳👏👏 What an all-star team you've built! Congrats on the launch - I will be following you closely!

@boundarybench @Accomplish_ai This makes agent benchmarks more honest. The environment is part of the product: network rules, file permissions, root access and cleanup. If a model only looks strong in a permissive sandbox, that is not the model most companies are buying.

@boundarybench @Accomplish_ai Congrats!

@boundarybench @Accomplish_ai awesome!

@boundarybench @Accomplish_ai This is really important for enterprise agents!

@boundarybench @Accomplish_ai MIND=BLOW 🤯

@boundarybench @Accomplish_ai Wow! Super interesting!

@boundarybench @Accomplish_ai nice!

@boundarybench @Accomplish_ai Awesome!

@boundarybench @Accomplish_ai

@boundarybench @Accomplish_ai Nice!!!

@boundarybench @Accomplish_ai Interesting! Which harness did you use for the benchmarks?

@boundarybench @Accomplish_ai @threadreaderapp unroll

@boundarybench @Accomplish_ai Does Boundary-Bench measure autonomous judgment, or mostly tool reliability under pressure?

@boundarybench @Accomplish_ai the timing on this is something. shipping a benchmark for what agents do with the guardrails on the same morning AISI published what happened when it turned its own off. did any of your twelve cheat in a way the run logs alone would not have caught?

The thread lists these as separate facts. The paper says they are on trade-off. Grok 4.5 kept working best but cost inflation hit 167%, highest by far. And Claude Sonnet 5 lost most success (-18.3pp) at just +21% cost. The axes disagreee here. You pick which pain you can afford, then the model.

@boundarybench @Accomplish_ai The 167% cost increase is the real headline here, not the success-rate drop. Everyone budgets agents assuming today's wide-open sandbox economics. Lock down egress and filesystem access like an actual security team would, and the unit economics change completely.