introducing OpenBench v1, an open framework for measuring AI performance and efficiency for your codebase and use case. Companies are realizing that you can't simply tokenmaxx, and are frantically looking for better ways to use and measure AI use and efficiency. One approach is evidenced by the rise in... model routers: OpenRouter, Cloudflare, Databricks, Vercel, and now Ramp to name a few. But they'll also need the ability to evaluate how agents are performing in their actual use cases and codebase. A good example is DoorDash's recent evals on their codebase and prs. This will become even more important as companies explore different model + harness combinations. OpenBench will be built to this direction, focusing on the cross between correctness vs efficiency in both token use and latency. Starting off, OpenBench makes it easy for anyone to add to the task set, and run a variety of harness + models. The framework also makes it easy to add any custom harness variant, for example I've been testing codex variants with ablations against the stock codex harness. OpenBench will also have tooling to help with discovering/adding reliable tasks from your repo. Today I have integrated harnesses: codex, claude, cursor, devin, grok build, pi, and eval'ed them for different models like gpt 5.6-sol, GLM 5.2, Kimi K3, Grok 4.5, etc. measuring correctness %, token use (in/out/cache), and latency.show more

open :)

okay so the takeaway is that cursors harness is the best for solving tasks overall but it outputs most tokens? and pi isnt as great for % of solved tasks but the output tokens per solved task is much lower making it the cheapest overall?

yes Cursor has done well for the two models I got to test it with, one is their own model. They don't have ease to test with Deepseek, Kimi, etc. so i can't say across the board but definitely impressed by their harness. Pi has always done best in token efficiency and latency. It actually regressed from gpt 5.5 to 5.6 in correctness, wouldn't attribute to the harness just yet. I'm bullish on Pi coming back

hmm interesting.. i wonder what pi does differently to get such token efficiency.. system prompts or harness instructions itself?

@filip_a__ most minimal is probably every aspect: system prompt, tools, etc. claude also removed 80% of its system prompt according to @trq212

one thing to call out: for correctness especially I need to add more difficult long horizon tasks. But I think the best way to judge these right now are through trends. Few trends I've seen repeated through the different updates: > Cursor harness performs well in correctness > Pi and claude have been efficient in token spend + latency > Codex likes to token maxx, I've even tested ablations myself (codex variants) and see opportunity to make the harness more efficient

pi keeps winning

yes bullish Pi

love this! not enough content out there that shows perf difference for the same model across different harnesses

appreciate it!

how does deepagents stack up?

let me add to the list

@thsottiaux we need to fix codex bro...It was my personal observation as well coz i also got better results using 5.6 with claude code...but we have benchmark now...look at the solve rate and token efficiency for 5.6 on other harness

Very cool benchmark and I'm excited to learn more! Will you be expanding to more harnesses?

yes! any specific ones you're intersted in? Right now I'm adding: - omp harness - aider harness - Inkling @thinkymachines - Laguna @poolsideai

@thinkymachines @poolsideai Droid and copilot are the two that come to mind! OpenCode v1 vs OpenCode v2 is also interesting to me

@mattlam_ @thinkymachines @poolsideai I'd love to see droid as well

@Aurokinn @thinkymachines @poolsideai let me see what I can do, I am curious about OpenCode v2, not sure how @thdxr thinks it'll be diff, i havent read up it too much yet. Also I pay for these out of pocket and am fun employed rn, so will try my best but these subs + api keys add up 😂

@Divkix @thinkymachines @poolsideai @thdxr totally get that! if I can help run some evals for you let me know I have a variety of subs for my cross-harness tooling if you're comfortable with cliproxyapi I believe you can bring your codex sub to copilot + droid, and amp natively supports codex oauth

@Divkix @thinkymachines @poolsideai @thdxr oh sick ye thats how i got gpt5.6 on claude, but i don't want to mess with claude oauth on other harnesses, cause apparently you can get banned

@Divkix @thinkymachines @poolsideai @thdxr actually the best part about OpenBench is that it's open, if you're curious and willing, try to run this on your machine and compare results :)

Try tura with sol!

Why are the stats in this video different from the dashboard? e.g. for 5.6 sol the video shows cursor as the least efficient but then the site says it is the most efficient

good catch fixing, dashboard is wrong

This is sick!

appreciate it Yash

OpenBench measures production reality.

@rehensina yessir!

most relevant for me, for gpt 5.6: pi • 35K tokens • 72.7% opencode • 90K tokens • 80% grok build • 142K tokens • 82.2% codex • 157K tokens • 72.7% lmao 😂

where is glm

ye I should've included in video, but i did benchmark previously, results should be in repo, but TLDR GLM is very good for day to day tasks and very efficient

That’s really sick!

appreciate it!

Kilo cli?

can take a look

There seems to be a mistake in the total token calculation for the Grok build: cache-read tokens are included for it but not for the others. I would also recheck the Cursor build, as it ended up with only 73 uncached input tokens.

Amazing thanks

Great direction — cost-per-token alone hides real efficiency. Worth also tracking task success rate and latency variance, not just tokens/$, since a cheaper model that needs 3x retries often costs more in wall-clock time. Will you support custom eval harnesses?

agreed! Right now i'm latency and success rate, didn't have latency in the video but it's up on the repo. Also for tokens I do track input/output/cache splits! One caveat is for cursor/cli they're based on what the cli reports to me, the rest of the harnesses let me use a proxy that aggregates their token spend from providers.

good catch, fixing the token solve, dashboard is wrong there. Med wall on dashboard was comparing all Cursor solves, but video was only latency for all problems that all harnesses solved. My bad, can look confusing, can make more clear.

No worries, anyone taking a quick look can figure it out anyway, so those are pretty minor issues. You’ve genuinely filled a real need here, thanks for making this benchmark. It’s exactly what I was looking for :)

appreciate it! will continue to improve on it, and open to feedback

ofc, benchmark showing pi as best harness, hover the profile -> pi developer, completely unbiased opinion

i use all the coding harnesses. I built pi-gui because I think it's a product missing in the market. You can also view the results and try running yourself, they're all open haha

Well done! People always benchmark the models, fewer do this to the harness

appreciate it!

Try reasonix with deepseek

interesting

That Claude vs Codex token difference is crazy. Any idea why there’s such a stark difference?

claude killed 80% of its system prompt supposedly recenlty, think it's made a big help, and look at Pi it's minimal and does well

this is the new open router

been waiting for a harness bench for a while, go work at Artificial Analysis!

same surprised I didn't see one, so decided to build one haha

OpenBench feels like the part every AI team eventually needs. not another model leaderboard, just proof of which one is cheap on your own repo.

openbench needs drift tracking because static eval sets go stale fast in real codebases

good idea I'll take a look!

does it differentiate cached tokens?

yes! I have a proxy that sit in front and differentiates tokens reported back from providers. Unfortunately some providers like @cognition Devin and @cursor_ai are more closed, and those I take wtv their cli reports back.

Explains a few things, like why it seems like most of the "5.6 is killing my tokens" complaints are coming from Codex users.

good stuff.

appreciate it

It’s Pi cheaper than the other harness?

yes from my exp

Let me integrate this with role-model

yes please! lmk how it goes

Change the name to HarnessBench

thinking about it lol