Skip navigation
introducing OpenBench v1, an open framework for measuring AI performance and efficiency for your…
0:00 / 0:20

introducing OpenBench v1, an open framework for measuring AI performance and efficiency for your…

Matthew Lam

@mattlam_3.8K followers

Visit on X
947 likesDownloadOriginal post

124,431 views

introducing OpenBench v1, an open framework for measuring AI performance and efficiency for your codebase and use case. Companies are realizing that you can't simply tokenmaxx, and are frantically looking for better ways to use and measure AI use and efficiency. One approach is evidenced by the rise in model routers: OpenRouter, Cloudflare, Databricks, Vercel, and now Ramp to name a few. But they'll also need the ability to evaluate how agents are performing in their actual use cases and codebase. A good example is DoorDash's recent evals on their codebase and prs. This will become even more important as companies explore different model + harness combinations. OpenBench will be built to this direction, focusing on the cross between correctness vs efficiency in both token use and latency. Starting off, OpenBench makes it easy for anyone to add to the task set, and run a variety of harness + models. The framework also makes it easy to add any custom harness variant, for example I've been testing codex variants with ablations against the stock codex harness. OpenBench will also have tooling to help with discovering/adding reliable tasks from your repo. Today I have integrated harnesses: codex, claude, cursor, devin, grok build, pi, and eval'ed them for different models like gpt 5.6-sol, GLM 5.2, Kimi K3, Grok 4.5, etc. measuring correctness %, token use (in/out/cache), and latency.

67 Comments

Comments from the original post on X (Twitter)

  • Matthew Lam's profile picture

    @mattlam_

    open :)

    26 likes

  • Filip's profile picture

    @aladicf

    okay so the takeaway is that cursors harness is the best for solving tasks overall but it outputs most tokens? and pi isnt as great for % of solved tasks but the output tokens per solved task is much lower making it the cheapest overall?

    15 likes

  • Matthew Lam's profile picture

    @mattlam_

    yes Cursor has done well for the two models I got to test it with, one is their own model. They don't have ease to test with Deepseek, Kimi, etc. so i can't say across the board but definitely impressed by their harness. Pi has always done best in token efficiency and latency. It actually regressed from gpt 5.5 to 5.6 in correctness, wouldn't attribute to the harness just yet. I'm bullish on Pi coming back

    7 likes

  • Filip's profile picture

    @aladicf

    hmm interesting.. i wonder what pi does differently to get such token efficiency.. system prompts or harness instructions itself?

    1 like

  • Matthew Lam's profile picture

    @mattlam_

    @filip_a__ most minimal is probably every aspect: system prompt, tools, etc. claude also removed 80% of its system prompt according to @trq212

    1 like

  • Matthew Lam's profile picture

    @mattlam_

    one thing to call out: for correctness especially I need to add more difficult long horizon tasks. But I think the best way to judge these right now are through trends. Few trends I've seen repeated through the different updates: > Cursor harness performs well in correctness > Pi and claude have been efficient in token spend + latency > Codex likes to token maxx, I've even tested ablations myself (codex variants) and see opportunity to make the harness more efficient

    8 likes

  • stridell's profile picture

    @emilstridell

    pi keeps winning

    5 likes

  • Matthew Lam's profile picture

    @mattlam_

    yes bullish Pi

    1 like

  • Chase's profile picture

    @ChaseMc67

    love this! not enough content out there that shows perf difference for the same model across different harnesses

    3 likes

  • Matthew Lam's profile picture

    @mattlam_

    appreciate it!

    1 like

  • Sydney Runkle's profile picture

    @sydneyrunkle

    how does deepagents stack up?

    3 likes

  • Matthew Lam's profile picture

    @mattlam_

    let me add to the list

    2 likes

  • Sezan's profile picture

    @Sezanjr

    @thsottiaux we need to fix codex bro...It was my personal observation as well coz i also got better results using 5.6 with claude code...but we have benchmark now...look at the solve rate and token efficiency for 5.6 on other harness

    2 likes

  • Auro's profile picture

    @Aurokinn

    Very cool benchmark and I'm excited to learn more! Will you be expanding to more harnesses?

    2 likes

  • Matthew Lam's profile picture

    @mattlam_

    yes! any specific ones you're intersted in? Right now I'm adding: - omp harness - aider harness - Inkling @thinkymachines - Laguna @poolsideai

    6 likes

  • Auro's profile picture

    @Aurokinn

    @thinkymachines @poolsideai Droid and copilot are the two that come to mind! OpenCode v1 vs OpenCode v2 is also interesting to me

    1 like

  • Divanshu Chauhan's profile picture

    @Divkix

    @mattlam_ @thinkymachines @poolsideai I'd love to see droid as well

    2 likes

  • Matthew Lam's profile picture

    @mattlam_

    @Aurokinn @thinkymachines @poolsideai let me see what I can do, I am curious about OpenCode v2, not sure how @thdxr thinks it'll be diff, i havent read up it too much yet. Also I pay for these out of pocket and am fun employed rn, so will try my best but these subs + api keys add up 😂

    2 likes

  • Auro's profile picture

    @Aurokinn

    @Divkix @thinkymachines @poolsideai @thdxr totally get that! if I can help run some evals for you let me know I have a variety of subs for my cross-harness tooling if you're comfortable with cliproxyapi I believe you can bring your codex sub to copilot + droid, and amp natively supports codex oauth

    1 like

  • Matthew Lam's profile picture

    @mattlam_

    @Divkix @thinkymachines @poolsideai @thdxr oh sick ye thats how i got gpt5.6 on claude, but i don't want to mess with claude oauth on other harnesses, cause apparently you can get banned

    1 like