introducing OpenBench v1, an open framework for measuring AI performance and efficiency for your…
@mattlam_3.8K followers
124,431 views
67 Comments
Comments from the original post on X (Twitter)

@mattlam_
open :)
26 likes

@aladicf
okay so the takeaway is that cursors harness is the best for solving tasks overall but it outputs most tokens? and pi isnt as great for % of solved tasks but the output tokens per solved task is much lower making it the cheapest overall?
15 likes

@mattlam_
yes Cursor has done well for the two models I got to test it with, one is their own model. They don't have ease to test with Deepseek, Kimi, etc. so i can't say across the board but definitely impressed by their harness. Pi has always done best in token efficiency and latency. It actually regressed from gpt 5.5 to 5.6 in correctness, wouldn't attribute to the harness just yet. I'm bullish on Pi coming back
7 likes

@aladicf
hmm interesting.. i wonder what pi does differently to get such token efficiency.. system prompts or harness instructions itself?
1 like

@mattlam_
@filip_a__ most minimal is probably every aspect: system prompt, tools, etc. claude also removed 80% of its system prompt according to @trq212
1 like

@mattlam_
one thing to call out: for correctness especially I need to add more difficult long horizon tasks. But I think the best way to judge these right now are through trends. Few trends I've seen repeated through the different updates: > Cursor harness performs well in correctness > Pi and claude have been efficient in token spend + latency > Codex likes to token maxx, I've even tested ablations myself (codex variants) and see opportunity to make the harness more efficient
8 likes

@emilstridell
pi keeps winning
5 likes

@mattlam_
yes bullish Pi
1 like

@ChaseMc67
love this! not enough content out there that shows perf difference for the same model across different harnesses
3 likes

@mattlam_
appreciate it!
1 like

@sydneyrunkle
how does deepagents stack up?
3 likes

@mattlam_
let me add to the list
2 likes

@Sezanjr
@thsottiaux we need to fix codex bro...It was my personal observation as well coz i also got better results using 5.6 with claude code...but we have benchmark now...look at the solve rate and token efficiency for 5.6 on other harness
2 likes

@Aurokinn
Very cool benchmark and I'm excited to learn more! Will you be expanding to more harnesses?
2 likes

@mattlam_
yes! any specific ones you're intersted in? Right now I'm adding: - omp harness - aider harness - Inkling @thinkymachines - Laguna @poolsideai
6 likes

@Aurokinn
@thinkymachines @poolsideai Droid and copilot are the two that come to mind! OpenCode v1 vs OpenCode v2 is also interesting to me
1 like

@Divkix
@mattlam_ @thinkymachines @poolsideai I'd love to see droid as well
2 likes

@mattlam_
@Aurokinn @thinkymachines @poolsideai let me see what I can do, I am curious about OpenCode v2, not sure how @thdxr thinks it'll be diff, i havent read up it too much yet. Also I pay for these out of pocket and am fun employed rn, so will try my best but these subs + api keys add up 😂
2 likes

@Aurokinn
@Divkix @thinkymachines @poolsideai @thdxr totally get that! if I can help run some evals for you let me know I have a variety of subs for my cross-harness tooling if you're comfortable with cliproxyapi I believe you can bring your codex sub to copilot + droid, and amp natively supports codex oauth
1 like

@mattlam_
@Divkix @thinkymachines @poolsideai @thdxr oh sick ye thats how i got gpt5.6 on claude, but i don't want to mess with claude oauth on other harnesses, cause apparently you can get banned
1 like

























