Video yükleniyor...
Video Yüklenemedi
OpenCode 2 far outscores OpenCode 1.19: from 8th of 9 harnesses to tied for first on all three models. It replaces 1.19 in the Hugging Face Harness Leaderboard, which compares coding agents on the same models. Like most harnesses there, it now uses the Chat Completions API directly, so... show more
147,575 görüntüleme • 7 gün önce •via X (Twitter)
36 Yorum

@huggingface Wow. And have you tried the DeepSeek Harness?

@huggingface It's coming soon, stay tuned! :)

@huggingface When I read "Chat Completions API", I was about to make a comment on the fact that it drops reasoning tokens (which the Responses API preserves), but it looks like you fixed that already ;) See e.g.

@huggingface Would you consider adding JetBrains' @junie_ai to the list?

@huggingface The harness was quietly dropping the reasoning all along.

@huggingface Can you add Zcode too

@huggingface I was always a true believer. OpenCode rocks!

@huggingface Fantastic, and what about OMP? Worth the comparison in your view?

@huggingface Can you add @PerchTerminal too

@huggingface Truly impressive jump.

@huggingface Fantastic. That was a large criticism originally when you did the old OpenCode harness. Patiently waiting now for Deepseek and Cursor! Thanks for your hard work and token spend @joelniklaus

@huggingface Please test Antigravity and Github Copilot also

@huggingface That jump from 8th of 9 to tied for first is a big swing for a harness change. Curious whether the gain comes mostly from the Chat Completions API switch or from how the harness handles tool calls and retries, since those tend to dominate coding-agent variance.

@huggingface Pi harness still feels significantly better

@huggingface this is why i stopped caring which model is "best". Opus 5.5 building and GPT 6.1 Sol reviewing in the same room is my whole setup now

@huggingface That's a quite jump

@huggingface Wow this is a cool benchmark afterall! Is there any possibility of testing different models from different companies and adding new harnesses like omp?

@huggingface Completely agree, moving state validation to schema definitions cuts out so much runtime headache.

@huggingface Evaluate @AmpCode too!

@huggingface If you were dropping reasoning, you should have run v1.19 with proper test setup, to separate the impact of incorrect test setup vs version change for Opencode. The graphs in your post are meaningless otherwise and the whole thing has lost a bit of credibility.

@huggingface big jump, super nice to see the updated results

@huggingface If all three models tie for first now, were the tasks too easy to separate them, or did the harness changes just hit the same ceiling?

@huggingface I'm done with OpenCode. I won't use other harnesses.

@huggingface Big leap, especially the cost drop. Shows the value of fixing that silent reasoning loss

@huggingface Bro mop

@huggingface I knew I was right for betting on opencode

@huggingface any idea where the improvement came from?

@huggingface Now it's time to add dsh, brother

@huggingface Thank you for your work on testing. 🙌 Is there any chance to have GitHub copilot harness tested for us who don’t have choice in our company ? 😅 I did my small test locally, and it did perform well vs opencode v2.

@huggingface Dsh

@huggingface 同一套条件下对比才有参考价值,榜单更贴近真实使用了,选工具也能少踩坑。

@huggingface where is @thdxr

@huggingface Also, there are no 95% CIs or other stats mentioned for any of the other metrics like cost, reliability, or noise. With any such stochastic process, a single point value for a metric is meaningless. (Or if there is, I may not be reading the charts very well.)

@huggingface thoughts on openchamber? recently switched to it from opencode v2 and i am absolutely loving it for someone who prefers desktop/gui over tui/terminal

@huggingface What about Copilot?

@huggingface I found this in my testing too, opencode had a higher solve rate.
