Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

OpenCode 2 far outscores OpenCode 1.19: from 8th of 9 harnesses to tied for first on all three models. It replaces 1.19 in the Hugging Face Harness Leaderboard, which compares coding agents on the same models. Like most harnesses there, it now uses the Chat Completions API directly, so...

147,575 görüntüleme • 7 gün önce •via X (Twitter)

36 Yorum

Jonas Hertner profil fotoğrafı
Jonas Hertner7 gün önce

@huggingface Wow. And have you tried the DeepSeek Harness?

Joël Niklaus profil fotoğrafı
Joël Niklaus7 gün önce

@huggingface It's coming soon, stay tuned! :)

Niels Rogge profil fotoğrafı
Niels Rogge7 gün önce

@huggingface When I read "Chat Completions API", I was about to make a comment on the fact that it drops reasoning tokens (which the Responses API preserves), but it looks like you fixed that already ;) See e.g.

Anton Arhipov profil fotoğrafı
Anton Arhipov7 gün önce

@huggingface Would you consider adding JetBrains' @junie_ai to the list?

Wei佳 profil fotoğrafı
Wei佳7 gün önce

@huggingface The harness was quietly dropping the reasoning all along.

hmziq profil fotoğrafı
hmziq7 gün önce

@huggingface Can you add Zcode too

Adriel Martins profil fotoğrafı
Adriel Martins7 gün önce

@huggingface I was always a true believer. OpenCode rocks!

Dima profil fotoğrafı
Dima7 gün önce

@huggingface Fantastic, and what about OMP? Worth the comparison in your view?

Axel ker profil fotoğrafı
Axel ker7 gün önce

@huggingface Can you add @PerchTerminal too

Aayush Khare profil fotoğrafı
Aayush Khare7 gün önce

@huggingface Truly impressive jump.

patapon profil fotoğrafı
patapon7 gün önce

@huggingface Fantastic. That was a large criticism originally when you did the old OpenCode harness. Patiently waiting now for Deepseek and Cursor! Thanks for your hard work and token spend @joelniklaus

Satoshi Nakamoto profil fotoğrafı
Satoshi Nakamoto7 gün önce

@huggingface Please test Antigravity and Github Copilot also

OnFinality profil fotoğrafı
OnFinality7 gün önce

@huggingface That jump from 8th of 9 to tied for first is a big swing for a harness change. Curious whether the gain comes mostly from the Chat Completions API switch or from how the harness handles tool calls and retries, since those tend to dominate coding-agent variance.

NameIsNotBlank profil fotoğrafı
NameIsNotBlank7 gün önce

@huggingface Pi harness still feels significantly better

Threadlines ADE profil fotoğrafı
Threadlines ADE7 gün önce

@huggingface this is why i stopped caring which model is "best". Opus 5.5 building and GPT 6.1 Sol reviewing in the same room is my whole setup now

Seti @ Home profil fotoğrafı
Seti @ Home7 gün önce

@huggingface That's a quite jump

Pawed profil fotoğrafı
Pawed7 gün önce

@huggingface Wow this is a cool benchmark afterall! Is there any possibility of testing different models from different companies and adding new harnesses like omp?

EDDY VU profil fotoğrafı
EDDY VU7 gün önce

@huggingface Completely agree, moving state validation to schema definitions cuts out so much runtime headache.

Praagya Joshi profil fotoğrafı
Praagya Joshi7 gün önce

@huggingface Evaluate @AmpCode too!

Rahul Raj profil fotoğrafı
Rahul Raj7 gün önce

@huggingface If you were dropping reasoning, you should have run v1.19 with proper test setup, to separate the impact of incorrect test setup vs version change for Opencode. The graphs in your post are meaningless otherwise and the whole thing has lost a bit of credibility.

Leonard profil fotoğrafı
Leonard7 gün önce

@huggingface big jump, super nice to see the updated results

Davis.AI.Explorer profil fotoğrafı
Davis.AI.Explorer7 gün önce

@huggingface If all three models tie for first now, were the tasks too easy to separate them, or did the harness changes just hit the same ceiling?

bukit 🇮🇩 🇵🇸 🔻 profil fotoğrafı
bukit 🇮🇩 🇵🇸 🔻7 gün önce

@huggingface I'm done with OpenCode. I won't use other harnesses.

Peter Dove profil fotoğrafı
Peter Dove7 gün önce

@huggingface Big leap, especially the cost drop. Shows the value of fixing that silent reasoning loss

Tay profil fotoğrafı
Tay7 gün önce

@huggingface Bro mop

🇪🇬 profil fotoğrafı
🇪🇬7 gün önce

@huggingface I knew I was right for betting on opencode

anson profil fotoğrafı
anson7 gün önce

@huggingface any idea where the improvement came from?

DeepCoded profil fotoğrafı
DeepCoded7 gün önce

@huggingface Now it's time to add dsh, brother

Social profil fotoğrafı
Social7 gün önce

@huggingface Thank you for your work on testing. 🙌 Is there any chance to have GitHub copilot harness tested for us who don’t have choice in our company ? 😅 I did my small test locally, and it did perform well vs opencode v2.

samuel Peñaloza profil fotoğrafı
samuel Peñaloza7 gün önce

@huggingface Dsh

Jhon Dennis profil fotoğrafı
Jhon Dennis7 gün önce

@huggingface 同一套条件下对比才有参考价值,榜单更贴近真实使用了,选工具也能少踩坑。

Vicenzo Masat profil fotoğrafı
Vicenzo Masat7 gün önce

@huggingface where is @thdxr

Roscoe A. Bartlett profil fotoğrafı
Roscoe A. Bartlett7 gün önce

@huggingface Also, there are no 95% CIs or other stats mentioned for any of the other metrics like cost, reliability, or noise. With any such stochastic process, a single point value for a metric is meaningless. (Or if there is, I may not be reading the charts very well.)

Taha profil fotoğrafı
Taha7 gün önce

@huggingface thoughts on openchamber? recently switched to it from opencode v2 and i am absolutely loving it for someone who prefers desktop/gui over tui/terminal

Filip Mares profil fotoğrafı
Filip Mares7 gün önce

@huggingface What about Copilot?

Dominic Deccoco profil fotoğrafı
Dominic Deccoco7 gün önce

@huggingface I found this in my testing too, opencode had a higher solve rate.

Benzer Videolar

GLM-5.3 Flash vs Qwen3.8 Flash-Next. Both local on my 4x 3090, both left alone with 1 prompt per task. 11 scenes, 18 hours, 2.4M tokens out. All of it is in the video. I wanted a clear winner. I didnt get one, but I know which one stays on my rig. GLM-5.3 Flash: Maya-M quant, 116 GB, runs in Project Maya on all 4 cards plus RAM. 35 to 48 tok/s while it works. Qwen3.8 Flash-Next: IQ3_S quant, 78 GB, runs in Strata on 2 cards. 85 to 118 tok/s. Both have 262K context and vision, so the agent opens its own page, takes a screenshot, looks at it and fixes what it sees. 1 prompt per scene. No follow ups and no hints from me. 120 minutes max per scene, 150 for the kart game. I ran them in 2 harnesses, OMP first and then DeepSeek Harness. Minutes until the model said it was done: Pelican in OMP: GLM 39, Qwen 66 Pelican in DeepSeek Harness: GLM 93, Qwen 86 Kart game in OMP: GLM 130, Qwen stopped by the 150 limit Millipede in OMP: both stopped by the 120 limit Millipede in DeepSeek Harness: Qwen stopped at 120 again Battle diorama in DeepSeek Harness: GLM 81, Qwen 113 Thats 11 of 16 combos. I ran out of time for the other 5, sorry. They work in a very different way and I didnt expect that. GLM thinks for a long time and then writes a lot at once. Its first pelican took only 36 turns. For the diorama it gave 1 answer of 47K tokens that ran for almost 18 minutes, with the full scene in a single file. Qwen goes in small steps and checks everything. The same pelican was 240 turns and 45 browser checks. In total GLM wrote 513K tokens in 5 runs and Qwen wrote 1.9M in 6 runs, so Qwen puts out 2.6x more per minute of real time, on half the cards. The harness also changed more than I thought it would. OMP keeps only 5 images in a request when the provider is local. So every new screenshot removes the oldest one, the prompt changes in the middle, and the server has to read the context again from the start. GLM lost 37 of its 130 kart minutes just on that. In DeepSeek Harness, on the diorama, it lost 3.5 minutes out of 81. Qwen reads a prompt almost 4x faster on my rig (84K tokens in 23 seconds, GLM needs 100 seconds for 96K), so in OMP it lost only 9 to 12 minutes per run. I cant give you a winner. GLM is the only one that finished the kart game inside the limit, and its scenes look cleaner to me. But its millipede is bad. Qwen puts more detail into everything, and its diorama and millipede from DeepSeek Harness are my favorite scenes in the video. It also got stopped by the time limit in 3 of its 6 runs, because it just keeps polishing. For my own use I'd take Qwen. It is a lot faster and it needs only 2 cards, so the other 2 stay free and I can run something else next to it. This is 1 run per combo, so dont read it as statistics. Both models ran with my refusal projection on, because thats how I use them every day. And in the kart clips the game's own AI is driving the player kart, I only recorded the screen. Music: YuE2

Alexey Fateev

13,945 görüntüleme • 1 ay önce