
Joël Niklaus
@joelniklaus • 4,289 subscribers
Harness Optimization @HuggingFace
Shorts
Videos

I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard. Which harness should you run your open model in? I ran 28,849 trials to find out. The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts. The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference. Averaged over the three models, mini-swe-agent (Kilian Lieret), a bash-only agent, solves 51.4% of tasks and OpenClaw🦞 27.3%. Four others are statistically tied with mini-swe-agent. One of them is Pi (Pi), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53. The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (Nous Research) on DeepSeek-V4.1-Flash and Kimi Code (Kimi.ai) on Flint. OpenCode costs $3.35 per solved task, 11.7 times as much as Pi. See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source. Leaderboard: Harness Lens:
Joël Niklaus336,314 просмотров • 11 дней назад

OpenCode 2 far outscores OpenCode 1.19: from 8th of 9 harnesses to tied for first on all three models. It replaces 1.19 in the Hugging Face Harness Leaderboard, which compares coding agents on the same models. Like most harnesses there, it now uses the Chat Completions API directly, so it's more comparable. At high effort it solves 65.7% of tasks instead of 43.3% on GLM-5.3-Flash and 69.4% instead of 57.0% on DeepSeek-V4.1-Flash. On Flint, 22.9% instead of 7.1%. Cost per solved task fell from $3.35 to $0.45. Part of the old gap was the setup. Harbor's standard route for OpenCode 1.19 is OpenAI's Responses API, which I translated to Chat Completions so it works with Hugging Face Inference Providers. It ran without errors but silently dropped the model's earlier reasoning. Thanks to dax and Aiden Cline from OpenCode for spotting it! Leaderboard:
Joël Niklaus147,091 просмотров • 7 дней назад

You asked for dsh, I added it DeepSeek Harness is the open-source coding agent by DeepSeek. It's now on the Hugging Face Harness Leaderboard, which compares ten coding agents on the same open models. DSH is statistically tied for first in all five evaluations: DeepSeek-V4.1-Flash and GLM-5.3-Flash at low and high effort, and Flint. It shines on DeepSWE, where agents change code in open-source repositories: 2nd of 10 averaged over the three models, 0.5 points behind Hermes Agent by Nous Research. On Terminal-Bench's command-line tasks it ranks 6th. One habit explains much of that gap: when DSH finishes a task, it shuts down every background program it started. Three Terminal-Bench tasks ask the agent to start a server, for example a model inference API. The grader tests that server after the agent has finished. With DSH, the server is gone by then. On DeepSeek-V4.1-Flash and GLM-5.3-Flash, DSH passed 2 of 36 attempts on these tasks. OpenCode, Pi and Hermes Agent passed all 36. Those three tasks cost DSH 1.5 to 1.7 points in every evaluation. On DeepSeek-V4.1-Flash at high effort, that is more than its 1.3-point gap to the leader. Developer preview: 0.2.2-rc.1. Web search is off because it needs a DeepSeek account key. It costs $0.53 per solved trial, averaged over the three models. Leaderboard:
Joël Niklaus61,855 просмотров • 5 дней назад
Больше нет контента для загрузки