Video yükleniyor...
Video Yüklenemedi
You asked for dsh, I added it DeepSeek Harness is the open-source coding agent by DeepSeek. It's now on the Hugging Face Harness Leaderboard, which compares ten coding agents on the same open models. DSH is statistically tied for first in all five evaluations: DeepSeek-V4.1-Flash and GLM-5.3-Flash at low... show more
61,855 görüntüleme • 5 gün önce •via X (Twitter)
38 Yorum

@TheZachMueller @deepseek_ai @huggingface Dude when can you re-run openclaw, its killing me to watch 😭

@TheZachMueller @deepseek_ai @huggingface It's running as we speak. Hopefully ready tomorrow.

would love to see omp by @stencil_labs

@stencil_labs omp is next on the list, will release results soon, stay tuned!

@deepseek_ai @huggingface Could you try copilot desktop harness?

@deepseek_ai @huggingface Did you fix the OpenClaw run with the poor methodology?

@deepseek_ai @huggingface And also please do Max effort too

@deepseek_ai @huggingface Do you wanna sponsor the compute for that? ;)

@deepseek_ai @huggingface O_O😵💫

@deepseek_ai @huggingface Hi Joël, would you mind adding Leon AI or share how can I run it on my side? I'm very curious to see how would our 2.0 behave

@deepseek_ai @huggingface Funny that cleaning up after itself is what cost it points. Would leaving the servers running close most of that gap?

@deepseek_ai @huggingface Now Droid Factory please!! We need it badd

@deepseek_ai @huggingface dsh losing terminal-bench points for shutting down its own background servers is kinda funny. tidy agent, wrong benchmark. do you flag those 3 tasks separately

@deepseek_ai @huggingface Comparing ten coding agents on the same open models separates harness quality from model quality. Which of the five evaluations showed the biggest gap between harnesses?

@deepseek_ai @huggingface Uses tons of tokens but it's very good

@deepseek_ai @huggingface Any findings on what makes a harness most effective?

@deepseek_ai @huggingface Currently focused on adding more harnesses. Will probably do some deeper analyses on it next week. Though all the traces are open, curious what you find :)

@deepseek_ai @huggingface Big tech Microsoft and AWS couldn't even make it to the listing 🙃 is it because you can't buy your way in?

@deepseek_ai @huggingface Which preset you used? Standard?

@deepseek_ai @huggingface Two of 36 on the server tasks, a steep price for good manners 😅

@deepseek_ai @huggingface Amazing! I've been thinking DS 4.1 Flash on High effort is, in some cases, better than Max

@deepseek_ai @huggingface The shutdown example is useful context behind the rank. It raises a good handoff question: which processes are temporary work, and which are the actual thing the user asked to keep running? That distinction should survive the final answer.

@deepseek_ai @huggingface Holding the model fixed and comparing harnesses is the comparison we'd want more of. For us the model hasn't been the system, and changing models hasn't made things more volatile.

@deepseek_ai @huggingface cleaning up is the polite thing in real life. losing points because the grader wanted the server still running feels like a hidden rule in the task, not a weaker agent. does the task text actually say "leave it running"?

@deepseek_ai @huggingface Devin & Capy please

@deepseek_ai @huggingface That shutdown habit is the most interesting detail in the whole thread. The agent is not failing at coding, it is failing at leaving things running for someone else to check, which is exactly the kind of invisible habit that decides whether an agent works in a real team.

@deepseek_ai @huggingface try with distill :D I'd love to see it in a battle:

@deepseek_ai @huggingface How does this compare to ? Cause they rank codex #1, and Opencode is way below.

@deepseek_ai @huggingface the shutdown habit is the tell. leftovers from task 3 still running during task 7, eating CPU and confusing the next test, so a harness that reaps its own children looks smarter on the same weights. i've chased a 'model regression' that turned out to be a leftover process.

@deepseek_ai @huggingface Aggressive process cleanup is usually good hygiene in production to prevent orphaned background jobs, even if it penalizes them on benchmarks that check state post-exit.

Is the tool-call budget held constant across harnesses? I’d want to see whether a score gap holds when each agent gets the same number of calls.

@deepseek_ai @huggingface Tying for first on open models at only $0.53 a solve is insane! It's hilarious that DSH loses the outright #1 spot just because it cleans up background servers too well before grading.

@deepseek_ai @huggingface Prime-agent jcode omo

@deepseek_ai @huggingface shutting down every background process after a task is smart until the grader still needs that server running.

@deepseek_ai @huggingface That server example makes the handoff part of the task: the next user still needs the service running. Does DSH expose a per-task cleanup policy, or would an operations team need to change the harness?

@deepseek_ai @huggingface damn, opencode my goat huh

@deepseek_ai @huggingface thx very interesting to see, dsh has an extremely lean system prompt ~1.5k only, there is almost nothing explicitly defined, and tool definitions have ~7.5k

@deepseek_ai @huggingface We actually showed how GLM 5.3 Flash became the benchmark standard, routing optimization made it faster and cheaper, so leaderboards are starting to reflect that.
