Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

You asked for dsh, I added it DeepSeek Harness is the open-source coding agent by DeepSeek. It's now on the Hugging Face Harness Leaderboard, which compares ten coding agents on the same open models. DSH is statistically tied for first in all five evaluations: DeepSeek-V4.1-Flash and GLM-5.3-Flash at low...

61,855 görüntüleme • 5 gün önce •via X (Twitter)

38 Yorum

Vincent Koc profil fotoğrafı
Vincent Koc5 gün önce

@TheZachMueller @deepseek_ai @huggingface Dude when can you re-run openclaw, its killing me to watch 😭

Joël Niklaus profil fotoğrafı
Joël Niklaus5 gün önce

@TheZachMueller @deepseek_ai @huggingface It's running as we speak. Hopefully ready tomorrow.

Yannick Monye profil fotoğrafı
Yannick Monye5 gün önce

would love to see omp by @stencil_labs

Joël Niklaus profil fotoğrafı
Joël Niklaus5 gün önce

@stencil_labs omp is next on the list, will release results soon, stay tuned!

Seti @ Home profil fotoğrafı
Seti @ Home5 gün önce

@deepseek_ai @huggingface Could you try copilot desktop harness?

Youssof Al Toukhi profil fotoğrafı
Youssof Al Toukhi5 gün önce

@deepseek_ai @huggingface Did you fix the OpenClaw run with the poor methodology?

AI_Luffy profil fotoğrafı
AI_Luffy5 gün önce

@deepseek_ai @huggingface And also please do Max effort too

Joël Niklaus profil fotoğrafı
Joël Niklaus5 gün önce

@deepseek_ai @huggingface Do you wanna sponsor the compute for that? ;)

AI_Luffy profil fotoğrafı
AI_Luffy5 gün önce

@deepseek_ai @huggingface O_O😵‍💫

Louis Grenard profil fotoğrafı
Louis Grenard5 gün önce

@deepseek_ai @huggingface Hi Joël, would you mind adding Leon AI or share how can I run it on my side? I'm very curious to see how would our 2.0 behave

Varun Chopra profil fotoğrafı
Varun Chopra5 gün önce

@deepseek_ai @huggingface Funny that cleaning up after itself is what cost it points. Would leaving the servers running close most of that gap?

AI_Luffy profil fotoğrafı
AI_Luffy5 gün önce

@deepseek_ai @huggingface Now Droid Factory please!! We need it badd

Mohith kumar profil fotoğrafı
Mohith kumar5 gün önce

@deepseek_ai @huggingface dsh losing terminal-bench points for shutting down its own background servers is kinda funny. tidy agent, wrong benchmark. do you flag those 3 tasks separately

ShadowSatoshi profil fotoğrafı
ShadowSatoshi5 gün önce

@deepseek_ai @huggingface Comparing ten coding agents on the same open models separates harness quality from model quality. Which of the five evaluations showed the biggest gap between harnesses?

Marcel Kaz profil fotoğrafı
Marcel Kaz5 gün önce

@deepseek_ai @huggingface Uses tons of tokens but it's very good

Jonas Hertner profil fotoğrafı
Jonas Hertner5 gün önce

@deepseek_ai @huggingface Any findings on what makes a harness most effective?

Joël Niklaus profil fotoğrafı
Joël Niklaus5 gün önce

@deepseek_ai @huggingface Currently focused on adding more harnesses. Will probably do some deeper analyses on it next week. Though all the traces are open, curious what you find :)

Ilpo Leppänen profil fotoğrafı
Ilpo Leppänen5 gün önce

@deepseek_ai @huggingface Big tech Microsoft and AWS couldn't even make it to the listing 🙃 is it because you can't buy your way in?

Marek Vrbka profil fotoğrafı
Marek Vrbka5 gün önce

@deepseek_ai @huggingface Which preset you used? Standard?

Aapakari profil fotoğrafı
Aapakari5 gün önce

@deepseek_ai @huggingface Two of 36 on the server tasks, a steep price for good manners 😅

Carlos Zuñiga profil fotoğrafı
Carlos Zuñiga5 gün önce

@deepseek_ai @huggingface Amazing! I've been thinking DS 4.1 Flash on High effort is, in some cases, better than Max

Talljack profil fotoğrafı
Talljack5 gün önce

@deepseek_ai @huggingface The shutdown example is useful context behind the rank. It raises a good handoff question: which processes are temporary work, and which are the actual thing the user asked to keep running? That distinction should survive the final answer.

Routekit Shell profil fotoğrafı
Routekit Shell5 gün önce

@deepseek_ai @huggingface Holding the model fixed and comparing harnesses is the comparison we'd want more of. For us the model hasn't been the system, and changing models hasn't made things more volatile.

Vinay Developer profil fotoğrafı
Vinay Developer5 gün önce

@deepseek_ai @huggingface cleaning up is the polite thing in real life. losing points because the grader wanted the server still running feels like a hidden rule in the task, not a weaker agent. does the task text actually say "leave it running"?

iamronaldo97 profil fotoğrafı
iamronaldo975 gün önce

@deepseek_ai @huggingface Devin & Capy please

Raúl Nogales 🇪🇸🇺🇸 profil fotoğrafı
Raúl Nogales 🇪🇸🇺🇸4 gün önce

@deepseek_ai @huggingface That shutdown habit is the most interesting detail in the whole thread. The agent is not failing at coding, it is failing at leaving things running for someone else to check, which is exactly the kind of invisible habit that decides whether an agent works in a real team.

Samuel Faj - The Token Saver 🙏 profil fotoğrafı
Samuel Faj - The Token Saver 🙏5 gün önce

@deepseek_ai @huggingface try with distill :D I'd love to see it in a battle:

Adhisu Sama profil fotoğrafı
Adhisu Sama5 gün önce

@deepseek_ai @huggingface How does this compare to ? Cause they rank codex #1, and Opencode is way below.

John Rood profil fotoğrafı
John Rood5 gün önce

@deepseek_ai @huggingface the shutdown habit is the tell. leftovers from task 3 still running during task 7, eating CPU and confusing the next test, so a harness that reaps its own children looks smarter on the same weights. i've chased a 'model regression' that turned out to be a leftover process.

John K profil fotoğrafı
John K5 gün önce

@deepseek_ai @huggingface Aggressive process cleanup is usually good hygiene in production to prevent orphaned background jobs, even if it penalizes them on benchmarks that check state post-exit.

Jace W. profil fotoğrafı
Jace W.5 gün önce

Is the tool-call budget held constant across harnesses? I’d want to see whether a score gap holds when each agent gets the same number of calls.

Inamul profil fotoğrafı
Inamul5 gün önce

@deepseek_ai @huggingface Tying for first on open models at only $0.53 a solve is insane! It's hilarious that DSH loses the outright #1 spot just because it cleans up background servers too well before grading.

Engel Der profil fotoğrafı
Engel Der4 gün önce

@deepseek_ai @huggingface Prime-agent jcode omo

Ofek Shaked | AI Engineer profil fotoğrafı
Ofek Shaked | AI Engineer4 gün önce

@deepseek_ai @huggingface shutting down every background process after a task is smart until the grader still needs that server running.

Ari 🦅 profil fotoğrafı
Ari 🦅5 gün önce

@deepseek_ai @huggingface That server example makes the handoff part of the task: the next user still needs the service running. Does DSH expose a per-task cleanup policy, or would an operations team need to change the harness?

Ernesto Davila Tirado profil fotoğrafı
Ernesto Davila Tirado5 gün önce

@deepseek_ai @huggingface damn, opencode my goat huh

silverqx profil fotoğrafı
silverqx5 gün önce

@deepseek_ai @huggingface thx very interesting to see, dsh has an extremely lean system prompt ~1.5k only, there is almost nothing explicitly defined, and tool definitions have ~7.5k

Martin Ronfort profil fotoğrafı
Martin Ronfort5 gün önce

@deepseek_ai @huggingface We actually showed how GLM 5.3 Flash became the benchmark standard, routing optimization made it faster and cheaper, so leaderboards are starting to reflect that.

Benzer Videolar

glm 5.3 vs qwen 3.8 vs gemini 3.7 vs deepseek v4 flash four models designed and built three structures each on a physics-backed site, with no dimensions anywhere in the brief the setup: our own agent loop on OpenRouter, a construction site as the tool set – footings, walls, arches, roofs, scaffold, a lamp. the site enforces physics and nothing else: unsupported brick falls, a roof needs walls under it, a worker reaches 3.2 m above whatever he stands on, an arch needs centring until the keystone is set, concrete cures before it carries. no budget ceiling – material cost is tallied and reported, never blocked. tasks: 1. house – a plot and a palette, no plan. shape, height and material are the model's call 2. lighthouse – a headland cut by a gully, with a rock stack standing 30 m offshore. the lamp must burn, it must be the highest thing built, and the keeper must be able to walk to it 3. bridge – a river with one islet and banks at different heights. cross it however you want models: Z.ai glm 5.3 flash, Qwen qwen 3.8 flash, Google DeepMind gemini 3.7 flash, DeepSeek v4 flash vision all twelve objects were finished and signed off by the models themselves. tallest lighthouse is qwen's at 38.4 m, planted on the offshore stack with a bridge run out to it – the only model that read the site that way. deepseek signed off its bridge on an empty riverbed: 0 bricks, 107 minutes, $1.16m of material tallied - total cost, three builds #1 glm 5.3 flash – $0.201 #2 gemini 3.7 flash – $0.871 #3 qwen 3.8 flash – $1.058 #4 deepseek v4 flash – $1.567 - wall clock, three builds #1 gemini 3.7 flash – 91m #2 glm 5.3 flash – 228m #3 deepseek v4 flash – 502m #4 qwen 3.8 flash – 912m - total tokens #1 gemini 3.7 flash – 3,567,052 #2 glm 5.3 flash – 4,732,748 #3 qwen 3.8 flash – 13,469,333 #4 deepseek v4 flash – 18,230,076 - defects logged by the site #1 deepseek v4 flash – 59 #2 gemini 3.7 flash – 132 #3 glm 5.3 flash – 221 #4 qwen 3.8 flash – 350 - material tallied across three builds #1 gemini 3.7 flash – $359,884 #2 glm 5.3 flash – $583,358 #3 deepseek v4 flash – $1,327,484 #4 qwen 3.8 flash – $2,188,625 observations: • glm is the cheap one and nothing here is close – $0.201 for three buildings, $0.042 per million tokens, 6x under gemini's rate • what glm spends it on is bulk, not care: 166,228 bricks in one house and 156 defect weight, the worst single object in the set • gemini is the efficiency line – 91 minutes and 3.57m tokens for all three and an eighth of qwen's clock • gemini also builds the smallest of everything. its lighthouse is 22.5 m against qwen's 38.4, its house 6.9 m against 19.3 • qwen is the maximalist: 1.18m bricks, $2.19m of material, tallest on all three tasks, and 912 minutes – 15 hours – to get there conclusion: twelve finished objects for $3.80 all in, and a 7.8x price spread between the cheapest model and the priciest! follow thehype. for 24/7 ai news, analysis and breakdowns

thehype.

26,360 görüntüleme • 2 ay önce