Загрузка видео...

Не удалось загрузить видео

На главную

You asked for dsh, I added it DeepSeek Harness is the open-source coding agent by DeepSeek. It's now on the Hugging Face Harness Leaderboard, which compares ten coding agents on the same open models. DSH is statistically tied for first in all five evaluations: DeepSeek-V4.1-Flash and GLM-5.3-Flash at low...

61,855 просмотров • 5 дней назад •via X (Twitter)

Комментарии: 38

Фото профиля Vincent Koc
Vincent Koc5 дней назад

@TheZachMueller @deepseek_ai @huggingface Dude when can you re-run openclaw, its killing me to watch 😭

Фото профиля Joël Niklaus
Joël Niklaus5 дней назад

@TheZachMueller @deepseek_ai @huggingface It's running as we speak. Hopefully ready tomorrow.

Фото профиля Yannick Monye
Yannick Monye5 дней назад

would love to see omp by @stencil_labs

Фото профиля Joël Niklaus
Joël Niklaus5 дней назад

@stencil_labs omp is next on the list, will release results soon, stay tuned!

Фото профиля Seti @ Home
Seti @ Home5 дней назад

@deepseek_ai @huggingface Could you try copilot desktop harness?

Фото профиля Youssof Al Toukhi
Youssof Al Toukhi5 дней назад

@deepseek_ai @huggingface Did you fix the OpenClaw run with the poor methodology?

Фото профиля AI_Luffy
AI_Luffy5 дней назад

@deepseek_ai @huggingface And also please do Max effort too

Фото профиля Joël Niklaus
Joël Niklaus5 дней назад

@deepseek_ai @huggingface Do you wanna sponsor the compute for that? ;)

Фото профиля AI_Luffy
AI_Luffy5 дней назад

@deepseek_ai @huggingface O_O😵‍💫

Фото профиля Louis Grenard
Louis Grenard5 дней назад

@deepseek_ai @huggingface Hi Joël, would you mind adding Leon AI or share how can I run it on my side? I'm very curious to see how would our 2.0 behave

Фото профиля Varun Chopra
Varun Chopra5 дней назад

@deepseek_ai @huggingface Funny that cleaning up after itself is what cost it points. Would leaving the servers running close most of that gap?

Фото профиля AI_Luffy
AI_Luffy5 дней назад

@deepseek_ai @huggingface Now Droid Factory please!! We need it badd

Фото профиля Mohith kumar
Mohith kumar5 дней назад

@deepseek_ai @huggingface dsh losing terminal-bench points for shutting down its own background servers is kinda funny. tidy agent, wrong benchmark. do you flag those 3 tasks separately

Фото профиля ShadowSatoshi
ShadowSatoshi5 дней назад

@deepseek_ai @huggingface Comparing ten coding agents on the same open models separates harness quality from model quality. Which of the five evaluations showed the biggest gap between harnesses?

Фото профиля Marcel Kaz
Marcel Kaz5 дней назад

@deepseek_ai @huggingface Uses tons of tokens but it's very good

Фото профиля Jonas Hertner
Jonas Hertner5 дней назад

@deepseek_ai @huggingface Any findings on what makes a harness most effective?

Фото профиля Joël Niklaus
Joël Niklaus5 дней назад

@deepseek_ai @huggingface Currently focused on adding more harnesses. Will probably do some deeper analyses on it next week. Though all the traces are open, curious what you find :)

Фото профиля Ilpo Leppänen
Ilpo Leppänen5 дней назад

@deepseek_ai @huggingface Big tech Microsoft and AWS couldn't even make it to the listing 🙃 is it because you can't buy your way in?

Фото профиля Marek Vrbka
Marek Vrbka5 дней назад

@deepseek_ai @huggingface Which preset you used? Standard?

Фото профиля Aapakari
Aapakari5 дней назад

@deepseek_ai @huggingface Two of 36 on the server tasks, a steep price for good manners 😅

Фото профиля Carlos Zuñiga
Carlos Zuñiga5 дней назад

@deepseek_ai @huggingface Amazing! I've been thinking DS 4.1 Flash on High effort is, in some cases, better than Max

Фото профиля Talljack
Talljack5 дней назад

@deepseek_ai @huggingface The shutdown example is useful context behind the rank. It raises a good handoff question: which processes are temporary work, and which are the actual thing the user asked to keep running? That distinction should survive the final answer.

Фото профиля Routekit Shell
Routekit Shell5 дней назад

@deepseek_ai @huggingface Holding the model fixed and comparing harnesses is the comparison we'd want more of. For us the model hasn't been the system, and changing models hasn't made things more volatile.

Фото профиля Vinay Developer
Vinay Developer5 дней назад

@deepseek_ai @huggingface cleaning up is the polite thing in real life. losing points because the grader wanted the server still running feels like a hidden rule in the task, not a weaker agent. does the task text actually say "leave it running"?

Фото профиля iamronaldo97
iamronaldo975 дней назад

@deepseek_ai @huggingface Devin & Capy please

Фото профиля Raúl Nogales 🇪🇸🇺🇸
Raúl Nogales 🇪🇸🇺🇸4 дней назад

@deepseek_ai @huggingface That shutdown habit is the most interesting detail in the whole thread. The agent is not failing at coding, it is failing at leaving things running for someone else to check, which is exactly the kind of invisible habit that decides whether an agent works in a real team.

Фото профиля Samuel Faj - The Token Saver 🙏
Samuel Faj - The Token Saver 🙏5 дней назад

@deepseek_ai @huggingface try with distill :D I'd love to see it in a battle:

Фото профиля Adhisu Sama
Adhisu Sama5 дней назад

@deepseek_ai @huggingface How does this compare to ? Cause they rank codex #1, and Opencode is way below.

Фото профиля John Rood
John Rood5 дней назад

@deepseek_ai @huggingface the shutdown habit is the tell. leftovers from task 3 still running during task 7, eating CPU and confusing the next test, so a harness that reaps its own children looks smarter on the same weights. i've chased a 'model regression' that turned out to be a leftover process.

Фото профиля John K
John K5 дней назад

@deepseek_ai @huggingface Aggressive process cleanup is usually good hygiene in production to prevent orphaned background jobs, even if it penalizes them on benchmarks that check state post-exit.

Фото профиля Jace W.
Jace W.5 дней назад

Is the tool-call budget held constant across harnesses? I’d want to see whether a score gap holds when each agent gets the same number of calls.

Фото профиля Inamul
Inamul5 дней назад

@deepseek_ai @huggingface Tying for first on open models at only $0.53 a solve is insane! It's hilarious that DSH loses the outright #1 spot just because it cleans up background servers too well before grading.

Фото профиля Engel Der
Engel Der4 дней назад

@deepseek_ai @huggingface Prime-agent jcode omo

Фото профиля Ofek Shaked | AI Engineer
Ofek Shaked | AI Engineer4 дней назад

@deepseek_ai @huggingface shutting down every background process after a task is smart until the grader still needs that server running.

Фото профиля Ari 🦅
Ari 🦅5 дней назад

@deepseek_ai @huggingface That server example makes the handoff part of the task: the next user still needs the service running. Does DSH expose a per-task cleanup policy, or would an operations team need to change the harness?

Фото профиля Ernesto Davila Tirado
Ernesto Davila Tirado5 дней назад

@deepseek_ai @huggingface damn, opencode my goat huh

Фото профиля silverqx
silverqx5 дней назад

@deepseek_ai @huggingface thx very interesting to see, dsh has an extremely lean system prompt ~1.5k only, there is almost nothing explicitly defined, and tool definitions have ~7.5k

Фото профиля Martin Ronfort
Martin Ronfort5 дней назад

@deepseek_ai @huggingface We actually showed how GLM 5.3 Flash became the benchmark standard, routing optimization made it faster and cheaper, so leaderboards are starting to reflect that.

Похожие видео

glm 5.3 vs qwen 3.8 vs gemini 3.7 vs deepseek v4 flash four models designed and built three structures each on a physics-backed site, with no dimensions anywhere in the brief the setup: our own agent loop on OpenRouter, a construction site as the tool set – footings, walls, arches, roofs, scaffold, a lamp. the site enforces physics and nothing else: unsupported brick falls, a roof needs walls under it, a worker reaches 3.2 m above whatever he stands on, an arch needs centring until the keystone is set, concrete cures before it carries. no budget ceiling – material cost is tallied and reported, never blocked. tasks: 1. house – a plot and a palette, no plan. shape, height and material are the model's call 2. lighthouse – a headland cut by a gully, with a rock stack standing 30 m offshore. the lamp must burn, it must be the highest thing built, and the keeper must be able to walk to it 3. bridge – a river with one islet and banks at different heights. cross it however you want models: Z.ai glm 5.3 flash, Qwen qwen 3.8 flash, Google DeepMind gemini 3.7 flash, DeepSeek v4 flash vision all twelve objects were finished and signed off by the models themselves. tallest lighthouse is qwen's at 38.4 m, planted on the offshore stack with a bridge run out to it – the only model that read the site that way. deepseek signed off its bridge on an empty riverbed: 0 bricks, 107 minutes, $1.16m of material tallied - total cost, three builds #1 glm 5.3 flash – $0.201 #2 gemini 3.7 flash – $0.871 #3 qwen 3.8 flash – $1.058 #4 deepseek v4 flash – $1.567 - wall clock, three builds #1 gemini 3.7 flash – 91m #2 glm 5.3 flash – 228m #3 deepseek v4 flash – 502m #4 qwen 3.8 flash – 912m - total tokens #1 gemini 3.7 flash – 3,567,052 #2 glm 5.3 flash – 4,732,748 #3 qwen 3.8 flash – 13,469,333 #4 deepseek v4 flash – 18,230,076 - defects logged by the site #1 deepseek v4 flash – 59 #2 gemini 3.7 flash – 132 #3 glm 5.3 flash – 221 #4 qwen 3.8 flash – 350 - material tallied across three builds #1 gemini 3.7 flash – $359,884 #2 glm 5.3 flash – $583,358 #3 deepseek v4 flash – $1,327,484 #4 qwen 3.8 flash – $2,188,625 observations: • glm is the cheap one and nothing here is close – $0.201 for three buildings, $0.042 per million tokens, 6x under gemini's rate • what glm spends it on is bulk, not care: 166,228 bricks in one house and 156 defect weight, the worst single object in the set • gemini is the efficiency line – 91 minutes and 3.57m tokens for all three and an eighth of qwen's clock • gemini also builds the smallest of everything. its lighthouse is 22.5 m against qwen's 38.4, its house 6.9 m against 19.3 • qwen is the maximalist: 1.18m bricks, $2.19m of material, tallest on all three tasks, and 912 minutes – 15 hours – to get there conclusion: twelve finished objects for $3.80 all in, and a 7.8x price spread between the cheapest model and the priciest! follow thehype. for 24/7 ai news, analysis and breakdowns

thehype.

26,360 просмотров • 2 месяцев назад