Joël Niklaus's banner
Joël Niklaus's profile picture

Joël Niklaus

@joelniklaus • 4,289 subscribers

Harness Optimization @HuggingFace

Shorts

High reasoning effort makes DeepSeek-V4.1-Flash 28% more expensive and not one point better. Reasoning effort sets how much the model thinks before it acts. On the Hugging Face Harness Leaderboard, I compared low and high effort with five coding agents: Claude Code, Codex, OpenCode, Pi and mini-swe-agent. The likely reason: DeepSeek-V4.1-Flash already reasons a lot at low effort, about 470 reasoning tokens per model call. High adds only 19% more, against 60% on GLM-5.3-Flash. On GLM-5.3-Flash, the extra thinking pays off: high effort lifts the same five agents by 5.8 percentage points on average. Each agent ran 198 coding tasks from the Terminal-Bench 2.1 and DeepSWE benchmarks at both efforts, three attempts each. The DeepSeek difference is within noise (95% CI -1.8 to +2.0 points). The GLM gain is not (+3.5 to +8.0). Leaderboard:

High reasoning effort makes DeepSeek-V4.1-Flash 28% more expensive and not one point better. Reasoning effort sets how much the model thinks before it acts. On the Hugging Face Harness Leaderboard, I compared low and high effort with five coding agents: Claude Code, Codex, OpenCode, Pi and mini-swe-agent. The likely reason: DeepSeek-V4.1-Flash already reasons a lot at low effort, about 470 reasoning tokens per model call. High adds only 19% more, against 60% on GLM-5.3-Flash. On GLM-5.3-Flash, the extra thinking pays off: high effort lifts the same five agents by 5.8 percentage points on average. Each agent ran 198 coding tasks from the Terminal-Bench 2.1 and DeepSWE benchmarks at both efforts, three attempts each. The DeepSeek difference is within noise (95% CI -1.8 to +2.0 points). The GLM gain is not (+3.5 to +8.0). Leaderboard:

26,946 次观看

Videos

没有更多内容可加载