正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Someone built a plugin that cuts Claude Code's cost by 54% on the same task, same model. Not a different model. Not a different harness. Just smarter tools underneath. It's called WOZCODE and the benchmark numbers are genuinely CRAZYYYY.

Hasan Toor

437,709 subscribers

20,754 次观看 • 6 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

same model. same budget. different harness. who won?

same model. same budget. different harness. who won?

Izkimar

264,791 次观看 • 3 个月前

Harness Engineering 2.0 is here! We benchmark models all the time but the harness running the model can dramatically improve the result, cost, and execution time too. And finally, devs just open-sourced Harness Arena, where you can test the same task across multiple harness and model combinations side by side. You provide one task and then run it across several model x harness configs that you want to test. Every config receives the same prompt, files, tools, skills, and evaluation checks. Harness Arena then returns: - total cost - execution time - whether the task passed - files and artifacts produced If you want to try it yourself, it is implemented within the open-source HarnessRouter project. Here's the repo: (don't forget to star it ⭐ ) HarnessRouter is a plug-and-play infrastructure layer that lets you run any harness under a single interface, like: - Codex - Hermes - Claude code - DeepSeek Harness - System One, powered by Jev - And 25 more agent harnesses In Harness Arena, the cheapest run does not automatically win. A config must complete the task correctly first. Cost and execution time are compared only among the configurations that pass. Here's the result from one Harness Arena test: - Hermes + GPT-5.2: 5/5 successful runs using 0.47 credits - Claude Code + Opus 4.8: 3/5 successful runs using 223 credits That is around 474x difference in credits, while the more expensive configuration also failed two runs. You can keep the model fixed to measure how much the harness changes the result. Or vary both to find the best harness and model pairing for that task. This helps answer practical questions such as: - Should this task run on Codex or Claude? - Can a cheaper config still pass every check? - Which setup is best to handle this workload? Once you find the best config, you can run future tasks through it using the same Unified Harness Protocol (UHP) interface for sessions, streaming, files, artifacts, cancellation, and failures. So you are no longer choosing an agent stack based only on a model leaderboard. Instead, you can benchmark the complete runtime on the task your product actually needs to solve. To dive deeper, my co-founder wrote an article explaining why model routing is different from harness routing, how UHP works, and what supporting multiple agent runtimes involves. It also covers the local setup, a working request, and how sessions and files move through the system. Read it below.

Harness Engineering 2.0 is here! We benchmark models all the time but the harness running the model can dramatically improve the result, cost, and execution time too. And finally, devs just open-sourced Harness Arena, where you can test the same task across multiple harness and model combinations side by side. You provide one task and then run it across several model x harness configs that you want to test. Every config receives the same prompt, files, tools, skills, and evaluation checks. Harness Arena then returns: - total cost - execution time - whether the task passed - files and artifacts produced If you want to try it yourself, it is implemented within the open-source HarnessRouter project. Here's the repo: (don't forget to star it ⭐ ) HarnessRouter is a plug-and-play infrastructure layer that lets you run any harness under a single interface, like: - Codex - Hermes - Claude code - DeepSeek Harness - System One, powered by Jev - And 25 more agent harnesses In Harness Arena, the cheapest run does not automatically win. A config must complete the task correctly first. Cost and execution time are compared only among the configurations that pass. Here's the result from one Harness Arena test: - Hermes + GPT-5.2: 5/5 successful runs using 0.47 credits - Claude Code + Opus 4.8: 3/5 successful runs using 223 credits That is around 474x difference in credits, while the more expensive configuration also failed two runs. You can keep the model fixed to measure how much the harness changes the result. Or vary both to find the best harness and model pairing for that task. This helps answer practical questions such as: - Should this task run on Codex or Claude? - Can a cheaper config still pass every check? - Which setup is best to handle this workload? Once you find the best config, you can run future tasks through it using the same Unified Harness Protocol (UHP) interface for sessions, streaming, files, artifacts, cancellation, and failures. So you are no longer choosing an agent stack based only on a model leaderboard. Instead, you can benchmark the complete runtime on the task your product actually needs to solve. To dive deeper, my co-founder wrote an article explaining why model routing is different from harness routing, how UHP works, and what supporting multiple agent runtimes involves. It also covers the local setup, a working request, and how sessions and files move through the system. Read it below.

Avi Chawla

30,196 次观看 • 2 天前

This completely changes how I look at AI benchmarks. Qwen 3.8 Max Preview felt nearly unusable in OpenCode and Qwen Code. With the Claude Code harness, it suddenly came surprisingly close to Kimi K3 and Fable 5. Same model. Same task. Different harness. Wildly different result. How is this even possible?

This completely changes how I look at AI benchmarks. Qwen 3.8 Max Preview felt nearly unusable in OpenCode and Qwen Code. With the Claude Code harness, it suddenly came surprisingly close to Kimi K3 and Fable 5. Same model. Same task. Different harness. Wildly different result. How is this even possible?

OmedTheVibeCoder

30,223 次观看 • 3 个月前

Your AI sucks. And it's not the model. It's you. "Make it better" isn't a prompt, it's a guessing game. I changed one step in my process and the output quality completely flipped. Same ideas, same models, totally different results.

Your AI sucks. And it's not the model. It's you. "Make it better" isn't a prompt, it's a guessing game. I changed one step in my process and the output quality completely flipped. Same ideas, same models, totally different results.

Flick

322,001 次观看 • 3 个月前

You can smell a big model. Not the parameter count. Not the benchmark score. It's that feeling when something is actually reasoning. Not just pattern matching. We call it "big model smell."

You can smell a big model. Not the parameter count. Not the benchmark score. It's that feeling when something is actually reasoning. Not just pattern matching. We call it "big model smell."

Arena.ai

112,227 次观看 • 7 个月前

how to use claude code mods like a top 1% user, step by step: give this to your agent before everyone catches on👇 1. set up Jev connect Jev to the model registry you want to use. check that the connection works and the listed models are available. 2. build your claude code mod open claude code 2.1.287 or later and paste this prompt: “build a mod called run-ledger. load plugin-authoring and use the API types for my installed version. create a dashboard that shows: - estimated cost per run, model, and source plugin where known - which model handles each task - each subagent’s status, latest action, and elapsed time include token counts, cache usage, and reported retries. count each request once. keep background tasks linked to their original run. use dated prices. label costs as API estimates, not subscription charges. show unknown when data is missing. connect the mod to my existing Jev setup. give Jev the task, available models, prices, budget, and relevant past results. ask it to recommend a model and explain why. start with recommendations. make automatic routing optional for eligible subagents. show the recommended model, actual model, result, and cost. include Jev’s own cost. keep state across hot reloads. add details and export. the dashboard itself must make no model calls. validate the plugin. test rendering, costs, attribution, and duplicate counting. give me steps for a live test.” 3. test the setup allow hot reload when prompted. run a simple task, a subagent task, and a mod-triggered model call. check that the dashboard updates, each request counts once, and Jev’s recommended model can actually run the task. 4. install the working mod ask claude to copy it out of the temporary folder and install it as a persistent plugin. see the cost. choose the model. check the result.

how to use claude code mods like a top 1% user, step by step: give this to your agent before everyone catches on👇 1. set up Jev connect Jev to the model registry you want to use. check that the connection works and the listed models are available. 2. build your claude code mod open claude code 2.1.287 or later and paste this prompt: “build a mod called run-ledger. load plugin-authoring and use the API types for my installed version. create a dashboard that shows: - estimated cost per run, model, and source plugin where known - which model handles each task - each subagent’s status, latest action, and elapsed time include token counts, cache usage, and reported retries. count each request once. keep background tasks linked to their original run. use dated prices. label costs as API estimates, not subscription charges. show unknown when data is missing. connect the mod to my existing Jev setup. give Jev the task, available models, prices, budget, and relevant past results. ask it to recommend a model and explain why. start with recommendations. make automatic routing optional for eligible subagents. show the recommended model, actual model, result, and cost. include Jev’s own cost. keep state across hot reloads. add details and export. the dashboard itself must make no model calls. validate the plugin. test rendering, costs, attribution, and duplicate counting. give me steps for a live test.” 3. test the setup allow hot reload when prompted. run a simple task, a subagent task, and a mod-triggered model call. check that the dashboard updates, each request counts once, and Jev’s recommended model can actually run the task. 4. install the working mod ask claude to copy it out of the temporary folder and install it as a persistent plugin. see the cost. choose the model. check the result.

Avid

47,276 次观看 • 1 个月前

π0.5 vs. MolmoAct2. Same task. Same object placements. Same training data regimen. Precise spatial placement lets us compare models side by side on any matching rollout in our Reality Check benchmark. Someone jokingly called it “Model Smash.” 👇 1/3

π0.5 vs. MolmoAct2. Same task. Same object placements. Same training data regimen. Precise spatial placement lets us compare models side by side on any matching rollout in our Reality Check benchmark. Someone jokingly called it “Model Smash.” 👇 1/3

Nicolas Keller

13,163 次观看 • 1 个月前

Most agent comparisons end up comparing models. We wanted to test the other half: the harness. That's the environment that decides whether a search actually runs, a source gets checked, and a file gets delivered. So we gave Minara, OpenClaw, Hermes and Claude Code the same tasks on the same base model: charts, research, code, and a prompt-injection test. Same model. Different outcomes. In the video, one AppLovin research note, GPT-5.6 Luna on both sides. Minara Harness delivered it in 1:51. Codex was still working when the recording ended. (5x speed) Full case studies 👇

Most agent comparisons end up comparing models. We wanted to test the other half: the harness. That's the environment that decides whether a search actually runs, a source gets checked, and a file gets delivered. So we gave Minara, OpenClaw, Hermes and Claude Code the same tasks on the same base model: charts, research, code, and a prompt-injection test. Same model. Different outcomes. In the video, one AppLovin research note, GPT-5.6 Luna on both sides. Minara Harness delivered it in 1:51. Codex was still working when the recording ended. (5x speed) Full case studies 👇

Minara AI

27,006 次观看 • 1 个月前

Sub-agent Model Selection — Different Tasks, Different Models Your main agent runs Qwen3.6-Plus for quality. But not every subtask needs a flagship model. Now sub-agents can use a different model. Create a skill file with model: openai:qwen3.5-plus and the sub-agent runs on that model. Powerful model for the hard parts, fast model for the easy parts. Save tokens without sacrificing quality on what matters.

Sub-agent Model Selection — Different Tasks, Different Models Your main agent runs Qwen3.6-Plus for quality. But not every subtask needs a flagship model. Now sub-agents can use a different model. Create a skill file with model: openai:qwen3.5-plus and the sub-agent runs on that model. Powerful model for the hard parts, fast model for the easy parts. Save tokens without sacrificing quality on what matters.

Qwen

21,333 次观看 • 7 个月前

Opus 4.7 - 400k vs 1m context - is there a difference? I've heard Theo - t3.gg talk about the fact that it is unlikely that Anthropic would have offered up a model with 1m context at the same cost, if it wasn't a different (i.e. cheaper to serve) model. I did a test where I toggled the 1m default model on & off in Claude Code (otherwise default settings, xHigh reasoning) and compared the outputs with 3x generations - same prompts etc. My observations: - Models feel DIFFERENT - often when you ask a model for the same generation, you get a somewhat different answer, but it feels & smells the same. Here 400k and 1m are very different every time - 400k model seems better - not that 1m is trash and 400k is amazing, but there are definitely issues with the level of ambition and accuracy that 1m model seems to have Examples of 1m failing: - Voxel Rome: the colosseum is nowhere near as impressive - Golden Gate: cars go sideways, waves not very high, bridge goes into land; though the structure of the bridge is a bit better - Stonehenge: structure is more 'wrong', lighting, shadows & textures are more flat and not as rich This isn't a conclusive evidence of course, but at least to me the two models do not behave the same way. Anecdotally as well when building 1m felt like it was doing more weird validation (e.g. going around in circles) and 400k was more straightforward. These sorts of things are harder to capture in tests, but you'd notice in Claude Code. You can review the hosted generations, see the code & prompts in the links below

Opus 4.7 - 400k vs 1m context - is there a difference? I've heard Theo - t3.gg talk about the fact that it is unlikely that Anthropic would have offered up a model with 1m context at the same cost, if it wasn't a different (i.e. cheaper to serve) model. I did a test where I toggled the 1m default model on & off in Claude Code (otherwise default settings, xHigh reasoning) and compared the outputs with 3x generations - same prompts etc. My observations: - Models feel DIFFERENT - often when you ask a model for the same generation, you get a somewhat different answer, but it feels & smells the same. Here 400k and 1m are very different every time - 400k model seems better - not that 1m is trash and 400k is amazing, but there are definitely issues with the level of ambition and accuracy that 1m model seems to have Examples of 1m failing: - Voxel Rome: the colosseum is nowhere near as impressive - Golden Gate: cars go sideways, waves not very high, bridge goes into land; though the structure of the bridge is a bit better - Stonehenge: structure is more 'wrong', lighting, shadows & textures are more flat and not as rich This isn't a conclusive evidence of course, but at least to me the two models do not behave the same way. Anecdotally as well when building 1m felt like it was doing more weird validation (e.g. going around in circles) and 400k was more straightforward. These sorts of things are harder to capture in tests, but you'd notice in Claude Code. You can review the hosted generations, see the code & prompts in the links below

Peter Gostev (SF: 22-26 June)

29,203 次观看 • 6 个月前

not every task needs a frontier model! we built a model router into our coding agent's harness and cut costs by 64% w/o reducing quality. this vid walks through how it works, and how you can build one into your own harness!

not every task needs a frontier model! we built a model router into our coding agent's harness and cut costs by 64% w/o reducing quality. this vid walks through how it works, and how you can build one into your own harness!

Sydney Runkle

12,620 次观看 • 1 个月前

MOST PEOPLE BUILD AI AGENTS BACKWARDS. They start with prompts, then add tools, memory, subagents, dashboards, and a graph nobody can debug. Then they wonder why the agent is expensive, confused, and unreliable. The better order is simple: harness first loop second graph only when needed A harness gives the model a workspace, tools, permissions, logs, and a definition of done. A loop lets it try, fail, read feedback, repair, and stop. A graph only makes sense when the workflow branches: approvals, parallel work, durable state, different routes, restart recovery. A graph is not a smarter prompt. A loop is not a production system. A harness is not just a system prompt. Most agents do not fail because the model is weak. They fail because nobody defined what success means.

MOST PEOPLE BUILD AI AGENTS BACKWARDS. They start with prompts, then add tools, memory, subagents, dashboards, and a graph nobody can debug. Then they wonder why the agent is expensive, confused, and unreliable. The better order is simple: harness first loop second graph only when needed A harness gives the model a workspace, tools, permissions, logs, and a definition of done. A loop lets it try, fail, read feedback, repair, and stop. A graph only makes sense when the workflow branches: approvals, parallel work, durable state, different routes, restart recovery. A graph is not a smarter prompt. A loop is not a production system. A harness is not just a system prompt. Most agents do not fail because the model is weak. They fail because nobody defined what success means.

Fluixo

12,244 次观看 • 3 个月前

Hit your usage limit in Codex/Claude Code mid-task? Orca now lets you resume the exact same session in a different agent harness — one click. Codex ⇄ Claude Code, full context intact🔥

Hit your usage limit in Codex/Claude Code mid-task? Orca now lets you resume the exact same session in a different agent harness — one click. Codex ⇄ Claude Code, full context intact🔥

Orca ADE

55,781 次观看 • 3 个月前

I made Claude compete against itself. The smartest model does not automatically make the best agent. To prove it, I took the same Opus 4.6 model, initial prompt, and empty repo and but it inside two different coding harnesses: Claude Code vs. Factory Droid. The task: clone Excalidraw from scratch, inspect the original in a browser, implement its key interactions, and verify the result. Factory Droid: 8 minutes, 20 tool calls, $1.60 Claude Code: 19 minutes, 40 tool calls, $1.87 The only difference was the harness. Get Free Factory credits here:

I made Claude compete against itself. The smartest model does not automatically make the best agent. To prove it, I took the same Opus 4.6 model, initial prompt, and empty repo and but it inside two different coding harnesses: Claude Code vs. Factory Droid. The task: clone Excalidraw from scratch, inspect the original in a browser, implement its key interactions, and verify the result. Factory Droid: 8 minutes, 20 tool calls, $1.60 Claude Code: 19 minutes, 40 tool calls, $1.87 The only difference was the harness. Get Free Factory credits here:

Sarah Chieng

106,993 次观看 • 2 个月前

wow... I just cut my claude code per-call cost to almost nothing with a 30-second config swap. Step 3.5 from StepFun plugs into your existing ~/.claude/settings.json by swapping three env vars. same workflow, different backend underneath. my agents have not stalled once since the switch. full setup below 🧵

wow... I just cut my claude code per-call cost to almost nothing with a 30-second config swap. Step 3.5 from StepFun plugs into your existing ~/.claude/settings.json by swapping three env vars. same workflow, different backend underneath. my agents have not stalled once since the switch. full setup below 🧵

Farhan

17,871 次观看 • 5 个月前

I ran the same task on Claude Code and DeepSeek's new agent harness. One cost $150. The other cost $2. Today we're launching (AgentSky), the "OpenRouter for Agents" — one API → Claude Code, Codex, DeepSeek, Kimi, OpenCode, and every major agent in the cloud. And Agent Playground on top: race them on your own task, with your real tools (GitHub, Gmail, more), side by side in a browser: time, cost, tokens burnt. Guess which one was $2.

I ran the same task on Claude Code and DeepSeek's new agent harness. One cost $150. The other cost $2. Today we're launching (AgentSky), the "OpenRouter for Agents" — one API → Claude Code, Codex, DeepSeek, Kimi, OpenCode, and every major agent in the cloud. And Agent Playground on top: race them on your own task, with your real tools (GitHub, Gmail, more), side by side in a browser: time, cost, tokens burnt. Guess which one was $2.

Xiaoyin Qu

2,227,612 次观看 • 2 个月前

The next task will almost certainly suit a different agent. Keep the context when you change tools → With Bring Your Own Agent, you can switch between Codex, Claude Code, Cursor, and other tools while keeping the same underlying context: • Switch model tools seamlessly • Reuse recorded briefs and Playbooks • Maintain strict permission boundaries Models will keep changing. Today it’s Claude, tomorrow it’s Cursor. What stays is the context you built. Save the rules, keep the history, and bring whichever agent fits the job.

The next task will almost certainly suit a different agent. Keep the context when you change tools → With Bring Your Own Agent, you can switch between Codex, Claude Code, Cursor, and other tools while keeping the same underlying context: • Switch model tools seamlessly • Reuse recorded briefs and Playbooks • Maintain strict permission boundaries Models will keep changing. Today it’s Claude, tomorrow it’s Cursor. What stays is the context you built. Save the rules, keep the history, and bring whichever agent fits the job.

Ethan Pierce

13,713 次观看 • 17 天前

Anthropic pays its ML engineers $500K+/year to deploy what Percy Liang just taught for free in Lecture 3 of CS221. 80 minutes. one whiteboard. everything between "a neuron" and the model running in your IDE right now. three things click after this one: / why two Claude prompts that "mean the same thing" land in different regions of weight space / why fine-tuning sometimes makes a model dumber, not smarter / what RLHF actually changes when Anthropic aligns Claude same $90K Stanford sequence. lecture 3 of ~20. free. the gap between "using Claude" and "understanding it" is exactly one lecture wide.

Anthropic pays its ML engineers $500K+/year to deploy what Percy Liang just taught for free in Lecture 3 of CS221. 80 minutes. one whiteboard. everything between "a neuron" and the model running in your IDE right now. three things click after this one: / why two Claude prompts that "mean the same thing" land in different regions of weight space / why fine-tuning sometimes makes a model dumber, not smarter / what RLHF actually changes when Anthropic aligns Claude same $90K Stanford sequence. lecture 3 of ~20. free. the gap between "using Claude" and "understanding it" is exactly one lecture wide.

dunik

24,455 次观看 • 5 个月前

AI isn't built by companies alone. It's built on the actions of millions of people. The problem? Millions contribute. A handful benefit. We're building a different future with Action Model.

AI isn't built by companies alone. It's built on the actions of millions of people. The problem? Millions contribute. A handful benefit. We're building a different future with Action Model.

Action Model

17,879 次观看 • 3 个月前