I ran the same task on Claude Code and DeepSeek's new agent harness. One cost $150. The other cost $2. Today we're launching (AgentSky), the "OpenRouter for Agents" — one API → Claude Code, Codex, DeepSeek, Kimi, OpenCode, and every major agent in the cloud. And Agent Playground on... top: race them on your own task, with your real tools (GitHub, Gmail, more), side by side in a browser: time, cost, tokens burnt. Guess which one was $2.show more

Race them yourself →

@agentsky_dev Seeing cost, time, and tokens together makes comparison much clearer.

@agentsky_dev I’m guessing DeepSeek handled the task for $2.

@agentsky_dev I love the Agent Playground idea. Very useful tool for AI devs building with agents.

@agentsky_dev This makes experimenting with a new agent much less of a project.

@agentsky_dev One API across major coding agents could simplify experimentation considerably.

@agentsky_dev I spent $80 on one rogue loop before adding circuit breakers. A two dollar run sounds like magic.

@agentsky_dev My last loop cost $800+. 😓😓😓

@agentsky_dev Congrats Xiaoyin, this is remarkable. Hopefully it goes as far, has similar outcomes, and goes big for acquisition.✨

@agentsky_dev lol haha thanks

@agentsky_dev Which agent burns the most tokens on the same job? That's the leaderboard I want

@agentsky_dev Interesting positioning. The cost gap is wild, but routing agents across harnesses is more about state and tool compatibility than just API calls. How do you handle session context when switching between Claude Code and DeepSeek mid-task?

@agentsky_dev from your question I know you are the pro! We handle tool state and tool compatibility! It's a lot of work. We are an infra company disguised as a routing one.

@agentsky_dev That makes sense. Tool state and compatibility are probably the real moat here. How do you keep context consistent when the same task moves between different harnesses?

@agentsky_dev My takeaway from AgentSky’s benchmark isn’t “always pick the cheapest.” It’s that agent choice is measurable: one pair may cost 3–10× more for only a few points of additional completion. Put cost, time, and outcome on the same screen.

@agentsky_dev Forget the API Give agents access to all the pools at once they instantly become 10x smarter and faster

@agentsky_dev congrats!!

@agentsky_dev I like not having to rebuild my workflow every time I want to try something different

@agentsky_dev the cost tail matters too: Claude Code × Fable 5 has a $16.54 p90 task cost overall; Hermes × DeepSeek V4 Pro is $1.79. Median comparisons already look wide. The expensive runs widen them further..

@agentsky_dev $150 vs $2 is the headline, but the number i track is how many of those runs i had to redo. most of my ~$840 a month at anthropic is buying not checking the work twice. cheap output i don't trust is the most expensive thing in the stack.

@agentsky_dev cool tool! seeing DeepSeek run 3.6x cheaper than GPT-5.6 Sol with the exact same median time is pretty wild 😅

@agentsky_dev Congrats on the launch! Will test it today.

@agentsky_dev thanks Eddie!!

@agentsky_dev In coding, Claude Code × Fable 5 costs $7.91 per task vs $2.91 for Codex × GPT-5.6 Sol: a 2.7× gap. Completion is 98.3% vs 97.4%. That extra 0.9 points is expensive

@agentsky_dev 祝贺！

@agentsky_dev 谢谢max

@agentsky_dev Harness also need routing!

@agentsky_dev Congrats! Very cool offering.

@agentsky_dev Like the 'OpenRouter for agents' idea. One request: surface success rate next to cost/tokens by default. Otherwise this just becomes a race to the cheapest, not the best.

@agentsky_dev 1m views in 90m, damn.

@agentsky_dev Curious how different the bills really are on identical tasks.

@agentsky_dev Even though deepseek is cheaper than claude, everything is harness dependent + thinking tokens used by deepseek, for example. That's been our experience at Headroom - curious what you're seeing

@agentsky_dev Anyone else immediately thinking of a terrible repo to test this on? 😂

@agentsky_dev Insane release

@agentsky_dev haha thanks!!

@agentsky_dev Although really like DeepSeek v4 Flash, it really cannot be compared to Claude Opus. I have spent 1 week troubleshooting a bad bug with DeepSeek, with orchestrator, multiple prompts working 24/7. A week later the bug was still there. Opus 5 High effort just fixed it in 3 hours.

@agentsky_dev As an adult we don't choose, we want them all 😈

@agentsky_dev Cost comparisons need a portable execution envelope. For a DSH race, can AgentSky export a replay bundle: harness/model versions, tool permissions, retained machine state, accepted output, tokens, compute time, and itemized cost? That would make the result auditable.

@agentsky_dev Congrats on the launch! This is really a brilliant idea! Harness also needs a route. btw Love your video style!

@agentsky_dev thanks Xinqi

@agentsky_dev Hangisi 2 dolar olan acaba

@agentsky_dev I’d trust a comparison more if every agent gets the exact same task

@agentsky_dev Same harness, different model: Hermes × DeepSeek V4 Pro costs $0.53; Hermes × Kimi K3 costs $2.26. DeepSeek is 4.3× cheaper, 256 seconds faster, and has higher completion in the public data.