AI agents are already everywhere -- but most devtools weren't built for them. So we decided to do something about it. We ran Claude through each tool's getting-started guide and measured: → How fast it completed → How many tool calls it took → How many errors it hit... → How much it cost ... that led to a final grade for how agent ready each tool is. See where your favorite tools landed →show more

Starting with video & image gen tools today, but over the next few days, we'll be sharing more on auth, AI infra, search and database. If you're missing a devtool you want to see evaluated, drop a comment below! DM for a full report.

Image gen results: Grade A: @bfl_ml Grade B: @openai, @togethercompute, @grok, @FireworksAI_HQ, @GoogleAIStudio

Video gen results: Grade A: none Grade B: @fal, @togethercompute, @replicate Grade C: @OpenAI, @grok Grade D: @GoogleAIStudio

"agents are gonna be the primary user of docs/libs" is so true

@samuelstros Your share of agent users has increased apparently!

Want to see more rankings!

Love This! Looking forward to see observability Tools in the Arena I’ve felt that especially without checking docs, agents sometimes tend to use old SDK versions (prbl because of Training data)

Thank you Annabell! Observability is coming later this week :) excited to share results! Super interesting, thanks for sharing this! What's your go to coding agent?

90% of the time Claude Code But giving Codex a Shot now Looking forward to more benchmarks!

Tysm for sharing!!!! Keep you posted 🫶🏼

fascinating to see Dall-E scoring lower on discoverability! congrats on the launch

Congrats on the launch! So true - docs are for humans but most are being read by agents now, crazy how fast the world is moving!

Tysm! Excited to be in this space :)

very cool, let's go team!

Merci! 🌟

LFG

🌟🫡

so cool

tysm! let us know if you’re missing a tool

Excited to use your tool!

Sick, let's go!

🙏🌟

Congrats on the launch and the agent readiness scoring, this should help us ship faster

yay! let us know if we can help

Agents will for sure be the main users for the future!

🫡

Congrats on the launch!!!

Thank you for the support!

I can barely work with tools that dont run on claude anymore

Agreed! What tools are your go to right now? We’re still running evals, adding more tools every day!

gmi

🫡

Let's go guys!!!

🫡🔥

amazing!

Jia 🌟

Nice!

congrats on the launch! Excited to use it 🚀

Thank you so much Kaylyn!

Congrats!

Thank you!

This is the benchmark format we needed - completion time, tool calls, errors, and cost. Agent ready should be measured, not claimed.

Congrats Mika!! 🚀

Thank you Bera!!!

Awesome! We spend a lot of time experimenting to see which dev tools agents are good at using, this would save us some time!

Thank you for sharing this Ben! How do you usually estimate ease of use for agents?

We manually experiment behavior with different prompts, different apis, and intuition. Then we work with users to see how they feel the agent is doing on the tasks they want it to be good at.

let’s gooo

Thank you for making sure we’re all fed and caffeinated for our launch! 🥞☕️

always happy to help 🤍

Congrats on the launch!

🫡 tysm!

this is wild!

🔥🔥🔥

Let’s go!

🫡

Congrats on the launch!

Thank you!

Interesting benchmark, but testing only the getting-started guide measures documentation quality more than actual agent-readiness. The real test is what happens on call #50, when the agent hits rate limits, weird edge cases in error responses, or needs to chain 3 API calls with auth token refresh in between. That's where most integrations break. Did you see a correlation between "good docs" and "good agent DX" or were there surprises?

Thanks for adding your view, very helpful! Agreed, getting started is just one layer. We started there because it's the first gate -- if agents can't even set up, nothing else matters. But the real stuff comes after that Curious what you'd add to the final "agent readiness" score? we're thinking - error message clarity - rate limit behavior...

This is amazing!! Congrats on the launch Mika ❤️

Thank you so much Lisa!!!

Let’s goo!

🔥

Congrats guys! 🚀🚀🚀

Thank you!!!! ✨