Unit tests check your code deterministically. But what happens when an LLM agent calls your WebMCP tools? 🎲 As Maud Nalpas explains, agent behavior varies. That's where evals come in! We built an experimental WebMCP Eval CLI tool to help you track pass rates and debug agent trajectories across... multiple runs. Have you tried running evals on your WebMCP tools?show more

The ordered/unordered example in your docs is useful: checking the jacket before the jeans should still pass, but reading either item's details before finding it shouldn't. A single expected sequence would penalize a valid route through the task.

Check out our evals CLI tool here: + File issues for feedback and feature requests!

Does the CLI record each tool call the agent makes?

Tracking agent trajectories with WebMCP Eval is a massive step forward for web automation. Raw pass rates give a solid baseline, but handling non-deterministic tool loops remains brutal. Do you evaluate agents using WebMCP or Braintrust?

how many runs behind each pass rate?

if the agent varies run to run, a failing run might be a bad tool description or just an odd route. what in a trajectory makes you call it one or the other?

Hey grok, remind me to check the garage for WebMCP tools.

WebMCP evals are important!

@maudnals great stuff!!!! 💪

how much do pass rates vary run to run?

evals for agent tool calls are the missing piece everyone hand-waves. pass rate tracking per tool is exactly the right primitive.