Factory benchmarks let you build your own model bench using your past coding agent runs. - Mirrors your environment, secrets, and MCPs for each run - Scores output on judging criteria you define - Generates a report with model recs using cost vs. quality Here's how it works:

You can get early access at 🔖

Love this custom benchmarks from your own agent runs is such a powerful idea

Factory benchmarks make coding agent evaluation smarter, faster, and incredibly insightful.

DM me, let's grow 🚀

A smart approach to evaluating coding agents—real-world benchmarks make model selection far more practical

Past runs are useful once the system that produced them is pinned. If the model, prompt, tools, or orchestration changed between runs, the cost-quality report is comparing different agents.

custom bench using your own runs is how you actually choose

Building benches from your own runs is the smart way

This is a smart way to benchmark coding agents against real-world workloads instead of generic tests.