Introducing ExtractBenchPro: the most challenging benchmark for real-world document extraction. We see a massive long tail of document complexity in terms of use cases brought on from our customers, that are completely unsolved from a price-performance standpoint. These include trial balances, well logs, canvass binders, site plan reviews, duty... reconciliations, loss runs, and more. Most existing extraction benchmarks are saturated by the frontier models (our cheapest modes now score 93%+ on ExtractBench). So there is enormous economic value in building a complex extraction benchmark that reflects this long-tail of knowledge work. We spent a long time curating the most complex elements that we saw from real world extraction use cases: ✅ Document conventions: containing legends or rules where the rest of the document should be interpreted ✅ Density of internal jargon: specific definitions that must be cross-referenced within a doc ✅ Aggregation and reasoning: beyond returning exact values on a page, infer values derived from aggregating multiple sources of information ✅ Drawn relations: complex diagrams like procedures and circuits ✅ Charts inside documents: read precise points within a chart ✅ Long chart sheets: some data series like well logs cram all this data into an extremely long page Within the industries where document extraction is relevant, even small mistakes can have enormous negative impacts. Therefore our headline evaluation metric is based on perfect documents: the proportion of docs with perfect (100%) value f1 score. (Besides this we also measure value f1 and grounded f1) It’s an extremely hard task for even the latest frontier models to solve, topping out at ~58% or so in Value F1 score for Codex with Astra - which is quite good but extremely expensive (almost $1 per page!) Our LlamaParse agentic/agentic plus modes are 20-40% and cover the left side of the Pareto frontier, at price ranges that are feasible for full production use. ExtractBenchPro is available today as a website/technical blog, and also on Github and HuggingFace. Check it out! Blog: Website: Hugging Face: Github (is ExtractBench):show more

I love benchmarks where the top guys are at ~60%, means we have so much work to do to still! good drop!

launch on launch on launch 🚀

yasssssh

A long-tail extraction bench is overdue. Price-performance on the ugly docs is where most demos fall apart. Which document type is the current high-water mark still failing on?

Finally a real benchmark on realistic real world task that is not saturated! Seems like best harness model (codex / GPT6 Astra) only score 58.0 ! Hopefully this will help frontier/other model to improve. Accurately extracting info from doc is a crucial task for agents.

Scoring on perfect documents is the right call. Below that bar a human still reviews every page, and a silently dropped row in a long table is the worst miss because precision stays high while recall falls and nothing flags it.

waiting for the 99% score benchmark so I can finally automate my taxes

The perfect-document metric is especially important for enterprise use cases. A high average F1 can hide a single critical error in an otherwise well-extracted document. Reliable enterprise knowledge requires not just accurate extraction, but traceable evidence and validation of the relationships and inferences built on top of it.

Document conventions is what breaks chunked pipelines: a legend on page 1 changes how page 40 reads, and that chunk never sees it. A first pass that extracts doc-level rules and injects them into every chunk helps. How do you score derived values: exact match or tolerance?

open-world docs with hundreds of interrelated tasks. the eval is the product, not the model card

scoring on fully perfect documents is the right call, one wrong number in a reconciliation costs more than the whole extraction

Its been super interesting to see this task evolve over time. At the end of the day, customers expect perfect output, so making it the core metric makes a lot

what's the nastiest doc in there?

okayyy 1 dollar a page to still fail half the docs is insane 👏

what kind of documents are the hardest ones for the models, scanned tables or messy layouts? curious what your customers send you

🔥

Which doc type tripped up frontier models the most?

I think all parsing vendors should fund an independent company to create benchmarks. It’s difficult to trust vendor maintained benchmarks due to obvious survivorship bias.

Saturated benchmarks were the same problem we hit with BI tools years ago. Every vendor looked great on the clean demo dataset and fell over on the customer’s actual tables. Loss runs and trial balances are a much more honest test.

doc extraction is one of those boring problems that's secretly everything. glad someone's finally benchmarking the real-world version lol

Can the models tell which documents they got wrong? I'd love to see whether they can reliably flag the ones that need a human, even before they can get every field right.

almost $1 a page at ~58% value f1 gets uglier once you divide by perfect docs instead. if only a slice of docs come out 100% clean, codex is probably a few dollars per doc you can ship untouched, would love that column next to the pareto chart

Perfect-document scoring compounds per-field error, so a long well log fails far more often than a short form at the same per-field accuracy. I'd want the scores split by fields per document.

imo merged cells in scanned tables are where most extractors still fall over

The charts-inside-documents case hits hardest — clients hand me 40-page PDFs with embedded charts and expect clean tables. Cheapest trick that works: crop chart regions first, then extract. Layout detection is the step most pipelines skip.

messy docs are where extraction pipelines quietly break. which type scores worst?

Scoring on perfect docs is the right call. At 95% per-field accuracy, a 50-field doc comes out fully correct under 10% of the time, and in a trial balance one wrong value is the whole problem. Same trap as a 98% test pass rate while the one flow that matters is broken.