Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with human judgment. We do this by painstakingly... assigning severity scores to 330+ error types, and tuning each metric to match the severity scores. Existing benchmarks either use brittle deterministic metrics (e.g. OmniDocBench) or LLM-as-a-judge (ParseArena), which can help recognize equivalent representations but are coarse/stochastic/subject to bias by the prompt+model. ParseBench-2 also contains: ✅ 4.3k+ document pages (double from ParseBench-1). ✅ An extra dimension (forms) ✅ Much better ground-truth across all dimensions compared to ParseBench-1. This is the next evolution of ParseBench, our hugely popular benchmark that has 50k+ monthly downloads, over 150+ OCR solutions benchmarked, and has been adopted by companies like NVIDIA, Cohere, Aleph Alpha, Perplexity, and others. Alongside ParseBench-2, we are also releasing OCRia-1.5 - a new family of document parsing models that powers all our modes in LlamaParse. They represent a massive step up in capabilities from our latest release, and are state of the art in parsing complex tables, forms, charts, formatting, and assigning bounding boxes to complex layouts. Come check out our blog post, technical report, and website. Blog: Technical Report: Website: If you’re interested in trying out our OCRia-1.5 capabilities, come sign up for LlamaParse:show more

My favourite piece is the "Metric Lab" where you can change how the severity aware scoring works

the people yearn for llamaindex's launches lets go TEAM!!! so proud of our execution

I love myself a good eval

Most accurate benchmark out there! Congrats to Pierre and the research team here.. Was awesome getting to see them build this out firsthand. And pumped to share OCRia 1.5 with our customers.

forms! forms! forms!

As we were developing the metrics, we start by looking how other metric in other benchmark work, and how good they are at spotting designed error type (such as a permutation of 2 digit in a number), and how much they penalize it. We did not find metric that penalized as we expect, so we build our owns. You can explore how different parse benchmark measure/penalize different scenarios here:

looks good tbh

Not THE biggest news of the day, but huge nonetheless!! Great progress @llama_index

Way to go @jerryjliu0 and team! 🦙

The next era of document parsing!

Massive - congrats!

thanks @ConorBronsdon 🙌

ParseBench-2🚀Accuracy!!!

Eval the evals. Eval all the way down. 🦙

Incredible stuff! Big step up from v1

really cool staff. Congrats Jerry and team.

thanks @rohanpaul_ai 🫡

congrats for it team!!!

The depth and scenarios covered by parsebench-2 is impressive. It can be really helpful benchmark to see the weakness spots of a document pipeline and continuously making improvements.

the severity part is the right call, a broken table border is cosmetic but a misread digit in an amount looks totally fine and an agent will act on it without blinking

the bbox improvements are wild on dense tables

Big ups to the team for an update on an already industry leading benchmark

Severity weights still have to move scores.

scoring 330+ error types by hand is tedious work, but it's what makes the final number mean something

doubling the pages is nice, but better ground truth matters more, bad labels cap every score

Glad blank form fields count too. An agent inventing a value where someone left the box empty is a very different problem from missing text.

severity-weighted errors is the right call, a typo and a broken table shouldn't cost the same

curious how ocria-1.5 does on charts vs tables, they usually fail in very different ways

Could you release the raw outputs and granular eval logs for every parser, not just aggregate scores? The new error taxonomy will make them far more useful. Tracking here:

Which languages are supported?

llamaparse itself supports 50+ languages! if you try it out let us know your feedback

Thank you. Will check it out

The nasty case is a swapped year header: perfect digits, wrong year. Save that as agent memory and the error can outlive the PDF.

Nice work @jerryjliu0 ! I’m curious about the annotation process and the distribution of human annotations vs synthetic ones

glad blank fields staying blank is scored, a parser that fills an empty box is worse than one that misses a filled one, the agent downstream has no way to know the value was never there

which parsing error type hurts the score most, does a human reviewer agree with that ranking?