We made a fair OCR benchmark. And we're not ranked first. Introducing OmniParseBench - 16k tests across 90 languages with consistent, interpretable scoring.

Read more at . Data at .

For over a year, we've been using olmOCR-bench, which we think is the most fair and consistent benchmark in the space. But it's saturated, and most remaining gains come from matching its formatting, not reading better.

ParseBench has emerged as an alternative, but it has many different types of tests that are hard to interpret, and sometimes conflicting.

OmniParseBench covers several OCR edge cases, including handwriting, tables, multiple languages, math, and forms. We'll keep it updated with new tests, to track progress in the field.

and @hheiden12 put in a lot of work to get this release out, and to make sure this benchmark is as reliable and fair as possible. More to come soon.

Where is lightOn???

We'll post open source models next week, we wanted to score a few different ones, and didn't have time today.

congrats on the release!

Thanks, you too!

who ranked first? curious what method beat yours on your own benchmark.

Great release, Vik. Did the other thing sort out ? There was a lot of noise around the chandra model being copied. Hope all settled in. Will look at the OCR benchmark !

A fair benchmark without ranking first is really rare, this benchmark looks promising for the OCR field.

Thank you @VikParuchuri for showing the right way to showcase a fair benchmark even your own product does not top it.

Did you benchmark new LightOnOCR-3 on this? @VikParuchuri

I would recommend to test falcon-ocr on this benchmark and lets us know your feedback.

Right number, wrong column is the invoice bug that still looks convincing.

This is really objective. Fair benchmarking is way more valuable than ranking first for boosting the whole OCR field.

the ranking is downstream of the metric. what breaks first across your 90 languages: cjk line segmentation or indic normalization inside the CER computation?

This is great news, I think the whole field needed a new benchmark. Glad to hear auditing is facilitated, the possibility to choose precisely which tags are kept in tests sounds very useful as well. Congratulations!