Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Let's talk parsing charts 📊📈. Last week we released ParseBench, the first document OCR benchmark for AI agents. New in ParseBench: ChartDataPointMatch. Most document look at a chart and OCR the caption. Agents need the actual numbers. That's the gap between "OCR'd the text around the chart" and "actually...

13,987 görüntüleme • 6 ay önce •via X (Twitter)

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

We’re open sourcing the first document OCR benchmark for the agentic era, ParseBench. Document parsing is the foundation of every AI agent that works with real-world files. ParseBench is a benchmark that measures parsing quality specifically for agent knowledge work: ✅ It optimizes for semantic correctness (instead of exact similarity) ✅ It has the most comprehensive distribution of real-world enterprise documents It contains ~2,000 human-verified enterprise document pages with 167,000+ test rules across five dimensions that matter most: tables, charts, content faithfulness, semantic formatting, and visual grounding. We benchmarked 14 known document parsers on ParseBench, from frontier/OSS VLMs to specialized parsers to LlamaParse. Here are some of our findings: 💡 Increasing compute budget yields diminishing returns - Gemini/gpt-5-mini/haiku gain 3-5 points from minimal to high thinking, at 4x the cost. 💡 Charts are the most polarizing dimension for evaluation. Most specialized parsers score below 6%, while some VLM-based parsers do a bit better. 💡 VLMs are great at visual understanding but terrible at layout extraction. GPT-5-mini/haiku score below 10% on our visual grounding task, all specialized parsers do much better. 💡 No method crushes all 5 dimensions at once, but LlamaParse achieves the highest overall score at 84.9%, and is the leader in 4 out of the 5 dimensions. This is by far the deepest technical work that we’ve published as a company. I would encourage you to start with our blog and explore our links to Hugging Face to GitHub. All the details are in our full 35-page (!!) ArXiv whitepaper. 🌐: Blog: 📄 Paper: 💻 Code: 📊 Dataset: 🎥 YouTube:

Jerry Liu

108,403 görüntüleme • 6 ay önce

Jev as a reviewer for OCR! OpenBMB and ModelBest just released MiniCPM-OCR 1.0 for document parsing and MiniCPM-OCR Jev for reviewing the reliability of those outputs. That second part is what I find more interesting. OCR errors are often hard to catch once the output looks plausible. A number can be slightly wrong, a formula can be misread, or a table cell can shift, and the result can still look completely reasonable when it moves into the next step of the workflow. MiniCPM-OCR Jev sits after the parser and evaluates how reliable those outputs are across text, formulas, and tables. It does not rewrite or correct the OCR result. Its job is to score the output and help decide which blocks can move forward and which ones should be sent for human review. So the workflow becomes: Document → Layout detection → OCR parser → Jev review → Human review where needed On the parsing side, MiniCPM-OCR 1.0 is a 1.3B model that supports scanned documents, photographed pages, curved inputs, tilted documents, tables, formulas, and key information extraction. OpenBMB reports 93.68 on Real5-OmniDocBench and 54.7 on OCRBench v2, while DSpark improves end-to-end throughput by 62% in their benchmark setup. On the review side, MiniCPM-OCR Jev reaches 93.5 AUROC on their JevOCRBench, close to Qwen3.5-397B-A17B at 93.8 and above JEV-35B at 90.1 in the official evaluation. The useful part here is that you no longer have to treat the whole OCR output as equally reliable. You can review it block by block and route only the uncertain parts for another look. I tested both models on my own documents to see how well Jev catches OCR outputs that look right but are actually wrong. I’ve shared the GitHub repo below!

Sumanth

21,117 görüntüleme • 6 gün önce

Should you rely on public benchmark scores when choosing a model? Vik Paruchuri argues "no" in this talk. To illustrate this, he shows specific problems found in popular OCR benchmarks: ParseBench: • A test expects 310 gigawatts of solar-panel capacity when the correct value is 303. Additionally, 73% of its chart tests ask for values that aren't printed on the chart. (4:51) • A test requires "Parish Priest:" to be bold even though those words aren't bold in the source document. (15:33) • The scorer fails to recognize heading tags placed on separate lines. (16:15) • dots.mocr ranks last among six models even though it beats the first-ranked model on content and tables. This is an example of how overall benchmark score can hide important differences in model capabilities. (14:48) olmOCR-bench: • A model correctly preserves a line break in a document and gets penalized for it. (10:30) • A test expects "Republican Party" when a document says "Republican party". Copying the capitalization correctly makes the model fail. (10:49) • A 700-line postprocessor that rewrites model output adds 1 to 2 points, more than many gaps between models. (12:12) OmniDocBench: • This benchmark uses edit distance, which penalizes a correct table written in HTML more heavily than a wrong number in Markdown. So the format matters more than the answer! (6:58) This is just the tip of the iceberg, the video contains more examples. Chapters: 0:00 Choosing a model when every vendor claims to be #1 1:54 How OCR benchmark scores improve without better OCR 5:30 Metric design and tests that reject correct answers 11:33 Postprocessing, saturation, and misleading model rankings 12:40 How to slice benchmarks by your task and check for bias 16:53 How to build fair tests with verified answers and consistent scoring 19:05 How to test OCR models on your own documents 24:10 Benchmark marketing, private evals, and how to choose a model

Hamel Husain

10,612 görüntüleme • 2 gün önce