Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

We tuned an AI agent that can do large-scale document extraction from long docs (50+ pages, some with 10k-100k fields) with 94%+ accuracy 📈 It uses a harness + model set that is tuned specifically for reasoning over extracting out complex information from complex docs. Each extracted field comes...

18,359 Aufrufe • vor 27 Tagen •via X (Twitter)

25 Kommentare

Profilbild von EMAN 💔
EMAN 💔vor 24 Tagen

You have the power to help, and we are eagerly awaiting it. Don't just pass by without doing something. [Campaign Link:

Profilbild von Alessandro Frau
Alessandro Frauvor 27 Tagen

This is cool, I see it as an expert agent that can be consulted by others via A2A or another protocol. I also need to remind myself to update LiteParse version in Agent Zero. Saw many improvements across the board : )

Profilbild von Off Script: Playable Series
Off Script: Playable Seriesvor 27 Tagen

ok that's genuinely cool. Bounding boxes on every field is the best part.

Profilbild von Forlais
Forlaisvor 26 Tagen

Provenance on every field is the right instinct. The EvaEsi Index does the same for claims, keeping the source attached and flagging you when a cited page changes underneath.

Profilbild von grim · vibe coding final boss
grim · vibe coding final bossvor 26 Tagen

94% on extraction is the demo number. In production it dies when the same field has two legal values in one packet: invoice total vs line total, both labeled Amount. The model picks one and never flags the collision.

Profilbild von 刘朝 Zhao Liu
刘朝 Zhao Liuvor 27 Tagen

94%+ extraction across 100k fields is the kind of result that deserves a reproducible corpus, dedup rules, bounding-box checks, and an error taxonomy. Long documents punish vague benchmarks.

Profilbild von Manish | Skygnosis
Manish | Skygnosisvor 26 Tagen

the confidence score + bounding box is what really stands out. accuracy tells you it works; provenance tells you where to look when it doesn't.

Profilbild von M.M.M.
M.M.M.vor 26 Tagen

94% on 50k+ fields is a solid leap, but enterprise compliance needs deterministic guarantees for the remaining 6%. Pairing the agent with schema-level cross-field validation rules and deterministic constraint solvers could bridge that gap without adding manual review.

Profilbild von Ella Tech & Tool
Ella Tech & Toolvor 27 Tagen

94%+ on 100k fields is insane 🔥 Big win for LlamaParse

Profilbild von Fady | AI Systems
Fady | AI Systemsvor 27 Tagen

The confidence score + bounding box combination is the part that actually matters for production use. General coding agents are optimized for “looks plausible.” Specialized harnesses that force evidence and location for every field are optimized for “can be audited.” That gap is still under-appreciated.

Profilbild von FlowOps Daily
FlowOps Dailyvor 27 Tagen

agentic plus is what makes the demo land. Showing the before and after in the same frame would make the change even easier to judge.

Profilbild von M.Camisani-Calzolari
M.Camisani-Calzolarivor 25 Tagen

Yes. The harness matters. In production, the model still needs a human to sign off when it improvises off-spec. I see that here in the States every day.

Profilbild von Jurly
Jurlyvor 27 Tagen

confidence scores and bounding boxes make automated extraction much easier to audit.

Profilbild von Adel Bucetta
Adel Bucettavor 27 Tagen

the reason most people miss this is that they don't realize the majority of human knowledge still lives in documents, not structured data. our job is to bridge that gap

Profilbild von Nico
Nicovor 27 Tagen

the 10k to 100k field range is where every extraction pipeline ive used starts double counting rows, hows yours handling dedupe

Profilbild von AI Mastery Guide
AI Mastery Guidevor 27 Tagen

94% accuracy on 100k fields is impressive 📈

Profilbild von Fajar M Reza
Fajar M Rezavor 27 Tagen

Long-document extraction reaching 94% accuracy makes structured evaluation the real breakthrough.

Profilbild von Henry Nguyen
Henry Nguyenvor 27 Tagen

94% on docs that long is honestly wild, we struggle with 10-page PDFs sometimes. What’s the biggest bottleneck you hit when tuning it?

Profilbild von RAJ
RAJvor 26 Tagen

On long docs the extraction was never the hard part, building ground truth to trust it was. Our worst error class on 50+ page contracts: confidently pulling the right field from the wrong table across a page break. Layout-aware chunking helped more than a bigger model.

Profilbild von AI Apps API
AI Apps APIvor 26 Tagen

The bounding box is what makes the confidence score usable. A score on its own only tells you to doubt a field. A score plus a location means the reviewer lands on the exact spot instead of rereading 50 pages, which is what makes a threshold policy affordable in the first place. 94% matters far less than whether the system knows which 6% it got wrong.

Profilbild von Aryans
Aryansvor 27 Tagen

Specialized > generalized for extraction

Profilbild von AgentVet.ai
AgentVet.aivor 26 Tagen

i'll be reading more on this

Profilbild von Nick
Nickvor 27 Tagen

narrow harness beats a general one

Profilbild von Lakshmi Narayana
Lakshmi Narayanavor 27 Tagen

How does wall clock look on a 100k field doc versus the coding agent harnesses? Accuracy lift is easy to sell, waiting an hour per doc less so.

Profilbild von Arash Rahimi
Arash Rahimivor 27 Tagen

confidence score plus bounding box is the right instinct. curious what happens below some threshold. does a low-confidence field get flagged for a human to check, or does it ship with the number attached and the burden's still on whoever reads the output.

Ähnliche Videos

The latest RAG trend for the current agent harnesses (Codex, Cowork) is to do two passes of document processing to solve a knowledge work task over a data room of documents: 1️⃣ A fast and light pass, oftentimes using a free/OSS doc parsing tool. This can be cheaply run across 10-100-1k’s of files, and enables the agent to then do retrieval (e.g. grep, semantic) to find relevant subsets of context. 2️⃣ A “just-in-time” VLM-based pass. Once the agent finds the relevant pages of context, it will screenshot the documents can call its own VLM (or write code) to dissect the pages. The issue with only using VLM-based OCR tools over massive ad-hoc customer file dumps is that it’s slow and expensive. Doing JIT VLM OCR allows the agent to filter through the data cheaply, but still preserve accuracy for the context that’s needed for the task. The agent harnesses do two-pass document processing by default using off the shelf-tools: pdf2text as the first pass, and using itself (Opus 5) as the second pass. See the below video where Cowork runs over a bunch of PDFs to answer a question about a benchmark graph in the Kimi k3 paper. The main issues here with the “out of the box” doc processing these agents offer are: * Opus 5 is not the best VLM for OCR. It is also way too expensive at scale and lacks grounding * The OSS tools like pypdf, pdf2text, may not be versatile enough as the first pass. * The agent will write a lot of throwaway code to rewrite things an OCR tool would’ve provided out of the box, like chart processing, bounding boxes, confidence scores, leading to increased cost and speed. We have all the tools within LlamaIndex 🦙 to help any agent do two-pass document processing with higher accuracy and lower cost. 1️⃣ We have liteparse for the first pass - a free/OSS parser written in Rust that’s faster/more accurate than other OSS parsers, and supports 50+ document types 2️⃣ We have LlamaParse for the second pass - an agentic document engine that uses VLMs+harnesses to achieve SOTA in accuracy and cost across various doc parsing and extraction tasks. It can be called from any agent harness as an MCP or skill. It takes in page numbers as input, so that the agent can choose to run LlamaParse over a subset of the doc instead of the full doc as a “zoom-in” pass. Come check it out! LiteParse: LlamaParse: All the relevant docs, including MCP, are here:

Jerry Liu

22,763 Aufrufe • vor 20 Tagen

Introducing ExtractBench, the most comprehensive benchmark for information extraction from complex enterprise documents. The latest models are pushing the frontier of coding and knowledge work, but surprisingly they still struggle on complex doc extraction tasks in production. A well-tuned extractor must parse multi-page filings without dropping rows, emit exact spatial citations for auditability, and handle messy scans. Also they must do all of this at a viable per-page cost so that you can scale this to millions of docs in production (you can’t be paying upwards of $1 in tokens per page!) Existing extraction benchmarks fall short: they are not large/diverse enough in document domain (finance, energy, gov, auto), elements (long records, scans, grounding), and schemas. So our applied research team built ExtractBench. We evaluated 14 systems: frontier VLMs, coding agents, and specialized extraction APIs, against 370 enterprise documents: 4,869 pages, 67 document types. Our biggest finding 🧪: Short documents mask critical system flaws. On files past 50 pages, commercial VLMs collapse below 35% recall due to silent list truncation. They hold high precision, but lose output attention and drop most of the table rows. ExtractBench evaluates value accuracy, long-record completeness, spatial grounding, and per-page cost with zero LLM judges. It is 100% deterministic and reproducible. In tandem with ExtractBench, we’re also introducing 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗣𝗹𝘂𝘀, a new Extract tier in LlamaParse that debuts at #1 on the leaderboard: 95.6% value accuracy, at less than a third the cost of the closest peer. Explore the findings, download the dataset, or run the harness: Blog: GitHub: HuggingFace: We will be actively evolving both our extraction benchmark as well as our extraction harness over time. If you check out either ExtractBench or LlamaParse, let us know your feedback!

Jerry Liu

75,415 Aufrufe • vor 1 Monat

We’re open sourcing the first document OCR benchmark for the agentic era, ParseBench. Document parsing is the foundation of every AI agent that works with real-world files. ParseBench is a benchmark that measures parsing quality specifically for agent knowledge work: ✅ It optimizes for semantic correctness (instead of exact similarity) ✅ It has the most comprehensive distribution of real-world enterprise documents It contains ~2,000 human-verified enterprise document pages with 167,000+ test rules across five dimensions that matter most: tables, charts, content faithfulness, semantic formatting, and visual grounding. We benchmarked 14 known document parsers on ParseBench, from frontier/OSS VLMs to specialized parsers to LlamaParse. Here are some of our findings: 💡 Increasing compute budget yields diminishing returns - Gemini/gpt-5-mini/haiku gain 3-5 points from minimal to high thinking, at 4x the cost. 💡 Charts are the most polarizing dimension for evaluation. Most specialized parsers score below 6%, while some VLM-based parsers do a bit better. 💡 VLMs are great at visual understanding but terrible at layout extraction. GPT-5-mini/haiku score below 10% on our visual grounding task, all specialized parsers do much better. 💡 No method crushes all 5 dimensions at once, but LlamaParse achieves the highest overall score at 84.9%, and is the leader in 4 out of the 5 dimensions. This is by far the deepest technical work that we’ve published as a company. I would encourage you to start with our blog and explore our links to Hugging Face to GitHub. All the details are in our full 35-page (!!) ArXiv whitepaper. 🌐: Blog: 📄 Paper: 💻 Code: 📊 Dataset: 🎥 YouTube:

Jerry Liu

108,093 Aufrufe • vor 5 Monaten