Loading video...

Video Failed to Load

Go Home

We've built the most accurate document extraction system on the planet. Extract Max Effort will spend dynamic tokens reasoning about every visual detail across hundreds of pages to ensure that you get the maximum accuracy possible. Check out this massive well log report below. It's more accurate and still...

17,788 views • 8 days ago •via X (Twitter)

27 Comments

Rob Stover's profile picture
Rob Stover8 days ago

@JSON_JEFF

Vraj Talati's profile picture
Vraj Talati7 days ago

can i cap the spend per document?

Federico Ramallo's profile picture
Federico Ramallo7 days ago

What decides whether this ships isn't accuracy, it's whether it flags which fields it's unsure of. At 98% on 500 pages you still get ~10 wrong values; unmarked, someone re-reads all 500. Does Max Effort emit per-field confidence, or just the answer?

AP's profile picture
AP8 days ago

A well log is a brutal stress test, those docs are a mess of tables and scans. What happens when the model is confident but wrong on a single cell? Is there a per-field confidence flag or do you lean on the reasoning pass?

Andrés Dancausa's profile picture
Andrés Dancausa8 days ago

Max effort makes sense when one wrong field is expensive, like in a well log. The lever in production is knowing which pages need it, so the deep pass runs on the hard sections and the rest stays cheap and fast.

Rayaan⚡'s profile picture
Rayaan⚡7 days ago

How are you scoring 'most accurate' on docs like that well log: field-level exact match, or something weighted to the numbers that matter? On long reports one wrong unit in a single table row costs more than ten formatting misses.

Marius Laurusevicius's profile picture
Marius Laurusevicius8 days ago

Spending more tokens only where the page is hard makes sense. I'd still want a rough cost per page before sending a big batch.

Adam Hicks's profile picture
Adam Hicks7 days ago

architectural drawings?

Serkan PALA's profile picture
Serkan PALA7 days ago

Max-effort extraction is still a reading, not a posting. In SAP a document number appears only after company code, document type, and a named authorization object all pass, and the change log is written in the same step. 61% accuracy is a queue, not a closed ledger.

Anıl Şen's profile picture
Anıl Şen7 days ago

Well logs are a brutal benchmark: dense tables, unit conversions, scanned pages and handwritten notes all in one file. The thing that matters most once this runs at scale is calibrated per-field confidence. If you are pushing millions of extracted values into a warehouse, average accuracy is less useful than knowing which 2% of cells need a human look. Curious whether Max Effort exposes that, and whether the extra reasoning tokens can be targeted only at the low-confidence regions to keep cost predictable.

Tommy Lee's profile picture
Tommy Lee7 days ago

Spending dynamic tokens per visual detail is the right instinct - extraction errors compound into every downstream decision. My read: as agents handle longer documents, verifiable accuracy per page becomes the actual product, not the model wrapper.

sophie z.'s profile picture
sophie z.7 days ago

Claiming it's the "most accurate on the planet" ignores the inherent cost vs. accuracy tradeoff shown in the chart. At what point does the marginal gain in accuracy justify the extreme effort?

Daksh Agarwal's profile picture
Daksh Agarwal8 days ago

a well log is a brutal test, dense tables plus scanned plots. does max effort flag where it was unsure? on reports that long the useful thing downstream is knowing which pages need a human look, not only a higher average score

Nathalie 😻's profile picture
Nathalie 😻7 days ago

are you still doing " verify" i can't see anywhere

Max Petrusenko's profile picture
Max Petrusenko8 days ago

on the well log report, is accuracy scored per field or per page, and who labeled the ground truth?

刘朝 Zhao Liu's profile picture
刘朝 Zhao Liu8 days ago

Can a field return conflicting candidates with page-level provenance, so a reviewer can distinguish source disagreement from extraction error?

Karan Handa's profile picture
Karan Handa7 days ago

Can it read a prescription?

Ignacio Ferro Picón's profile picture
Ignacio Ferro Picón8 days ago

Answer keys are the slow part of a benchmark like this. A typesetter can produce them: I build one, Postext, and it returns every line and figure with its coordinates. Same values, different columns, floats and footnotes: you'd see exactly which layout breaks extraction.

Chloe Bennett's profile picture
Chloe Bennett7 days ago

Spending tokens dynamically per page makes cost hard to predict. Do users get an estimate before a big run, or a cap they can set per document?

SocialWatch's profile picture
SocialWatch7 days ago

Wow! Incredible work 👏🏻👏🏻👏🏻🫡

arun singh's profile picture
arun singh8 days ago

Spending more tokens reasoning didn't catch errors for us. Building a consensus from multiple reader did better.

kabishou11's profile picture
kabishou117 days ago

"Max Effort" on every page is the opposite of cheap RAG - accuracy wins, token bill follows. Would you flip this on for contracts, or only when a human already flagged the doc?

macblackstuff's profile picture
macblackstuff8 days ago

when a multi-page table splits across page breaks, summing column totals against the footer catches dropped rows before any retry

Agentik's profile picture
Agentik8 days ago

What does a max effort run cost on a few hundred pages? Well logs are brutal to parse, half the tables are scanned at an angle and the units change halfway through the report

Shanto Mathew | AI Builder's profile picture
Shanto Mathew | AI Builder7 days ago

curious how you score accuracy on something like a 300 page well log. per field checks against a labeled set, or spot checks?

Justin Yablonski's profile picture
Justin Yablonski7 days ago

Extraction accuracy is half of it for invoices. The other half is matching what you pulled against what's already in the system, like vendor, amount and dates, so a person only sees the real mismatches. That's where most of my build time has gone.

Kala's profile picture
Kala7 days ago

does page order affect the extraction? a correction tucked into an appendix could change a value that looked settled near the start.

Related Videos

The latest RAG trend for the current agent harnesses (Codex, Cowork) is to do two passes of document processing to solve a knowledge work task over a data room of documents: 1️⃣ A fast and light pass, oftentimes using a free/OSS doc parsing tool. This can be cheaply run across 10-100-1k’s of files, and enables the agent to then do retrieval (e.g. grep, semantic) to find relevant subsets of context. 2️⃣ A “just-in-time” VLM-based pass. Once the agent finds the relevant pages of context, it will screenshot the documents can call its own VLM (or write code) to dissect the pages. The issue with only using VLM-based OCR tools over massive ad-hoc customer file dumps is that it’s slow and expensive. Doing JIT VLM OCR allows the agent to filter through the data cheaply, but still preserve accuracy for the context that’s needed for the task. The agent harnesses do two-pass document processing by default using off the shelf-tools: pdf2text as the first pass, and using itself (Opus 5) as the second pass. See the below video where Cowork runs over a bunch of PDFs to answer a question about a benchmark graph in the Kimi k3 paper. The main issues here with the “out of the box” doc processing these agents offer are: * Opus 5 is not the best VLM for OCR. It is also way too expensive at scale and lacks grounding * The OSS tools like pypdf, pdf2text, may not be versatile enough as the first pass. * The agent will write a lot of throwaway code to rewrite things an OCR tool would’ve provided out of the box, like chart processing, bounding boxes, confidence scores, leading to increased cost and speed. We have all the tools within LlamaIndex 🦙 to help any agent do two-pass document processing with higher accuracy and lower cost. 1️⃣ We have liteparse for the first pass - a free/OSS parser written in Rust that’s faster/more accurate than other OSS parsers, and supports 50+ document types 2️⃣ We have LlamaParse for the second pass - an agentic document engine that uses VLMs+harnesses to achieve SOTA in accuracy and cost across various doc parsing and extraction tasks. It can be called from any agent harness as an MCP or skill. It takes in page numbers as input, so that the agent can choose to run LlamaParse over a subset of the doc instead of the full doc as a “zoom-in” pass. Come check it out! LiteParse: LlamaParse: All the relevant docs, including MCP, are here:

Jerry Liu

23,239 views • 2 months ago

Introducing ExtractBench, the most comprehensive benchmark for information extraction from complex enterprise documents. The latest models are pushing the frontier of coding and knowledge work, but surprisingly they still struggle on complex doc extraction tasks in production. A well-tuned extractor must parse multi-page filings without dropping rows, emit exact spatial citations for auditability, and handle messy scans. Also they must do all of this at a viable per-page cost so that you can scale this to millions of docs in production (you can’t be paying upwards of $1 in tokens per page!) Existing extraction benchmarks fall short: they are not large/diverse enough in document domain (finance, energy, gov, auto), elements (long records, scans, grounding), and schemas. So our applied research team built ExtractBench. We evaluated 14 systems: frontier VLMs, coding agents, and specialized extraction APIs, against 370 enterprise documents: 4,869 pages, 67 document types. Our biggest finding 🧪: Short documents mask critical system flaws. On files past 50 pages, commercial VLMs collapse below 35% recall due to silent list truncation. They hold high precision, but lose output attention and drop most of the table rows. ExtractBench evaluates value accuracy, long-record completeness, spatial grounding, and per-page cost with zero LLM judges. It is 100% deterministic and reproducible. In tandem with ExtractBench, we’re also introducing 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗣𝗹𝘂𝘀, a new Extract tier in LlamaParse that debuts at #1 on the leaderboard: 95.6% value accuracy, at less than a third the cost of the closest peer. Explore the findings, download the dataset, or run the harness: Blog: GitHub: HuggingFace: We will be actively evolving both our extraction benchmark as well as our extraction harness over time. If you check out either ExtractBench or LlamaParse, let us know your feedback!

Jerry Liu

75,787 views • 3 months ago

THIS IS STRAIGHT F*CKING GOLD Anthropic just shared one of the best tricks for Claude Code. The idea: one knob with five settings, and almost nobody ever moves it. Every routine turn runs at the same effort as your hardest decision, and you pay for thinking you never read. You just start Claude Code like this: claude --model opus --effort low here's the split: → low runs the routine turns, renames, imports, fixtures, formatting → /effort xhigh the moment an actual decision shows up → every subagent carries its own effort line in its frontmatter → you stop paying Opus 5.5 to think about which file to open you only climb the ladder at three moments: → before a plan that touches a lot of files → when the same bug survives two fixes → before a release you cannot roll back Low for the routine. High for the fork. Max for the rescue. JEV engineering is the same move one layer down: the forks that need no thinker (which file, which tool, retry or stop) go to JEV in milliseconds with zero thinking tokens, and the big models only see the ones that split the full tree > Opus 5.5 runs the main session at low effort > /effort xhigh the moment a real decision shows up, then back down > search runs on Haiku 5.5 with effort: low in its frontmatter > review runs on Sonnet 5.5 with effort: high in its frontmatter > every skill carries its own effort line, so the level follows the task > JEV routes micro-forks in milliseconds and thinks about none of them paste the tree and this prompt into Claude Code ↓ "Rebuild my Claude Code setup around this tree: 1. Run the main session on Opus 5.5 at low effort. 2. Set effort: low on every read-only subagent and effort: high on the review subagent. 3. Add one rule to CLAUDE.md: stay at low, climb to xhigh only before a multi-file plan, after the same error twice, and before an irreversible change, then drop back. 4. Route fast deterministic forks (tool choice, path selection, retries) through JEV. Show me every change as a diff first. No edits until I say go" ↳ save it, it'll come in handy.

delost

19,355 views • 23 days ago