正在加载视频...

视频加载失败

We've built the most accurate document extraction system on the planet. Extract Max Effort will spend dynamic tokens reasoning about every visual detail across hundreds of pages to ensure that you get the maximum accuracy possible. Check out this massive well log report below. It's more accurate and still...

17,788 次观看 • 8 天前 •via X (Twitter)

27 条评论

Rob Stover 的头像
Rob Stover8 天前

@JSON_JEFF

Vraj Talati 的头像
Vraj Talati7 天前

can i cap the spend per document?

Federico Ramallo 的头像
Federico Ramallo7 天前

What decides whether this ships isn't accuracy, it's whether it flags which fields it's unsure of. At 98% on 500 pages you still get ~10 wrong values; unmarked, someone re-reads all 500. Does Max Effort emit per-field confidence, or just the answer?

AP 的头像
AP8 天前

A well log is a brutal stress test, those docs are a mess of tables and scans. What happens when the model is confident but wrong on a single cell? Is there a per-field confidence flag or do you lean on the reasoning pass?

Andrés Dancausa 的头像
Andrés Dancausa8 天前

Max effort makes sense when one wrong field is expensive, like in a well log. The lever in production is knowing which pages need it, so the deep pass runs on the hard sections and the rest stays cheap and fast.

Rayaan⚡ 的头像
Rayaan⚡7 天前

How are you scoring 'most accurate' on docs like that well log: field-level exact match, or something weighted to the numbers that matter? On long reports one wrong unit in a single table row costs more than ten formatting misses.

Marius Laurusevicius 的头像
Marius Laurusevicius8 天前

Spending more tokens only where the page is hard makes sense. I'd still want a rough cost per page before sending a big batch.

Adam Hicks 的头像
Adam Hicks7 天前

architectural drawings?

Serkan PALA 的头像
Serkan PALA7 天前

Max-effort extraction is still a reading, not a posting. In SAP a document number appears only after company code, document type, and a named authorization object all pass, and the change log is written in the same step. 61% accuracy is a queue, not a closed ledger.

Anıl Şen 的头像
Anıl Şen7 天前

Well logs are a brutal benchmark: dense tables, unit conversions, scanned pages and handwritten notes all in one file. The thing that matters most once this runs at scale is calibrated per-field confidence. If you are pushing millions of extracted values into a warehouse, average accuracy is less useful than knowing which 2% of cells need a human look. Curious whether Max Effort exposes that, and whether the extra reasoning tokens can be targeted only at the low-confidence regions to keep cost predictable.

Tommy Lee 的头像
Tommy Lee7 天前

Spending dynamic tokens per visual detail is the right instinct - extraction errors compound into every downstream decision. My read: as agents handle longer documents, verifiable accuracy per page becomes the actual product, not the model wrapper.

sophie z. 的头像
sophie z.7 天前

Claiming it's the "most accurate on the planet" ignores the inherent cost vs. accuracy tradeoff shown in the chart. At what point does the marginal gain in accuracy justify the extreme effort?

Daksh Agarwal 的头像
Daksh Agarwal8 天前

a well log is a brutal test, dense tables plus scanned plots. does max effort flag where it was unsure? on reports that long the useful thing downstream is knowing which pages need a human look, not only a higher average score

Nathalie 😻 的头像
Nathalie 😻7 天前

are you still doing " verify" i can't see anywhere

Max Petrusenko 的头像
Max Petrusenko8 天前

on the well log report, is accuracy scored per field or per page, and who labeled the ground truth?

刘朝 Zhao Liu 的头像
刘朝 Zhao Liu8 天前

Can a field return conflicting candidates with page-level provenance, so a reviewer can distinguish source disagreement from extraction error?

Karan Handa 的头像
Karan Handa7 天前

Can it read a prescription?

Ignacio Ferro Picón 的头像
Ignacio Ferro Picón8 天前

Answer keys are the slow part of a benchmark like this. A typesetter can produce them: I build one, Postext, and it returns every line and figure with its coordinates. Same values, different columns, floats and footnotes: you'd see exactly which layout breaks extraction.

Chloe Bennett 的头像
Chloe Bennett7 天前

Spending tokens dynamically per page makes cost hard to predict. Do users get an estimate before a big run, or a cap they can set per document?

SocialWatch 的头像
SocialWatch8 天前

Wow! Incredible work 👏🏻👏🏻👏🏻🫡

arun singh 的头像
arun singh8 天前

Spending more tokens reasoning didn't catch errors for us. Building a consensus from multiple reader did better.

kabishou11 的头像
kabishou117 天前

"Max Effort" on every page is the opposite of cheap RAG - accuracy wins, token bill follows. Would you flip this on for contracts, or only when a human already flagged the doc?

macblackstuff 的头像
macblackstuff8 天前

when a multi-page table splits across page breaks, summing column totals against the footer catches dropped rows before any retry

Agentik 的头像
Agentik8 天前

What does a max effort run cost on a few hundred pages? Well logs are brutal to parse, half the tables are scanned at an angle and the units change halfway through the report

Shanto Mathew | AI Builder 的头像
Shanto Mathew | AI Builder8 天前

curious how you score accuracy on something like a 300 page well log. per field checks against a labeled set, or spot checks?

Justin Yablonski 的头像
Justin Yablonski7 天前

Extraction accuracy is half of it for invoices. The other half is matching what you pulled against what's already in the system, like vendor, amount and dates, so a person only sees the real mismatches. That's where most of my build time has gone.

Kala 的头像
Kala7 天前

does page order affect the extraction? a correction tucked into an appendix could change a value that looked settled near the start.

相关视频

The latest RAG trend for the current agent harnesses (Codex, Cowork) is to do two passes of document processing to solve a knowledge work task over a data room of documents: 1️⃣ A fast and light pass, oftentimes using a free/OSS doc parsing tool. This can be cheaply run across 10-100-1k’s of files, and enables the agent to then do retrieval (e.g. grep, semantic) to find relevant subsets of context. 2️⃣ A “just-in-time” VLM-based pass. Once the agent finds the relevant pages of context, it will screenshot the documents can call its own VLM (or write code) to dissect the pages. The issue with only using VLM-based OCR tools over massive ad-hoc customer file dumps is that it’s slow and expensive. Doing JIT VLM OCR allows the agent to filter through the data cheaply, but still preserve accuracy for the context that’s needed for the task. The agent harnesses do two-pass document processing by default using off the shelf-tools: pdf2text as the first pass, and using itself (Opus 5) as the second pass. See the below video where Cowork runs over a bunch of PDFs to answer a question about a benchmark graph in the Kimi k3 paper. The main issues here with the “out of the box” doc processing these agents offer are: * Opus 5 is not the best VLM for OCR. It is also way too expensive at scale and lacks grounding * The OSS tools like pypdf, pdf2text, may not be versatile enough as the first pass. * The agent will write a lot of throwaway code to rewrite things an OCR tool would’ve provided out of the box, like chart processing, bounding boxes, confidence scores, leading to increased cost and speed. We have all the tools within LlamaIndex 🦙 to help any agent do two-pass document processing with higher accuracy and lower cost. 1️⃣ We have liteparse for the first pass - a free/OSS parser written in Rust that’s faster/more accurate than other OSS parsers, and supports 50+ document types 2️⃣ We have LlamaParse for the second pass - an agentic document engine that uses VLMs+harnesses to achieve SOTA in accuracy and cost across various doc parsing and extraction tasks. It can be called from any agent harness as an MCP or skill. It takes in page numbers as input, so that the agent can choose to run LlamaParse over a subset of the doc instead of the full doc as a “zoom-in” pass. Come check it out! LiteParse: LlamaParse: All the relevant docs, including MCP, are here:

Jerry Liu

23,239 次观看 • 2 个月前

Introducing ExtractBench, the most comprehensive benchmark for information extraction from complex enterprise documents. The latest models are pushing the frontier of coding and knowledge work, but surprisingly they still struggle on complex doc extraction tasks in production. A well-tuned extractor must parse multi-page filings without dropping rows, emit exact spatial citations for auditability, and handle messy scans. Also they must do all of this at a viable per-page cost so that you can scale this to millions of docs in production (you can’t be paying upwards of $1 in tokens per page!) Existing extraction benchmarks fall short: they are not large/diverse enough in document domain (finance, energy, gov, auto), elements (long records, scans, grounding), and schemas. So our applied research team built ExtractBench. We evaluated 14 systems: frontier VLMs, coding agents, and specialized extraction APIs, against 370 enterprise documents: 4,869 pages, 67 document types. Our biggest finding 🧪: Short documents mask critical system flaws. On files past 50 pages, commercial VLMs collapse below 35% recall due to silent list truncation. They hold high precision, but lose output attention and drop most of the table rows. ExtractBench evaluates value accuracy, long-record completeness, spatial grounding, and per-page cost with zero LLM judges. It is 100% deterministic and reproducible. In tandem with ExtractBench, we’re also introducing 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗣𝗹𝘂𝘀, a new Extract tier in LlamaParse that debuts at #1 on the leaderboard: 95.6% value accuracy, at less than a third the cost of the closest peer. Explore the findings, download the dataset, or run the harness: Blog: GitHub: HuggingFace: We will be actively evolving both our extraction benchmark as well as our extraction harness over time. If you check out either ExtractBench or LlamaParse, let us know your feedback!

Jerry Liu

75,787 次观看 • 3 个月前

THIS IS STRAIGHT F*CKING GOLD Anthropic just shared one of the best tricks for Claude Code. The idea: one knob with five settings, and almost nobody ever moves it. Every routine turn runs at the same effort as your hardest decision, and you pay for thinking you never read. You just start Claude Code like this: claude --model opus --effort low here's the split: → low runs the routine turns, renames, imports, fixtures, formatting → /effort xhigh the moment an actual decision shows up → every subagent carries its own effort line in its frontmatter → you stop paying Opus 5.5 to think about which file to open you only climb the ladder at three moments: → before a plan that touches a lot of files → when the same bug survives two fixes → before a release you cannot roll back Low for the routine. High for the fork. Max for the rescue. JEV engineering is the same move one layer down: the forks that need no thinker (which file, which tool, retry or stop) go to JEV in milliseconds with zero thinking tokens, and the big models only see the ones that split the full tree > Opus 5.5 runs the main session at low effort > /effort xhigh the moment a real decision shows up, then back down > search runs on Haiku 5.5 with effort: low in its frontmatter > review runs on Sonnet 5.5 with effort: high in its frontmatter > every skill carries its own effort line, so the level follows the task > JEV routes micro-forks in milliseconds and thinks about none of them paste the tree and this prompt into Claude Code ↓ "Rebuild my Claude Code setup around this tree: 1. Run the main session on Opus 5.5 at low effort. 2. Set effort: low on every read-only subagent and effort: high on the review subagent. 3. Add one rule to CLAUDE.md: stay at low, climb to xhigh only before a multi-file plan, after the same error twice, and before an irreversible change, then drop back. 4. Route fast deterministic forks (tool choice, path selection, retries) through JEV. Show me every change as a diff first. No edits until I say go" ↳ save it, it'll come in handy.

delost

19,355 次观看 • 23 天前