Loading video...
Video Failed to Load
The latest RAG trend for the current agent harnesses (Codex, Cowork) is to do two passes of document processing to solve a knowledge work task over a data room of documents: 1️⃣ A fast and light pass, oftentimes using a free/OSS doc parsing tool. This can be cheaply run... show more
22,763 views • 20 days ago •via X (Twitter)
36 Comments
Recall loss in pass 1 is the killer: if cheap text parsing mangles a chart or table, the agent never triggers the VLM pass. How do you catch those silent misses?
Genuine question from evaluating LlamaParse for a legal/land doc layer where citations are contractual: is the bbox coordinate system (units/origin/page-basis/rotation) a documented stable contract anywhere? And is the agentic escalation per-page observable in output? Those two were our blockers.
1. the coordinates are in pdf points, you can figure out how to convert to image pixels here: 2. if you toggle cost optimizer, the metadata will contain which pages have been "cost optimized" vs not
@stov3r How do your costs and accuracy compare to reducto? Also, i am planning to offer to customers to take photos of documents (older demographic with phone), would your vlm process the images with high accuracy comparable to reducto?
@stov3r we spend a lot of time benchmarking, you can check the parsebench results here: i like to think we're at the pareto frontier of accuracy and cost w.r.t diff docs but feel free to surface feedback if you test it over your docs!
This shape generalizes past documents. Web extraction landed on the same split: a cheap fetch and HTML parse across everything, then a real browser render only for the pages the cheap pass flagged as JS dependent or ambiguous. Same economics, the expensive path earns its cost by being rare. What decided quality for us was not the second pass, it was the routing rule. Too loose and you pay VLM prices on the whole corpus anyway. Too tight and the cheap pass silently drops the pages that mattered. Logging which pass produced each field is what lets you tune that boundary from real misses instead of guesses.
Love this approach. Agent codebases get way too messy when they have to write throwaway vision logic just to dissect a single graph.
@jerryjliu0 reminds me of how I batch process logs before diving deeper. saves so much time!
Does flattening structure in the cheap pass hurt retrieval enough to occasionally send the VLM to the wrong pages? We ran into the same thing with Kindle highlights: flat exports are cheap to store but useless to navigate.
The two-pass pattern makes sense: use a cheap pass to map the corpus, then spend stronger reasoning only where uncertainty or relevance is high. The real product advantage becomes routing, evidence tracking, and evaluation—not raw context size.
In document workflows, the fast pass becomes the failure boundary: anything it misses is invisible downstream. We’ve found coverage/exclusion receipts and source evidence more useful than confidence alone. Are you considering completeness receipts in this two-pass pattern?
The two-pass pattern is definitely becoming standard. Doing the heavy VLM inference only on the filtered subset saves a ton on token costs and latency compared to blind full-doc OCR.
Great breakdown! Quick question: how do you handle information loss or context degradation between the fast light pass and the deep extraction? Do you pass the raw chunks or a structured summary to the second agent?
I love this approach
Watching agents write 40 lines of Pillow code just to read one graph is the new tax of 2026 knowledge work. Two-pass is the right architecture. The missing piece was always a first-pass parser that doesn’t silently drop charts/tables + a second-pass that actually understands layout. LiteParse + LlamaParse feels like the first serious fix for this.
cheap breadth first, expensive vision only where uncertainty survives feels like the right default
Matches what we see on legal document dumps. The two-pass split matters because accuracy only really decides the outcome on the second pass. Missing a file in the cheap pass is recoverable. Getting a clause boundary or a defined term wrong in the zoom-in pass isn't.
Please add top-up credits to your cloud service, very small initial projects can't afford $50/month
The two-pass structure is the pattern worth stealing generally, cheap and broad first, expensive and targeted only where the first pass says it's needed. Most agent cost blowouts happen when people reach for the expensive tool on everything.
The part that decides accuracy is not either pass, it is the trigger that chooses when to zoom in. Cheap first passes fail quietly on scanned tables and rotated pages, so the agent never learns it needed a second look. Logging the pages the light pass got wrong and tuning the trigger on that bought us more accuracy than swapping parsers ever did.
两遍式思路对，但第一遍省不出recall，漏了文件第二遍再强也白搭。
如果是大型企业 / 极度敏感业务：这套“两次处理（Two-Pass）”的思路 100% 值得借鉴，但核心的第二步（精读）可能会选择在本地自研或私有化部署更小型的视觉开源模型，而不是直接依赖 LlamaParse 的云端服务。
two-pass works because relevance and extraction have opposite cost curves. pass one is cheap and only needs recall, don't miss the right doc. pass two is expensive and needs precision. collapsing them is what made naive rag either miss context or blow the token budget.
Context most people miss here: two-pass fixes fidelity per document, which is the right call and the expensive part. The failure that survives both passes is between documents. In a data room the amended agreement and the original both parse cleanly, both score high, and retrieval returns whichever is more semantically similar to the question. Confidence scores tell you how well you read a page, not that the page is superseded. We've been building that adjudication at Nucleus as a layer above parsing, consuming parser output rather than competing with it, which is why a strong second pass matters more to us than it might look. Does LlamaParse surface anything at the document level that could carry a version or supersession signal, or is that strictly downstream?
The two passes are doing different jobs. The cheap pass can decide where deeper attention belongs. But if its compressed representation becomes the basis for the final answer, information discarded for navigation can quietly become information the system no longer knows it lost. The deeper pass has to reconnect the selected material to enough of the original source, surrounding context, provenance, and constraints to support the claim being made. The first pass routes attention. The second has to restore the evidence basis.
The two-pass approach is smart, but it raises the same question every "cheap-then-precise" pipeline runs into, how do you know when the fast pass is wrong before it ever reaches the accurate pass? Grounding and confidence scores aren't just nice-to-haves here, they're what tells the agent whether it needs the JIT VLM step at all. Without that signal, you're either over-calling the expensive pass or under-catching bad extractions.
cheap recall first -> expensive precision second. Parse broadly enough to find candidate pages, then spend VLM compute only where visual understanding actually matters.
Staged retrieval makes agent RAG cheaper and clearer than giant context dumps.
JIT parsing makes sense
This makes sense, let the cheap parser narrow down 1K documents, then spend the expensive vision on the few pages that actually need it. That should scale much better.
this two pass setup makes way more sense than throwing every page at a vlm and hoping for the best
The two-pass pattern is solid, but the fast pass often becomes the bottleneck when parsing quality varies. Do you find that a cheap parser misses enough structure to force a slower, more expensive second pass?
Same shape as skill routing. First pass to route, second to execute. Forcing the routing pass to emit structured output instead of prose is what made it reliable for me.
For those light passes, I've found something like unstructured is pretty solid. Avoids a lot of the parsing hell. For heavier lifts, then yeah, a more solid approach makes sense.
两遍式本质是把成本压力都推给第二遍。关键在第一遍只要 recall 不要精度——便宜 parser 加粗 chunk 就够，别过度设计。但要小心召回上限是被第一遍定死的：漏掉的文档第二遍永远找不回来，所以它可以便宜，不能敷衍。
Two-pass maps well to provenance: the cheap first pass yields file/page anchors; the JIT pass pays VLM cost only where retrieval pointed. Does LlamaParse's zoom-in output carry region-level confidence, so downstream validation can separate extraction errors from retrieval misses?
