正在加载视频...
视频加载失败
We’re open sourcing the first document OCR benchmark for the agentic era, ParseBench. Document parsing is the foundation of every AI agent that works with real-world files. ParseBench is a benchmark that measures parsing quality specifically for agent knowledge work: ✅ It optimizes for semantic correctness (instead of exact... show more
30 条评论
🔥
finally, a scientific way to quantify my daily trauma with pdfs.
@llama_index What about datalab and Chandra OCR? I had been testing llama parser and datalab, and datalab provided best results for my sample. If you compare with datalab I am immediately in!
document parsing is the unsexy bottleneck that breaks most agent pipelines in production. garbage in, garbage out — doesn't matter how smart your reasoning layer is. glad someone is finally benchmarking this properly for the agentic use case.
Benchmarks like this matter because parsing errors aren’t just OCR noise, they become reasoning errors downstream. Glad to see evaluation framed around agent knowledge work instead of generic document OCR.
document parsing is the silent bottleneck that kills most RAG and agentic pipelines in production. you can have perfect retrieval logic but if tables, headers, and multi-column layouts get mangled on ingest, your agent is reasoning on garbage. a proper benchmark for this is overdue — what document types showed the biggest parsing quality gaps?
almost every agent failure I've debugged this year started with a misparsed PDF. about time.
This is great - love open benchmarks
Impressive optimization of retrieval augmentation in LlamaIndex. Yes, and have you considered the semantic security implication of model inversion attacks on your embedding store?
Love this. Feels like the ecosystem is getting much better at pre-release evals and benchmarks. The next gap is post-release behavior. How these systems actually perform once real users and agents start leaning on them.
document parsing is tricky. benchmarks like this help. hope it pushes the field forward and sparks more innovation.
the open source OCR benchmark is the kind of thing that actually moves agents forward benchmarks that test one narrow capability are noise. benchmarks that test real world document understanding are signal more of this please
Benchmarks that define a category usually outlast the product companies built around them — smart OSS wedge. One gap I'd flag: how does ParseBench score on mixed CJK+English scanned PDFs? That's the hardest parsing case for any agent deployed in Asian enterprise markets.
only 85% accuracy? so on a 100 page doc 15 pages are compeltely wrong?
Document parsing is one of those underrated bottlenecks in agent workflows. Most agents just pass raw PDF text to an LLM and hope for the best. Having a proper benchmark for OCR quality in agentic contexts is badly needed - the downstream impact on retrieval accuracy and reasoning over documents is huge.
Awesome and keep it up!
this feels like one of the most underappreciated bottlenecks in agent pipelines - garbage in at the parsing layer means expensive garbage out. we run into this constantly at @ImplicitCloudAI... really curious how ParseBench scores on messy real-world enterprise docs vs cleaner web PDFs
Cool
So who’s the best
most agent failures I've seen on real-world docs are OCR, not reasoning. good to finally have a benchmark that separates the two.
Interesting, will need to sniff out the biases ie. benchmarkmaxxing but otherwise will test my docling ETL on it
This is excellent work — semantic correctness over raw similarity is exactly what agentic systems need. Question: Once a document is parsed with high semantic fidelity, how do you currently prevent the *downstream agent* from turning that correct parse into an incorrect or out-of-policy action? In other words, do you see governance as something that should sit *after* parsing (in the agent loop) or as a structural layer that intercepts the parsed output before it ever reaches memory or execution? Genuinely curious — this feels like the next critical layer after great parsing.
Cool 👀
Semantic correctness over exact match is the right call for agent work, exact match penalizes paraphrases the downstream agent doesn't even care about.
finally a benchmark that proves you can't just throw more compute at bad layout extraction. llamaparse wins but your workflow is still broken if you're manually checking those 167k rules
Preston is a 🏆 dudes brilliant and yet talks and treats everyone equally. Thanks for being so cool at the @StanfordLaw LLM hack yesterday 🙏
@grok compare them against reducto product.
agentic era needs its own benchmarks. standard NLP metrics don't capture whether an agent can recover from a broken pipeline at 3am. this is the right direction
How does MinerU compare against liteparse?
parsing quality is the read foundation; the write gap is underbuilt. once the agent reads that doc and decides to file/delete/send, most harnesses have no runtime gate between the decision and the action. that's the other half of agent infra:
