Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Document OCR benchmarks are still an open problem Existing document OCR benchmarks are either too narrowly focused on a specific type (e.g. FinTabNet, ChartQA), or on documents that aren’t reflective of real-world tasks (e.g. OmniDocBench, OlmOCR-bench on over academic papers) ParseBench is a step towards solving this problem. ... It tries to comprehensively cover real-world document distributions within the enterprise. It contains comprehensive evaluations across 5 different dimensions (tables, charts, content faithfulness, formatting, grounding). * It tries to use metrics that optimize for agent semantic understanding rather than structural similarity. We released this yesterday, and there’s a TON of content: 1. Whitepaper 2. HF dataset 3. Github repo 4. Blog 5. Video And today, we’re excited to feature our home page website for ParseBench 💫 come check it out! Take a look at some of our other materials if you’re interested: Blog: Paper:show more

Jerry Liu

75,628 subscribers

21,657 views • 3 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

We’re open sourcing the first document OCR benchmark for the agentic era, ParseBench. Document parsing is the foundation of every AI agent that works with real-world files. ParseBench is a benchmark that measures parsing quality specifically for agent knowledge work: ✅ It optimizes for semantic correctness (instead of exact similarity) ✅ It has the most comprehensive distribution of real-world enterprise documents It contains ~2,000 human-verified enterprise document pages with 167,000+ test rules across five dimensions that matter most: tables, charts, content faithfulness, semantic formatting, and visual grounding. We benchmarked 14 known document parsers on ParseBench, from frontier/OSS VLMs to specialized parsers to LlamaParse. Here are some of our findings: 💡 Increasing compute budget yields diminishing returns - Gemini/gpt-5-mini/haiku gain 3-5 points from minimal to high thinking, at 4x the cost. 💡 Charts are the most polarizing dimension for evaluation. Most specialized parsers score below 6%, while some VLM-based parsers do a bit better. 💡 VLMs are great at visual understanding but terrible at layout extraction. GPT-5-mini/haiku score below 10% on our visual grounding task, all specialized parsers do much better. 💡 No method crushes all 5 dimensions at once, but LlamaParse achieves the highest overall score at 84.9%, and is the leader in 4 out of the 5 dimensions. This is by far the deepest technical work that we’ve published as a company. I would encourage you to start with our blog and explore our links to Hugging Face to GitHub. All the details are in our full 35-page (!!) ArXiv whitepaper. 🌐: Blog: 📄 Paper: 💻 Code: 📊 Dataset: 🎥 YouTube:

We’re open sourcing the first document OCR benchmark for the agentic era, ParseBench. Document parsing is the foundation of every AI agent that works with real-world files. ParseBench is a benchmark that measures parsing quality specifically for agent knowledge work: ✅ It optimizes for semantic correctness (instead of exact similarity) ✅ It has the most comprehensive distribution of real-world enterprise documents It contains ~2,000 human-verified enterprise document pages with 167,000+ test rules across five dimensions that matter most: tables, charts, content faithfulness, semantic formatting, and visual grounding. We benchmarked 14 known document parsers on ParseBench, from frontier/OSS VLMs to specialized parsers to LlamaParse. Here are some of our findings: 💡 Increasing compute budget yields diminishing returns - Gemini/gpt-5-mini/haiku gain 3-5 points from minimal to high thinking, at 4x the cost. 💡 Charts are the most polarizing dimension for evaluation. Most specialized parsers score below 6%, while some VLM-based parsers do a bit better. 💡 VLMs are great at visual understanding but terrible at layout extraction. GPT-5-mini/haiku score below 10% on our visual grounding task, all specialized parsers do much better. 💡 No method crushes all 5 dimensions at once, but LlamaParse achieves the highest overall score at 84.9%, and is the leader in 4 out of the 5 dimensions. This is by far the deepest technical work that we’ve published as a company. I would encourage you to start with our blog and explore our links to Hugging Face to GitHub. All the details are in our full 35-page (!!) ArXiv whitepaper. 🌐: Blog: 📄 Paper: 💻 Code: 📊 Dataset: 🎥 YouTube:

Jerry Liu

107,866 views • 3 months ago

We’re excited to officially launch LlamaParse, the first genAI-native document parsing solution. Not only is it better at parsing out images/tables/charts 📊📈 than virtually every other parser, it is now steerable through natural language instructions - output the document in whatever format you desire! It is also the only parsing solution that seamlessly allows you to build accurate RAG over complex documents, free of hallucinations 🔥 We launched it in private preview a few weeks ago and hit 2k users, 1M total PDF pages parsed. And now it’s better than ever. LlamaParse contains the following killer features: ✅ SOTA table/chart extraction ✅ Seamless integration with LlamaIndex 🦙 advanced RAG/agents ✅✨ Natural language Parsing Instructions ✅✨JSON mode and image extraction ✅✨Support for ~10 document types (.pdf, .pptx, .docx, .xml) and more Our pricing is simple: 1k free per day, and additional pages at 0.3c a page, or $3 for 1k pages. If you want advanced document RAG and/or private deployments, come get in touch with us to chat about LlamaCloud. Check out our full blog post here: LlamaParse client repo: Signup at 🦙☁️: Come talk to us:

We’re excited to officially launch LlamaParse, the first genAI-native document parsing solution. Not only is it better at parsing out images/tables/charts 📊📈 than virtually every other parser, it is now steerable through natural language instructions - output the document in whatever format you desire! It is also the only parsing solution that seamlessly allows you to build accurate RAG over complex documents, free of hallucinations 🔥 We launched it in private preview a few weeks ago and hit 2k users, 1M total PDF pages parsed. And now it’s better than ever. LlamaParse contains the following killer features: ✅ SOTA table/chart extraction ✅ Seamless integration with LlamaIndex 🦙 advanced RAG/agents ✅✨ Natural language Parsing Instructions ✅✨JSON mode and image extraction ✅✨Support for ~10 document types (.pdf, .pptx, .docx, .xml) and more Our pricing is simple: 1k free per day, and additional pages at 0.3c a page, or $3 for 1k pages. If you want advanced document RAG and/or private deployments, come get in touch with us to chat about LlamaCloud. Check out our full blog post here: LlamaParse client repo: Signup at 🦙☁️: Come talk to us:

LlamaIndex 🦙

143,136 views • 2 years ago

There’s been two papers released in the past couple months, one by Google and one by NVIDIA, that argue that ordering the documents retrieved by RAG systems can enhance performance. However, they both give two different strategies on HOW these documents should be ordered 🤔 Both papers agree on two main points: 1️⃣ There’s a fundamental issue in RAG - as more documents are retrieved, more irrelevant context (e.g., hard negatives) are introduced, which leads to confusion for the LLM and eventually degrades the quality of the generated output. This is called an inverted-U performance curve. 2️⃣ Ordering the retrieved documents is a key lever for optimizing RAG performance. Google Cloud researchers proposed ordering results based on relevance scores: The authors in this paper argue for relevance-based reordering, or ordering the retrieved chunks based on their similarity scores, so the most relevant documents are at the beginning and the end of the inputs to counter the “lost in the middle” effect. NVIDIA researchers proposed ordering results based on the original sequence of document chunks: The authors of this paper argue for Order-Preserving Reordering, or Order-Preserve RAG (OP-RAG), to maintain the logically coherent content flow of the document. So they preserved the original order of retrieved document chunks in the source text, instead of ranking them by relevance scores. So which one is right? It probably depends on the specific use case and dataset - relevance-based reordering could perform better in tasks where you need fast access to the most critical information (e.g., fact retrieval, QA systems), while order-preserving RAG might be better where you need to understand the sequential structure of information (e.g., narrative or legal documents). There are still so many uncertainties in AI - we don’t actually know what we’re doing, and it takes awhile to figure out the best strategies for most things! Excited to see more research about this.

There’s been two papers released in the past couple months, one by Google and one by NVIDIA, that argue that ordering the documents retrieved by RAG systems can enhance performance. However, they both give two different strategies on HOW these documents should be ordered 🤔 Both papers agree on two main points: 1️⃣ There’s a fundamental issue in RAG - as more documents are retrieved, more irrelevant context (e.g., hard negatives) are introduced, which leads to confusion for the LLM and eventually degrades the quality of the generated output. This is called an inverted-U performance curve. 2️⃣ Ordering the retrieved documents is a key lever for optimizing RAG performance. Google Cloud researchers proposed ordering results based on relevance scores: The authors in this paper argue for relevance-based reordering, or ordering the retrieved chunks based on their similarity scores, so the most relevant documents are at the beginning and the end of the inputs to counter the “lost in the middle” effect. NVIDIA researchers proposed ordering results based on the original sequence of document chunks: The authors of this paper argue for Order-Preserving Reordering, or Order-Preserve RAG (OP-RAG), to maintain the logically coherent content flow of the document. So they preserved the original order of retrieved document chunks in the source text, instead of ranking them by relevance scores. So which one is right? It probably depends on the specific use case and dataset - relevance-based reordering could perform better in tasks where you need fast access to the most critical information (e.g., fact retrieval, QA systems), while order-preserving RAG might be better where you need to understand the sequential structure of information (e.g., narrative or legal documents). There are still so many uncertainties in AI - we don’t actually know what we’re doing, and it takes awhile to figure out the best strategies for most things! Excited to see more research about this.

Victoria Slocum

15,333 views • 1 year ago

Over 1 billion PDFs are created every day, but your agents still can’t read them reliably. Today we’re releasing Parse 2.0, the most accurate document parsing API in the world. Extend already processes millions of pages daily for leading AI teams like Brex, Mercury, Opendoor, Flatiron Health, and hundreds of others. Now, its even better. Parse 2.0 is SOTA quality on RealDoc-Bench, our open source benchmark that measures agent success rate on real world docs that agents actually encounter in production. We trained Parse 2.0 on 1M+ pages of the hardest documents seen in production. Here’s how it stacks up: - #1 in healthcare, real estate, logistics, and financial services - 95.7% agent Q&A accuracy on 581 docs (next best: 92%) - 0.847 F1 on layout (next best: 0.759) Give it a try today and build production-ready document agents with Extend.

Over 1 billion PDFs are created every day, but your agents still can’t read them reliably. Today we’re releasing Parse 2.0, the most accurate document parsing API in the world. Extend already processes millions of pages daily for leading AI teams like Brex, Mercury, Opendoor, Flatiron Health, and hundreds of others. Now, its even better. Parse 2.0 is SOTA quality on RealDoc-Bench, our open source benchmark that measures agent success rate on real world docs that agents actually encounter in production. We trained Parse 2.0 on 1M+ pages of the hardest documents seen in production. Here’s how it stacks up: - #1 in healthcare, real estate, logistics, and financial services - 95.7% agent Q&A accuracy on 581 docs (next best: 92%) - 0.847 F1 on layout (next best: 0.759) Give it a try today and build production-ready document agents with Extend.

Kushal Byatnal

585,143 views • 1 month ago

Today, Box is announcing major new AI agent capabilities to let customers tap into the full value of their unstructured data. First, we’re announcing all new updates to the Box AI Studio to make it even easier to build AI agents that tap into your enterprise content for any job function, business process, or industry specific use case. We are also expanding our set of foundational agents that customers will be able to use to work with their enterprise content, including new features like search and research on unstructured data. Next, we’re announcing Box Extract to enable customers to use AI agents seamlessly for complex data extraction from any type of document or content. This makes it easier than ever to pull out data from contracts, invoices, research data, marketing assets, medical charts, and more. Finally, we’re introducing Box Automate, a new workflow automation solution within Box that lets you deploy AI agents across enterprise content-centric workflows. With Box Automate, you can design your business process in a simple drag and drop builder and then drop in AI agents at any step in the process. This ensures agents execute tasks at the right steps in a workflow every time. Best of all, our AI agents and workflow tools are designed to work across any system our customers work within, whether it’s leveraging pre-built integrations, Box APIs, or the new Box MCP Server. Ultimately, all of these capabilities come together to transform how companies can work with their enterprise content. Software has historically only been good at automating work that deals with structured data, which is why ERP, CRM, and HR systems have been mainstays of enterprise software for so long. The data in these systems fits neatly into a database, and the workflows are very ripe for automation. But it turns out most of the work in the world deals with unstructured data. It’s ideating through research documents, working with a client on contracts, reviewing details for a new product launch, looking at a patient’s healthcare record to make a diagnosis, working through due diligence documents for an M&A deal, and so on. For the first time ever, we can begin to bring all new insights and automation to this work with AI agents. At Box, we’re incredibly excited to be on this journey to help customers transform how they work with their most important data.

Today, Box is announcing major new AI agent capabilities to let customers tap into the full value of their unstructured data. First, we’re announcing all new updates to the Box AI Studio to make it even easier to build AI agents that tap into your enterprise content for any job function, business process, or industry specific use case. We are also expanding our set of foundational agents that customers will be able to use to work with their enterprise content, including new features like search and research on unstructured data. Next, we’re announcing Box Extract to enable customers to use AI agents seamlessly for complex data extraction from any type of document or content. This makes it easier than ever to pull out data from contracts, invoices, research data, marketing assets, medical charts, and more. Finally, we’re introducing Box Automate, a new workflow automation solution within Box that lets you deploy AI agents across enterprise content-centric workflows. With Box Automate, you can design your business process in a simple drag and drop builder and then drop in AI agents at any step in the process. This ensures agents execute tasks at the right steps in a workflow every time. Best of all, our AI agents and workflow tools are designed to work across any system our customers work within, whether it’s leveraging pre-built integrations, Box APIs, or the new Box MCP Server. Ultimately, all of these capabilities come together to transform how companies can work with their enterprise content. Software has historically only been good at automating work that deals with structured data, which is why ERP, CRM, and HR systems have been mainstays of enterprise software for so long. The data in these systems fits neatly into a database, and the workflows are very ripe for automation. But it turns out most of the work in the world deals with unstructured data. It’s ideating through research documents, working with a client on contracts, reviewing details for a new product launch, looking at a patient’s healthcare record to make a diagnosis, working through due diligence documents for an M&A deal, and so on. For the first time ever, we can begin to bring all new insights and automation to this work with AI agents. At Box, we’re incredibly excited to be on this journey to help customers transform how they work with their most important data.

Aaron Levie

91,863 views • 10 months ago

How to extract data from papers for literature review in seconds? 1. Go to 2. Upload 2-3 papers you already know are relevant 3. Start with clicking on one paper 4. ResearchCollab do the following for the paper ✦ Provide a short overview of the paper ✦ Identify weaknesses in the paper ✦ Share papers that contrast the existing paper ✦ Extract meta data about the paper ✦ Evaluate each part (e.g., methodology) of the paper ✦ Provide AI chat to extract any other data 5. Now click on the Related Papers button at top right 6. You will get papers related to the seed paper 7. You can click on each paper and read its abstract 8. If you find it relevant, just add it to your list of papers 9. Do the same for the other 2 seed papers 10. This way you will collect all relevant papers 11. And extract data from those papers After this, analyse the data and report the findings. Try ResearchCollab today:

How to extract data from papers for literature review in seconds? 1. Go to 2. Upload 2-3 papers you already know are relevant 3. Start with clicking on one paper 4. ResearchCollab do the following for the paper ✦ Provide a short overview of the paper ✦ Identify weaknesses in the paper ✦ Share papers that contrast the existing paper ✦ Extract meta data about the paper ✦ Evaluate each part (e.g., methodology) of the paper ✦ Provide AI chat to extract any other data 5. Now click on the Related Papers button at top right 6. You will get papers related to the seed paper 7. You can click on each paper and read its abstract 8. If you find it relevant, just add it to your list of papers 9. Do the same for the other 2 seed papers 10. This way you will collect all relevant papers 11. And extract data from those papers After this, analyse the data and report the findings. Try ResearchCollab today:

Faheem Ullah

26,494 views • 5 months ago

🚀New Amazon Q Developer agent for software development is available to customers: This agent is based on a new agent architecture that has exciting results coming from the SWE-bench scores (on the full and verified benchmarks) representing AI models’ ability to resolve real-world coding problems. Interesting aspect of Q Agent is that with these newest updates, Q drove nearly 50% more successful coding tasks completed. What makes Q Dev Agent remarkable? The agent architecture is not just about using the best LLMs (which we do), but also giving the agent the ability to constantly explore multiple paths to find the best way to resolve a particular problem (and back tracking when it has reached dead end like a developer would do). Needless to say, we are just getting started on the developer agent and we are constantly pushing to advance our AI capabilities while maintaining quality, security, privacy, and reliability to keep Amazon Q Developer an innovative and trusted option available to our customers using agents for software development. We highlighted the results of our first SWE-bench submission of Amazon Q Developer back in June blog post; with these updates, our new agent resolves 51% more coding tasks than its previous iteration on the SWE-bench verified dataset, and 43% more on the full dataset. That’s the difference a few months make, and I can’t wait to share what our teams will deliver at re:Invent this December. Here's a quick demo showcasing our new Agent in action:

🚀New Amazon Q Developer agent for software development is available to customers: This agent is based on a new agent architecture that has exciting results coming from the SWE-bench scores (on the full and verified benchmarks) representing AI models’ ability to resolve real-world coding problems. Interesting aspect of Q Agent is that with these newest updates, Q drove nearly 50% more successful coding tasks completed. What makes Q Dev Agent remarkable? The agent architecture is not just about using the best LLMs (which we do), but also giving the agent the ability to constantly explore multiple paths to find the best way to resolve a particular problem (and back tracking when it has reached dead end like a developer would do). Needless to say, we are just getting started on the developer agent and we are constantly pushing to advance our AI capabilities while maintaining quality, security, privacy, and reliability to keep Amazon Q Developer an innovative and trusted option available to our customers using agents for software development. We highlighted the results of our first SWE-bench submission of Amazon Q Developer back in June blog post; with these updates, our new agent resolves 51% more coding tasks than its previous iteration on the SWE-bench verified dataset, and 43% more on the full dataset. That’s the difference a few months make, and I can’t wait to share what our teams will deliver at re:Invent this December. Here's a quick demo showcasing our new Agent in action:

Swami Sivasubramanian

28,946 views • 1 year ago

Researchers built a new RAG approach that: - does not need a vector DB. - does not embed data. - involves no chunking. - performs no similarity search. And it hit 98.7% accuracy on a financial benchmark (SOTA). Here's the core problem with RAG that this new approach solves: Traditional RAG chunks documents, embeds them into vectors, and retrieves based on semantic similarity. But similarity ≠ relevance. When you ask "What were the debt trends in 2023?", a vector search returns chunks that look similar. But the actual answer might be buried in some Appendix, referenced on some page, in a section that shares zero semantic overlap with your query. Traditional RAG would likely never find it. PageIndex (open-source) solves this. Instead of chunking and embedding, PageIndex builds a hierarchical tree structure from your documents, like an intelligent table of contents. Then it uses reasoning to traverse that tree. For instance, the model doesn't ask: "What text looks similar to this query?" Instead, it asks: "Based on this document's structure, where would a human expert look for this answer?" That's a fundamentally different approach with: - No arbitrary chunking that breaks context. - No vector DB infrastructure to maintain. - Traceable retrieval to see exactly why it chose a specific section. - The ability to see in-document references ("see Table 5.3") the way a human would. But here's the deeper issue that it solves. Vector search treats every query as independent. But documents have structure and logic, like sections that reference other sections and context that builds across pages. PageIndex respects that structure instead of flattening it into embeddings. Do note that this approach may not make sense in every use case since traditional vector search is still fast, simple, and works well for many applications. But for professional documents that require domain expertise and multi-step reasoning, this tree-based, reasoning-first approach shines. For instance, PageIndex achieved 98.7% accuracy on FinanceBench, significantly outperforming traditional vector-based RAG systems on complex financial document analysis. Everything is fully open-source, so you can see the full implementation in GitHub and try it yourself. I have shared the GitHub repo in the replies!

Researchers built a new RAG approach that: - does not need a vector DB. - does not embed data. - involves no chunking. - performs no similarity search. And it hit 98.7% accuracy on a financial benchmark (SOTA). Here's the core problem with RAG that this new approach solves: Traditional RAG chunks documents, embeds them into vectors, and retrieves based on semantic similarity. But similarity ≠ relevance. When you ask "What were the debt trends in 2023?", a vector search returns chunks that look similar. But the actual answer might be buried in some Appendix, referenced on some page, in a section that shares zero semantic overlap with your query. Traditional RAG would likely never find it. PageIndex (open-source) solves this. Instead of chunking and embedding, PageIndex builds a hierarchical tree structure from your documents, like an intelligent table of contents. Then it uses reasoning to traverse that tree. For instance, the model doesn't ask: "What text looks similar to this query?" Instead, it asks: "Based on this document's structure, where would a human expert look for this answer?" That's a fundamentally different approach with: - No arbitrary chunking that breaks context. - No vector DB infrastructure to maintain. - Traceable retrieval to see exactly why it chose a specific section. - The ability to see in-document references ("see Table 5.3") the way a human would. But here's the deeper issue that it solves. Vector search treats every query as independent. But documents have structure and logic, like sections that reference other sections and context that builds across pages. PageIndex respects that structure instead of flattening it into embeddings. Do note that this approach may not make sense in every use case since traditional vector search is still fast, simple, and works well for many applications. But for professional documents that require domain expertise and multi-step reasoning, this tree-based, reasoning-first approach shines. For instance, PageIndex achieved 98.7% accuracy on FinanceBench, significantly outperforming traditional vector-based RAG systems on complex financial document analysis. Everything is fully open-source, so you can see the full implementation in GitHub and try it yourself. I have shared the GitHub repo in the replies!

Avi Chawla

972,347 views • 5 months ago

Today, we’re pushing a major update to Edison Analysis, our data analysis agent, which is tuned for scientific research and SOTA across data analysis benchmarks. In contrast to Kosmos, which runs for 6-12 hours and produces tens of thousands of lines of code, Edison Analysis runs for seconds to minutes and is best for specific, well-defined computational tasks. It is available both on our platform under the Analysis tab, and via API, and costs only one credit per run, so it is available to users on both free and paid tiers. Edison Analysis is a modified version of the data analysis agent Kosmos uses in its trajectories. Try it out! One of the most important improvements over our previous data analysis agents has been the addition of a specialized data retrieval tool. Edison Analysis can either use this tool to access data, or can pull data down directly via API. To evaluate this tool, we ranked the most commonly used public data repositories across recent papers from BioRxiv, and created a new benchmark that measures the ability of a language agent system to retrieve raw data from those sources. Edison Analysis gets 71% on this benchmark, and we’ll be working to increase this over time. You can read more about our benchmarks in the our blog post, link below. Some features worth highlighting: 1. Edison Analysis produces a report on the analysis it runs, along with a Jupyter notebook that you can download to reproduce the analysis yourself. Every figure it produces is linked back to the specific lines of code used to produce the figure, to make it easy to reproduce. 2. It works well with both Python and R. 3. One of the best uses for Edison Analysis is to use it to retrieve datasets that you can then analyze with Kosmos. We have a bunch of major improvements to Edison Analysis coming in the next few months that we’re excited to share. In the meantime, congratulations to the team, especially Ludovico Mitchener, Jon Laurent, Conor Igoe , Alex Andonian, and many more.

Today, we’re pushing a major update to Edison Analysis, our data analysis agent, which is tuned for scientific research and SOTA across data analysis benchmarks. In contrast to Kosmos, which runs for 6-12 hours and produces tens of thousands of lines of code, Edison Analysis runs for seconds to minutes and is best for specific, well-defined computational tasks. It is available both on our platform under the Analysis tab, and via API, and costs only one credit per run, so it is available to users on both free and paid tiers. Edison Analysis is a modified version of the data analysis agent Kosmos uses in its trajectories. Try it out! One of the most important improvements over our previous data analysis agents has been the addition of a specialized data retrieval tool. Edison Analysis can either use this tool to access data, or can pull data down directly via API. To evaluate this tool, we ranked the most commonly used public data repositories across recent papers from BioRxiv, and created a new benchmark that measures the ability of a language agent system to retrieve raw data from those sources. Edison Analysis gets 71% on this benchmark, and we’ll be working to increase this over time. You can read more about our benchmarks in the our blog post, link below. Some features worth highlighting: 1. Edison Analysis produces a report on the analysis it runs, along with a Jupyter notebook that you can download to reproduce the analysis yourself. Every figure it produces is linked back to the specific lines of code used to produce the figure, to make it easy to reproduce. 2. It works well with both Python and R. 3. One of the best uses for Edison Analysis is to use it to retrieve datasets that you can then analyze with Kosmos. We have a bunch of major improvements to Edison Analysis coming in the next few months that we’re excited to share. In the meantime, congratulations to the team, especially Ludovico Mitchener, Jon Laurent, Conor Igoe , Alex Andonian, and many more.

Sam Rodriques

61,860 views • 8 months ago

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Santiago

72,306 views • 2 years ago

Introducing LiteParse - the best model-free document parsing tool for AI agents 💫 ✅ It’s completely open-source and free. ✅ No GPU required, will process ~500 pages in 2 seconds on commodity hardware ✅ More accurate than PyPDF, PyMuPDF, Markdown. Also way more readable - see below for how we parse tables!! ✅ Supports 50+ file formats, from PDFs to Office docs to images ✅ Is designed to plug and play with Claude Code, OpenClaw, and any other AI agent with a one-line skills install. Supports native screenshotting capabilities. We spent years building up LlamaParse by orchestrating state-of-the-art VLMs over the most complex documents. Along the way we realized that you could get quite far on most docs through fast and cheap text parsing. Take a look at the video below. For really complex tables within PDFs, we output them in a spatial grid that’s both AI and human-interpretable. Any other free/light parser light PyPDF will destroy the representation of this table and output a sequential list. This is not a replacement for a VLM-based OCR tool (it requires 0 GPUs and doesn’t use models), but it is shocking how good it is to parse most documents. Huge shoutout to Logan Markewich and Clelia Bertelli (🦙/acc) for all the work here. Come check it out: Repo:

Introducing LiteParse - the best model-free document parsing tool for AI agents 💫 ✅ It’s completely open-source and free. ✅ No GPU required, will process ~500 pages in 2 seconds on commodity hardware ✅ More accurate than PyPDF, PyMuPDF, Markdown. Also way more readable - see below for how we parse tables!! ✅ Supports 50+ file formats, from PDFs to Office docs to images ✅ Is designed to plug and play with Claude Code, OpenClaw, and any other AI agent with a one-line skills install. Supports native screenshotting capabilities. We spent years building up LlamaParse by orchestrating state-of-the-art VLMs over the most complex documents. Along the way we realized that you could get quite far on most docs through fast and cheap text parsing. Take a look at the video below. For really complex tables within PDFs, we output them in a spatial grid that’s both AI and human-interpretable. Any other free/light parser light PyPDF will destroy the representation of this table and output a sequential list. This is not a replacement for a VLM-based OCR tool (it requires 0 GPUs and doesn’t use models), but it is shocking how good it is to parse most documents. Huge shoutout to Logan Markewich and Clelia Bertelli (🦙/acc) for all the work here. Come check it out: Repo:

Jerry Liu

256,399 views • 4 months ago

Introducing ALE-Bench, ALE-Agent! Towards Automating Long-Horizon Algorithm Engineering for Hard Optimization Problems Blog: Paper: ALE-Bench is a coding benchmark primarily focused on hard optimization (NP-hard) problems. We developed this benchmark with AtCoder Inc., a leading coding contest platform company. What makes ALE-Bench unique is its focus on hard optimization problems that demand long-horizon and creative reasoning. It’s open-ended, in the sense that true optima are out of reach (NP-hard) and scores can continuously improve. We believe this benchmark has the potential to become one of the key benchmarks for reasoning and coding in the next generation. ALE-Agent is our end-to-end agent that we specifically designed for this challenging domain. In fact, our ALE-Agent has already built an impressive track record in the wild! In May 2025, our agent participated in a live AtCoder Heuristic Competition (AHC), alongside 1,000 other participants in real-time. AHC is considered to be one of the most challenging coding competitions in this domain. Our ALE-Agent achieved an impressive ranking of 21st out of 1,000 human participants in the competition (top 2%), marking a turning point for AI discovery of solutions to hard optimization problems with a wide spectrum of important real world applications such as logistics, routing, packing, factory production planning, power-grid balancing. We look forward to applying this technology to real industrial optimization opportunities. Building on the insights from this study, Sakana AI will continue to tackle the challenge of developing AI with even greater algorithm engineering capabilities. ALE-Bench Dataset: ALE-Bench Code: This research was conducted in collaboration with AtCoder Inc. (AtCoder). We are deeply grateful for their outstanding expertise and contributions in optimization and algorithms, which were invaluable in providing data, analyzing results, and enabling our AI agent’s participation in their contests.

Introducing ALE-Bench, ALE-Agent! Towards Automating Long-Horizon Algorithm Engineering for Hard Optimization Problems Blog: Paper: ALE-Bench is a coding benchmark primarily focused on hard optimization (NP-hard) problems. We developed this benchmark with AtCoder Inc., a leading coding contest platform company. What makes ALE-Bench unique is its focus on hard optimization problems that demand long-horizon and creative reasoning. It’s open-ended, in the sense that true optima are out of reach (NP-hard) and scores can continuously improve. We believe this benchmark has the potential to become one of the key benchmarks for reasoning and coding in the next generation. ALE-Agent is our end-to-end agent that we specifically designed for this challenging domain. In fact, our ALE-Agent has already built an impressive track record in the wild! In May 2025, our agent participated in a live AtCoder Heuristic Competition (AHC), alongside 1,000 other participants in real-time. AHC is considered to be one of the most challenging coding competitions in this domain. Our ALE-Agent achieved an impressive ranking of 21st out of 1,000 human participants in the competition (top 2%), marking a turning point for AI discovery of solutions to hard optimization problems with a wide spectrum of important real world applications such as logistics, routing, packing, factory production planning, power-grid balancing. We look forward to applying this technology to real industrial optimization opportunities. Building on the insights from this study, Sakana AI will continue to tackle the challenge of developing AI with even greater algorithm engineering capabilities. ALE-Bench Dataset: ALE-Bench Code: This research was conducted in collaboration with AtCoder Inc. (AtCoder). We are deeply grateful for their outstanding expertise and contributions in optimization and algorithms, which were invaluable in providing data, analyzing results, and enabling our AI agent’s participation in their contests.

Sakana AI

237,195 views • 1 year ago

“Intelligent Document Processing” 📑🧪 as an industry is gone . With our latest release this week, *anyone* can build and deploy a specialized document agent in seconds ⚡️🤖, and customize the steps via code. Let’s take a tour through our invoice processing and contract matching agent: given an invoice, extract out vendor details and line items, and match it against the corresponding MSA with the vendor. 1️⃣ Put in your name and API key, and deploy the agent in 5 seconds 2️⃣ Upload some sample contracts and invoices, and watch the workflow run. 3️⃣ If you want to customize it, you can clone our source repository, modify the internals, and deploy the agent! It is both more accurate and more customizable than existing IDP solutions. With coding agents today, the ease of use is equivalent too. Click on the “agents” tab in LlamaCloud to check it out! Invoice processing repo: LlamaAgents Docs:

“Intelligent Document Processing” 📑🧪 as an industry is gone . With our latest release this week, anyone can build and deploy a specialized document agent in seconds ⚡️🤖, and customize the steps via code. Let’s take a tour through our invoice processing and contract matching agent: given an invoice, extract out vendor details and line items, and match it against the corresponding MSA with the vendor. 1️⃣ Put in your name and API key, and deploy the agent in 5 seconds 2️⃣ Upload some sample contracts and invoices, and watch the workflow run. 3️⃣ If you want to customize it, you can clone our source repository, modify the internals, and deploy the agent! It is both more accurate and more customizable than existing IDP solutions. With coding agents today, the ease of use is equivalent too. Click on the “agents” tab in LlamaCloud to check it out! Invoice processing repo: LlamaAgents Docs:

Jerry Liu

34,480 views • 7 months ago

PhD Students – How to automatically identify 90% of the issues in your research paper before you submit it to a journal? This is possible through manual or automated paper review. First, let’s understand the following. 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐚 𝐩𝐚𝐩𝐞𝐫 𝐫𝐞𝐯𝐢𝐞𝐰? Paper review is a process in which subject matter experts evaluate your paper based on the following criteria: 1. Significance – Is this research important? 2. Novelty – Is this research new? 3. Methodology – Is this research carried out in the correct way? 4. Verifiability – Can other researchers verify this research? 5. Presentation – Is the research presented in the right way? 𝐖𝐡𝐲 𝐭𝐨 𝐡𝐚𝐯𝐞 𝐲𝐨𝐮𝐫 𝐩𝐚𝐩𝐞𝐫 𝐫𝐞𝐯𝐢𝐞𝐰𝐞𝐝 𝐛𝐞𝐟𝐨𝐫𝐞 𝐬𝐮𝐛𝐦𝐢𝐬𝐬𝐢𝐨𝐧? ➟ Identify the critical issues in your paper ➟ Fix those issues to increase the chances of your paper acceptance 𝐇𝐨𝐰 𝐭𝐨 𝐚𝐮𝐭𝐨𝐦𝐚𝐭𝐞 “𝐬𝐞𝐥𝐟-𝐫𝐞𝐯𝐢𝐞𝐰” 𝐨𝐟 𝐲𝐨𝐮𝐫 𝐩𝐚𝐩𝐞𝐫? Paperpal just launched an amazing feature – AI Review. Using this feature, you can get instant self-feedback. This feature will help you in the following ways. ➝ Check for gaps in your logic ➝ Get feedback on the structure and flow of your writing ➝ Review your research questions ➝ Identify opportunities to strengthen your paper ➝ Increase the chances of your paper acceptance Here is a step-by-step process for using AI Review feature. Step 1: Go to and login. Step 2: Open an existing document or make a new document Step 3: Go to the right-side bar and click on checks | AI Review. Step 4: For this feature to work there should be more than 150 words. Step 5: Copy and paste your paper. Step 6: Now go to the right side and check the prompts Step 7: With these prompts, you will evaluate your paper. Step 8: You will find various prompts e.g., suggest writing feedback, check flow and structure etc. Step 9: You can select a prompt from the existing prompts or write your custom prompt and execute Step 10: Paperpal will generate feedback as per the prompt. Step 11: Read through the feedback and save it for further use. Use other specific prompts for tailored feedback. Step 12: This way you can evaluate various aspects of your paper yourself. This is a very customized and efficient way of automatically reviewing your paper. You can also go one step further to work on the feedback and improve your paper based on suggestions. Please note that AI Review feature does not replace human or expert reviewers in any way. This feature only aims to provide you with quick self-feedback. Try the AI Review feature of Paperpal. Paperpal link:

PhD Students – How to automatically identify 90% of the issues in your research paper before you submit it to a journal? This is possible through manual or automated paper review. First, let’s understand the following. 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐚 𝐩𝐚𝐩𝐞𝐫 𝐫𝐞𝐯𝐢𝐞𝐰? Paper review is a process in which subject matter experts evaluate your paper based on the following criteria: 1. Significance – Is this research important? 2. Novelty – Is this research new? 3. Methodology – Is this research carried out in the correct way? 4. Verifiability – Can other researchers verify this research? 5. Presentation – Is the research presented in the right way? 𝐖𝐡𝐲 𝐭𝐨 𝐡𝐚𝐯𝐞 𝐲𝐨𝐮𝐫 𝐩𝐚𝐩𝐞𝐫 𝐫𝐞𝐯𝐢𝐞𝐰𝐞𝐝 𝐛𝐞𝐟𝐨𝐫𝐞 𝐬𝐮𝐛𝐦𝐢𝐬𝐬𝐢𝐨𝐧? ➟ Identify the critical issues in your paper ➟ Fix those issues to increase the chances of your paper acceptance 𝐇𝐨𝐰 𝐭𝐨 𝐚𝐮𝐭𝐨𝐦𝐚𝐭𝐞 “𝐬𝐞𝐥𝐟-𝐫𝐞𝐯𝐢𝐞𝐰” 𝐨𝐟 𝐲𝐨𝐮𝐫 𝐩𝐚𝐩𝐞𝐫? Paperpal just launched an amazing feature – AI Review. Using this feature, you can get instant self-feedback. This feature will help you in the following ways. ➝ Check for gaps in your logic ➝ Get feedback on the structure and flow of your writing ➝ Review your research questions ➝ Identify opportunities to strengthen your paper ➝ Increase the chances of your paper acceptance Here is a step-by-step process for using AI Review feature. Step 1: Go to and login. Step 2: Open an existing document or make a new document Step 3: Go to the right-side bar and click on checks | AI Review. Step 4: For this feature to work there should be more than 150 words. Step 5: Copy and paste your paper. Step 6: Now go to the right side and check the prompts Step 7: With these prompts, you will evaluate your paper. Step 8: You will find various prompts e.g., suggest writing feedback, check flow and structure etc. Step 9: You can select a prompt from the existing prompts or write your custom prompt and execute Step 10: Paperpal will generate feedback as per the prompt. Step 11: Read through the feedback and save it for further use. Use other specific prompts for tailored feedback. Step 12: This way you can evaluate various aspects of your paper yourself. This is a very customized and efficient way of automatically reviewing your paper. You can also go one step further to work on the feedback and improve your paper based on suggestions. Please note that AI Review feature does not replace human or expert reviewers in any way. This feature only aims to provide you with quick self-feedback. Try the AI Review feature of Paperpal. Paperpal link:

Faheem Ullah

15,270 views • 1 year ago

Today I’m excited to announce Spreadsheet Agents 📊🤖 - a brand-new LlamaIndex feature that allows users to both do data transformation and QA over unnormalized Excel sheets. A lot of knowledge work happens in Excel (or Sheets, Numbers, etc.) - from insurance to tax to corporate finance. It is an extremely nonstandard format that is unsolved by LLMs - neither text-to-CSV nor RAG nor code interpreter work well over these use cases. We built an agent that has really advanced technical capabilities: 1. We do RL-based semantic structure parsing to understand the structure of the Excel sheet 2. We expose these data models through specialized tools to allow an Excel agent to interact with these sheets Huge shoutout to Daniel B. and Zhaoqi Li for working on this 🙌 Currently available in private preview, check out our blog: Come talk to us if you’re interested: Come talk to us if you’re interested:

Jerry Liu

37,901 views • 1 year ago

UPDATE: Ended up with this cool little tool to both build and verify the training data quality; it turned out very accurate! So I am at least happy about that (tho it obvisouly was the easy part) The pipeline to produce this kind of training data (from any Warsh recitation) is in the GitHub below. The repo also documents my full exploration of the Tarteel-for-Warsh problem (and quite frankly, of the offline open source tarteel problem in general): what I tried, what's promising, what's not, and what I think the next steps worth taking are: from a non-engineer with little technical expertiste trying to figure it out. I tried to document everything so the repo might be excessively comprehensive at times. It's basically my entire convo with Claude on the subject, condensed in one repo. Please someone take this and run with it! I genuinely believe the right engineers who can make something real out of this are out there First public repo I've ever shared, so if I made dumb mistakes, please tell me! 🔗

UPDATE: Ended up with this cool little tool to both build and verify the training data quality; it turned out very accurate! So I am at least happy about that (tho it obvisouly was the easy part) The pipeline to produce this kind of training data (from any Warsh recitation) is in the GitHub below. The repo also documents my full exploration of the Tarteel-for-Warsh problem (and quite frankly, of the offline open source tarteel problem in general): what I tried, what's promising, what's not, and what I think the next steps worth taking are: from a non-engineer with little technical expertiste trying to figure it out. I tried to document everything so the repo might be excessively comprehensive at times. It's basically my entire convo with Claude on the subject, condensed in one repo. Please someone take this and run with it! I genuinely believe the right engineers who can make something real out of this are out there First public repo I've ever shared, so if I made dumb mistakes, please tell me! 🔗

Yousr

37,936 views • 4 months ago

The private markets didn’t have a purpose-built AI agent for portfolio analysis, so we decided to build one. Today, I’m excited to announce the launch of Standard Metrics’ AI Analyst. When we first launched a portfolio-company-specific agent last year, it was an experiment: would investors trust AI to answer questions and provide reporting and analytics for an individual portfolio company? The answer turned out to be a resounding yes. The missing piece was portfolio-wide analysis to answer broader queries like “which of my companies have accelerating revenue growth?” or “create a report of our companies in Europe that are getting low on runway.” These types of analyses are now possible in seconds with our new AI Analyst, which allows for multi-company, portfolio-wide questions across quantitative and qualitative data on Standard Metrics. It’s been incredibly rewarding working with a small set of Beta customers to hone the analyst before today’s release. Many use cases have already emerged, including monitoring risks, creating materials for follow-on investment decisions, and summarizing portfolio performance for LPs. The AI Analyst was a labor of love across every department at Standard Metrics. We’re just getting started and continuing to invest aggressively in new capabilities. Check it out in action at the video below. More on the launch can be found at our blog in comments. 👇

The private markets didn’t have a purpose-built AI agent for portfolio analysis, so we decided to build one. Today, I’m excited to announce the launch of Standard Metrics’ AI Analyst. When we first launched a portfolio-company-specific agent last year, it was an experiment: would investors trust AI to answer questions and provide reporting and analytics for an individual portfolio company? The answer turned out to be a resounding yes. The missing piece was portfolio-wide analysis to answer broader queries like “which of my companies have accelerating revenue growth?” or “create a report of our companies in Europe that are getting low on runway.” These types of analyses are now possible in seconds with our new AI Analyst, which allows for multi-company, portfolio-wide questions across quantitative and qualitative data on Standard Metrics. It’s been incredibly rewarding working with a small set of Beta customers to hone the analyst before today’s release. Many use cases have already emerged, including monitoring risks, creating materials for follow-on investment decisions, and summarizing portfolio performance for LPs. The AI Analyst was a labor of love across every department at Standard Metrics. We’re just getting started and continuing to invest aggressively in new capabilities. Check it out in action at the video below. More on the launch can be found at our blog in comments. 👇

John Melas-Kyriazi

66,105 views • 5 months ago

As detailed in the Meta Movie Gen technical report, today we’re open sourcing Movie Gen Bench: two new media generation benchmarks that we hope will help to enable the AI research community to progress work on more capable audio and video generation models. Movie Gen Video Bench is the largest and most comprehensive benchmark ever released for evaluating text-to-video generation. It includes a collection of 1,000+ prompts that cover concepts ranging from detailed human activity to animals, physics, unusual subjects and more — with broad coverage across different motion levels. Movie Gen Audio Bench is a first-of-its-kind benchmark aimed at evaluating video-to-audio and (text+video)-to-audio generation. It includes 527 generated videos and associated sound effects and music prompts covering a diverse set of ambient environments and sound effects. To enable fair and easy comparison to our models for future works, these new benchmarks include non cherry-picked generated videos and audio from Movie Gen. In releasing these new benchmarks we hope to promote fair & extensive evaluations in media generation research to enable greater progress in this field.

As detailed in the Meta Movie Gen technical report, today we’re open sourcing Movie Gen Bench: two new media generation benchmarks that we hope will help to enable the AI research community to progress work on more capable audio and video generation models. Movie Gen Video Bench is the largest and most comprehensive benchmark ever released for evaluating text-to-video generation. It includes a collection of 1,000+ prompts that cover concepts ranging from detailed human activity to animals, physics, unusual subjects and more — with broad coverage across different motion levels. Movie Gen Audio Bench is a first-of-its-kind benchmark aimed at evaluating video-to-audio and (text+video)-to-audio generation. It includes 527 generated videos and associated sound effects and music prompts covering a diverse set of ambient environments and sound effects. To enable fair and easy comparison to our models for future works, these new benchmarks include non cherry-picked generated videos and audio from Movie Gen. In releasing these new benchmarks we hope to promote fair & extensive evaluations in media generation research to enable greater progress in this field.

AI at Meta

156,273 views • 1 year ago

Introducing EdgeBench, a benchmark designed to study how agents learn from environments over at least 12~72-hour runs. We find that performance follows a log-sigmoid function of environment interaction time with high precision. EdgeBench is built with three ingredients: - 🌍 Real & Diverse: 134 real-world tasks across 6 task categories, spanning scientific problems, professional knowledge work, software engineering, optimization, formal math, and games. - ⏳ Ultra-Long-Horizon: Each task supports 12–72 hours of agent work. Recorded human effort averages 57.2 hours. - 🔁 Informative Feedback: Agents receive real-world feedback for continuous improvement. After 38,000 hours of agent runs on EdgeBench, a scaling law for learning from environments emerges: - 📈 As agents interact with task environments over time, their aggregate performance is precisely fit by a log-sigmoid function. - 🧠 This phenomenon can be explained by an elegant theory of graph exploration. We are releasing an initial 51 of the 134 tasks, together with the full evaluation framework, to help advance long-horizon agent research. Check our blog & paper for more findings! Blog Paper GitHub Dataset Details below 👇🧵

Introducing EdgeBench, a benchmark designed to study how agents learn from environments over at least 12~72-hour runs. We find that performance follows a log-sigmoid function of environment interaction time with high precision. EdgeBench is built with three ingredients: - 🌍 Real & Diverse: 134 real-world tasks across 6 task categories, spanning scientific problems, professional knowledge work, software engineering, optimization, formal math, and games. - ⏳ Ultra-Long-Horizon: Each task supports 12–72 hours of agent work. Recorded human effort averages 57.2 hours. - 🔁 Informative Feedback: Agents receive real-world feedback for continuous improvement. After 38,000 hours of agent runs on EdgeBench, a scaling law for learning from environments emerges: - 📈 As agents interact with task environments over time, their aggregate performance is precisely fit by a log-sigmoid function. - 🧠 This phenomenon can be explained by an elegant theory of graph exploration. We are releasing an initial 51 of the 134 tasks, together with the full evaluation framework, to help advance long-horizon agent research. Check our blog & paper for more findings! Blog Paper GitHub Dataset Details below 👇🧵

Deyao Zhu

351,046 views • 16 days ago