Loading video...

Video Failed to Load

Go Home

New course: Document AI: From OCR to Agentic Doc Extraction, built with LandingAI, where I'm executive chairman, and taught by David Park and Andrea Kropp. Much of the world's data is locked in PDFs, JPEGs, and other documents. This short course shows you how to build agentic workflows that...

AndrewYNg's profile picture

Andrew Ng

1,833,071 subscribers

200,844 views • 8 months ago •via X (Twitter)

EducationHealth & WellnessScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

OCR can process characters but it doesn’t understand pixels. OCR has no way to reason about the headers, totals, or checkboxes found in tables, invoices, or forms. In our course with LandingAI, "Document AI: From OCR to Agentic Doc Extraction," we build agents to address these failure modes by breaking documents into pieces, applying the right tools, and mapping information to expected formats. Learn more and enroll today:
1:16

OCR can process characters but it doesn’t understand pixels. OCR has no way to reason about the headers, totals, or checkboxes found in tables, invoices, or forms. In our course with LandingAI, "Document AI: From OCR to Agentic Doc Extraction," we build agents to address these failure modes by breaking documents into pieces, applying the right tools, and mapping information to expected formats. Learn more and enroll today:

DeepLearning.AI

21,554 views • 8 months ago

Agentic Document Extraction now supports field extraction! Many doc extraction use cases extract specific fields from forms and other structured documents. You can now input a picture or PDF of an invoice, request the vendor name, item list, and prices, and get back the extracted fields. Or input a medical form and specify a schema to extract patient name, patient ID, insurance number, etc. One cool feature: If you don't feel like writing a schema (json specification of what fields to extract) yourself, upload one sample document and write a natural language prompt saying what you want, and we automatically generate a schema for you. See the video for details!
3:16

Agentic Document Extraction now supports field extraction! Many doc extraction use cases extract specific fields from forms and other structured documents. You can now input a picture or PDF of an invoice, request the vendor name, item list, and prices, and get back the extracted fields. Or input a medical form and specify a schema to extract patient name, patient ID, insurance number, etc. One cool feature: If you don't feel like writing a schema (json specification of what fields to extract) yourself, upload one sample document and write a natural language prompt saying what you want, and we automatically generate a schema for you. See the video for details!

Andrew Ng

193,139 views • 1 year ago

Last week we brought LlamaParse to Claude. Now it’s available in ChatGPT, too 🦙 ChatGPT can already read PDFs. But document work gets harder when the information you need is buried in messy scans, dense tables, spreadsheets, and charts. Strike the right balance between burning tokens and accuracy, with a specialized API for document understanding. Turn complex documents into clean, structured context and let Astra run its magic. Use our connector to: ✅ Parse documents into clean Markdown, JSON, or HTML. ✅ Extract specific fields into a schema you define. ✅ Search across document collections to find the information you need. ✅ Classify files and split documents into logical sections. Live in the ChatGPT plugin directory now! →
0:31

Last week we brought LlamaParse to Claude. Now it’s available in ChatGPT, too 🦙 ChatGPT can already read PDFs. But document work gets harder when the information you need is buried in messy scans, dense tables, spreadsheets, and charts. Strike the right balance between burning tokens and accuracy, with a specialized API for document understanding. Turn complex documents into clean, structured context and let Astra run its magic. Use our connector to: ✅ Parse documents into clean Markdown, JSON, or HTML. ✅ Extract specific fields into a schema you define. ✅ Search across document collections to find the information you need. ✅ Classify files and split documents into logical sections. Live in the ChatGPT plugin directory now! →

LlamaIndex 🦙

17,601 views • 29 days ago

Data preprocessing is critical for building effective RAG systems. Our new short course, Preprocessing Unstructured Data for LLM Applications, taught by Matt Robinson of Unstructured, demonstrates important but sometimes overlooked aspects of RAG systems: - How to extract and normalize content from diverse formats like PDF, Powerpoint, and HTML to expand your LLM's knowledge - Enriching data with metadata to enable more powerful retrieval and reasoning - Applying document layout analysis and vision transforms to process embedded images and tables Then you’ll use all these skills and build a RAG bot that draws from a corpus that includes PDF, PowerPoint, and Markdown documents. Please sign up here:
3:07

Data preprocessing is critical for building effective RAG systems. Our new short course, Preprocessing Unstructured Data for LLM Applications, taught by Matt Robinson of Unstructured, demonstrates important but sometimes overlooked aspects of RAG systems: - How to extract and normalize content from diverse formats like PDF, Powerpoint, and HTML to expand your LLM's knowledge - Enriching data with metadata to enable more powerful retrieval and reasoning - Applying document layout analysis and vision transforms to process embedded images and tables Then you’ll use all these skills and build a RAG bot that draws from a corpus that includes PDF, PowerPoint, and Markdown documents. Please sign up here:

Andrew Ng

150,391 views • 2 years ago

LangChain: Chat with Your Data, a new free short course created with Harrison Chase, is now available! In this 1 hour course, you’ll learn how to build one of the most requested LLM-based applications: Answering questions using information from a document or collection of documents (often called Retrieval Augmented Generation). You'll also learn how to use vector stores and embeddings to retrieve document chunks relevant to a query. I hope you enjoy the course!
3:17

LangChain: Chat with Your Data, a new free short course created with Harrison Chase, is now available! In this 1 hour course, you’ll learn how to build one of the most requested LLM-based applications: Answering questions using information from a document or collection of documents (often called Retrieval Augmented Generation). You'll also learn how to use vector stores and embeddings to retrieve document chunks relevant to a query. I hope you enjoy the course!

Andrew Ng

384,524 views • 3 years ago

LlamaParse is officially a verified connector on Claude! 🦙 Claude is perfectly fine for the occasional text-only PDF. But most of the context your agent needs lives in messy PDFs, spreadsheets, scanned forms, dense tables, and charts. Feeding those docs straight to the model burns through tokens and can lead to hallucinations when layouts get scrambled, chart values are missed, or numbers are pulled from the wrong column. LlamaParse converts those documents into clean, structured context Claude can work with. With the connector you can ✅️ Parse documents into clean Markdown, JSON, or HTML ✅️ Extract specific fields directly into your schema ✅️ Search across document collections with filesystem-style tools ✅️ Classify documents or split them into predefined sections Live in the Claude Connectors Directory now !
0:44

LlamaParse is officially a verified connector on Claude! 🦙 Claude is perfectly fine for the occasional text-only PDF. But most of the context your agent needs lives in messy PDFs, spreadsheets, scanned forms, dense tables, and charts. Feeding those docs straight to the model burns through tokens and can lead to hallucinations when layouts get scrambled, chart values are missed, or numbers are pulled from the wrong column. LlamaParse converts those documents into clean, structured context Claude can work with. With the connector you can ✅️ Parse documents into clean Markdown, JSON, or HTML ✅️ Extract specific fields directly into your schema ✅️ Search across document collections with filesystem-style tools ✅️ Classify documents or split them into predefined sections Live in the Claude Connectors Directory now !

LlamaIndex 🦙

13,467 views • 1 month ago

Tutorial: Automating KYC with AI agents 🪪🕵 I’m creating a new tutorial series of automating practical document workflows with agents. Every financial institution needs to perform KYC (know your customer) to verify a customer’s identity, and this involves manually sifting through IDs, bank statements, etc. and doing the cross-checking by hand. This is a great first use case for agentic document workflows: 1. Extract identification information from the user supplied ID (license, passport) 2. Extract fields from utility bills/bank statements and then use LLMs to cross-validate extracted fields with the extracted ID fields It obviously doesn’t cover the full e2e process and uses publicly available online data, but should be a good reference guide to get started. To make this work well, you do need high-quality document extraction with confidence scores and citations! Check out the tutorial: If you’re interested come check out LlamaParse:
1:18

Tutorial: Automating KYC with AI agents 🪪🕵 I’m creating a new tutorial series of automating practical document workflows with agents. Every financial institution needs to perform KYC (know your customer) to verify a customer’s identity, and this involves manually sifting through IDs, bank statements, etc. and doing the cross-checking by hand. This is a great first use case for agentic document workflows: 1. Extract identification information from the user supplied ID (license, passport) 2. Extract fields from utility bills/bank statements and then use LLMs to cross-validate extracted fields with the extracted ID fields It obviously doesn’t cover the full e2e process and uses publicly available online data, but should be a good reference guide to get started. To make this work well, you do need high-quality document extraction with confidence scores and citations! Check out the tutorial: If you’re interested come check out LlamaParse:

Jerry Liu

30,062 views • 6 months ago

Build better RAG by letting a team of agents extract and connect your reference materials into a knowledge graph. Our new short course, “Agentic Knowledge Graph Construction,” taught by Neo4j Innovation Lead Andreas Kollegger, shows you how. Knowledge graphs are an important way to store information accurately but they are a lot of work to build manually. In this course you’ll learn how to build a team of agents that turn data– in this case product reviews and invoices from suppliers–into structured graphs of entities and relationships for RAG. Learn how agents can automatically handle the time-consuming work of building graphs — extracting entities and relationships (e.g., Product "contains" Assembly, Part "supplied_by" Supplier, Customer review "mentions" Product), deduplicating them, fact-checking them, and committing them to a graph database — so your retrieval system can find right information to generate accurate output. For example, you can use agents to help trace customer complaints directly to specific suppliers, manufacturing processes, and product hierarchies, thus turning fragmented information into queryable business intelligence. Skills you’ll gain: - Build, store, and access knowledge graphs using the Neo4j graph database - Build multi-agent systems using Google’s Agent Development Kit (ADK) - Set up a loop of agentic workflows to propose and refine a graph schema through fact-checking - Connect agent-generated graphs of unstructured and structured data into a unified knowledge graph This course gets into the practicum of why knowledge graphs give more accurate information retrieval than vector search alone, especially for high-stakes applications where precision matters more than fuzzy similarity matching. Sign up here:
2:26

Build better RAG by letting a team of agents extract and connect your reference materials into a knowledge graph. Our new short course, “Agentic Knowledge Graph Construction,” taught by Neo4j Innovation Lead Andreas Kollegger, shows you how. Knowledge graphs are an important way to store information accurately but they are a lot of work to build manually. In this course you’ll learn how to build a team of agents that turn data– in this case product reviews and invoices from suppliers–into structured graphs of entities and relationships for RAG. Learn how agents can automatically handle the time-consuming work of building graphs — extracting entities and relationships (e.g., Product "contains" Assembly, Part "supplied_by" Supplier, Customer review "mentions" Product), deduplicating them, fact-checking them, and committing them to a graph database — so your retrieval system can find right information to generate accurate output. For example, you can use agents to help trace customer complaints directly to specific suppliers, manufacturing processes, and product hierarchies, thus turning fragmented information into queryable business intelligence. Skills you’ll gain: - Build, store, and access knowledge graphs using the Neo4j graph database - Build multi-agent systems using Google’s Agent Development Kit (ADK) - Set up a loop of agentic workflows to propose and refine a graph schema through fact-checking - Connect agent-generated graphs of unstructured and structured data into a unified knowledge graph This course gets into the practicum of why knowledge graphs give more accurate information retrieval than vector search alone, especially for high-stakes applications where precision matters more than fuzzy similarity matching. Sign up here:

Andrew Ng

168,153 views • 1 year ago

We built an AI agent that lets you vibe-code document extraction - high accuracy and citations over the most complex documents. Our latest release lets you upload documents as context. All you then have to do is describe what you want extracted in natural language. 💡 Our agent will then read the document with file tools to infer the right schema, validation rules, and other pre/postprocessing logic. ✅ It will give you back a workflow that can extract over thousands/millions of documents at scale. You can still of course review and edit every output before approving. Stop handling paperwork manually; just upload files, describe your task, and let our agent handle the rest. Our vision for LlamaAgents is to provide the most advanced and easy-to-use way for you to orchestrate document work. Walkthrough: Check it out: If you’re interested in reducing the operational burden of document extraction (invoices, claims, onboarding forms), come talk to us!
0:58

We built an AI agent that lets you vibe-code document extraction - high accuracy and citations over the most complex documents. Our latest release lets you upload documents as context. All you then have to do is describe what you want extracted in natural language. 💡 Our agent will then read the document with file tools to infer the right schema, validation rules, and other pre/postprocessing logic. ✅ It will give you back a workflow that can extract over thousands/millions of documents at scale. You can still of course review and edit every output before approving. Stop handling paperwork manually; just upload files, describe your task, and let our agent handle the rest. Our vision for LlamaAgents is to provide the most advanced and easy-to-use way for you to orchestrate document work. Walkthrough: Check it out: If you’re interested in reducing the operational burden of document extraction (invoices, claims, onboarding forms), come talk to us!

Jerry Liu

20,857 views • 7 months ago

Announcing a new Coursera course: Retrieval Augmented Generation (RAG) You'll learn to build high performance, production-ready RAG systems in this hands-on, in-depth course created by and taught by , experienced AI and ML engineer, researcher, and educator. RAG is a critical component today of many LLM-based applications in customer support, internal company Q&A systems, even many of the leading chatbots that use web search to answer your questions. This course teaches you in-depth how to make RAG work well. LLMs can produce generic or outdated responses, especially when asked specialized questions not covered in its training data. RAG is the most widely used technique for addressing this. It brings in data from new data sources, such as internal documents or recent news, to give the LLM the relevant context to private, recent, or specialized information. This lets it generate more grounded and accurate responses. In this course, you’ll learn to design and implement every part of a RAG system, from retrievers to vector databases to generation to evals. You’ll learn about the fundamental principles behind RAG and how to optimize it at both the component and whole-system levels. As AI evolves, RAG is evolving too. New models can handle longer context windows, reason more effectively, and can be parts of complex agentic workflows. One exciting growth area is Agentic RAG, in which an AI agent at runtime (rather than it being hardcoded at development time) autonomously decides what data to retrieve, and when/how to go deeper. Even with this evolution, access to high-quality data at runtime is essential, which is why RAG is a key part of so many applications. You'll learn via hands-on experiences to: - Build a RAG system with retrieval and prompt augmentation - Compare retrieval methods like BM25, semantic search, and Reciprocal Rank Fusion - Chunk, index, and retrieve documents using a Weaviate vector database and a news dataset - Develop a chatbot, using open-source LLMs hosted by Together AI, for a fictional store that answers product and FAQ questions - Use evals to drive improving reliability, and incorporate multi-modal data RAG is an important foundational technique. Become good at it through this course! Please sign up here:
1:39

Announcing a new Coursera course: Retrieval Augmented Generation (RAG) You'll learn to build high performance, production-ready RAG systems in this hands-on, in-depth course created by and taught by , experienced AI and ML engineer, researcher, and educator. RAG is a critical component today of many LLM-based applications in customer support, internal company Q&A systems, even many of the leading chatbots that use web search to answer your questions. This course teaches you in-depth how to make RAG work well. LLMs can produce generic or outdated responses, especially when asked specialized questions not covered in its training data. RAG is the most widely used technique for addressing this. It brings in data from new data sources, such as internal documents or recent news, to give the LLM the relevant context to private, recent, or specialized information. This lets it generate more grounded and accurate responses. In this course, you’ll learn to design and implement every part of a RAG system, from retrievers to vector databases to generation to evals. You’ll learn about the fundamental principles behind RAG and how to optimize it at both the component and whole-system levels. As AI evolves, RAG is evolving too. New models can handle longer context windows, reason more effectively, and can be parts of complex agentic workflows. One exciting growth area is Agentic RAG, in which an AI agent at runtime (rather than it being hardcoded at development time) autonomously decides what data to retrieve, and when/how to go deeper. Even with this evolution, access to high-quality data at runtime is essential, which is why RAG is a key part of so many applications. You'll learn via hands-on experiences to: - Build a RAG system with retrieval and prompt augmentation - Compare retrieval methods like BM25, semantic search, and Reciprocal Rank Fusion - Chunk, index, and retrieve documents using a Weaviate vector database and a news dataset - Develop a chatbot, using open-source LLMs hosted by Together AI, for a fictional store that answers product and FAQ questions - Use evals to drive improving reliability, and incorporate multi-modal data RAG is an important foundational technique. Become good at it through this course! Please sign up here:

Andrew Ng

124,656 views • 1 year ago

New short course: Build AI Apps with MCP Servers: Working with Box Files, built with Box and taught by Ben Kus , their CTO. Many AI applications require custom code for basic file operations. The Model Context Protocol (MCP) standardizes this by letting you offload file tasks to dedicated servers that provide tools an LLM can use directly. In this course, you'll process documents stored in a Box folder using the Box MCP server. Rather than writing custom integration code to connect to the Box API and download files, you'll design your application to use the tools provided via MCP. Skills you'll gain: - Build an LLM-powered document processing app, using the Box MCP server to access files - Design a multi-agent system using Google's Agent Development Kit (ADK), consisting of specialized agents for file operations - Coordinate the multi-agent workflow through an orchestrator that uses the Agent2Agent (A2A) protocol to connect to the agents You'll start with a local file-processing app, refactor it to work with Box's MCP server, then evolve it into a multi-agent system. Sign up here:
1:48

New short course: Build AI Apps with MCP Servers: Working with Box Files, built with Box and taught by Ben Kus , their CTO. Many AI applications require custom code for basic file operations. The Model Context Protocol (MCP) standardizes this by letting you offload file tasks to dedicated servers that provide tools an LLM can use directly. In this course, you'll process documents stored in a Box folder using the Box MCP server. Rather than writing custom integration code to connect to the Box API and download files, you'll design your application to use the tools provided via MCP. Skills you'll gain: - Build an LLM-powered document processing app, using the Box MCP server to access files - Design a multi-agent system using Google's Agent Development Kit (ADK), consisting of specialized agents for file operations - Coordinate the multi-agent workflow through an orchestrator that uses the Agent2Agent (A2A) protocol to connect to the agents You'll start with a local file-processing app, refactor it to work with Box's MCP server, then evolve it into a multi-agent system. Sign up here:

Andrew Ng

82,067 views • 1 year ago

New Agentic AI short course! AI Agentic Design Patterns with AutoGen, taught by Microsoft Research's Chi Wang and Penn State's Qingyun Wu (Hiring), shows you how to use AutoGen to implement agentic design patterns like multi-agent collaboration, sequential and nested chat, reflection, tool use, and planning. Learn how to build and combine multiple specialized agents – such as researchers, planners, coders, writers, and critics – that interact to execute complex workflows, like generating detailed financial reports, that would otherwise have taken extensive manual effort. This course illustrates key agentic design principles with many fun demonstrations. For example, you'll build a conversational chess game using two player agents, each of which can use a tool to validate moves and update the board state, while engaging in lively banter about the game! I've enjoyed using AutoGen, and think you will too. Sign up to get started here:
5:14

New Agentic AI short course! AI Agentic Design Patterns with AutoGen, taught by Microsoft Research's Chi Wang and Penn State's Qingyun Wu (Hiring), shows you how to use AutoGen to implement agentic design patterns like multi-agent collaboration, sequential and nested chat, reflection, tool use, and planning. Learn how to build and combine multiple specialized agents – such as researchers, planners, coders, writers, and critics – that interact to execute complex workflows, like generating detailed financial reports, that would otherwise have taken extensive manual effort. This course illustrates key agentic design principles with many fun demonstrations. For example, you'll build a conversational chess game using two player agents, each of which can use a tool to validate moves and update the board state, while engaging in lively banter about the game! I've enjoyed using AutoGen, and think you will too. Sign up to get started here:

Andrew Ng

218,895 views • 2 years ago

New short course: Serverless Agentic Workflows with Amazon Bedrock. Learn to build and deploy serverless agents in this course created with Amazon Web Services and taught by Mike G Chambers, a Senior Developer Advocate at AWS specializing in GenAI. (Disclosure: I serve on Amazon's board.) Generative AI applications are becoming more complex, sophisticated, and agentic. Agentic applications have workloads that can be hard to predict in advance -- for example, what tools will it decide to call? -- and a serverless architecture helps you efficiently providing on-demand resources. This course teaches you to build and deploy a serverless agentic application. You’ll learn to create agents with tools, code execution, and guardrails, and build responsible agents for business use cases: - Build a customer service bot for a fictional tea mug business that can answering questions, retrieve information, and process orders. - Connect your customer service agent to a CRM to get customer info and log support tickets in real-time. - Explore how you invoke the agent, and see the trace to review the agent’s thought process and observation loop until it reaches its final output. - Attach a code interpreter to your agent, giving it the ability to perform accurate calculations by writing and running its own Python code. - Implement guardrails to prevent your agent from revealing sensitive information or using inappropriate language. By the end, you will have built a sophisticated AI agent capable of handling real-world customer support scenarios. Please sign up here!
3:02

New short course: Serverless Agentic Workflows with Amazon Bedrock. Learn to build and deploy serverless agents in this course created with Amazon Web Services and taught by Mike G Chambers, a Senior Developer Advocate at AWS specializing in GenAI. (Disclosure: I serve on Amazon's board.) Generative AI applications are becoming more complex, sophisticated, and agentic. Agentic applications have workloads that can be hard to predict in advance -- for example, what tools will it decide to call? -- and a serverless architecture helps you efficiently providing on-demand resources. This course teaches you to build and deploy a serverless agentic application. You’ll learn to create agents with tools, code execution, and guardrails, and build responsible agents for business use cases: - Build a customer service bot for a fictional tea mug business that can answering questions, retrieve information, and process orders. - Connect your customer service agent to a CRM to get customer info and log support tickets in real-time. - Explore how you invoke the agent, and see the trace to review the agent’s thought process and observation loop until it reaches its final output. - Attach a code interpreter to your agent, giving it the ability to perform accurate calculations by writing and running its own Python code. - Implement guardrails to prevent your agent from revealing sensitive information or using inappropriate language. By the end, you will have built a sophisticated AI agent capable of handling real-world customer support scenarios. Please sign up here!

Andrew Ng

81,243 views • 2 years ago

The latest RAG trend for the current agent harnesses (Codex, Cowork) is to do two passes of document processing to solve a knowledge work task over a data room of documents: 1️⃣ A fast and light pass, oftentimes using a free/OSS doc parsing tool. This can be cheaply run across 10-100-1k’s of files, and enables the agent to then do retrieval (e.g. grep, semantic) to find relevant subsets of context. 2️⃣ A “just-in-time” VLM-based pass. Once the agent finds the relevant pages of context, it will screenshot the documents can call its own VLM (or write code) to dissect the pages. The issue with only using VLM-based OCR tools over massive ad-hoc customer file dumps is that it’s slow and expensive. Doing JIT VLM OCR allows the agent to filter through the data cheaply, but still preserve accuracy for the context that’s needed for the task. The agent harnesses do two-pass document processing by default using off the shelf-tools: pdf2text as the first pass, and using itself (Opus 5) as the second pass. See the below video where Cowork runs over a bunch of PDFs to answer a question about a benchmark graph in the Kimi k3 paper. The main issues here with the “out of the box” doc processing these agents offer are: * Opus 5 is not the best VLM for OCR. It is also way too expensive at scale and lacks grounding * The OSS tools like pypdf, pdf2text, may not be versatile enough as the first pass. * The agent will write a lot of throwaway code to rewrite things an OCR tool would’ve provided out of the box, like chart processing, bounding boxes, confidence scores, leading to increased cost and speed. We have all the tools within LlamaIndex 🦙 to help any agent do two-pass document processing with higher accuracy and lower cost. 1️⃣ We have liteparse for the first pass - a free/OSS parser written in Rust that’s faster/more accurate than other OSS parsers, and supports 50+ document types 2️⃣ We have LlamaParse for the second pass - an agentic document engine that uses VLMs+harnesses to achieve SOTA in accuracy and cost across various doc parsing and extraction tasks. It can be called from any agent harness as an MCP or skill. It takes in page numbers as input, so that the agent can choose to run LlamaParse over a subset of the doc instead of the full doc as a “zoom-in” pass. Come check it out! LiteParse: LlamaParse: All the relevant docs, including MCP, are here:
1:24

The latest RAG trend for the current agent harnesses (Codex, Cowork) is to do two passes of document processing to solve a knowledge work task over a data room of documents: 1️⃣ A fast and light pass, oftentimes using a free/OSS doc parsing tool. This can be cheaply run across 10-100-1k’s of files, and enables the agent to then do retrieval (e.g. grep, semantic) to find relevant subsets of context. 2️⃣ A “just-in-time” VLM-based pass. Once the agent finds the relevant pages of context, it will screenshot the documents can call its own VLM (or write code) to dissect the pages. The issue with only using VLM-based OCR tools over massive ad-hoc customer file dumps is that it’s slow and expensive. Doing JIT VLM OCR allows the agent to filter through the data cheaply, but still preserve accuracy for the context that’s needed for the task. The agent harnesses do two-pass document processing by default using off the shelf-tools: pdf2text as the first pass, and using itself (Opus 5) as the second pass. See the below video where Cowork runs over a bunch of PDFs to answer a question about a benchmark graph in the Kimi k3 paper. The main issues here with the “out of the box” doc processing these agents offer are: * Opus 5 is not the best VLM for OCR. It is also way too expensive at scale and lacks grounding * The OSS tools like pypdf, pdf2text, may not be versatile enough as the first pass. * The agent will write a lot of throwaway code to rewrite things an OCR tool would’ve provided out of the box, like chart processing, bounding boxes, confidence scores, leading to increased cost and speed. We have all the tools within LlamaIndex 🦙 to help any agent do two-pass document processing with higher accuracy and lower cost. 1️⃣ We have liteparse for the first pass - a free/OSS parser written in Rust that’s faster/more accurate than other OSS parsers, and supports 50+ document types 2️⃣ We have LlamaParse for the second pass - an agentic document engine that uses VLMs+harnesses to achieve SOTA in accuracy and cost across various doc parsing and extraction tasks. It can be called from any agent harness as an MCP or skill. It takes in page numbers as input, so that the agent can choose to run LlamaParse over a subset of the doc instead of the full doc as a “zoom-in” pass. Come check it out! LiteParse: LlamaParse: All the relevant docs, including MCP, are here:

Jerry Liu

23,119 views • 1 month ago

Important new course: Agent Skills with Anthropic, built with Anthropic and taught by Elie Schoppik! Skills are constructed as folders of instructions that equip agents with on-demand knowledge and workflows. This short course teaches you how to create them following best practices. Because skills follow an open standard format, you can build them once and deploy across any skills-compatible agent, like Claude Code. What you'll learn: - Create custom skills for code generation and review, data analysis, and research - Build complex workflows using Anthropic's pre-built skills (Excel, PowerPoint, skill creation) and custom skills - Combine skills with MCP and subagents to create agentic systems with specialized knowledge - Deploy the same skills across Claude Code, the Claude API, and the Claude Agent SDK Join and learn to equip agents with the specialized knowledge they need for reliable, repeatable workflows.
1:59

Important new course: Agent Skills with Anthropic, built with Anthropic and taught by Elie Schoppik! Skills are constructed as folders of instructions that equip agents with on-demand knowledge and workflows. This short course teaches you how to create them following best practices. Because skills follow an open standard format, you can build them once and deploy across any skills-compatible agent, like Claude Code. What you'll learn: - Create custom skills for code generation and review, data analysis, and research - Build complex workflows using Anthropic's pre-built skills (Excel, PowerPoint, skill creation) and custom skills - Combine skills with MCP and subagents to create agentic systems with specialized knowledge - Deploy the same skills across Claude Code, the Claude API, and the Claude Agent SDK Join and learn to equip agents with the specialized knowledge they need for reliable, repeatable workflows.

Andrew Ng

890,490 views • 8 months ago

I’m excited to kick off the first of our short courses focused on agents, starting with Building Agentic RAG with LlamaIndex, taught by Jerry Liu, CEO of LlamaIndex 🦙. This covers an important shift in RAG (retrieval augmented generation), in which rather than having the developer write explicit routines to retrieve information to feed into the LLM context, we instead build a RAG agent that that has access to tools for retrieving information. This lets the agent decide what information to fetch, and enables it to answer more complex questions using multi-step reasoning. In detail, you'll learn about: - Routing: Where your agent will use decision-making to route requests to multiple tools. - Tool Use: Where you'll create an interface for agents to select what tool (function call) to use as well as generate the right arguments. - Multi-step reasoning with tool use: Where you'll use an LLM to carry out multiple steps of reasoning, while retaining memory throughout the process. You’ll also learn how to step through what your agent is doing to debug and improve it iteratively. It’s an exciting time to build agents. Sign up and get started here!
2:06

I’m excited to kick off the first of our short courses focused on agents, starting with Building Agentic RAG with LlamaIndex, taught by Jerry Liu, CEO of LlamaIndex 🦙. This covers an important shift in RAG (retrieval augmented generation), in which rather than having the developer write explicit routines to retrieve information to feed into the LLM context, we instead build a RAG agent that that has access to tools for retrieving information. This lets the agent decide what information to fetch, and enables it to answer more complex questions using multi-step reasoning. In detail, you'll learn about: - Routing: Where your agent will use decision-making to route requests to multiple tools. - Tool Use: Where you'll create an interface for agents to select what tool (function call) to use as well as generate the right arguments. - Multi-step reasoning with tool use: Where you'll use an LLM to carry out multiple steps of reasoning, while retaining memory throughout the process. You’ll also learn how to step through what your agent is doing to debug and improve it iteratively. It’s an exciting time to build agents. Sign up and get started here!

Andrew Ng

297,303 views • 2 years ago

everyone is sleeping on this new JSON extraction model! - open-weights - matches Gemini Flash 3.5 - while being 3x faster pulling structured data out of documents usually takes a chain of OCR, an LLM, and extra code to repair broken JSON. lift, from Datalab, does all of this in a single step. you give lift a PDF or image along with a JSON schema, and get back a JSON object that matches that schema. the model reads every page of a document in one pass, so values that span multiple pages still come out right. any document you can describe in a schema works, from invoices to research papers. here's what you get with lift: - 90.2% field accuracy on 11,000 fields - 9B model, nearly matches Gemini Flash 3.5 at 3x the speed - built not to hallucinate; returns null if missing - self-hostable with vLLM or Hugging Face GitHub repo: (don't forget to star 🌟)
1:10

everyone is sleeping on this new JSON extraction model! - open-weights - matches Gemini Flash 3.5 - while being 3x faster pulling structured data out of documents usually takes a chain of OCR, an LLM, and extra code to repair broken JSON. lift, from Datalab, does all of this in a single step. you give lift a PDF or image along with a JSON schema, and get back a JSON object that matches that schema. the model reads every page of a document in one pass, so values that span multiple pages still come out right. any document you can describe in a schema works, from invoices to research papers. here's what you get with lift: - 90.2% field accuracy on 11,000 fields - 9B model, nearly matches Gemini Flash 3.5 at 3x the speed - built not to hallucinate; returns null if missing - self-hostable with vLLM or Hugging Face GitHub repo: (don't forget to star 🌟)

Akshay 🚀

57,515 views • 5 days ago

Finally, a RAG solution that works with complex documents! Real-world documents can be messy, filled with text, tables, images, and intricate flow charts. Traditional parsing and chunking methods struggle to handle these. So, what’s the solution? We need smart techniques that can intuitively chunk relevant context and understand what’s inside each chunk, whether it's text, images, or diagrams. In this video, I’ll walk you through a breakthrough technique for extracting structured information from complex documents. It's unlike any other technique you've seen before.✨ It takes any unstructured (text, tables, images, flow-charts) input and parses it into a JSON format that LLMs can easily process. I used EyeLevel.AI's GroundX platform for this – a powerful tool that allows you to build a RAG application in just 3 steps. It also comes with a nice Python SDK and can be easily deployed on-premise (K8s cluster)! Try it yourself:
6:45

Finally, a RAG solution that works with complex documents! Real-world documents can be messy, filled with text, tables, images, and intricate flow charts. Traditional parsing and chunking methods struggle to handle these. So, what’s the solution? We need smart techniques that can intuitively chunk relevant context and understand what’s inside each chunk, whether it's text, images, or diagrams. In this video, I’ll walk you through a breakthrough technique for extracting structured information from complex documents. It's unlike any other technique you've seen before.✨ It takes any unstructured (text, tables, images, flow-charts) input and parses it into a JSON format that LLMs can easily process. I used EyeLevel.AI's GroundX platform for this – a powerful tool that allows you to build a RAG application in just 3 steps. It also comes with a nice Python SDK and can be easily deployed on-premise (K8s cluster)! Try it yourself:

Akshay 🚀

102,660 views • 1 year ago

Traditional data pipelines don't work for RAG applications. There are 3 issues with them: ​ 1. Traditional data engineering solutions are optimized to handle structured data. RAG applications rely primarily on unstructured data. ​ 2. The connector ecosystem to load data from unstructured data sources is very immature. ​ 3. Traditional solutions do not offer any way to transform unstructured data into an optimized vector search index. ​ The goal of a RAG Pipeline is to solve these problems. ​ The number one objective is to create a reliable vector search index using factual knowledge and relevant context. This sounds easy, but it's one of the biggest challenges we face when building RAG applications. ​ At a high level, there are four different stages in the architecture of a RAG pipeline: ​ 1. Ingestion: Here is where the pipeline loads the information from the data source. ​ 2. Extraction: Where the pipeline processes the input data and decides how to retrieve the text contained inside them. ​ 3. Transform: Where the pipeline chunks the data and generates document embeddings. ​ 4. Load: Where the pipeline creates a search index in a vector database and loads the document embeddings. ​ There are different rabbit holes at each one of these stages. Here are three of them: ​ 1. Ingesting data once is simple. The hard part is refreshing the vector database whenever the original data source changes. ​ 2. Extracting the content of a plain text document is simple. The hard part is to extract content from complex documents containing tables, images, or cross-references. ​ 3. A simple continual chunking strategy with an overlap is simple. The hard part is to find the optimal strategy for your specific knowledge base and the way you are planning to query it. ​ In the attached video, I'll show you how you can build an enterprise-grade RAG Pipeline that solves every one of the above problems. ​ I'll use Vectorize. They partnered with me on this post. You can use them to build RAG pipelines optimized for accurate context retrieval. ​ ​ If you have a few documents lying around, set up a free account and give it a try.
3:33

Traditional data pipelines don't work for RAG applications. There are 3 issues with them: ​ 1. Traditional data engineering solutions are optimized to handle structured data. RAG applications rely primarily on unstructured data. ​ 2. The connector ecosystem to load data from unstructured data sources is very immature. ​ 3. Traditional solutions do not offer any way to transform unstructured data into an optimized vector search index. ​ The goal of a RAG Pipeline is to solve these problems. ​ The number one objective is to create a reliable vector search index using factual knowledge and relevant context. This sounds easy, but it's one of the biggest challenges we face when building RAG applications. ​ At a high level, there are four different stages in the architecture of a RAG pipeline: ​ 1. Ingestion: Here is where the pipeline loads the information from the data source. ​ 2. Extraction: Where the pipeline processes the input data and decides how to retrieve the text contained inside them. ​ 3. Transform: Where the pipeline chunks the data and generates document embeddings. ​ 4. Load: Where the pipeline creates a search index in a vector database and loads the document embeddings. ​ There are different rabbit holes at each one of these stages. Here are three of them: ​ 1. Ingesting data once is simple. The hard part is refreshing the vector database whenever the original data source changes. ​ 2. Extracting the content of a plain text document is simple. The hard part is to extract content from complex documents containing tables, images, or cross-references. ​ 3. A simple continual chunking strategy with an overlap is simple. The hard part is to find the optimal strategy for your specific knowledge base and the way you are planning to query it. ​ In the attached video, I'll show you how you can build an enterprise-grade RAG Pipeline that solves every one of the above problems. ​ I'll use Vectorize. They partnered with me on this post. You can use them to build RAG pipelines optimized for accurate context retrieval. ​ ​ If you have a few documents lying around, set up a free account and give it a try.

Santiago

40,643 views • 2 years ago