Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Building “RAG 2.0” is just making Claude Code running over your filesystem 🤖🗂️ To make this work well, you need to solve three things 1️⃣ Virtualize your filesystem to prevent the agent from messing stuff up. AgentFS by Turso is a nice example of how you can give the... agent access to a copy of all your files without messing up your raw data. 2️⃣ Parse unstructured documents like PDFs, pptx, Word into an LLM-ready format. Agentic OCR solutions like LlamaParse can help here 3️⃣ Creating an agentic loop with human-in-the-loop. If you want to control the agent implementation instead of using Claude Code out of the box, you can use LlamaIndex 🦙 workflows to help orchestrate these long-running agent tasks. Shoutout Clelia Bertelli (🦙/acc), check it out! Blog: Repo:show more

Jerry Liu

73,855 subscribers

55,620 Aufrufe • vor 10 Monaten •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

We built a neat tool that lets you convert a directory of Powerpoint files into clean, structured markdown - that Claude Code / agent SDK / any generalized agent wrapper can easily understand. The pptx skill in Claude Code is quite basic and doesn’t have high-fidelity understanding over graphics/charts/tables. Our project Surreal Slides uses LlamaParse to convert presentations into clean structured data that you can put into a db (SurrealDB) for simple retrieval, without having to take screenshots of the data on the fly. Thanks to Clelia Bertelli (🦙/acc) for this project, check it out:

We built a neat tool that lets you convert a directory of Powerpoint files into clean, structured markdown - that Claude Code / agent SDK / any generalized agent wrapper can easily understand. The pptx skill in Claude Code is quite basic and doesn’t have high-fidelity understanding over graphics/charts/tables. Our project Surreal Slides uses LlamaParse to convert presentations into clean structured data that you can put into a db (SurrealDB) for simple retrieval, without having to take screenshots of the data on the fly. Thanks to Clelia Bertelli (🦙/acc) for this project, check it out:

Jerry Liu

39,872 Aufrufe • vor 7 Monaten

Introducing RAGs, a Streamlit app that allows you to create and customize your own RAG agent and then use it over your own data, all with natural language 🔥 Directly inspired by OpenAI GPTs, you can converse with an agent to help you do search/retrieval over any data you specify. The app contains three main pages: 🏠 Home Page : Have a “builder agent” build your RAG agent through natural language (you specify the data). ⚙️ RAG Config: Look at configured parameters 🤖 Use your RAG agent! Check out details below 👇 Blog: Repo:

Introducing RAGs, a Streamlit app that allows you to create and customize your own RAG agent and then use it over your own data, all with natural language 🔥 Directly inspired by OpenAI GPTs, you can converse with an agent to help you do search/retrieval over any data you specify. The app contains three main pages: 🏠 Home Page : Have a “builder agent” build your RAG agent through natural language (you specify the data). ⚙️ RAG Config: Look at configured parameters 🤖 Use your RAG agent! Check out details below 👇 Blog: Repo:

LlamaIndex 🦙

475,732 Aufrufe • vor 2 Jahren

AGENT ID FOR CLAUDE AGENT issue an agent ID to your claude code agent in 2 min - so it can commit to github as a verified, human-authorized agent not just a random bot this is the first step towards building trust in the agentic internet, powered by Alien

AGENT ID FOR CLAUDE AGENT issue an agent ID to your claude code agent in 2 min - so it can commit to github as a verified, human-authorized agent not just a random bot this is the first step towards building trust in the agentic internet, powered by Alien

kirill avery

16,742 Aufrufe • vor 6 Monaten

Claude R Perrin Code SDK is under-rated One cool part with Claude Code SDK is that it supports full features out of box: - Skills - Sub agents - Hooks - ... Which means you can use claude code CLI as a debug tool to build agent, and then import to your Claude code powered agent Here is a quick example of a pptx presentation agent using Claude Code SDK + doc skills Code example from AI Builder Club by Kai:

Claude R Perrin Code SDK is under-rated One cool part with Claude Code SDK is that it supports full features out of box: - Skills - Sub agents - Hooks - ... Which means you can use claude code CLI as a debug tool to build agent, and then import to your Claude code powered agent Here is a quick example of a pptx presentation agent using Claude Code SDK + doc skills Code example from AI Builder Club by Kai:

Jason Zhou

11,981 Aufrufe • vor 11 Monaten

Give Claude Code a semantic filesystem 🗃️🛠️ Giving Claude Code access to the right CLI tools over your filesystem turns it into a general agent capable of automating far more knowledge work beyond code - it can do dynamic financial/legal/medical/technical/backoffice analysis over any subset of documents. With our latest release of semtools 💫, you can now manually or *agentically* create a persistent workspace over any subset of files. This gives Claude Code the ability to get blazing-fast, local semantic search over any data, while still allowing it to chain with commands like grep/cat/etc. so that it can load in dynamic context instead of naive top-k vector search. The coding agent can dynamically index data and use those indexes, instead of having to rebuild it every time. So you get the benefits of fast search along with agentic reasoning over CLI tools mentioned above. Come check it out!

Give Claude Code a semantic filesystem 🗃️🛠️ Giving Claude Code access to the right CLI tools over your filesystem turns it into a general agent capable of automating far more knowledge work beyond code - it can do dynamic financial/legal/medical/technical/backoffice analysis over any subset of documents. With our latest release of semtools 💫, you can now manually or agentically create a persistent workspace over any subset of files. This gives Claude Code the ability to get blazing-fast, local semantic search over any data, while still allowing it to chain with commands like grep/cat/etc. so that it can load in dynamic context instead of naive top-k vector search. The coding agent can dynamically index data and use those indexes, instead of having to rebuild it every time. So you get the benefits of fast search along with agentic reasoning over CLI tools mentioned above. Come check it out!

Jerry Liu

77,373 Aufrufe • vor 1 Jahr

I vibe-coded an AI agent for making presentations 🤖🖼️ - accessible to everyone and open-source! You can upload both your context files, as well as a style template (a file that captures the style of the presentation you want to create). Then you launch a prompt and watch the agent one-shot your presentation. After the presentation is generated, you can edit the text inline, chat with the agent to add/modify/delete slides, and export to Powerpoint or PDF. Built with Claude Code, powered by Claude Agent SDK + LlamaParse. Hosted on Vercel/Render. To use it, you do need to specify your LlamaCloud API key and Anthropic API key (see below). App: Repo: Sign up to LlamaCloud:

I vibe-coded an AI agent for making presentations 🤖🖼️ - accessible to everyone and open-source! You can upload both your context files, as well as a style template (a file that captures the style of the presentation you want to create). Then you launch a prompt and watch the agent one-shot your presentation. After the presentation is generated, you can edit the text inline, chat with the agent to add/modify/delete slides, and export to Powerpoint or PDF. Built with Claude Code, powered by Claude Agent SDK + LlamaParse. Hosted on Vercel/Render. To use it, you do need to specify your LlamaCloud API key and Anthropic API key (see below). App: Repo: Sign up to LlamaCloud:

Jerry Liu

21,073 Aufrufe • vor 8 Monaten

Here is how simple it is to build a UI front-end for an AI agent (assuming you are using . The video here is 90 seconds and will show you everything you need to connect an agent to a web app: 1. Set up CopilotKit🪁 in your app 1. Copy the agent's localhost URL from LangGraph Studio 2. Paste that URL into a CopilotKit🪁 remote endpoint That's all you need to get a front-end to interact with the agent! The secret sauce here is CoAgents, an open-source library. You can use CoAgents to integrate any agent with your web application. Here is what you get: • Human-in-the-loop to steer and correct the agent • Stream intermediate agent state • Real-time state sharing between the agent and the application • Agentic generative UI to build trust that the agent is on the right path Watch the attached demo so you see it in action.

Here is how simple it is to build a UI front-end for an AI agent (assuming you are using . The video here is 90 seconds and will show you everything you need to connect an agent to a web app: 1. Set up CopilotKit🪁 in your app 1. Copy the agent's localhost URL from LangGraph Studio 2. Paste that URL into a CopilotKit🪁 remote endpoint That's all you need to get a front-end to interact with the agent! The secret sauce here is CoAgents, an open-source library. You can use CoAgents to integrate any agent with your web application. Here is what you get: • Human-in-the-loop to steer and correct the agent • Stream intermediate agent state • Real-time state sharing between the agent and the application • Agentic generative UI to build trust that the agent is on the right path Watch the attached demo so you see it in action.

Santiago

46,142 Aufrufe • vor 1 Jahr

Build a loan underwriting agent workflow in ~5 mins 💵🤖 (No code, just English) I built an agentic workflow that can process incoming PDFs of financial statements: checking statements, bank statements, brokerage accounts, and extract/normalize the financials in each source of data. I did this by specifying the schemas and classification logic in plain English; our agent compiles this into a deterministic workflow with calls to agentic parsing to extract out all the information that you need. Our agent builder is now in LlamaCloud, you can sign up and try it yourself! Blog: Sign up:

Build a loan underwriting agent workflow in ~5 mins 💵🤖 (No code, just English) I built an agentic workflow that can process incoming PDFs of financial statements: checking statements, bank statements, brokerage accounts, and extract/normalize the financials in each source of data. I did this by specifying the schemas and classification logic in plain English; our agent compiles this into a deterministic workflow with calls to agentic parsing to extract out all the information that you need. Our agent builder is now in LlamaCloud, you can sign up and try it yourself! Blog: Sign up:

Jerry Liu

30,015 Aufrufe • vor 8 Monaten

Bash is all you need! Which is why I'm introducing my holiday project: just-bash just-bash is a pretty complete implementation of bash in TypeScript designed to be used as a bash tool by AI agents. Because it turns out agents love exploring data via shell scripts, even beyond coding. It comes with grep, sed, awk and the 99th percentile features that an agent like Claude Code or Cursor would use. In fact, Claude Code can use it for secure bash execution. In the package - A bash-tool for AI SDK - A binary for use by yourself or your coding agents - An overlay filesystem to feed files to your agent securely - A Vercel Sandbox compatible API, so you can quickly upgrade to a real VM if you need to run binaries - An example AI agent that explores the just-bash code base using just-bash - I imported the Oils shell bash compatibility suite and just-bash passes a very good chunk What is interesting about this codebase: It was essentially entirely written by Opus 4.5. Coding agents love bash and they are good at reproducing it. They are also great at text-book recursive descent parsers and AST tweet-walk interpreters. That said, it is, like, a lot of code and I didn't read it all 😅. This is very much a hack, but it also seems to be _really_ useful. I haven't really found anything agents want to use that it doesn't support and it's fast and secure (caveats apply). It doesn't have write access to your computer and the filesystem is given a root that the agent cannot escape from. Find it at Related: Our recent blog post how we migrated our data analysis agent to bash tools and achieved incredible quality improvements The video shows the example agent investigating the just-bash code base

Bash is all you need! Which is why I'm introducing my holiday project: just-bash just-bash is a pretty complete implementation of bash in TypeScript designed to be used as a bash tool by AI agents. Because it turns out agents love exploring data via shell scripts, even beyond coding. It comes with grep, sed, awk and the 99th percentile features that an agent like Claude Code or Cursor would use. In fact, Claude Code can use it for secure bash execution. In the package - A bash-tool for AI SDK - A binary for use by yourself or your coding agents - An overlay filesystem to feed files to your agent securely - A Vercel Sandbox compatible API, so you can quickly upgrade to a real VM if you need to run binaries - An example AI agent that explores the just-bash code base using just-bash - I imported the Oils shell bash compatibility suite and just-bash passes a very good chunk What is interesting about this codebase: It was essentially entirely written by Opus 4.5. Coding agents love bash and they are good at reproducing it. They are also great at text-book recursive descent parsers and AST tweet-walk interpreters. That said, it is, like, a lot of code and I didn't read it all 😅. This is very much a hack, but it also seems to be _really_ useful. I haven't really found anything agents want to use that it doesn't support and it's fast and secure (caveats apply). It doesn't have write access to your computer and the filesystem is given a root that the agent cannot escape from. Find it at Related: Our recent blog post how we migrated our data analysis agent to bash tools and achieved incredible quality improvements The video shows the example agent investigating the just-bash code base

Malte Ubl

125,326 Aufrufe • vor 9 Monaten

this is f**king dangerous. someone just open sourced an entire library of ready-made JEV skills for building insane AI agent workflows. instead of building common agent patterns from scratch, you can grab skills for things like: → jev-route for routing tasks → jev-guardrail for adding checks and verification → jev-fanout for running multiple tasks in parallel → reusable skills you can plug into your own workflows the interesting part is that these solve problems you'll run into once your agents start doing more than simple tasks. you don't have to reinvent routing, verification, or parallel execution every time you build a new workflow. save this now. you'll probably need one of these skills the next time you're building an agent and don't want to start from scratch. repo below:

this is f**king dangerous. someone just open sourced an entire library of ready-made JEV skills for building insane AI agent workflows. instead of building common agent patterns from scratch, you can grab skills for things like: → jev-route for routing tasks → jev-guardrail for adding checks and verification → jev-fanout for running multiple tasks in parallel → reusable skills you can plug into your own workflows the interesting part is that these solve problems you'll run into once your agents start doing more than simple tasks. you don't have to reinvent routing, verification, or parallel execution every time you build a new workflow. save this now. you'll probably need one of these skills the next time you're building an agent and don't want to start from scratch. repo below:

MIKE

131,315 Aufrufe • vor 22 Tagen

Introducing FlowMaker 🌊🤖 A fully open-source, low-code way of building custom agent workflows. Build agents via a drag and drop interface, run it directly in the app, and also directly export it into a deeply custom workflow backed by LlamaIndex 🦙.TS. It’s a fantastic visual tool to help you get started with agents but also gives you full customizability to build the deepest, most advanced workflows, since everything is backed by code. There’s a lot of great low-code agent tools already, but this is a fully-open source template that you can use or clone/customize for your own needs. It was a fun project by Laurie Voss and is intended to give users an easy-on ramp onto our powerful agent orchestration tooling. Also integrates with LlamaCloud in case you want to integrate your own knowledge base into your agent. FlowMaker app: FlowMaker Repo: LlamaCloud:

Introducing FlowMaker 🌊🤖 A fully open-source, low-code way of building custom agent workflows. Build agents via a drag and drop interface, run it directly in the app, and also directly export it into a deeply custom workflow backed by LlamaIndex 🦙.TS. It’s a fantastic visual tool to help you get started with agents but also gives you full customizability to build the deepest, most advanced workflows, since everything is backed by code. There’s a lot of great low-code agent tools already, but this is a fully-open source template that you can use or clone/customize for your own needs. It was a fun project by Laurie Voss and is intended to give users an easy-on ramp onto our powerful agent orchestration tooling. Also integrates with LlamaCloud in case you want to integrate your own knowledge base into your agent. FlowMaker app: FlowMaker Repo: LlamaCloud:

Jerry Liu

20,457 Aufrufe • vor 1 Jahr

I built a form-filling agent that anyone can use 💫 This is an extremely simple but useful (I hope) app. Upload a fillable form 📋, some context files, and chat with the agent to fill the form out automatically ✍️ 1️⃣ Yes it is a Claude Code SDK wrapper 2️⃣ It is better and faster than ChatGPT/Claude UI out of the box 3️⃣ We use LlamaParse to parse the context files, so you can have more trust that we are able to read context without hallucinations (e.g. messy scanned handwriting, drivers license photo, and more). This was one of my holiday Claude Code vibe-coding projects. Built with Opus 4.5, and also powered by Opus 4.5. Feeling the AGI 🫡 App is here: Repo is open-source:

I built a form-filling agent that anyone can use 💫 This is an extremely simple but useful (I hope) app. Upload a fillable form 📋, some context files, and chat with the agent to fill the form out automatically ✍️ 1️⃣ Yes it is a Claude Code SDK wrapper 2️⃣ It is better and faster than ChatGPT/Claude UI out of the box 3️⃣ We use LlamaParse to parse the context files, so you can have more trust that we are able to read context without hallucinations (e.g. messy scanned handwriting, drivers license photo, and more). This was one of my holiday Claude Code vibe-coding projects. Built with Opus 4.5, and also powered by Opus 4.5. Feeling the AGI 🫡 App is here: Repo is open-source:

Jerry Liu

48,479 Aufrufe • vor 9 Monaten

happy friday. sneak peek at some new features i'm building for the bankr agent: > sandboxed filesystem (your agent gets its own secure file system in the browser. you control what it can access. no agent running loose on your actual computer) > skill uploads (plugin new capabilities so your agent learns new things) > cli download & access > secure environment variables for api keys > github integration (connect your repo for reads and writes directly from your agent) all of this runs in a secure browser environment. no desktop app. no messy configuration. no downloading agent harnesses from the terminal. just a sandbox you control. in the video: i download an audit skill, run it against a smart contract, and save the report straight to the filesystem. then i ask the agent to pull all my 2026 transactions and write a csv for my accountant.

happy friday. sneak peek at some new features i'm building for the bankr agent: > sandboxed filesystem (your agent gets its own secure file system in the browser. you control what it can access. no agent running loose on your actual computer) > skill uploads (plugin new capabilities so your agent learns new things) > cli download & access > secure environment variables for api keys > github integration (connect your repo for reads and writes directly from your agent) all of this runs in a secure browser environment. no desktop app. no messy configuration. no downloading agent harnesses from the terminal. just a sandbox you control. in the video: i download an audit skill, run it against a smart contract, and save the report straight to the filesystem. then i ask the agent to pull all my 2026 transactions and write a csv for my accountant.

deployer

45,746 Aufrufe • vor 6 Monaten

Introducing the new Box Agent. The Box Agent works across your entire Box file system, maintaining all your security and access controls, and is hyper tuned for working with enterprise content. This means you can now ask questions from all your enterprise content, search for files that were impossible to find before, deploy an agent on specific tasks on subsets of documents, analyze complex data sets, and generate or edit documents and spreadsheets via the agent. You can have the Box Agent search across your Box account to prepare for a sales meeting, analyze customer sentiment reports, process a large set of contracts for legal risk, provide insights into product development, leverage existing knowledge to answer RFPs, and thousands of other use-cases. 90% of enterprise data is unstructured data. This means most enterprise knowledge is sitting in inside of research reports, marketing assets, presentations, roadmap files, contracts, HR documents, and more. This is the critical context that agents need to be able to answer questions about a business, automate workflows, or serve up to other agents. We’ve been grinding on this for a quite a bit, and due to recent AI model advancements we’re now ready to release it to customers. Previous model generations had a difficult time knowing when to give up or keep going on a search, when to browse for files vs. use queries, how to rank files appropriately to know which version of content to use, how to handle large amounts of context to comb through, and more. Due to recent breakthroughs from models like GPT-5.4, Opus 4.6, and Gemini 3, we’ve seen major gains in tool calling, code execution, advanced reasoning, and more. Combined with an agent harness tuned to Box context, now it’s finally possible to have an agent that can work across your file system on long running tasks and actually deliver high quality results. Best of all, because the Box Agent works with any leading AI model, you’ll quickly get the gains coming out of the major labs as major new models are released. Further, openness at Box is key, so you’ll be able to call up the Box Agent from Box’s APIs and MCP server, so you can interact with Box intelligently from any other AI system. We know work happens everywhere, and we want to ensure you can access to the content you need from those places. The new Box Agent is available starting today, rolling out now for Enterprise Plus and Enterprise Advanced customers.

Introducing the new Box Agent. The Box Agent works across your entire Box file system, maintaining all your security and access controls, and is hyper tuned for working with enterprise content. This means you can now ask questions from all your enterprise content, search for files that were impossible to find before, deploy an agent on specific tasks on subsets of documents, analyze complex data sets, and generate or edit documents and spreadsheets via the agent. You can have the Box Agent search across your Box account to prepare for a sales meeting, analyze customer sentiment reports, process a large set of contracts for legal risk, provide insights into product development, leverage existing knowledge to answer RFPs, and thousands of other use-cases. 90% of enterprise data is unstructured data. This means most enterprise knowledge is sitting in inside of research reports, marketing assets, presentations, roadmap files, contracts, HR documents, and more. This is the critical context that agents need to be able to answer questions about a business, automate workflows, or serve up to other agents. We’ve been grinding on this for a quite a bit, and due to recent AI model advancements we’re now ready to release it to customers. Previous model generations had a difficult time knowing when to give up or keep going on a search, when to browse for files vs. use queries, how to rank files appropriately to know which version of content to use, how to handle large amounts of context to comb through, and more. Due to recent breakthroughs from models like GPT-5.4, Opus 4.6, and Gemini 3, we’ve seen major gains in tool calling, code execution, advanced reasoning, and more. Combined with an agent harness tuned to Box context, now it’s finally possible to have an agent that can work across your file system on long running tasks and actually deliver high quality results. Best of all, because the Box Agent works with any leading AI model, you’ll quickly get the gains coming out of the major labs as major new models are released. Further, openness at Box is key, so you’ll be able to call up the Box Agent from Box’s APIs and MCP server, so you can interact with Box intelligently from any other AI system. We know work happens everywhere, and we want to ensure you can access to the content you need from those places. The new Box Agent is available starting today, rolling out now for Enterprise Plus and Enterprise Advanced customers.

Aaron Levie

44,624 Aufrufe • vor 6 Monaten

You probably know OpenClaw🦞, one of the fastest-growing open-source projects around. An OpenClaw harness lets you build, code, analyze, and take real control of your machine. But the more access an agent has to your machine, the more it matters where your data goes & who can see it. With QVAC, you can run your OpenClaw agent fully on your own machine in a few minutes. Nothing leaves your computer. Full setup guide on the blog, see first comment below. 💡 Pro tip: already running OpenClaw? Send the blog article to your agent and let the it do the set up.

You probably know OpenClaw🦞, one of the fastest-growing open-source projects around. An OpenClaw harness lets you build, code, analyze, and take real control of your machine. But the more access an agent has to your machine, the more it matters where your data goes & who can see it. With QVAC, you can run your OpenClaw agent fully on your own machine in a few minutes. Nothing leaves your computer. Full setup guide on the blog, see first comment below. 💡 Pro tip: already running OpenClaw? Send the blog article to your agent and let the it do the set up.

QVAC

15,019,651 Aufrufe • vor 4 Monaten

The latest RAG trend for the current agent harnesses (Codex, Cowork) is to do two passes of document processing to solve a knowledge work task over a data room of documents: 1️⃣ A fast and light pass, oftentimes using a free/OSS doc parsing tool. This can be cheaply run across 10-100-1k’s of files, and enables the agent to then do retrieval (e.g. grep, semantic) to find relevant subsets of context. 2️⃣ A “just-in-time” VLM-based pass. Once the agent finds the relevant pages of context, it will screenshot the documents can call its own VLM (or write code) to dissect the pages. The issue with only using VLM-based OCR tools over massive ad-hoc customer file dumps is that it’s slow and expensive. Doing JIT VLM OCR allows the agent to filter through the data cheaply, but still preserve accuracy for the context that’s needed for the task. The agent harnesses do two-pass document processing by default using off the shelf-tools: pdf2text as the first pass, and using itself (Opus 5) as the second pass. See the below video where Cowork runs over a bunch of PDFs to answer a question about a benchmark graph in the Kimi k3 paper. The main issues here with the “out of the box” doc processing these agents offer are: * Opus 5 is not the best VLM for OCR. It is also way too expensive at scale and lacks grounding * The OSS tools like pypdf, pdf2text, may not be versatile enough as the first pass. * The agent will write a lot of throwaway code to rewrite things an OCR tool would’ve provided out of the box, like chart processing, bounding boxes, confidence scores, leading to increased cost and speed. We have all the tools within LlamaIndex 🦙 to help any agent do two-pass document processing with higher accuracy and lower cost. 1️⃣ We have liteparse for the first pass - a free/OSS parser written in Rust that’s faster/more accurate than other OSS parsers, and supports 50+ document types 2️⃣ We have LlamaParse for the second pass - an agentic document engine that uses VLMs+harnesses to achieve SOTA in accuracy and cost across various doc parsing and extraction tasks. It can be called from any agent harness as an MCP or skill. It takes in page numbers as input, so that the agent can choose to run LlamaParse over a subset of the doc instead of the full doc as a “zoom-in” pass. Come check it out! LiteParse: LlamaParse: All the relevant docs, including MCP, are here:

The latest RAG trend for the current agent harnesses (Codex, Cowork) is to do two passes of document processing to solve a knowledge work task over a data room of documents: 1️⃣ A fast and light pass, oftentimes using a free/OSS doc parsing tool. This can be cheaply run across 10-100-1k’s of files, and enables the agent to then do retrieval (e.g. grep, semantic) to find relevant subsets of context. 2️⃣ A “just-in-time” VLM-based pass. Once the agent finds the relevant pages of context, it will screenshot the documents can call its own VLM (or write code) to dissect the pages. The issue with only using VLM-based OCR tools over massive ad-hoc customer file dumps is that it’s slow and expensive. Doing JIT VLM OCR allows the agent to filter through the data cheaply, but still preserve accuracy for the context that’s needed for the task. The agent harnesses do two-pass document processing by default using off the shelf-tools: pdf2text as the first pass, and using itself (Opus 5) as the second pass. See the below video where Cowork runs over a bunch of PDFs to answer a question about a benchmark graph in the Kimi k3 paper. The main issues here with the “out of the box” doc processing these agents offer are: * Opus 5 is not the best VLM for OCR. It is also way too expensive at scale and lacks grounding * The OSS tools like pypdf, pdf2text, may not be versatile enough as the first pass. * The agent will write a lot of throwaway code to rewrite things an OCR tool would’ve provided out of the box, like chart processing, bounding boxes, confidence scores, leading to increased cost and speed. We have all the tools within LlamaIndex 🦙 to help any agent do two-pass document processing with higher accuracy and lower cost. 1️⃣ We have liteparse for the first pass - a free/OSS parser written in Rust that’s faster/more accurate than other OSS parsers, and supports 50+ document types 2️⃣ We have LlamaParse for the second pass - an agentic document engine that uses VLMs+harnesses to achieve SOTA in accuracy and cost across various doc parsing and extraction tasks. It can be called from any agent harness as an MCP or skill. It takes in page numbers as input, so that the agent can choose to run LlamaParse over a subset of the doc instead of the full doc as a “zoom-in” pass. Come check it out! LiteParse: LlamaParse: All the relevant docs, including MCP, are here:

Jerry Liu

23,239 Aufrufe • vor 1 Monat

New JavaScript short course: Build a full-stack web application that uses RAG in JavaScript RAG Web Apps with LlamaIndex, taught by Laurie Voss, VP of Developer Relations at LlamaIndex 🦙 and npm co-founder. - Build a RAG application for querying your own data - Develop tools to interact with multiple data sources using an agent that intelligently selects the right tool for your queries - Create a full-stack web app that can chat with your data - Dig further into production-ready techniques, like how to persist your data so you aren’t constantly reindexing, and try the create-llama command line tool from LlamaIndex You can sign up here:

New JavaScript short course: Build a full-stack web application that uses RAG in JavaScript RAG Web Apps with LlamaIndex, taught by Laurie Voss, VP of Developer Relations at LlamaIndex 🦙 and npm co-founder. - Build a RAG application for querying your own data - Develop tools to interact with multiple data sources using an agent that intelligently selects the right tool for your queries - Create a full-stack web app that can chat with your data - Dig further into production-ready techniques, like how to persist your data so you aren’t constantly reindexing, and try the create-llama command line tool from LlamaIndex You can sign up here:

Andrew Ng

218,703 Aufrufe • vor 2 Jahren

Our new RAG CLI tool lets you RAG any file on your filesystem, using only on-device LLMs/embeddings! 💾🔎 Using LlamaIndex 🦙, Mistral-7B on ollama, and bge-m3, you have access to an LLM-powered grep that you can easily install, use, and customize. Can input any files including glob strings. LlamaPack: We also have the recipe for you to build your own RAG CLI from scratch. Check out this excellent guide by Sourabh Desai:

Our new RAG CLI tool lets you RAG any file on your filesystem, using only on-device LLMs/embeddings! 💾🔎 Using LlamaIndex 🦙, Mistral-7B on ollama, and bge-m3, you have access to an LLM-powered grep that you can easily install, use, and customize. Can input any files including glob strings. LlamaPack: We also have the recipe for you to build your own RAG CLI from scratch. Check out this excellent guide by Sourabh Desai:

LlamaIndex 🦙

40,124 Aufrufe • vor 2 Jahren

I’m excited to kick off the first of our short courses focused on agents, starting with Building Agentic RAG with LlamaIndex, taught by Jerry Liu, CEO of LlamaIndex 🦙. This covers an important shift in RAG (retrieval augmented generation), in which rather than having the developer write explicit routines to retrieve information to feed into the LLM context, we instead build a RAG agent that that has access to tools for retrieving information. This lets the agent decide what information to fetch, and enables it to answer more complex questions using multi-step reasoning. In detail, you'll learn about: - Routing: Where your agent will use decision-making to route requests to multiple tools. - Tool Use: Where you'll create an interface for agents to select what tool (function call) to use as well as generate the right arguments. - Multi-step reasoning with tool use: Where you'll use an LLM to carry out multiple steps of reasoning, while retaining memory throughout the process. You’ll also learn how to step through what your agent is doing to debug and improve it iteratively. It’s an exciting time to build agents. Sign up and get started here!

I’m excited to kick off the first of our short courses focused on agents, starting with Building Agentic RAG with LlamaIndex, taught by Jerry Liu, CEO of LlamaIndex 🦙. This covers an important shift in RAG (retrieval augmented generation), in which rather than having the developer write explicit routines to retrieve information to feed into the LLM context, we instead build a RAG agent that that has access to tools for retrieving information. This lets the agent decide what information to fetch, and enables it to answer more complex questions using multi-step reasoning. In detail, you'll learn about: - Routing: Where your agent will use decision-making to route requests to multiple tools. - Tool Use: Where you'll create an interface for agents to select what tool (function call) to use as well as generate the right arguments. - Multi-step reasoning with tool use: Where you'll use an LLM to carry out multiple steps of reasoning, while retaining memory throughout the process. You’ll also learn how to step through what your agent is doing to debug and improve it iteratively. It’s an exciting time to build agents. Sign up and get started here!

Andrew Ng

297,303 Aufrufe • vor 2 Jahren