Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

RAG vs. Agents Interview with Tanishq Singh. This is a real interview question from a big tech company, asked to a candidate in their technical interview round. The video explains the answer in roughly 4 minutes. 00:00 Question - RAG vs Agents 00:35 Difference between RAG and Agents 01:45... show more

Gaurav Sen

73,669 subscribers

28,949 views • 11 days ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

How do we run evals with LLMs? Interview with Tanishq Singh. This is a real interview question from a big tech company, asked to a candidate in their technical interview round. The video explains the answer in roughly 11 minutes. 00:00 Question - Reliability with LLMs 01:00 Observability & Evals 02:50 What are evals? 06:41 Who does evals? 08:11 Answer relevancy, groundedness, tool sequence 10:39 Bloopers :p AI engineering program for software engineers:

How do we run evals with LLMs? Interview with Tanishq Singh. This is a real interview question from a big tech company, asked to a candidate in their technical interview round. The video explains the answer in roughly 11 minutes. 00:00 Question - Reliability with LLMs 01:00 Observability & Evals 02:50 What are evals? 06:41 Who does evals? 08:11 Answer relevancy, groundedness, tool sequence 10:39 Bloopers :p AI engineering program for software engineers:

Gaurav Sen

35,079 views • 20 days ago

What's better? LLMs or Agents? This is a real interview question from a big tech company. For more questions, subscribe to the channel! AI engineering program for software engineers: 00:00 Question - LLMs vs. Agents 01:02 Basic Answer 02:16 Follow-up question 03:47 Context Management 05:08 Bloopers #LLMs #AI #InterviewReady

What's better? LLMs or Agents? This is a real interview question from a big tech company. For more questions, subscribe to the channel! AI engineering program for software engineers: 00:00 Question - LLMs vs. Agents 01:02 Basic Answer 02:16 Follow-up question 03:47 Context Management 05:08 Bloopers #LLMs #AI #InterviewReady

Gaurav Sen

18,889 views • 1 month ago

How do we cache prompts with LLMs? Interview with Tanishq Singh. This is a real interview question from a big tech company, asked to a candidate in their technical interview round. The video explains the answer in roughly 6 minutes. 00:00 Question - Prompt Caching with LLMs 00:39 Exact Query Caching 02:04 Semantic Caching 04:14 Tradeoffs 05:52 Blooper :p AI engineering program for software engineers: #LLMs #AI #InterviewReady

How do we cache prompts with LLMs? Interview with Tanishq Singh. This is a real interview question from a big tech company, asked to a candidate in their technical interview round. The video explains the answer in roughly 6 minutes. 00:00 Question - Prompt Caching with LLMs 00:39 Exact Query Caching 02:04 Semantic Caching 04:14 Tradeoffs 05:52 Blooper :p AI engineering program for software engineers: #LLMs #AI #InterviewReady

Gaurav Sen

94,211 views • 29 days ago

Google just showed how AI engineers are moving from simple "RAG" to "graphs", memory and multimodal agents • 15:32 - setting up the production agent stack • 28:00 - turning disconnected data into a knowledge graph • 41:00 - Graph RAG with semantic + hybrid search • 58:00 - extracting graph context from images, text and video • 1:09:00 - orchestrating specialized agents with ADK • 1:20:00 - giving agents persistent memory across sessions 94-minute Google Cloud workshop, and it’s one of the clearest hands-on examples of the shift from -> RAG → Graph RAG → Memory → Multimodal Agent Graphs. The workshop uses Spanner Graph, Gemini, ADK and Memory Bank to build that stack. Watch it today, then read the full “From RAG to Context Graphs” roadmap below.

Google just showed how AI engineers are moving from simple "RAG" to "graphs", memory and multimodal agents • 15:32 - setting up the production agent stack • 28:00 - turning disconnected data into a knowledge graph • 41:00 - Graph RAG with semantic + hybrid search • 58:00 - extracting graph context from images, text and video • 1:09:00 - orchestrating specialized agents with ADK • 1:20:00 - giving agents persistent memory across sessions 94-minute Google Cloud workshop, and it’s one of the clearest hands-on examples of the shift from -> RAG → Graph RAG → Memory → Multimodal Agent Graphs. The workshop uses Spanner Graph, Gemini, ADK and Memory Bank to build that stack. Watch it today, then read the full “From RAG to Context Graphs” roadmap below.

Morlex

74,134 views • 21 days ago

I'm excited to share my talk on building Agentic RAG systems at last week's event with Arize AI and Google! 🥳 My talk covered: 1. The differences between Vanilla RAG and Agentic RAG 2. The agent ecosystem and how you can build agents today 3. How Weaviate AI Database is building agents with Generative Feedback Loops (GFLs) This snippet presents the multi-agent paradigm for Agentic RAG. I hope you find this talk interesting and would love to know what you think! The slide deck and full video links are below.

I'm excited to share my talk on building Agentic RAG systems at last week's event with Arize AI and Google! 🥳 My talk covered: 1. The differences between Vanilla RAG and Agentic RAG 2. The agent ecosystem and how you can build agents today 3. How Weaviate AI Database is building agents with Generative Feedback Loops (GFLs) This snippet presents the multi-agent paradigm for Agentic RAG. I hope you find this talk interesting and would love to know what you think! The slide deck and full video links are below.

Erika Shorten

61,317 views • 1 year ago

You need 6 key skills to become an AI PM. Here's a demo of all 6 with Paweł Huryn: 1. Prompting - 02:39 2. AI PRDs - 11:54 3. Fine-Tuning - 16:42 4. RAG - 30:03 5. MCP - 59:00 6. AI Agents - 01:18:31

You need 6 key skills to become an AI PM. Here's a demo of all 6 with Paweł Huryn: 1. Prompting - 02:39 2. AI PRDs - 11:54 3. Fine-Tuning - 16:42 4. RAG - 30:03 5. MCP - 59:00 6. AI Agents - 01:18:31

Aakash Gupta

110,166 views • 1 year ago

Google just released a 1-hour course on building agentic knowledge Graphs from scratch: • 00:00 - Introduction to AI agents • 08:09 - Foundations of multi-agent systems • 13:25 - Introduction to Graph engineering • 23:27 - Environment setup for agentic Graphs • 47:43 - Building Graph RAG agents • 1:25:14 - Agent memory bank with graphs Worth more than 10 articles on loop engineering. Watch it today, then read how to become a graph engineer in the article below.

Google just released a 1-hour course on building agentic knowledge Graphs from scratch: • 00:00 - Introduction to AI agents • 08:09 - Foundations of multi-agent systems • 13:25 - Introduction to Graph engineering • 23:27 - Environment setup for agentic Graphs • 47:43 - Building Graph RAG agents • 1:25:14 - Agent memory bank with graphs Worth more than 10 articles on loop engineering. Watch it today, then read how to become a graph engineer in the article below.

Codez

196,064 views • 1 month ago

Ex-Google engineer just released a free 3-hour course on building and monetizing AI agents. How to go from one agent to a full system that finds leads and makes money: 00:00 - Design an AI agent system 07:38 - Add human handoffs 19:27 - Understand RAG and vector databases 52:38 - Deploy agents to Google Cloud 1:25:57 - Turn agents into a paid WhatsApp business 1:33:00 - Convert conversations into leads 2:24:01 - Learn loops vs graphs 2:33:08 - Build a multi-tool agent graph Most people are still building AI demos. This course shows the full stack: Agents → RAG → Deployment → Leads → Revenue Building agents is the old workflow. Monetizing agent systems is the new one. This 3-hour watch covers more than most $500 paid courses. Save it before everyone starts selling the same thing.

Ex-Google engineer just released a free 3-hour course on building and monetizing AI agents. How to go from one agent to a full system that finds leads and makes money: 00:00 - Design an AI agent system 07:38 - Add human handoffs 19:27 - Understand RAG and vector databases 52:38 - Deploy agents to Google Cloud 1:25:57 - Turn agents into a paid WhatsApp business 1:33:00 - Convert conversations into leads 2:24:01 - Learn loops vs graphs 2:33:08 - Build a multi-tool agent graph Most people are still building AI demos. This course shows the full stack: Agents → RAG → Deployment → Leads → Revenue Building agents is the old workflow. Monetizing agent systems is the new one. This 3-hour watch covers more than most $500 paid courses. Save it before everyone starts selling the same thing.

Lunar

348,649 views • 27 days ago

Anthropic engineer just released a 2-hour workshop on "Graph Engineering" for agentic systems: “80% of our engineers are using self-improving loops. Now everyone is building agentic graphs.” • 00:00 - Introduction to RAG & Graphs • 06:39 - Core of "Graph Engineering" (state, nodes) • 14:29 - 3 feedback loops of Graph agents • 23:06 - Agent evaluation with Graphs • 36:29 - Agent cycles in graphs • 1:15:22 - Agentic RAG & agent context • 1:41:20 - Evaluation datasets based on Graphs This 2-hour workshop will replace 10 paid courses on agentic engineering. Watch it today, then learn how to become a Graph Engineer in the article below.

Anthropic engineer just released a 2-hour workshop on "Graph Engineering" for agentic systems: “80% of our engineers are using self-improving loops. Now everyone is building agentic graphs.” • 00:00 - Introduction to RAG & Graphs • 06:39 - Core of "Graph Engineering" (state, nodes) • 14:29 - 3 feedback loops of Graph agents • 23:06 - Agent evaluation with Graphs • 36:29 - Agent cycles in graphs • 1:15:22 - Agentic RAG & agent context • 1:41:20 - Evaluation datasets based on Graphs This 2-hour workshop will replace 10 paid courses on agentic engineering. Watch it today, then learn how to become a Graph Engineer in the article below.

Codez

400,915 views • 1 month ago

Today, we’re excited to announce the general availability of the Contextual AI Platform. This is the first enterprise platform designed for building specialized RAG agents to support expert knowledge work. What is a specialized RAG agent? First, a general-purpose AI agent is one designed to automate simple daily tasks like scheduling a meeting or responding to an email. On the other hand, a specialized RAG agent is one designed to augment subject-matter experts performing complex domain-specific work. The Contextual AI Platform allows you to create these agents easily and achieve SOTA accuracy right out of the box. Check out what the Contextual AI Platform can do.

Today, we’re excited to announce the general availability of the Contextual AI Platform. This is the first enterprise platform designed for building specialized RAG agents to support expert knowledge work. What is a specialized RAG agent? First, a general-purpose AI agent is one designed to automate simple daily tasks like scheduling a meeting or responding to an email. On the other hand, a specialized RAG agent is one designed to augment subject-matter experts performing complex domain-specific work. The Contextual AI Platform allows you to create these agents easily and achieve SOTA accuracy right out of the box. Check out what the Contextual AI Platform can do.

Contextual AI

879,586 views • 1 year ago

My friend applied to 200 tech jobs in two years. No MIT. No Stanford. Last month Anthropic offered him $750,000. I asked him how he broke in from zero. He sent me the exact video that got him in. Anthropic's 2 hour course on "How to become an AI engineer in 2026" Their core team shows exactly how to architect & build AI agents from scratch. I watched it last night. Halfway through, I realized I could break into an AI lab in weeks, not years. Bookmark this and read the article below. • 00:00 - AI agents with graph engineering • 06:41 - AI agent architecture • 14:31 - building AI agent loops live • 1:15:24 - AI agentic RAG • 2:20:42 - Anthropic interview process

My friend applied to 200 tech jobs in two years. No MIT. No Stanford. Last month Anthropic offered him $750,000. I asked him how he broke in from zero. He sent me the exact video that got him in. Anthropic's 2 hour course on "How to become an AI engineer in 2026" Their core team shows exactly how to architect & build AI agents from scratch. I watched it last night. Halfway through, I realized I could break into an AI lab in weeks, not years. Bookmark this and read the article below. • 00:00 - AI agents with graph engineering • 06:41 - AI agent architecture • 14:31 - building AI agent loops live • 1:15:24 - AI agentic RAG • 2:20:42 - Anthropic interview process

Roan

143,175 views • 1 month ago

Free 3-hour course from an ex-Google engineer. AI agents, graph engineering, harness, agent loops. Covers more than most $500 paid courses. This is the exact playbook to learn how to actually build and monetize AI agents. Don't miss your shot, save it now. • 0:00 - Design an AI agent system from scratch • 7:38 - When your agent should hand off to a human • 19:27 - RAG and vector databases explained simply • 52:38 - Deploy your AI agent to Google Cloud • 1:25:57 - Turn AI agents into a paid business via WhatsApp • 1:33:00 - Turn conversations into real leads automatically • 2:24:01 - Loop engineering vs graphs, the real difference • 2:33:08 - Build your own multi-tool agent graph Save this video, because it's the only one that can actually take you to your $2,000+ profit.

Free 3-hour course from an ex-Google engineer. AI agents, graph engineering, harness, agent loops. Covers more than most $500 paid courses. This is the exact playbook to learn how to actually build and monetize AI agents. Don't miss your shot, save it now. • 0:00 - Design an AI agent system from scratch • 7:38 - When your agent should hand off to a human • 19:27 - RAG and vector databases explained simply • 52:38 - Deploy your AI agent to Google Cloud • 1:25:57 - Turn AI agents into a paid business via WhatsApp • 1:33:00 - Turn conversations into real leads automatically • 2:24:01 - Loop engineering vs graphs, the real difference • 2:33:08 - Build your own multi-tool agent graph Save this video, because it's the only one that can actually take you to your $2,000+ profit.

unicode

142,596 views • 29 days ago

Announcing a new Coursera course: Retrieval Augmented Generation (RAG) You'll learn to build high performance, production-ready RAG systems in this hands-on, in-depth course created by and taught by , experienced AI and ML engineer, researcher, and educator. RAG is a critical component today of many LLM-based applications in customer support, internal company Q&A systems, even many of the leading chatbots that use web search to answer your questions. This course teaches you in-depth how to make RAG work well. LLMs can produce generic or outdated responses, especially when asked specialized questions not covered in its training data. RAG is the most widely used technique for addressing this. It brings in data from new data sources, such as internal documents or recent news, to give the LLM the relevant context to private, recent, or specialized information. This lets it generate more grounded and accurate responses. In this course, you’ll learn to design and implement every part of a RAG system, from retrievers to vector databases to generation to evals. You’ll learn about the fundamental principles behind RAG and how to optimize it at both the component and whole-system levels. As AI evolves, RAG is evolving too. New models can handle longer context windows, reason more effectively, and can be parts of complex agentic workflows. One exciting growth area is Agentic RAG, in which an AI agent at runtime (rather than it being hardcoded at development time) autonomously decides what data to retrieve, and when/how to go deeper. Even with this evolution, access to high-quality data at runtime is essential, which is why RAG is a key part of so many applications. You'll learn via hands-on experiences to: - Build a RAG system with retrieval and prompt augmentation - Compare retrieval methods like BM25, semantic search, and Reciprocal Rank Fusion - Chunk, index, and retrieve documents using a Weaviate vector database and a news dataset - Develop a chatbot, using open-source LLMs hosted by Together AI, for a fictional store that answers product and FAQ questions - Use evals to drive improving reliability, and incorporate multi-modal data RAG is an important foundational technique. Become good at it through this course! Please sign up here:

Announcing a new Coursera course: Retrieval Augmented Generation (RAG) You'll learn to build high performance, production-ready RAG systems in this hands-on, in-depth course created by and taught by , experienced AI and ML engineer, researcher, and educator. RAG is a critical component today of many LLM-based applications in customer support, internal company Q&A systems, even many of the leading chatbots that use web search to answer your questions. This course teaches you in-depth how to make RAG work well. LLMs can produce generic or outdated responses, especially when asked specialized questions not covered in its training data. RAG is the most widely used technique for addressing this. It brings in data from new data sources, such as internal documents or recent news, to give the LLM the relevant context to private, recent, or specialized information. This lets it generate more grounded and accurate responses. In this course, you’ll learn to design and implement every part of a RAG system, from retrievers to vector databases to generation to evals. You’ll learn about the fundamental principles behind RAG and how to optimize it at both the component and whole-system levels. As AI evolves, RAG is evolving too. New models can handle longer context windows, reason more effectively, and can be parts of complex agentic workflows. One exciting growth area is Agentic RAG, in which an AI agent at runtime (rather than it being hardcoded at development time) autonomously decides what data to retrieve, and when/how to go deeper. Even with this evolution, access to high-quality data at runtime is essential, which is why RAG is a key part of so many applications. You'll learn via hands-on experiences to: - Build a RAG system with retrieval and prompt augmentation - Compare retrieval methods like BM25, semantic search, and Reciprocal Rank Fusion - Chunk, index, and retrieve documents using a Weaviate vector database and a news dataset - Develop a chatbot, using open-source LLMs hosted by Together AI, for a fictional store that answers product and FAQ questions - Use evals to drive improving reliability, and incorporate multi-modal data RAG is an important foundational technique. Become good at it through this course! Please sign up here:

Andrew Ng

124,656 views • 1 year ago

Let's build a dashboard to evaluate and monitor your Agentic and RAG apps! . . In this video, I'll guide you through creating an evaluation and observability pipeline for your AI apps using a 100% open-source tool! Tech Stack: - Comet's Opik to eval and monitor - LlamaIndex to build a RAG pipeline - Ragas for synthetic datagen You'll learn: - Setting up Opik - Building a RAG pipeline - Creating an eval dataset - Evaluating the RAG pipeline - Monitoring all activities during the process It's a hands on demo with code and step-by-step guide to do everything listed above. CometML's Opik is fully open-source, offering the most Pythonic and easiest way to monitor LLM apps. I have shared link to their repo in next tweet!

Let's build a dashboard to evaluate and monitor your Agentic and RAG apps! . . In this video, I'll guide you through creating an evaluation and observability pipeline for your AI apps using a 100% open-source tool! Tech Stack: - Comet's Opik to eval and monitor - LlamaIndex to build a RAG pipeline - Ragas for synthetic datagen You'll learn: - Setting up Opik - Building a RAG pipeline - Creating an eval dataset - Evaluating the RAG pipeline - Monitoring all activities during the process It's a hands on demo with code and step-by-step guide to do everything listed above. CometML's Opik is fully open-source, offering the most Pythonic and easiest way to monitor LLM apps. I have shared link to their repo in next tweet!

Akshay 🚀

20,058 views • 1 year ago

In this video, we break down the Model Context Protocol — a massively underrated concept in AI that's quietly redefining what large language models can do. While most people are distracted by the hype, MCP is solving a core problem: LLMs that can perform actions. We dive into: 1. The limitations of current AI systems 2. What makes MCP different 3. How MCP can reduce engineering overhead 4. Real-world scenarios where MCP shines 00:00 What is MCP? 01:48 Usecase - SEO 03:50 Usecase - RAG 04:50 Usecase - Apps 06:30 Conclusion 07:00 The future? #MCP #AI #LLM

In this video, we break down the Model Context Protocol — a massively underrated concept in AI that's quietly redefining what large language models can do. While most people are distracted by the hype, MCP is solving a core problem: LLMs that can perform actions. We dive into: 1. The limitations of current AI systems 2. What makes MCP different 3. How MCP can reduce engineering overhead 4. Real-world scenarios where MCP shines 00:00 What is MCP? 01:48 Usecase - SEO 03:50 Usecase - RAG 04:50 Usecase - Apps 06:30 Conclusion 07:00 The future? #MCP #AI #LLM

Gaurav Sen

41,154 views • 1 year ago

Box CEO Aaron Levie on the AI Adoption Gap Aaron Levie joins Steven Sinofsky, Martin Casado, and Erik Torenberg to discuss how AI agents will revolutionize work, the growing pains of building software for the agent economy, what Wall Street gets wrong about AI, and more. 00:00 Intro 00:51 Building software for agents vs. humans 02:10 Can non-technical workers actually use AI agents? 14:31 CFO/CIO pushback: the real fear of agents doing integration 18:39 Treating agents like employees and why it breaks down 27:35 Diffusion gap: startups vs. enterprises 42:53 What Wall Street gets wrong Aaron Levie Steven Sinofsky martin_casado Erik Torenberg

Box CEO Aaron Levie on the AI Adoption Gap Aaron Levie joins Steven Sinofsky, Martin Casado, and Erik Torenberg to discuss how AI agents will revolutionize work, the growing pains of building software for the agent economy, what Wall Street gets wrong about AI, and more. 00:00 Intro 00:51 Building software for agents vs. humans 02:10 Can non-technical workers actually use AI agents? 14:31 CFO/CIO pushback: the real fear of agents doing integration 18:39 Treating agents like employees and why it breaks down 27:35 Diffusion gap: startups vs. enterprises 42:53 What Wall Street gets wrong Aaron Levie Steven Sinofsky martin_casado Erik Torenberg

a16z

276,989 views • 4 months ago

chat, is starknet gaming cooked? for episode two, loaf has joined me to talk about ai agents in gaming, eliza and Daydreams: Taskmarket . timestamps loaf introduction 00:46 what are ai agents? 01:16 what will gaming look like with ai agents? 02:02 whats the point of gaming with ai agents? 04:06 why is onchain gaming the ideal playground for AI agents? 05:11 is a gaming revolution imminent? 07:00 eliza & daydreams 08:36 endgame of ai agents and onchain games 13:33 when and where to play using agents 16:58 final remarks 18:47

chat, is starknet gaming cooked? for episode two, loaf has joined me to talk about ai agents in gaming, eliza and Daydreams: Taskmarket . timestamps loaf introduction 00:46 what are ai agents? 01:16 what will gaming look like with ai agents? 02:02 whats the point of gaming with ai agents? 04:06 why is onchain gaming the ideal playground for AI agents? 05:11 is a gaming revolution imminent? 07:00 eliza & daydreams 08:36 endgame of ai agents and onchain games 13:33 when and where to play using agents 16:58 final remarks 18:47

shadowfax

37,582 views • 1 year ago

Andrew Ng just released a 1-hour course on what comes after RAG. How to turn raw documents into agent memory: 00:00 - Why larger context windows are not enough 02:18 - Extract entities, relationships, and sources 14:31 - Coordinate specialized agents inside one graph 30:43 - Build it with Google ADK and Neo4j 47:25 - Turn knowledge graphs into persistent memory Most people are still learning RAG. Andrew Ng is already teaching the layer that comes after it. Documents → Entities → Relationships → Agents → Knowledge Graph More context is the old workflow. Persistent graphs are the new memory layer. This 1-hour watch is worth more than most paid GraphRAG courses. Bookmark and watch it today Then read the full graph engineering roadmap below

Andrew Ng just released a 1-hour course on what comes after RAG. How to turn raw documents into agent memory: 00:00 - Why larger context windows are not enough 02:18 - Extract entities, relationships, and sources 14:31 - Coordinate specialized agents inside one graph 30:43 - Build it with Google ADK and Neo4j 47:25 - Turn knowledge graphs into persistent memory Most people are still learning RAG. Andrew Ng is already teaching the layer that comes after it. Documents → Entities → Relationships → Agents → Knowledge Graph More context is the old workflow. Persistent graphs are the new memory layer. This 1-hour watch is worth more than most paid GraphRAG courses. Bookmark and watch it today Then read the full graph engineering roadmap below

Lunar

42,118 views • 1 month ago

I'm often asked for the best public example of AI evals done right for a real, production product. I finally have an answer. Teresa Torres shares how she shipped an AI interview coach, and used evals to rapidly squash bugs and improve the product. Teresa shows how she: 1. did error analysis FIRST to find real issues (instead of using generic metrics) 😍 2. used Jupyter notebooks to analyze errors 3. built custom annotation tools + custom widgets in notebooks 4. built a LLM-judge and assertions to test for specific errors 5. iterated through this feedback loop until it worked. 6. kept things simple the whole time It's also probably the best commercial for Jupyter notebooks you can imagine. 🥰 Chapter summary below. Link to YT in next thread 00:00:00 - Intro 00:01:45 - The Product: Building an AI Interview Coach 00:06:34 - The Problem: How Do I Know if My AI Coach is Any Good? 00:10:15 - Using Airtable for Traces and Annotation 00:12:15 - Discovering Jupyter Notebooks and Designing the First Evals 00:15:15 - Example Evals: LLM-as-Judge vs. Code-Based Assertions 00:21:00 - Learning Python with ChatGPT to Analyze Eval Results 00:31:00 - VS Code, Custom Tools, and an Eval Investigation Notebook 00:39:45 - Building a Custom Annotation Tool with Claude 00:41:00 - From Personal Project to Production App 00:46:02 - How Should PMs and Engineers Collaborate on AI Products? 00:55:45 - Q&A: Capturing Feedback and Annotations from End Users 00:58:11 - Q&A: Is a Technical Background Necessary to Build AI? 01:02:28 - Q&A: What's Next for Teresa? 01:03:13 - Q&A: Unpacking the Micro-Decisions of Building an AI App

I'm often asked for the best public example of AI evals done right for a real, production product. I finally have an answer. Teresa Torres shares how she shipped an AI interview coach, and used evals to rapidly squash bugs and improve the product. Teresa shows how she: 1. did error analysis FIRST to find real issues (instead of using generic metrics) 😍 2. used Jupyter notebooks to analyze errors 3. built custom annotation tools + custom widgets in notebooks 4. built a LLM-judge and assertions to test for specific errors 5. iterated through this feedback loop until it worked. 6. kept things simple the whole time It's also probably the best commercial for Jupyter notebooks you can imagine. 🥰 Chapter summary below. Link to YT in next thread 00:00:00 - Intro 00:01:45 - The Product: Building an AI Interview Coach 00:06:34 - The Problem: How Do I Know if My AI Coach is Any Good? 00:10:15 - Using Airtable for Traces and Annotation 00:12:15 - Discovering Jupyter Notebooks and Designing the First Evals 00:15:15 - Example Evals: LLM-as-Judge vs. Code-Based Assertions 00:21:00 - Learning Python with ChatGPT to Analyze Eval Results 00:31:00 - VS Code, Custom Tools, and an Eval Investigation Notebook 00:39:45 - Building a Custom Annotation Tool with Claude 00:41:00 - From Personal Project to Production App 00:46:02 - How Should PMs and Engineers Collaborate on AI Products? 00:55:45 - Q&A: Capturing Feedback and Annotations from End Users 00:58:11 - Q&A: Is a Technical Background Necessary to Build AI? 01:02:28 - Q&A: What's Next for Teresa? 01:03:13 - Q&A: Unpacking the Micro-Decisions of Building an AI App

Hamel Husain

51,376 views • 1 year ago