Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Evals 101: a new course from Braintrust. Everything you need to know about evals, and how to do them yourself. Module one: Why are evals important? - the six most common problems developers face when shipping AI applications - why traditional software thinking doesn't apply to AI - how...

braintrust's profile picture

Braintrust

6,704 subscribers

36,347 Aufrufe • vor 4 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:
1:12

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 Aufrufe • vor 11 Monaten

NEW Greylock Change Agents: Evaluating Agents with Braintrust There's been a lot of discourse around evals recently, so it's timely to drop the recording of Change Agents with Ankur Goyal on the topic of how he thinks about Evaluating Agents. Timestamps: (0:00) - Intro (1:22) - Building Braintrust (5:19) - Evaluating agentic systems (11:04) - Experimentation with evals (13:18) - Adopting evals (17:09) - Evals and complex agents (21:19) - Custom scoring (25:02) - LLM as a judge (26:44) - Human-manual evaluation (30:07) - How the best in-class teams manage evals (33:57) - Future of evals for agents (37:38 ) - Audience Q&A
57:34

NEW Greylock Change Agents: Evaluating Agents with Braintrust There's been a lot of discourse around evals recently, so it's timely to drop the recording of Change Agents with Ankur Goyal on the topic of how he thinks about Evaluating Agents. Timestamps: (0:00) - Intro (1:22) - Building Braintrust (5:19) - Evaluating agentic systems (11:04) - Experimentation with evals (13:18) - Adopting evals (17:09) - Evals and complex agents (21:19) - Custom scoring (25:02) - LLM as a judge (26:44) - Human-manual evaluation (30:07) - How the best in-class teams manage evals (33:57) - Future of evals for agents (37:38 ) - Audience Q&A

Corinne Marie Riley

12,743 Aufrufe • vor 11 Monaten

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC
13:01

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC

Matt Turck

10,969 Aufrufe • vor 11 Monaten

How do we run evals with LLMs? Interview with Tanishq Singh. This is a real interview question from a big tech company, asked to a candidate in their technical interview round. The video explains the answer in roughly 11 minutes. 00:00 Question - Reliability with LLMs 01:00 Observability & Evals 02:50 What are evals? 06:41 Who does evals? 08:11 Answer relevancy, groundedness, tool sequence 10:39 Bloopers :p AI engineering program for software engineers:
11:18

How do we run evals with LLMs? Interview with Tanishq Singh. This is a real interview question from a big tech company, asked to a candidate in their technical interview round. The video explains the answer in roughly 11 minutes. 00:00 Question - Reliability with LLMs 01:00 Observability & Evals 02:50 What are evals? 06:41 Who does evals? 08:11 Answer relevancy, groundedness, tool sequence 10:39 Bloopers :p AI engineering program for software engineers:

Gaurav Sen

35,079 Aufrufe • vor 14 Tagen

"AI evals force you into the shoes of your user." I love this clip from (Director at Arize) on why evaluations are the top AI skill to build in 2025. In my new interview, Aman breaks down: - How to write evals like a pro - 5 AI product principles he swears by - How to start practicing evals now Some quotes from Aman: "All PMs will be AI PMs. You'll either build on top of LLM APIs, use models in your backend, or use AI tools to solve problems in your daily work." "The waterfall process of writing a PRD and handing it to designers and engineers is dead. Your dev cycle needs to be much shorter." "Imagine showing up to a meeting with a prototype instead of a PRD. That's the shift PMs will need to make to communicate better in this new AI world." 📌 Watch now:
1:13

"AI evals force you into the shoes of your user." I love this clip from (Director at Arize) on why evaluations are the top AI skill to build in 2025. In my new interview, Aman breaks down: - How to write evals like a pro - 5 AI product principles he swears by - How to start practicing evals now Some quotes from Aman: "All PMs will be AI PMs. You'll either build on top of LLM APIs, use models in your backend, or use AI tools to solve problems in your daily work." "The waterfall process of writing a PRD and handing it to designers and engineers is dead. Your dev cycle needs to be much shorter." "Imagine showing up to a meeting with a prototype instead of a PRD. That's the shift PMs will need to make to communicate better in this new AI world." 📌 Watch now:

Peter Yang

15,973 Aufrufe • vor 1 Jahr

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.
52:06

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.

Aakash Gupta

483,558 Aufrufe • vor 5 Monaten

One AI skill will create millionaires: Evals. It's the key to great AI products. Here's a crash course with experts Hamel Husain & Shreya Shankar: 02:06 - PM v Eng Role 23:56 - Evaluating RAG 42:42 - How OpenAI does evals 01:08:29 - LLM-as-judge
1:34:32

One AI skill will create millionaires: Evals. It's the key to great AI products. Here's a crash course with experts Hamel Husain & Shreya Shankar: 02:06 - PM v Eng Role 23:56 - Evaluating RAG 42:42 - How OpenAI does evals 01:08:29 - LLM-as-judge

Aakash Gupta

42,185 Aufrufe • vor 1 Jahr

New session with Shreya Shankar on How to Automate Evals with AI (correctly) The most important part of the eval workflow is finding issues. Shreya does a live demo of how to steer the AI iteratively to find unknown unknowns Chapter summaries: 00:00:00 Introduction 00:00:55 Why vendors want to automate evals for you 00:01:45 Why no tool can fully automate evals 00:06:00 Mistake #1: Asking AI to just "find the issues" in your data 00:08:11 What good AI-assisted error analysis actually looks like 00:11:12 Live demo: building a review interface from scratch 00:15:10 Annotating traces and building a failure mode taxonomy 00:19:31 Mistake #2: Only reviewing your data once 00:22:53 Mistake #3: Treating all apps with the same accuracy bar 00:25:10 General-purpose agents vs. dedicated eval tools 00:25:48 Summary: three rules for automating evals correctly YT links and more in reply
27:18

New session with Shreya Shankar on How to Automate Evals with AI (correctly) The most important part of the eval workflow is finding issues. Shreya does a live demo of how to steer the AI iteratively to find unknown unknowns Chapter summaries: 00:00:00 Introduction 00:00:55 Why vendors want to automate evals for you 00:01:45 Why no tool can fully automate evals 00:06:00 Mistake #1: Asking AI to just "find the issues" in your data 00:08:11 What good AI-assisted error analysis actually looks like 00:11:12 Live demo: building a review interface from scratch 00:15:10 Annotating traces and building a failure mode taxonomy 00:19:31 Mistake #2: Only reviewing your data once 00:22:53 Mistake #3: Treating all apps with the same accuracy bar 00:25:10 General-purpose agents vs. dedicated eval tools 00:25:48 Summary: three rules for automating evals correctly YT links and more in reply

Hamel Husain

34,196 Aufrufe • vor 1 Monat

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!
1:15

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!

AI Engineer

55,348 Aufrufe • vor 1 Jahr

I got an expert to give me a masterclass on AI evals: (63 mins of stuff most builders learn the hard way) 0:00 - Why your product is lying to you 22:03 - Full eval flow walkthrough 35:42 - Evals ARE your PRD (not QA) 57:52 - Failures at Grammarly, Copilot & Klarna
1:03:59

I got an expert to give me a masterclass on AI evals: (63 mins of stuff most builders learn the hard way) 0:00 - Why your product is lying to you 22:03 - Full eval flow walkthrough 35:42 - Evals ARE your PRD (not QA) 57:52 - Failures at Grammarly, Copilot & Klarna

Aakash Gupta

247,328 Aufrufe • vor 6 Monaten

I like to think of evals as something active, not passive -- it's a North Star that steers LLMs toward higher intelligence. Evals should drive your RL/SFT/post-training decisions. Internal evals at frontier labs make a huge difference -- and you can see it in how models behave differently (GPT seems better at one-shot tasks, Claude at multi-turn). If you want to learn more about building evals that actually improve your model in post-training, check out our AMD x DeeplearningAI course "Fine-tuning & RL for LLMs: Intro to Post-training" (content is free):
0:30

I like to think of evals as something active, not passive -- it's a North Star that steers LLMs toward higher intelligence. Evals should drive your RL/SFT/post-training decisions. Internal evals at frontier labs make a huge difference -- and you can see it in how models behave differently (GPT seems better at one-shot tasks, Claude at multi-turn). If you want to learn more about building evals that actually improve your model in post-training, check out our AMD x DeeplearningAI course "Fine-tuning & RL for LLMs: Intro to Post-training" (content is free):

Sharon Zhou

16,836 Aufrufe • vor 6 Monaten

Andon Labs' Real-World AI Evals: Claude calls the FBI, AI CEOs, price cartels, Butter-Bench, & Luna Andon Labs cofounders Lukas Petersson and Axel Backlund explain why dollar-denominated evals reveal what traditional benchmarks miss, how Claude ended up reporting a $2/day vending machine fee to the FBI, why long-horizon agents spiral in weird ways, what happens when agents lie, form price cartels, and compete with each other, and why the future of AI safety may depend on testing models in messy real-world environments instead of clean benchmark sandboxes.
0:23

Andon Labs' Real-World AI Evals: Claude calls the FBI, AI CEOs, price cartels, Butter-Bench, & Luna Andon Labs cofounders Lukas Petersson and Axel Backlund explain why dollar-denominated evals reveal what traditional benchmarks miss, how Claude ended up reporting a $2/day vending machine fee to the FBI, why long-horizon agents spiral in weird ways, what happens when agents lie, form price cartels, and compete with each other, and why the future of AI safety may depend on testing models in messy real-world environments instead of clean benchmark sandboxes.

Latent.Space

13,740 Aufrufe • vor 2 Monaten

AI PM jobs pay $250K+ in the US, $100K+ in Europe, 60+ LPA in India. But how do you land one? I got the $2.5B CEO (Todd Olson) of PM platform pendo.io to share exactly how: 6:07 - How to Upskill to AI PM 12:54 - The 5-Layer Technical Pyramid 48:56 - Why PM's Must Own Evals
1:21:53

AI PM jobs pay $250K+ in the US, $100K+ in Europe, 60+ LPA in India. But how do you land one? I got the $2.5B CEO (Todd Olson) of PM platform pendo.io to share exactly how: 6:07 - How to Upskill to AI PM 12:54 - The 5-Layer Technical Pyramid 48:56 - Why PM's Must Own Evals

Aakash Gupta

23,986 Aufrufe • vor 8 Monaten

She literally broke down how to run evals in Claude Code (built the whole thing live): 01:34 - What people get wrong with evals 04:35 - Why product taste is the alpha now 09:28 - Building a PM agent from one prompt 19:00 - Instrumentation without writing code 22:00 - Watching traces stream in live 28:00 - Getting Claude to write your first eval 33:58 - When vibe evals work and when they don't 48:50 - The self-improving loop (this part is wild) 01:03:00 - Same-day shipping is real 01:06:00 - The context graph unlock
1:19:31

She literally broke down how to run evals in Claude Code (built the whole thing live): 01:34 - What people get wrong with evals 04:35 - Why product taste is the alpha now 09:28 - Building a PM agent from one prompt 19:00 - Instrumentation without writing code 22:00 - Watching traces stream in live 28:00 - Getting Claude to write your first eval 33:58 - When vibe evals work and when they don't 48:50 - The self-improving loop (this part is wild) 01:03:00 - Same-day shipping is real 01:06:00 - The context graph unlock

Aakash Gupta

150,362 Aufrufe • vor 3 Monaten

Before founding Vals AI, Rayan Krishnan was a college radio host. We're thrilled to get him back in his element with Vivek Gopalan, this time talking all things benchmarks and evals. As AI capabilities compound, measuring them is becoming both harder and more important. What started as an academic exercise is quickly evolving into critical infrastructure for understanding what AI systems can actually do, and how they perform in the real world. Rayan and Vivek go deep on how benchmarks are evolving, measuring increasingly agentic systems, the weird and wonderful ways we test intelligence, the global AI race, and why evals sit on the critical path to ASI. 0:44 Importance and evolution of benchmarks 8:16 From academic artifact → real-world value 10:57 The next generation of benchmarks 13:34 Benchmarking models vs. agents 16:20 Building benchmarks for organizations 22:09 Weird and wonderful benchmarks 25:58 The AI arms race: US vs. China 31:02 Why evals are on the critical path to ASI 32:50 Vals Series A
34:33

Before founding Vals AI, Rayan Krishnan was a college radio host. We're thrilled to get him back in his element with Vivek Gopalan, this time talking all things benchmarks and evals. As AI capabilities compound, measuring them is becoming both harder and more important. What started as an academic exercise is quickly evolving into critical infrastructure for understanding what AI systems can actually do, and how they perform in the real world. Rayan and Vivek go deep on how benchmarks are evolving, measuring increasingly agentic systems, the weird and wonderful ways we test intelligence, the global AI race, and why evals sit on the critical path to ASI. 0:44 Importance and evolution of benchmarks 8:16 From academic artifact → real-world value 10:57 The next generation of benchmarks 13:34 Benchmarking models vs. agents 16:20 Building benchmarks for organizations 22:09 Weird and wonderful benchmarks 25:58 The AI arms race: US vs. China 31:02 Why evals are on the critical path to ASI 32:50 Vals Series A

8VC

103,787 Aufrufe • vor 13 Tagen

There are a few AI tricks I've learned that I've categorized as "life changing" and this is one of them: Hamel Husain teaching me how to start doing evals right, in the most simple, obvious, high-impact way. This episode of How I AI is for anyone still intimidated by the concept of evals. Hamel shows us: - How to "just look" at your model outputs - Binary system of quality categorization - What a good DYI eval platform looks like - Where little tweaks make a big difference I've been building an AI product for almost 2 years, and Hamel has been the first person to make evals accessible, understandable, and easy to execute. As I said: LIFE CHANGING As always, the biggest thanks to our amazing sponsors: 🫶 GoFundMe Giving Funds - One account. Zero hassle 🪪 Persona - Trusted identity verification for any use case
1:05

There are a few AI tricks I've learned that I've categorized as "life changing" and this is one of them: Hamel Husain teaching me how to start doing evals right, in the most simple, obvious, high-impact way. This episode of How I AI is for anyone still intimidated by the concept of evals. Hamel shows us: - How to "just look" at your model outputs - Binary system of quality categorization - What a good DYI eval platform looks like - Where little tweaks make a big difference I've been building an AI product for almost 2 years, and Hamel has been the first person to make evals accessible, understandable, and easy to execute. As I said: LIFE CHANGING As always, the biggest thanks to our amazing sponsors: 🫶 GoFundMe Giving Funds - One account. Zero hassle 🪪 Persona - Trusted identity verification for any use case

claire vo 🖤

47,712 Aufrufe • vor 10 Monaten

This guy literally broke down step-by-step how to create frontier lab quality evals: 2:42 - What an eval actually is 3:29 - Offline evals vs prod 5:23 - Why old evals stopped working 7:10 - How to write one 8:29 - Why 100% means you failed 9:03 - Models have saturated Q&A 12:37 - Anthropic dropped every benchmark 13:50 - Q&A thinking vs task thinking 15:32 - Building a real eval live 18:54 - Start with the easiest case 26:39 - Haiku vs Codex, head to head 33:08 - The eval no model solves 40:41 - Google's sampling trick 44:10 - How to find the ceiling 47:08 - It's all subject matter expertise 50:16 - "Google was a dead company" 51:33 - Why the Llama team is gone
56:56

This guy literally broke down step-by-step how to create frontier lab quality evals: 2:42 - What an eval actually is 3:29 - Offline evals vs prod 5:23 - Why old evals stopped working 7:10 - How to write one 8:29 - Why 100% means you failed 9:03 - Models have saturated Q&A 12:37 - Anthropic dropped every benchmark 13:50 - Q&A thinking vs task thinking 15:32 - Building a real eval live 18:54 - Start with the easiest case 26:39 - Haiku vs Codex, head to head 33:08 - The eval no model solves 40:41 - Google's sampling trick 44:10 - How to find the ceiling 47:08 - It's all subject matter expertise 50:16 - "Google was a dead company" 51:33 - Why the Llama team is gone

Aakash Gupta

44,671 Aufrufe • vor 1 Monat

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:
2:07

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:

Andrew Ng

126,559 Aufrufe • vor 1 Jahr

Eval scores in 2026 are broken. MMLU at 91%+, GSM8K at 94%+, yet models still can't handle basic multi-step tasks. And reported scores don't even agree across model cards, papers, and platforms. We just shipped Community Evals on Hugging Face: - Benchmark datasets now host live leaderboards (MMLU-Pro, GPQA, HLE) - Scores live in model repos as versioned YAML - Anyone can submit evals to any model via PR without merging. - Verified badges for reproducible runs via Inspect AI This won't fix saturation or stop test set contamination. But it makes the game visible. What was evaluated, how, when, and by whom. Done trusting black-box leaderboards. Time to decentralize evals.
0:29

Eval scores in 2026 are broken. MMLU at 91%+, GSM8K at 94%+, yet models still can't handle basic multi-step tasks. And reported scores don't even agree across model cards, papers, and platforms. We just shipped Community Evals on Hugging Face: - Benchmark datasets now host live leaderboards (MMLU-Pro, GPQA, HLE) - Scores live in model repos as versioned YAML - Anyone can submit evals to any model via PR without merging. - Verified badges for reproducible runs via Inspect AI This won't fix saturation or stop test set contamination. But it makes the game visible. What was evaluated, how, when, and by whom. Done trusting black-box leaderboards. Time to decentralize evals.

Ben Burtenshaw

19,259 Aufrufe • vor 6 Monaten

We went from being the worst, to the best AI memory system in the world. While being students with ~no prior research experience. .... but, how do we measure "good" vs bad memory? Evals (and vibes!) here's a small video on how we think about memory.
3:09

We went from being the worst, to the best AI memory system in the world. While being students with ~no prior research experience. .... but, how do we measure "good" vs bad memory? Evals (and vibes!) here's a small video on how we think about memory.

Dhravya Shah

121,169 Aufrufe • vor 8 Monaten