Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Evals 101: a new course from Braintrust. Everything you need to know about evals, and how to do them yourself. Module one: Why are evals important? - the six most common problems developers face when shipping AI applications - why traditional software thinking doesn't apply to AI - how...

braintrust's profile picture

Braintrust

6,704 subscribers

36,347 görüntüleme • 2 ay önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:
1:12

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 görüntüleme • 9 ay önce

NEW Greylock Change Agents: Evaluating Agents with Braintrust There's been a lot of discourse around evals recently, so it's timely to drop the recording of Change Agents with Ankur Goyal on the topic of how he thinks about Evaluating Agents. Timestamps: (0:00) - Intro (1:22) - Building Braintrust (5:19) - Evaluating agentic systems (11:04) - Experimentation with evals (13:18) - Adopting evals (17:09) - Evals and complex agents (21:19) - Custom scoring (25:02) - LLM as a judge (26:44) - Human-manual evaluation (30:07) - How the best in-class teams manage evals (33:57) - Future of evals for agents (37:38 ) - Audience Q&A
57:34

NEW Greylock Change Agents: Evaluating Agents with Braintrust There's been a lot of discourse around evals recently, so it's timely to drop the recording of Change Agents with Ankur Goyal on the topic of how he thinks about Evaluating Agents. Timestamps: (0:00) - Intro (1:22) - Building Braintrust (5:19) - Evaluating agentic systems (11:04) - Experimentation with evals (13:18) - Adopting evals (17:09) - Evals and complex agents (21:19) - Custom scoring (25:02) - LLM as a judge (26:44) - Human-manual evaluation (30:07) - How the best in-class teams manage evals (33:57) - Future of evals for agents (37:38 ) - Audience Q&A

Corinne Marie Riley

12,743 görüntüleme • 9 ay önce

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC
13:01

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC

Matt Turck

10,969 görüntüleme • 9 ay önce

Let’s talk about evals. We’re always looking for better ways to measure and forecast model progress, especially as benchmarks get saturated or gamed. Tejal Patwardhan, who leads our frontier evals team, spoke to Andrew Mayne about why evals matter and what models need to be judged on next.
44:22

Let’s talk about evals. We’re always looking for better ways to measure and forecast model progress, especially as benchmarks get saturated or gamed. Tejal Patwardhan, who leads our frontier evals team, spoke to Andrew Mayne about why evals matter and what models need to be judged on next.

OpenAI

275,048 görüntüleme • 24 gün önce

"AI evals force you into the shoes of your user." I love this clip from (Director at Arize) on why evaluations are the top AI skill to build in 2025. In my new interview, Aman breaks down: - How to write evals like a pro - 5 AI product principles he swears by - How to start practicing evals now Some quotes from Aman: "All PMs will be AI PMs. You'll either build on top of LLM APIs, use models in your backend, or use AI tools to solve problems in your daily work." "The waterfall process of writing a PRD and handing it to designers and engineers is dead. Your dev cycle needs to be much shorter." "Imagine showing up to a meeting with a prototype instead of a PRD. That's the shift PMs will need to make to communicate better in this new AI world." 📌 Watch now:
1:13

"AI evals force you into the shoes of your user." I love this clip from (Director at Arize) on why evaluations are the top AI skill to build in 2025. In my new interview, Aman breaks down: - How to write evals like a pro - 5 AI product principles he swears by - How to start practicing evals now Some quotes from Aman: "All PMs will be AI PMs. You'll either build on top of LLM APIs, use models in your backend, or use AI tools to solve problems in your daily work." "The waterfall process of writing a PRD and handing it to designers and engineers is dead. Your dev cycle needs to be much shorter." "Imagine showing up to a meeting with a prototype instead of a PRD. That's the shift PMs will need to make to communicate better in this new AI world." 📌 Watch now:

Peter Yang

15,973 görüntüleme • 1 yıl önce

One AI skill will create millionaires: Evals. It's the key to great AI products. Here's a crash course with experts Hamel Husain & Shreya Shankar: 02:06 - PM v Eng Role 23:56 - Evaluating RAG 42:42 - How OpenAI does evals 01:08:29 - LLM-as-judge
1:34:32

One AI skill will create millionaires: Evals. It's the key to great AI products. Here's a crash course with experts Hamel Husain & Shreya Shankar: 02:06 - PM v Eng Role 23:56 - Evaluating RAG 42:42 - How OpenAI does evals 01:08:29 - LLM-as-judge

Aakash Gupta

42,185 görüntüleme • 1 yıl önce

How do we understand & evaluate the fuzzy space of LLM outputs? We clone your Subject Matter Expert annotator into a Judge. Introducing EVALS EVALS EVALS Create a custom Judge that works for you
1:26

How do we understand & evaluate the fuzzy space of LLM outputs? We clone your Subject Matter Expert annotator into a Judge. Introducing EVALS EVALS EVALS Create a custom Judge that works for you

Leonard Tang

36,496 görüntüleme • 1 yıl önce

Hosted evals are finally live!! Smal vid showing how to use them, more to come : )
3:59

Hosted evals are finally live!! Smal vid showing how to use them, more to come : )

Florian Brand

32,593 görüntüleme • 1 ay önce

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.
52:06

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.

Aakash Gupta

483,558 görüntüleme • 3 ay önce

We now run evals to help you know what AI models perform best to write Nuxt code.
0:21

We now run evals to help you know what AI models perform best to write Nuxt code.

Nuxt

28,019 görüntüleme • 4 ay önce

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!
1:15

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!

AI Engineer

55,348 görüntüleme • 1 yıl önce

I got an expert to give me a masterclass on AI evals: (63 mins of stuff most builders learn the hard way) 0:00 - Why your product is lying to you 22:03 - Full eval flow walkthrough 35:42 - Evals ARE your PRD (not QA) 57:52 - Failures at Grammarly, Copilot & Klarna
1:03:59

I got an expert to give me a masterclass on AI evals: (63 mins of stuff most builders learn the hard way) 0:00 - Why your product is lying to you 22:03 - Full eval flow walkthrough 35:42 - Evals ARE your PRD (not QA) 57:52 - Failures at Grammarly, Copilot & Klarna

Aakash Gupta

247,208 görüntüleme • 4 ay önce

Announcing skillgrade - the easiest way to evaluate your agent skills All you need is two commands: skillgrade init # create evals skillgrade # run them By default evals run in a safe sandboxed docker container
0:56

Announcing skillgrade - the easiest way to evaluate your agent skills All you need is two commands: skillgrade init # create evals skillgrade # run them By default evals run in a safe sandboxed docker container

Minko Gechev

45,051 görüntüleme • 3 ay önce

I like to think of evals as something active, not passive -- it's a North Star that steers LLMs toward higher intelligence. Evals should drive your RL/SFT/post-training decisions. Internal evals at frontier labs make a huge difference -- and you can see it in how models behave differently (GPT seems better at one-shot tasks, Claude at multi-turn). If you want to learn more about building evals that actually improve your model in post-training, check out our AMD x DeeplearningAI course "Fine-tuning & RL for LLMs: Intro to Post-training" (content is free):
0:30

I like to think of evals as something active, not passive -- it's a North Star that steers LLMs toward higher intelligence. Evals should drive your RL/SFT/post-training decisions. Internal evals at frontier labs make a huge difference -- and you can see it in how models behave differently (GPT seems better at one-shot tasks, Claude at multi-turn). If you want to learn more about building evals that actually improve your model in post-training, check out our AMD x DeeplearningAI course "Fine-tuning & RL for LLMs: Intro to Post-training" (content is free):

Sharon Zhou

16,613 görüntüleme • 5 ay önce

Andon Labs' Real-World AI Evals: Claude calls the FBI, AI CEOs, price cartels, Butter-Bench, & Luna Andon Labs cofounders Lukas Petersson and Axel Backlund explain why dollar-denominated evals reveal what traditional benchmarks miss, how Claude ended up reporting a $2/day vending machine fee to the FBI, why long-horizon agents spiral in weird ways, what happens when agents lie, form price cartels, and compete with each other, and why the future of AI safety may depend on testing models in messy real-world environments instead of clean benchmark sandboxes.
0:23

Andon Labs' Real-World AI Evals: Claude calls the FBI, AI CEOs, price cartels, Butter-Bench, & Luna Andon Labs cofounders Lukas Petersson and Axel Backlund explain why dollar-denominated evals reveal what traditional benchmarks miss, how Claude ended up reporting a $2/day vending machine fee to the FBI, why long-horizon agents spiral in weird ways, what happens when agents lie, form price cartels, and compete with each other, and why the future of AI safety may depend on testing models in messy real-world environments instead of clean benchmark sandboxes.

Latent.Space

13,294 görüntüleme • 1 ay önce

✅ Shared Human+AI governance ✅ End-to-end audit systems ✅ Granular access controls See how Palantir’s AIP Evals takes AI from prototype to production - where trust, compliance, and impact are non-negotiable.
32:12

✅ Shared Human+AI governance ✅ End-to-end audit systems ✅ Granular access controls See how Palantir’s AIP Evals takes AI from prototype to production - where trust, compliance, and impact are non-negotiable.

Palantir

50,441 görüntüleme • 1 yıl önce

AI PM jobs pay $250K+ in the US, $100K+ in Europe, 60+ LPA in India. But how do you land one? I got the $2.5B CEO (Todd Olson) of PM platform pendo.io to share exactly how: 6:07 - How to Upskill to AI PM 12:54 - The 5-Layer Technical Pyramid 48:56 - Why PM's Must Own Evals
1:21:53

AI PM jobs pay $250K+ in the US, $100K+ in Europe, 60+ LPA in India. But how do you land one? I got the $2.5B CEO (Todd Olson) of PM platform pendo.io to share exactly how: 6:07 - How to Upskill to AI PM 12:54 - The 5-Layer Technical Pyramid 48:56 - Why PM's Must Own Evals

Aakash Gupta

23,986 görüntüleme • 7 ay önce

Feedback 🔁 Fix See how Palantir’s AIP Evals power high-fidelity feedback loops with AI agents, enabling human operators to encode their acumen and deliver compounding value for the enterprise.
6:22

Feedback 🔁 Fix See how Palantir’s AIP Evals power high-fidelity feedback loops with AI agents, enabling human operators to encode their acumen and deliver compounding value for the enterprise.

Palantir

31,310 görüntüleme • 11 ay önce

Create datasets, run evals, and even train models directly in Cursor with the Hugging Face plugin. Here's Ben Burtenshaw to show you how:
1:53

Create datasets, run evals, and even train models directly in Cursor with the Hugging Face plugin. Here's Ben Burtenshaw to show you how:

edwin

17,119 görüntüleme • 4 ay önce