Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Evals are hard. Good eval research should be accessible. At Braintrust, we have unique access to how leading AI teams think about quality. We study the best existing research and conduct our own experiments. Today we're making this work available in one place, with an open source library of... show more

Braintrust

7,535 subscribers

23,243 views • 17 days ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Evals 101: a new course from Braintrust. Everything you need to know about evals, and how to do them yourself. Module one: Why are evals important? - the six most common problems developers face when shipping AI applications - why traditional software thinking doesn't apply to AI - how evals can fix these problems

Evals 101: a new course from Braintrust. Everything you need to know about evals, and how to do them yourself. Module one: Why are evals important? - the six most common problems developers face when shipping AI applications - why traditional software thinking doesn't apply to AI - how evals can fix these problems

Braintrust

36,347 views • 4 months ago

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.

Aakash Gupta

483,558 views • 5 months ago

Braintrust has raised an $80M Series B. We're building the infrastructure that helps teams measure, evaluate, and improve their AI products. Don't take our word for it. Hear how Notion, @Vercel, Navan, and BILL use Braintrust to ship quality AI.

Braintrust has raised an $80M Series B. We're building the infrastructure that helps teams measure, evaluate, and improve their AI products. Don't take our word for it. Hear how Notion, @Vercel, Navan, and BILL use Braintrust to ship quality AI.

Braintrust

130,349 views • 6 months ago

We went from being the worst, to the best AI memory system in the world. While being students with ~no prior research experience. .... but, how do we measure "good" vs bad memory? Evals (and vibes!) here's a small video on how we think about memory.

We went from being the worst, to the best AI memory system in the world. While being students with ~no prior research experience. .... but, how do we measure "good" vs bad memory? Evals (and vibes!) here's a small video on how we think about memory.

Dhravya Shah

121,169 views • 8 months ago

Today we're announcing Base Labs, a dedicated research organization focused on advancing open-source AI. We believe in a healthy, open frontier model ecosystem. To enable this, we are working on: - Blue-sky research on continual learning, the science of RL, and how models learn across their full lifecycle, with every experiment and recipe shared openly. - The BaseHub Data Foundry: the highest-quality open RL environments, training data, and real-world benchmarks, built for anyone to train and benchmark on. - Post-post training: taking open-source models and making them better, safer, and more aligned through continual post-training, built on our research, and deployed with our frontier safety stack so organizations can use open-source models with confidence. - Making models cheaper and more performant through our model performance research. This is a mission-driven research effort, not a commercial product. We believe the health of the open-source AI ecosystem matters and that the best way to advance it is to do science in the open. We’re hiring engineers, researchers, and research fellows to advance this mission.

Today we're announcing Base Labs, a dedicated research organization focused on advancing open-source AI. We believe in a healthy, open frontier model ecosystem. To enable this, we are working on: - Blue-sky research on continual learning, the science of RL, and how models learn across their full lifecycle, with every experiment and recipe shared openly. - The BaseHub Data Foundry: the highest-quality open RL environments, training data, and real-world benchmarks, built for anyone to train and benchmark on. - Post-post training: taking open-source models and making them better, safer, and more aligned through continual post-training, built on our research, and deployed with our frontier safety stack so organizations can use open-source models with confidence. - Making models cheaper and more performant through our model performance research. This is a mission-driven research effort, not a commercial product. We believe the health of the open-source AI ecosystem matters and that the best way to advance it is to do science in the open. We’re hiring engineers, researchers, and research fellows to advance this mission.

Base Labs

194,762 views • 1 day ago

An $800M company exists because evals were so broken that the founder had to build the same internal tool twice, at two different companies, before anyone would pay for it. First at his own startup. Then again leading the AI team at Figma. Same problem both times: teams shipping AI features had no structured way to know if the outputs were getting better or worse. They were vibe-checking. Manually reading outputs. Guessing. > That's how BrainTrust started. And now Vercel, Replit, Ramp, Zapier, Notion, and Airtable all use it. The number that reframes this: the companies whose AI products actually work are running 12.8 eval experiments per day. Think about that cadence. Most AI teams I talk to aren't running 12.8 per month. The framework is simpler than people expect. Every eval is three things: a set of inputs your product handles, a task that generates outputs, and a scoring function that produces a number between 0 and 1. In this episode, we built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes. Evals are becoming the new PRD. The PMs who build eval infrastructure now are going to compound product quality in a way that PMs who keep vibe-checking simply cannot match. The gap is already opening.

An $800M company exists because evals were so broken that the founder had to build the same internal tool twice, at two different companies, before anyone would pay for it. First at his own startup. Then again leading the AI team at Figma. Same problem both times: teams shipping AI features had no structured way to know if the outputs were getting better or worse. They were vibe-checking. Manually reading outputs. Guessing. > That's how BrainTrust started. And now Vercel, Replit, Ramp, Zapier, Notion, and Airtable all use it. The number that reframes this: the companies whose AI products actually work are running 12.8 eval experiments per day. Think about that cadence. Most AI teams I talk to aren't running 12.8 per month. The framework is simpler than people expect. Every eval is three things: a set of inputs your product handles, a task that generates outputs, and a scoring function that produces a number between 0 and 1. In this episode, we built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes. Evals are becoming the new PRD. The PMs who build eval infrastructure now are going to compound product quality in a way that PMs who keep vibe-checking simply cannot match. The gap is already opening.

Aakash Gupta

21,624 views • 5 months ago

$We partnered with Harvey to train a model for Review Table, achieving state-of-the-art accuracy at a fraction of the cost and latency of frontier alternatives. Review Table is one of Harvey’s highest inference volume products. Training an open source model for production traffic required building a representative eval and understanding how real lawyers use the product. Our teams discuss the research process.$

We partnered with Harvey to train a model for Review Table, achieving state-of-the-art accuracy at a fraction of the cost and latency of frontier alternatives. Review Table is one of Harvey’s highest inference volume products. Training an open source model for production traffic required building a representative eval and understanding how real lawyers use the product. Our teams discuss the research process.

Applied Compute

33,491 views • 21 days ago

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 views • 11 months ago

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.

Respan

127,680 views • 4 months ago

Ivanka Ivanka Trump and Elad Gil are working on a project that uses Al to translate the world's great public-domain books into every major language, making them accessible for free to anyone: “What are some of the positive use cases for AI? And we started talking about how so much of history's great works of information and literature are not accessible to so many people due to lack of access. AI has gotten so good that we could create high-fidelity translations of these incredible literary works. So you think about Dostoevsky, you think about Bronte, you think about Marcus Aurelius, or Epictetus. All of these works are available in the public domain. We can use AI to translate them into all the world's commonly spoken languages and make them accessible and available for free if you have internet access. So we're democratizing access to this incredible knowledge. We're calling it Alexandria Library”

Ivanka Ivanka Trump and Elad Gil are working on a project that uses Al to translate the world's great public-domain books into every major language, making them accessible for free to anyone: “What are some of the positive use cases for AI? And we started talking about how so much of history's great works of information and literature are not accessible to so many people due to lack of access. AI has gotten so good that we could create high-fidelity translations of these incredible literary works. So you think about Dostoevsky, you think about Bronte, you think about Marcus Aurelius, or Epictetus. All of these works are available in the public domain. We can use AI to translate them into all the world's commonly spoken languages and make them accessible and available for free if you have internet access. So we're democratizing access to this incredible knowledge. We're calling it Alexandria Library”

David Senra

663,406 views • 3 months ago

There are a few AI tricks I've learned that I've categorized as "life changing" and this is one of them: Hamel Husain teaching me how to start doing evals right, in the most simple, obvious, high-impact way. This episode of How I AI is for anyone still intimidated by the concept of evals. Hamel shows us: - How to "just look" at your model outputs - Binary system of quality categorization - What a good DYI eval platform looks like - Where little tweaks make a big difference I've been building an AI product for almost 2 years, and Hamel has been the first person to make evals accessible, understandable, and easy to execute. As I said: LIFE CHANGING As always, the biggest thanks to our amazing sponsors: 🫶 GoFundMe Giving Funds - One account. Zero hassle 🪪 Persona - Trusted identity verification for any use case

There are a few AI tricks I've learned that I've categorized as "life changing" and this is one of them: Hamel Husain teaching me how to start doing evals right, in the most simple, obvious, high-impact way. This episode of How I AI is for anyone still intimidated by the concept of evals. Hamel shows us: - How to "just look" at your model outputs - Binary system of quality categorization - What a good DYI eval platform looks like - Where little tweaks make a big difference I've been building an AI product for almost 2 years, and Hamel has been the first person to make evals accessible, understandable, and easy to execute. As I said: LIFE CHANGING As always, the biggest thanks to our amazing sponsors: 🫶 GoFundMe Giving Funds - One account. Zero hassle 🪪 Persona - Trusted identity verification for any use case

claire vo 🖤

47,712 views • 10 months ago

🔥 Pokémon Red is becoming a go-to benchmark for testing advanced AIs such as Gemini. But is Pokémon Red really a good eval? We study this problem and identify three issues: 1️⃣ Navigation tasks are too hard. 2️⃣ Combat control is too simple. 3️⃣ Raising a strong Pokémon team is slow and expensive as an eval. We find most of the problems are not fundamental to games themselves, but how they have been used. We believe game-as-an-eval remains a compelling and underutilized evaluation strategy. We introduce Lmgame Bench to standardize game-as-an-eval. More details and findings in our blogpost:

🔥 Pokémon Red is becoming a go-to benchmark for testing advanced AIs such as Gemini. But is Pokémon Red really a good eval? We study this problem and identify three issues: 1️⃣ Navigation tasks are too hard. 2️⃣ Combat control is too simple. 3️⃣ Raising a strong Pokémon team is slow and expensive as an eval. We find most of the problems are not fundamental to games themselves, but how they have been used. We believe game-as-an-eval remains a compelling and underutilized evaluation strategy. We introduce Lmgame Bench to standardize game-as-an-eval. More details and findings in our blogpost:

Hao AI Lab

69,037 views • 1 year ago

"One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals. And you look at the evals and you go, 'Those are pretty hard evals.' But the economic impact seems to be dramatically behind. There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data. When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"

"One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals. And you look at the evals and you go, 'Those are pretty hard evals.' But the economic impact seems to be dramatically behind. There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data. When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"

Dwarkesh Patel

502,484 views • 9 months ago

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!

AI Engineer

55,348 views • 1 year ago

Are you interested in peptides? Have you heard about Elite Research? We have the highest quality, third party laboratory tested compounds For the best prices With fast, safe, secure, discreet shipping Visit our website to browse our large selection of research compounds Use the link below with code YUKON10 at checkout to save 10% off your entire order

Are you interested in peptides? Have you heard about Elite Research? We have the highest quality, third party laboratory tested compounds For the best prices With fast, safe, secure, discreet shipping Visit our website to browse our large selection of research compounds Use the link below with code YUKON10 at checkout to save 10% off your entire order

BowTiedYukon

109,241 views • 1 year ago

We're launching the microagi Research Fellowship. Fellows get up to $2M in compute, robotics hardware, our evals, and one of the largest physical AI datasets ever assembled. You build in our lab, with our team, alongside partners like Unitree, Nvidia, and Google Cloud. The hard part of AI left is physical. That's the part we're working on. Come build with us. One more thing: know someone who belongs here? Reply with their name. If they get in, we send you 10.000 USD

We're launching the microagi Research Fellowship. Fellows get up to $2M in compute, robotics hardware, our evals, and one of the largest physical AI datasets ever assembled. You build in our lab, with our team, alongside partners like Unitree, Nvidia, and Google Cloud. The hard part of AI left is physical. That's the part we're working on. Come build with us. One more thing: know someone who belongs here? Reply with their name. If they get in, we send you 10.000 USD

Bercan

175,427 views • 3 months ago

Earlier this week at GTC, we announced our partnership with Nvidia. We will work with Nvidia to build strong, American open-source models that are at the frontier of scientific reasoning. These models will be essential for the US to compete with China on science in the coming decades. Jensen is committing to spend tens of billions of dollars developing open-source models, and we are excited to be a partner with them in figuring out how to benchmark, train and use those agents to accelerate scientific research. We have already open-sourced some of the work we have done with them, and are looking forward to open-sourcing more. There are few things today that are more important. See our blog post below, and watch the video to learn more, narrated by the man himself.

Earlier this week at GTC, we announced our partnership with Nvidia. We will work with Nvidia to build strong, American open-source models that are at the frontier of scientific reasoning. These models will be essential for the US to compete with China on science in the coming decades. Jensen is committing to spend tens of billions of dollars developing open-source models, and we are excited to be a partner with them in figuring out how to benchmark, train and use those agents to accelerate scientific research. We have already open-sourced some of the work we have done with them, and are looking forward to open-sourcing more. There are few things today that are more important. See our blog post below, and watch the video to learn more, narrated by the man himself.

Sam Rodriques

23,763 views • 5 months ago

I'm thrilled to announce the launch of ⚡️Flash Diffusion from Jasper! Earlier this year, with our acquisition of Clipdrop, we launched the Jasper AI Research Lab in Paris. Today, we are excited to release our first piece of groundbreaking research: the open-source distillation method, "Flash Diffusion". Flash Diffusion accelerates inference by 500%, reduces computing costs, and produces higher-quality image outputs. Dive into the details and discover how Flash Diffusion is set to revolutionize the field of AI and image synthesis. Read all about it here: Try a demo on Hugging Face:

I'm thrilled to announce the launch of ⚡️Flash Diffusion from Jasper! Earlier this year, with our acquisition of Clipdrop, we launched the Jasper AI Research Lab in Paris. Today, we are excited to release our first piece of groundbreaking research: the open-source distillation method, "Flash Diffusion". Flash Diffusion accelerates inference by 500%, reduces computing costs, and produces higher-quality image outputs. Dive into the details and discover how Flash Diffusion is set to revolutionize the field of AI and image synthesis. Read all about it here: Try a demo on Hugging Face:

Timothy Young

10,093 views • 2 years ago

In this episode, OpenAI Chief Research Officer Mark Chen joins allen to flambé shrimp, cook Korean stew, and chat about being at the frontier of AI research: why scaling laws and pre-training still matter, how OpenAI chooses research bets and allocates compute, what it means to develop research taste, why evals are in crisis, how to avoid benchmark-maxing, and what it will take for models to handle long-horizon real-world work, multimodal reasoning, and eventually end-to-end AI research. Timestamps: 0:00 Intro 0:28 The Soup Story 1:52 From Trading to AI Research 3:21 How to Develop Research Taste 5:23 RL, Evals, and Superhuman Benchmarks 8:17 Cooking Begins on the Impulse Stove 8:53 Scaling Laws, Pre-Training, and Reasoning 12:33 OpenAI’s Research Roadmap and Compute Allocation 15:48 What Makes a Great Researcher 19:33 The Evals Crisis and Benchmark-Maxing 24:34 Jagged Intelligence, Context, and Long-Horizon Learning 27:14 Shrimp Flambé and New Research Bets 31:32 Multimodal Models and One Architecture 32:36 Vibe Researching and End-to-End AI Research 34:36 Failed Bets, Postmortems, and OpenAI’s Alpha 37:07 Final Taste Test 37:53 Overrated vs. Underrated AI Research 41:00 Closing

In this episode, OpenAI Chief Research Officer Mark Chen joins allen to flambé shrimp, cook Korean stew, and chat about being at the frontier of AI research: why scaling laws and pre-training still matter, how OpenAI chooses research bets and allocates compute, what it means to develop research taste, why evals are in crisis, how to avoid benchmark-maxing, and what it will take for models to handle long-horizon real-world work, multimodal reasoning, and eventually end-to-end AI research. Timestamps: 0:00 Intro 0:28 The Soup Story 1:52 From Trading to AI Research 3:21 How to Develop Research Taste 5:23 RL, Evals, and Superhuman Benchmarks 8:17 Cooking Begins on the Impulse Stove 8:53 Scaling Laws, Pre-Training, and Reasoning 12:33 OpenAI’s Research Roadmap and Compute Allocation 15:48 What Makes a Great Researcher 19:33 The Evals Crisis and Benchmark-Maxing 24:34 Jagged Intelligence, Context, and Long-Horizon Learning 27:14 Shrimp Flambé and New Research Bets 31:32 Multimodal Models and One Architecture 32:36 Vibe Researching and End-to-End AI Research 34:36 Failed Bets, Postmortems, and OpenAI’s Alpha 37:07 Final Taste Test 37:53 Overrated vs. Underrated AI Research 41:00 Closing

Latent.Space

62,796 views • 2 months ago

World Foundation Models (WFM) are the future of Physical AI, and we believe this future should be open-sourced. We are partnering with Valeo Group to build one of the largest open-source, multi-camera WFMs — redefining how Physical AI interacts with the real world.

World Foundation Models (WFM) are the future of Physical AI, and we believe this future should be open-sourced. We are partnering with Valeo Group to build one of the largest open-source, multi-camera WFMs — redefining how Physical AI interacts with the real world.

NATIX Network

261,597 views • 7 months ago