Loading video...

Video Failed to Load

Go Home

There are a few AI tricks I've learned that I've categorized as "life changing" and this is one of them: Hamel Husain teaching me how to start doing evals right, in the most simple, obvious, high-impact way. This episode of How I AI is for anyone still intimidated by...

clairevo's profile picture

claire vo 🖤

37,547 subscribers

47,712 views • 9 months ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:
1:12

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 views • 10 months ago

I have been hyping Brian Scanlan and Intercom to every eng leader I've spoken to since getting a behind the scenes look at how they 2x'd their productivity in their R&D org. In this week's episode of How I AI, Brian pulls up dashboards and tools and shows us how it's done: - explicit targets + reporting on leading indicators of AI adoption - how they measured how quality increased with more AI usage - centralized skills repo for the entire company - telemetry on claude code + evals on internal success - building for an agent-first onboarding flow Sorry to be that bro, but truly: STOP what you are doing and watch this right now if you are looking to inflect your eng + product team with AI. As always, a huge thanks to our sponsors: 🧠Celigo - Intelligent automation built for AI: Cursor - The best (and my fave!) way to code with AI: On YT now 👉
1:03

I have been hyping Brian Scanlan and Intercom to every eng leader I've spoken to since getting a behind the scenes look at how they 2x'd their productivity in their R&D org. In this week's episode of How I AI, Brian pulls up dashboards and tools and shows us how it's done: - explicit targets + reporting on leading indicators of AI adoption - how they measured how quality increased with more AI usage - centralized skills repo for the entire company - telemetry on claude code + evals on internal success - building for an agent-first onboarding flow Sorry to be that bro, but truly: STOP what you are doing and watch this right now if you are looking to inflect your eng + product team with AI. As always, a huge thanks to our sponsors: 🧠Celigo - Intelligent automation built for AI: Cursor - The best (and my fave!) way to code with AI: On YT now 👉

claire vo 🖤

12,834 views • 3 months ago

Episode 5 of How I AI with some practical AI hacks for managers from hilary gridley, Head of Core Product at WHOOP. This one is all about feedback: how to scale it, how to make it specific, how to give it to AI tools. You’ll learn how to: - build rubrics for what “good” looks like - create GPTs to “think like you” and use them to scale quality feedback to your team - use prompting techniques to improve the insights you get from AI - improve your quality of writing and strategic thinking by bringing it all together Plus we gab about getting more women in AI, girly use cases for chatGPT, and whether or not getting invited to meetings is a good thing. Check it out in the links below 👇🔗 As always, a big thanks to our sponsors: 🔀 Orkes enterprise platform for genetic workflows ☑️ Vanta automate compliance and simplify security
0:41

Episode 5 of How I AI with some practical AI hacks for managers from hilary gridley, Head of Core Product at WHOOP. This one is all about feedback: how to scale it, how to make it specific, how to give it to AI tools. You’ll learn how to: - build rubrics for what “good” looks like - create GPTs to “think like you” and use them to scale quality feedback to your team - use prompting techniques to improve the insights you get from AI - improve your quality of writing and strategic thinking by bringing it all together Plus we gab about getting more women in AI, girly use cases for chatGPT, and whether or not getting invited to meetings is a good thing. Check it out in the links below 👇🔗 As always, a big thanks to our sponsors: 🔀 Orkes enterprise platform for genetic workflows ☑️ Vanta automate compliance and simplify security

claire vo 🖤

19,127 views • 1 year ago

Tired: Writing PRDs by hand Wired: Writing PRDs with AI Inspired: letting AI reply to all PRD comments left by your coworkers Dennis Yang takes AI product management to the nth degree in this week's episode of How I AI, showing us how he: - uses Cursor for ALL his product work (yes, including commenting back to his team) - maxes out the notion and jira MCPs to automate work across his team's source of truth - prototypes AI products using cursor as the foundation This one is a 11 out of 10 for any PM trying to take their AI use to the next level and inspired me in a bunch of ways. And yes, his colleagues know all about it 😎 As always, a BIG thanks to our wonderful sponsors: ⚡️ Zapier - the most connected AI orchestration platform 🤑 Brex - the intelligent finance platform built for founders
1:03

Tired: Writing PRDs by hand Wired: Writing PRDs with AI Inspired: letting AI reply to all PRD comments left by your coworkers Dennis Yang takes AI product management to the nth degree in this week's episode of How I AI, showing us how he: - uses Cursor for ALL his product work (yes, including commenting back to his team) - maxes out the notion and jira MCPs to automate work across his team's source of truth - prototypes AI products using cursor as the foundation This one is a 11 out of 10 for any PM trying to take their AI use to the next level and inspired me in a bunch of ways. And yes, his colleagues know all about it 😎 As always, a BIG thanks to our wonderful sponsors: ⚡️ Zapier - the most connected AI orchestration platform 🤑 Brex - the intelligent finance platform built for founders

claire vo 🖤

45,455 views • 9 months ago

(Episode 2) I'm building an app to make $10,000/Month and I'll record everything... This episode I start building the AI Portrait app only to learn that none of my friends want it, but I still believe... Link to first episode: Why this series? I want to document my WHOLE journey building an app, marketing it, and making money off of it. Is it really that easy? How hard is it? I've been seeing so many X users online talking about how easy it is to make money online with AI now. I want to test it and show people the reality of making a business online.
10:44

(Episode 2) I'm building an app to make $10,000/Month and I'll record everything... This episode I start building the AI Portrait app only to learn that none of my friends want it, but I still believe... Link to first episode: Why this series? I want to document my WHOLE journey building an app, marketing it, and making money off of it. Is it really that easy? How hard is it? I've been seeing so many X users online talking about how easy it is to make money online with AI now. I want to test it and show people the reality of making a business online.

Joma Tech

42,176 views • 1 day ago

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:
2:07

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:

Andrew Ng

126,462 views • 1 year ago

"AI evals force you into the shoes of your user." I love this clip from (Director at Arize) on why evaluations are the top AI skill to build in 2025. In my new interview, Aman breaks down: - How to write evals like a pro - 5 AI product principles he swears by - How to start practicing evals now Some quotes from Aman: "All PMs will be AI PMs. You'll either build on top of LLM APIs, use models in your backend, or use AI tools to solve problems in your daily work." "The waterfall process of writing a PRD and handing it to designers and engineers is dead. Your dev cycle needs to be much shorter." "Imagine showing up to a meeting with a prototype instead of a PRD. That's the shift PMs will need to make to communicate better in this new AI world." 📌 Watch now:
1:13

"AI evals force you into the shoes of your user." I love this clip from (Director at Arize) on why evaluations are the top AI skill to build in 2025. In my new interview, Aman breaks down: - How to write evals like a pro - 5 AI product principles he swears by - How to start practicing evals now Some quotes from Aman: "All PMs will be AI PMs. You'll either build on top of LLM APIs, use models in your backend, or use AI tools to solve problems in your daily work." "The waterfall process of writing a PRD and handing it to designers and engineers is dead. Your dev cycle needs to be much shorter." "Imagine showing up to a meeting with a prototype instead of a PRD. That's the shift PMs will need to make to communicate better in this new AI world." 📌 Watch now:

Peter Yang

15,973 views • 1 year ago

"More context is not always better for AI." Adam (Meta) has helped 100+ companies build AI products. In my new interview, he shares a complete beginner-friendly AI course on: - 2 AI optimizations every PM should know - 4 types of evals and why points beat ratings - How to avoid RAG making your AI worse Some quotes from Adam Loving: "My eval tip is instead of asking an LLM to rate something 1-10, use a point system..." "Instead of a one giant prompt, use a series of prompts and run this each step of the way..." "RAG can actually make your product worse, here's how to avoid this..." 📌 Watch now:
0:48

"More context is not always better for AI." Adam (Meta) has helped 100+ companies build AI products. In my new interview, he shares a complete beginner-friendly AI course on: - 2 AI optimizations every PM should know - 4 types of evals and why points beat ratings - How to avoid RAG making your AI worse Some quotes from Adam Loving: "My eval tip is instead of asking an LLM to rate something 1-10, use a point system..." "Instead of a one giant prompt, use a series of prompts and run this each step of the way..." "RAG can actually make your product worse, here's how to avoid this..." 📌 Watch now:

Peter Yang

38,066 views • 1 year ago

Episode 10 of How I AI is my personal favorite so far: we talk to John Blackman, 91 years old, about using AI to code a web app for the first time in his life. John and his grandson Brandon show us: - Claude for writing user stories - Replit agents for coding a HUGE volunteer support app - the challenges of going from dev to production - user feedback for Anthropic and Replit ⠕ as a beginner If you’ve been sitting on the sidelines, this is your call to action! No excuses, at any age. As always, thanks to our fab sponsors! 💼 WorkOS: make your app enterprise ready today 🔀 Orkes: the platform for reliable apps and agentic workflows
1:17

Episode 10 of How I AI is my personal favorite so far: we talk to John Blackman, 91 years old, about using AI to code a web app for the first time in his life. John and his grandson Brandon show us: - Claude for writing user stories - Replit agents for coding a HUGE volunteer support app - the challenges of going from dev to production - user feedback for Anthropic and Replit ⠕ as a beginner If you’ve been sitting on the sidelines, this is your call to action! No excuses, at any age. As always, thanks to our fab sponsors! 💼 WorkOS: make your app enterprise ready today 🔀 Orkes: the platform for reliable apps and agentic workflows

claire vo 🖤

138,420 views • 1 year ago

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!
1:15

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!

AI Engineer

55,348 views • 1 year ago

Episode 5 of How I AI is how I learned I was using Cursor all wrong. My friend Ryan Carson shows us the way with his genius 3-step AI coding workflow. If you’re trying to build bigger projects (or even whole companies!) this one is for you. You’ll learn how to: - use 3 cursorrules to bring structure - generate PRDs and task lists - make good use of context windows - build as a solo founder Ryan is not only using AI IDEs better than most people I know, he’s generous in open sourcing his work and sharing insights online. I learned a lot of practical things and you will too. Check it out in the links below 👇🔗 As always, a big thanks to our sponsors: 🤖 Notion the best AI tools for work 💕 ChatPRD AI copilot for PMs and their teams
0:51

Episode 5 of How I AI is how I learned I was using Cursor all wrong. My friend Ryan Carson shows us the way with his genius 3-step AI coding workflow. If you’re trying to build bigger projects (or even whole companies!) this one is for you. You’ll learn how to: - use 3 cursorrules to bring structure - generate PRDs and task lists - make good use of context windows - build as a solo founder Ryan is not only using AI IDEs better than most people I know, he’s generous in open sourcing his work and sharing insights online. I learned a lot of practical things and you will too. Check it out in the links below 👇🔗 As always, a big thanks to our sponsors: 🤖 Notion the best AI tools for work 💕 ChatPRD AI copilot for PMs and their teams

claire vo 🖤

208,356 views • 1 year ago

A highlight at the Sohn Conference Foundation this week for me was one of the best investors I've met (Daniel Gross) interviewing one of the leading AI researchers (Eric Steinberger) for 20 minutes. They discuss the path to AGI through first building AI software engineer bots and how there will be a step function change once an AI model becomes 99% reliable and intelligence is suddenly abundant for the first time. Recommend as a high signal guide to what's coming.
20:11

A highlight at the Sohn Conference Foundation this week for me was one of the best investors I've met (Daniel Gross) interviewing one of the leading AI researchers (Eric Steinberger) for 20 minutes. They discuss the path to AGI through first building AI software engineer bots and how there will be a step function change once an AI model becomes 99% reliable and intelligence is suddenly abundant for the first time. Recommend as a high signal guide to what's coming.

Graham Duncan

83,512 views • 2 years ago

.Scott Wu to Joe Lonsdale on why Cognition uses AI in job interviews: "One or two years ago, people had this mentality of 'How do we make sure people aren't using AI while we're interviewing them?' And I think that has totally flipped. Honestly, I think that was wrong... If you're asking the question of like, how do we evaluate people on exactly the thing that AI can already do, that seems kind of the wrong. Our interview process has always been... you can use as much AI as you want to use. We're going to give you a few hours. Just build your whole own product surface. A lot of these are projects that, frankly, is like, if you were trying to do this by hand, you would not be able to get done in a few hours. So you kind of have to use AI for this. But in reality, what we actually want to test is, in addition to how familiar you are with these tools, what we actually want to test is, what do you think is the right thing to build? Or how do you make these product decisions? How do you make these trade-offs? How do you decide or collect information about what you should be doing?"
0:58

.Scott Wu to Joe Lonsdale on why Cognition uses AI in job interviews: "One or two years ago, people had this mentality of 'How do we make sure people aren't using AI while we're interviewing them?' And I think that has totally flipped. Honestly, I think that was wrong... If you're asking the question of like, how do we evaluate people on exactly the thing that AI can already do, that seems kind of the wrong. Our interview process has always been... you can use as much AI as you want to use. We're going to give you a few hours. Just build your whole own product surface. A lot of these are projects that, frankly, is like, if you were trying to do this by hand, you would not be able to get done in a few hours. So you kind of have to use AI for this. But in reality, what we actually want to test is, in addition to how familiar you are with these tools, what we actually want to test is, what do you think is the right thing to build? Or how do you make these product decisions? How do you make these trade-offs? How do you decide or collect information about what you should be doing?"

American Optimist

13,182 views • 4 months ago

Episode 7 of How I AI is going to inspire the marketers and scare all of SaaS. is head of growth at and ambitiously embraces AI for marketing, to build internal tools, and to hack on his personal data. You’ll learn how to: - take Granola transcripts and turn them into multiple marketing assets - replace expensive saas tools by building a custom solution in Cursor - use Luke’s WhatsApp MCP to query your group chat Luke gives one of the most fun and broad tours of using AI as someone who is technically not in a technical role. Plus, he teaches me “everything is a launch.” Check it out in the links below 👇🔗 As always, a big thanks to our sponsors: 🔀 Orkes enterprise platform for agentic workflows 🧰 Retool AI for developers, built for the enterprise
0:34

Episode 7 of How I AI is going to inspire the marketers and scare all of SaaS. is head of growth at and ambitiously embraces AI for marketing, to build internal tools, and to hack on his personal data. You’ll learn how to: - take Granola transcripts and turn them into multiple marketing assets - replace expensive saas tools by building a custom solution in Cursor - use Luke’s WhatsApp MCP to query your group chat Luke gives one of the most fun and broad tours of using AI as someone who is technically not in a technical role. Plus, he teaches me “everything is a launch.” Check it out in the links below 👇🔗 As always, a big thanks to our sponsors: 🔀 Orkes enterprise platform for agentic workflows 🧰 Retool AI for developers, built for the enterprise

claire vo 🖤

24,914 views • 1 year ago

A great executive assistant is a superpower for CEOs. An AI-powered EA is something else entirely. Cortney Hickey, Executive Business Partner to Wade Foster at Zapier has leaned in HARD to AI + automations to drive operational efficiency, leadership alignment, and cultural clarity across the company. In this week's How I AI, she shows us some of her tricks, including: - a meeting prep agent that makes sure she + the CEO are 100% ready for any meeting - an AI "meeting coach" that ensures the exec team is staying true to their principles and commitments - her pre-exec doc review system that makes sure anyone in the company can get CEO-level feedback...before meeting with the CEO This is one of the most practical and aspirational episodes we've had so far. Cortney is an AI star. As always, big thanks to our sponsors! 💼 WorkOS Make your app enterprise-ready today 🤑 Brex The intelligent finance platform built for founders
1:00

A great executive assistant is a superpower for CEOs. An AI-powered EA is something else entirely. Cortney Hickey, Executive Business Partner to Wade Foster at Zapier has leaned in HARD to AI + automations to drive operational efficiency, leadership alignment, and cultural clarity across the company. In this week's How I AI, she shows us some of her tricks, including: - a meeting prep agent that makes sure she + the CEO are 100% ready for any meeting - an AI "meeting coach" that ensures the exec team is staying true to their principles and commitments - her pre-exec doc review system that makes sure anyone in the company can get CEO-level feedback...before meeting with the CEO This is one of the most practical and aspirational episodes we've had so far. Cortney is an AI star. As always, big thanks to our sponsors! 💼 WorkOS Make your app enterprise-ready today 🤑 Brex The intelligent finance platform built for founders

claire vo 🖤

16,022 views • 7 months ago

Klarna CEO Sebastian Siemiatkowski never learned how to code. But after a summer using Cursor, he now brings working prototypes to meetings instead of asking engineers to interpret his ideas: “Because I'm not an engineer, I've been vibe coding my whole life.” “I come into the meeting room, there's tons of product people and engineers. I say, ‘I think we should build an app that kind of does this, and it should work like that.’ They try to make sense of what is saying.” “I've never seen the code. I don't know how they made it. It's beautiful, I like our app. I think it's nice but, I don't know how it works.” “So to me, I've been vibe coding for 20 years. The only difference is now I'm sitting with a computer, and I'm doing it with AI and it's coming back with the prototype in 20 minutes.”
3:02

Klarna CEO Sebastian Siemiatkowski never learned how to code. But after a summer using Cursor, he now brings working prototypes to meetings instead of asking engineers to interpret his ideas: “Because I'm not an engineer, I've been vibe coding my whole life.” “I come into the meeting room, there's tons of product people and engineers. I say, ‘I think we should build an app that kind of does this, and it should work like that.’ They try to make sense of what is saying.” “I've never seen the code. I don't know how they made it. It's beautiful, I like our app. I think it's nice but, I don't know how it works.” “So to me, I've been vibe coding for 20 years. The only difference is now I'm sitting with a computer, and I'm doing it with AI and it's coming back with the prototype in 20 minutes.”

sourcery

149,352 views • 6 months ago

New episode with Kevin Weil 🇺🇸, CPO at OpenAI, former Head of Product at Instagram and Twitter, VP of Product at Facebook, President at Planet, co-creator of Libra What you'll learn: 🔸 How OpenAI structures its product teams 🔸 Why writing effective evals (AI evaluation tests) is becoming a critical skill for product builders 🔸 The power of model ensembles—using multiple specialized models together like a group of humans with different skills 🔸 What OpenAI looks for when hiring (hint: high agency) 🔸 “Model maximalism” and why today’s AI is the worst you’ll ever use 🔸 How “vibe coding” is changing how companies operate 🔸 The surprisingly enduring value of chat as an interface for AI, despite predictions of its obsolescence 🔸 Practical prompting techniques that improve AI interactions, including example-based prompting Links to listen below 👇
1:28

New episode with Kevin Weil 🇺🇸, CPO at OpenAI, former Head of Product at Instagram and Twitter, VP of Product at Facebook, President at Planet, co-creator of Libra What you'll learn: 🔸 How OpenAI structures its product teams 🔸 Why writing effective evals (AI evaluation tests) is becoming a critical skill for product builders 🔸 The power of model ensembles—using multiple specialized models together like a group of humans with different skills 🔸 What OpenAI looks for when hiring (hint: high agency) 🔸 “Model maximalism” and why today’s AI is the worst you’ll ever use 🔸 How “vibe coding” is changing how companies operate 🔸 The surprisingly enduring value of chat as an interface for AI, despite predictions of its obsolescence 🔸 Practical prompting techniques that improve AI interactions, including example-based prompting Links to listen below 👇

Lenny Rachitsky

135,926 views • 1 year ago

"One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals. And you look at the evals and you go, 'Those are pretty hard evals.' But the economic impact seems to be dramatically behind. There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data. When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"
2:21

"One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals. And you look at the evals and you go, 'Those are pretty hard evals.' But the economic impact seems to be dramatically behind. There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data. When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"

Dwarkesh Patel

502,249 views • 8 months ago