正在加载视频...

视频加载失败

返回首页

"Everyone says AI evaluations are important, so let's actually build one live from scratch.” Here's my new episode with (Arize) where we build AI evals for a customer support agent live, including: ✅ Creating the eval criteria ✅ Labeling the golden dataset ✅ Aligning LLM judges with human scores...

petergyang's profile picture

Peter Yang

215,613 subscribers

37,177 次观看 • 11 个月前 •via X (Twitter)

教育健康养生科学技术
LIVE

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

"AI evals force you into the shoes of your user." I love this clip from (Director at Arize) on why evaluations are the top AI skill to build in 2025. In my new interview, Aman breaks down: - How to write evals like a pro - 5 AI product principles he swears by - How to start practicing evals now Some quotes from Aman: "All PMs will be AI PMs. You'll either build on top of LLM APIs, use models in your backend, or use AI tools to solve problems in your daily work." "The waterfall process of writing a PRD and handing it to designers and engineers is dead. Your dev cycle needs to be much shorter." "Imagine showing up to a meeting with a prototype instead of a PRD. That's the shift PMs will need to make to communicate better in this new AI world." 📌 Watch now:
1:13

"AI evals force you into the shoes of your user." I love this clip from (Director at Arize) on why evaluations are the top AI skill to build in 2025. In my new interview, Aman breaks down: - How to write evals like a pro - 5 AI product principles he swears by - How to start practicing evals now Some quotes from Aman: "All PMs will be AI PMs. You'll either build on top of LLM APIs, use models in your backend, or use AI tools to solve problems in your daily work." "The waterfall process of writing a PRD and handing it to designers and engineers is dead. Your dev cycle needs to be much shorter." "Imagine showing up to a meeting with a prototype instead of a PRD. That's the shift PMs will need to make to communicate better in this new AI world." 📌 Watch now:

Peter Yang

15,973 次观看 • 1 年前

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:
1:12

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 次观看 • 10 个月前

Notion's co-founder Akshay Kothari confessed to me that he couldn't get his wife to use Notion for 7 years until just 3 weeks ago: "She didn't want to deal with all the manual stuff. Now for her, Notion is just a chat thing. She just chats with the agent and it does all the work." Here's my new episode with Akshay and (AI lead) covering their top lessons from building the best AI agents for work: ✅ Why it took 2 years and mistakes to avoid ✅ How they learned to "channel the model" ✅ How to avoid AI slop to build useful agents 📌 Watch now: Also available on: Spotify: Apple: Newsletter:
0:41

Notion's co-founder Akshay Kothari confessed to me that he couldn't get his wife to use Notion for 7 years until just 3 weeks ago: "She didn't want to deal with all the manual stuff. Now for her, Notion is just a chat thing. She just chats with the agent and it does all the work." Here's my new episode with Akshay and (AI lead) covering their top lessons from building the best AI agents for work: ✅ Why it took 2 years and mistakes to avoid ✅ How they learned to "channel the model" ✅ How to avoid AI slop to build useful agents 📌 Watch now: Also available on: Spotify: Apple: Newsletter:

Peter Yang

153,175 次观看 • 9 个月前

"Vibe coding is NOT the same as using AI to write code that's reviewed, tested, and maintained." Here's my new episode with Lee Robinson (Cursor's Head of AI Education) where he shows how to build AI apps properly using planning, test-driven development, agent workflows, and more. Some tips from Lee: 1. Work with AI to create a detailed technical plan upfront including requirements, design preferences, and your preferred tech stack. 2. Ask AI to write tests so that it can check its own work. Add this to the plan: "Write unit tests for business logic, e2e tests for core user journeys." 3. Add "make descriptive commits" to your initial plan so that you can track what AI changed and why. 📌 Watch our beginner-friendly tutorial now: Also available on: Spotify: Apple: Newsletter:
0:52

"Vibe coding is NOT the same as using AI to write code that's reviewed, tested, and maintained." Here's my new episode with Lee Robinson (Cursor's Head of AI Education) where he shows how to build AI apps properly using planning, test-driven development, agent workflows, and more. Some tips from Lee: 1. Work with AI to create a detailed technical plan upfront including requirements, design preferences, and your preferred tech stack. 2. Ask AI to write tests so that it can check its own work. Add this to the plan: "Write unit tests for business logic, e2e tests for core user journeys." 3. Add "make descriptive commits" to your initial plan so that you can track what AI changed and why. 📌 Watch our beginner-friendly tutorial now: Also available on: Spotify: Apple: Newsletter:

Peter Yang

108,418 次观看 • 11 个月前

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.
2:13

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.

Respan

127,656 次观看 • 3 个月前

New session with Shreya Shankar on How to Automate Evals with AI (correctly) The most important part of the eval workflow is finding issues. Shreya does a live demo of how to steer the AI iteratively to find unknown unknowns Chapter summaries: 00:00:00 Introduction 00:00:55 Why vendors want to automate evals for you 00:01:45 Why no tool can fully automate evals 00:06:00 Mistake #1: Asking AI to just "find the issues" in your data 00:08:11 What good AI-assisted error analysis actually looks like 00:11:12 Live demo: building a review interface from scratch 00:15:10 Annotating traces and building a failure mode taxonomy 00:19:31 Mistake #2: Only reviewing your data once 00:22:53 Mistake #3: Treating all apps with the same accuracy bar 00:25:10 General-purpose agents vs. dedicated eval tools 00:25:48 Summary: three rules for automating evals correctly YT links and more in reply
27:18

New session with Shreya Shankar on How to Automate Evals with AI (correctly) The most important part of the eval workflow is finding issues. Shreya does a live demo of how to steer the AI iteratively to find unknown unknowns Chapter summaries: 00:00:00 Introduction 00:00:55 Why vendors want to automate evals for you 00:01:45 Why no tool can fully automate evals 00:06:00 Mistake #1: Asking AI to just "find the issues" in your data 00:08:11 What good AI-assisted error analysis actually looks like 00:11:12 Live demo: building a review interface from scratch 00:15:10 Annotating traces and building a failure mode taxonomy 00:19:31 Mistake #2: Only reviewing your data once 00:22:53 Mistake #3: Treating all apps with the same accuracy bar 00:25:10 General-purpose agents vs. dedicated eval tools 00:25:48 Summary: three rules for automating evals correctly YT links and more in reply

Hamel Husain

34,118 次观看 • 1 个月前

"More context is not always better for AI." Adam (Meta) has helped 100+ companies build AI products. In my new interview, he shares a complete beginner-friendly AI course on: - 2 AI optimizations every PM should know - 4 types of evals and why points beat ratings - How to avoid RAG making your AI worse Some quotes from Adam Loving: "My eval tip is instead of asking an LLM to rate something 1-10, use a point system..." "Instead of a one giant prompt, use a series of prompts and run this each step of the way..." "RAG can actually make your product worse, here's how to avoid this..." 📌 Watch now:
0:48

"More context is not always better for AI." Adam (Meta) has helped 100+ companies build AI products. In my new interview, he shares a complete beginner-friendly AI course on: - 2 AI optimizations every PM should know - 4 types of evals and why points beat ratings - How to avoid RAG making your AI worse Some quotes from Adam Loving: "My eval tip is instead of asking an LLM to rate something 1-10, use a point system..." "Instead of a one giant prompt, use a series of prompts and run this each step of the way..." "RAG can actually make your product worse, here's how to avoid this..." 📌 Watch now:

Peter Yang

38,066 次观看 • 1 年前

Tasker AI is now live, an AI assistant for work and life to-dos. With Tasker you control a team of AI agents to get recurring tasks done and win back your time. Try Tasker out and tell us what you think. Here are just some of the things Tasker could do for you: Y Combinator ✅ Coordinate or reschedule a meeting on your behalf ✅ Morning and Evening email debrief and news digest ✅ Track and reserve deals on flights for dates you actually want to travel ✅ Put the ingredients from a recipe video into a shopping cart ✅ Book an entire date night – from rideshare to dinner to movie tickets – from one prompt. ✅ Build editable landing pages with a Google Sheet backend
3:46

Tasker AI is now live, an AI assistant for work and life to-dos. With Tasker you control a team of AI agents to get recurring tasks done and win back your time. Try Tasker out and tell us what you think. Here are just some of the things Tasker could do for you: Y Combinator ✅ Coordinate or reschedule a meeting on your behalf ✅ Morning and Evening email debrief and news digest ✅ Track and reserve deals on flights for dates you actually want to travel ✅ Put the ingredients from a recipe video into a shopping cart ✅ Book an entire date night – from rideshare to dinner to movie tickets – from one prompt. ✅ Build editable landing pages with a Google Sheet backend

Tasker AI

418,471 次观看 • 1 年前

McKinsey surveyed 2,000 companies in 2025. 51% said AI backfired on them. Top reason? Inaccuracy. From what I can tell, most of these systems weren't broken. They were unreliable. And unreliable is wayyyy worse because you can't predict when it fails. So I got Ash Tilawat (the Mr. Miyagi of teaching AI) from Gauntlet AI to walk me through the solution. Here's his 2026 framework for evaluating if your AI is trustworthy, reliable, and production-ready: 1. build your golden set Identify 30–50 core requests your AI must handle correctly. The stuff that, if broken, makes the whole system useless. And sit with the person whose job this AI is doing/automating/replacing/helping with. 2. test the weird stuff Your golden set covers common requests. But in production, users don't only ask common requests. So build a matrix of categories (topic x complexity) and fill the gaps. Every gap is a corner where failures can hide behind. 3. build a replay harness Record the exact state of every interaction so you can test prompt changes without burning API calls. Think of it like game film... you don't put players back on the field just to review the play. 4. create your rubric Use an LLM to grade outputs on accuracy, completeness, and tone. But calibrate it first -> run 50–100 examples through human and LLM scoring, find disagreements, fix the rubric, repeat until they match. 5. run experiments New model? Prompt rewrite? Run your eval suite against both versions. Ship if the golden set passes, no regressions, and the cost is acceptable. The teams still running production AI on vibes will be f***** in 2026. But the teams building eval libraries are compounding an advantage that gets harder to catch every month. Competitors can copy your product. They can't copy your test cases. h/t Austen Allred for helping put this together. Full playbook + vid below 👇
1:05:36

McKinsey surveyed 2,000 companies in 2025. 51% said AI backfired on them. Top reason? Inaccuracy. From what I can tell, most of these systems weren't broken. They were unreliable. And unreliable is wayyyy worse because you can't predict when it fails. So I got Ash Tilawat (the Mr. Miyagi of teaching AI) from Gauntlet AI to walk me through the solution. Here's his 2026 framework for evaluating if your AI is trustworthy, reliable, and production-ready: 1. build your golden set Identify 30–50 core requests your AI must handle correctly. The stuff that, if broken, makes the whole system useless. And sit with the person whose job this AI is doing/automating/replacing/helping with. 2. test the weird stuff Your golden set covers common requests. But in production, users don't only ask common requests. So build a matrix of categories (topic x complexity) and fill the gaps. Every gap is a corner where failures can hide behind. 3. build a replay harness Record the exact state of every interaction so you can test prompt changes without burning API calls. Think of it like game film... you don't put players back on the field just to review the play. 4. create your rubric Use an LLM to grade outputs on accuracy, completeness, and tone. But calibrate it first -> run 50–100 examples through human and LLM scoring, find disagreements, fix the rubric, repeat until they match. 5. run experiments New model? Prompt rewrite? Run your eval suite against both versions. Ship if the golden set passes, no regressions, and the cost is acceptable. The teams still running production AI on vibes will be f***** in 2026. But the teams building eval libraries are compounding an advantage that gets harder to catch every month. Competitors can copy your product. They can't copy your test cases. h/t Austen Allred for helping put this together. Full playbook + vid below 👇

Alex Lieberman

71,920 次观看 • 6 个月前

"We (Anthropic) are now creating entire features in days, not weeks." Here's my new episode with jenny wen (Claude's Head of Design) where she gave me a rare look at how Anthropic operates, including: ✅ How she uses Cowork to build products ✅ The real story behind Cowork's creation (including screens of early Cowork prototypes) ✅ How Anthropic is able to ship every day Some quotes from Jenny: "The specs we used to make with milestones ...we don't really do that anymore." "People think we built Cowork in 10 days. The actual story is we've been prototyping this direction for a year." "Designers, if you feel like the ground is shifting beneath your feet, it's because it is." 📌 Watch now: Thanks to our sponsors: Replit ⠕: Plan, design, and build with AI agents Linear: The AI agent platform for modern teams
0:39

"We (Anthropic) are now creating entire features in days, not weeks." Here's my new episode with jenny wen (Claude's Head of Design) where she gave me a rare look at how Anthropic operates, including: ✅ How she uses Cowork to build products ✅ The real story behind Cowork's creation (including screens of early Cowork prototypes) ✅ How Anthropic is able to ship every day Some quotes from Jenny: "The specs we used to make with milestones ...we don't really do that anymore." "People think we built Cowork in 10 days. The actual story is we've been prototyping this direction for a year." "Designers, if you feel like the ground is shifting beneath your feet, it's because it is." 📌 Watch now: Thanks to our sponsors: Replit ⠕: Plan, design, and build with AI agents Linear: The AI agent platform for modern teams

Peter Yang

427,284 次观看 • 4 个月前

"We used to say build the MVP. Now you should build the system that builds the MVP first." Here's my new episode with Ryan Carson where he shared how he runs his startup solo with AI agents: ✅ OpenClaw as his AI chief of staff to triage emails, book meetings, and do sales outreach ✅ Codex and Devin as his AI eng team to ship features while he sleeps Some quotes from Ryan: "Spend a lot of time upfront setting up your skills + documentation. Then you've suddenly unlocked the work of 10 people." "Treat your agent like a real employee. Give it a real email address, calendar access, and GitHub account." "Pay a designer to set up your design system and brand. After that, you can use AI to generate on-brand assets." 📌 Watch now: Thanks to our sponsors: Wispr Flow: Don't type, just speak Linear: The AI agent platform for modern teams
0:39

"We used to say build the MVP. Now you should build the system that builds the MVP first." Here's my new episode with Ryan Carson where he shared how he runs his startup solo with AI agents: ✅ OpenClaw as his AI chief of staff to triage emails, book meetings, and do sales outreach ✅ Codex and Devin as his AI eng team to ship features while he sleeps Some quotes from Ryan: "Spend a lot of time upfront setting up your skills + documentation. Then you've suddenly unlocked the work of 10 people." "Treat your agent like a real employee. Give it a real email address, calendar access, and GitHub account." "Pay a designer to set up your design system and brand. After that, you can use AI to generate on-brand assets." 📌 Watch now: Thanks to our sponsors: Wispr Flow: Don't type, just speak Linear: The AI agent platform for modern teams

Peter Yang

74,393 次观看 • 2 个月前

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:
2:07

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:

Andrew Ng

126,478 次观看 • 1 年前

Everyone keeps calling everything an "AI agent." Wade Foster (Zapier CEO) and I are tired of the hype, so we had some real talk about what actually works in my new episode. We covered: ✅ The workflows → agents spectrum and why the middle is where the magic is ✅ Live demo: His email workflow that triages 100+ emails to 10 that need his attention ✅ How to find AI automation wins on your calendar ✅ His reaction to AI influencers tweeting "RIP Zapier" after OpenAI's agentic builder launch Some quotes from Wade: "What's working today are AI workflows—not pure agents. You get more determinism, reliability, and cost advantages with the former." "Look at your schedule and say, 'I spent a lot of time interviewing. What could I automate to make this work better?'" "There's a lot of influencers out there who just want clicks. You can tell who is using the tools and who's just regurgitating headlines." 📌 Watch now: Also available on: Spotify: Apple: Newsletter:
0:46

Everyone keeps calling everything an "AI agent." Wade Foster (Zapier CEO) and I are tired of the hype, so we had some real talk about what actually works in my new episode. We covered: ✅ The workflows → agents spectrum and why the middle is where the magic is ✅ Live demo: His email workflow that triages 100+ emails to 10 that need his attention ✅ How to find AI automation wins on your calendar ✅ His reaction to AI influencers tweeting "RIP Zapier" after OpenAI's agentic builder launch Some quotes from Wade: "What's working today are AI workflows—not pure agents. You get more determinism, reliability, and cost advantages with the former." "Look at your schedule and say, 'I spent a lot of time interviewing. What could I automate to make this work better?'" "There's a lot of influencers out there who just want clicks. You can tell who is using the tools and who's just regurgitating headlines." 📌 Watch now: Also available on: Spotify: Apple: Newsletter:

Peter Yang

125,832 次观看 • 9 个月前

"Claude Code is not just for coding. It's an AI agent that can do whatever you want it to do." Here's my new tutorial with where he showed me exactly how to build a life OS with Claude Code in 25 minutes. He saves 2+ hours daily using: ✅ /researcher to analyze competitors ✅ /daily_brief to get AI news every morning ✅ /brain_dump to find patterns from notes His setup uses slash commands and sub-agents and honestly blew my mind 🔥 Watch the full tutorial to learn how to set up a life operating system for yourself. 📌 Watch now: Also available on: Spotify: Newsletter:
0:40

"Claude Code is not just for coding. It's an AI agent that can do whatever you want it to do." Here's my new tutorial with where he showed me exactly how to build a life OS with Claude Code in 25 minutes. He saves 2+ hours daily using: ✅ /researcher to analyze competitors ✅ /daily_brief to get AI news every morning ✅ /brain_dump to find patterns from notes His setup uses slash commands and sub-agents and honestly blew my mind 🔥 Watch the full tutorial to learn how to set up a life operating system for yourself. 📌 Watch now: Also available on: Spotify: Newsletter:

Peter Yang

190,052 次观看 • 11 个月前

Our latest LangChain Academy course – Building Ambient Agents with LangGraph – is now available! Most agents today handle one request at a time through chat interfaces. But as models have improved, agents can now run in the background – and take on long-running, complex tasks. LangGraph is built for these “ambient agents,” with support for human-in-the-loop workflows and memory. LangGraph Platform provides the infrastructure to run these agents at scale, and LangSmith helps you observe, evaluate, and improve your agents. Together, they make it easier to build reliable, production-ready agents. In this course, you’ll: 📩 Learn the fundamentals of LangGraph and how to build your own email agent 📊 Evaluate your agent using LangSmith ✅ Add human-in-the-loop for reviews, and memory so your agent adapts over time 🚀 Deploy your agent with LangGraph Platform and connect it to Gmail
3:02

Our latest LangChain Academy course – Building Ambient Agents with LangGraph – is now available! Most agents today handle one request at a time through chat interfaces. But as models have improved, agents can now run in the background – and take on long-running, complex tasks. LangGraph is built for these “ambient agents,” with support for human-in-the-loop workflows and memory. LangGraph Platform provides the infrastructure to run these agents at scale, and LangSmith helps you observe, evaluate, and improve your agents. Together, they make it easier to build reliable, production-ready agents. In this course, you’ll: 📩 Learn the fundamentals of LangGraph and how to build your own email agent 📊 Evaluate your agent using LangSmith ✅ Add human-in-the-loop for reviews, and memory so your agent adapts over time 🚀 Deploy your agent with LangGraph Platform and connect it to Gmail

LangChain

33,192 次观看 • 1 年前

"AI gets you to average quickly. Your job is to push past that." Here's my new episode with Dylan Field (Figma CEO) where I asked him some tough questions, including: → Can you teach AI design taste? → Do design systems hurt creativity? → What's Figma's role when code is free? Two-thirds of Figma users are now non-designers and it was super interesting to hear Dylan talk about Figma's future. Some quotes from Dylan: "Design is the new code. You'll be designing in a visual-first way before pushing a pull request right to production." "If you're a PM and you think your job is to make docs and slide decks, you're going to love this new world. You get to make things too." "If everyone agrees with your point of view, then you probably don't have one." 📌 Watch now: Thanks to our sponsors: Replit ⠕: Plan, design, and build with AI agents Linear: The AI agent platform for modern teams
0:41

"AI gets you to average quickly. Your job is to push past that." Here's my new episode with Dylan Field (Figma CEO) where I asked him some tough questions, including: → Can you teach AI design taste? → Do design systems hurt creativity? → What's Figma's role when code is free? Two-thirds of Figma users are now non-designers and it was super interesting to hear Dylan talk about Figma's future. Some quotes from Dylan: "Design is the new code. You'll be designing in a visual-first way before pushing a pull request right to production." "If you're a PM and you think your job is to make docs and slide decks, you're going to love this new world. You get to make things too." "If everyone agrees with your point of view, then you probably don't have one." 📌 Watch now: Thanks to our sponsors: Replit ⠕: Plan, design, and build with AI agents Linear: The AI agent platform for modern teams

Peter Yang

103,627 次观看 • 4 个月前

New short course on Reinforcement Learning from Human Feedback! RLHF is one of the key techniques that led to the rise of modern LLMs. It is used to align LLMs with human preferences, to make them more honest, helpful and harmless, by (i) learning a reward function that mimics human preferences, as expressed in human-provided labels, then, (ii) tuning an LLM to generate outputs that receive a high reward. In this course, taught by Nikita Namjoshi, Developer Advocate for GenAI at Google Cloud, you'll learn the details of how RLHF works, including how to apply it to tune an LLM for your own applications. You'll also use an open source library to tune a base LLM to align with human preferences expressed in a training set, and evaluate the tuned model by comparing its responses before and after RLHF-tuning. Please sign up here!
3:26

New short course on Reinforcement Learning from Human Feedback! RLHF is one of the key techniques that led to the rise of modern LLMs. It is used to align LLMs with human preferences, to make them more honest, helpful and harmless, by (i) learning a reward function that mimics human preferences, as expressed in human-provided labels, then, (ii) tuning an LLM to generate outputs that receive a high reward. In this course, taught by Nikita Namjoshi, Developer Advocate for GenAI at Google Cloud, you'll learn the details of how RLHF works, including how to apply it to tune an LLM for your own applications. You'll also use an open source library to tune a base LLM to align with human preferences expressed in a training set, and evaluate the tuned model by comparing its responses before and after RLHF-tuning. Please sign up here!

Andrew Ng

205,558 次观看 • 2 年前

“You cannot make a choice and then sit back and wait for what God will put in your hands. While you are waiting for God to place things in your hands, he has already given you those hands to use, he gave you arms to work and sustain yourself, and be able to live as the dignified human being he created you to be. That is what is supposed to guide us, that is what should determine the path we follow. We were able to overcome our bad history, but what was the goal? It was for people to live well, and make progress; for the country to develop, and to build a country where Rwandans are proud of who they are and where they are going. And where they are the ones to decide what to do with their lives. Everything else, the support we receive, the partnerships, can only build on that.” President Kagame | Interview with RBA.
1:15

“You cannot make a choice and then sit back and wait for what God will put in your hands. While you are waiting for God to place things in your hands, he has already given you those hands to use, he gave you arms to work and sustain yourself, and be able to live as the dignified human being he created you to be. That is what is supposed to guide us, that is what should determine the path we follow. We were able to overcome our bad history, but what was the goal? It was for people to live well, and make progress; for the country to develop, and to build a country where Rwandans are proud of who they are and where they are going. And where they are the ones to decide what to do with their lives. Everything else, the support we receive, the partnerships, can only build on that.” President Kagame | Interview with RBA.

Presidency | Rwanda

36,675 次观看 • 2 年前

"I have prompts to simulate every exec I meet with so I can anticipate their questions." Here's my new episode with Morgan Brown (VP AI at Dropbox) where he shared his: - Top 3 AI workflows for PM work - AI product to save workers 160 hours a year - Top lessons from Drew (Dropbox), Tobi (Shopify), and Zuck (Meta) Some quotes from Morgan: "The best PMs know how to orchestrate a council of LLMs so that they can free up time to think." "Shipping is what puts your ideas in contact with reality. Progress should be measured by how fast you're learning." "We run a robust evaluation process that includes golden datasets, LLM as a judge, and real-time feedback. Here's how it works..." 📌 Watch now:
1:04

"I have prompts to simulate every exec I meet with so I can anticipate their questions." Here's my new episode with Morgan Brown (VP AI at Dropbox) where he shared his: - Top 3 AI workflows for PM work - AI product to save workers 160 hours a year - Top lessons from Drew (Dropbox), Tobi (Shopify), and Zuck (Meta) Some quotes from Morgan: "The best PMs know how to orchestrate a council of LLMs so that they can free up time to think." "Shipping is what puts your ideas in contact with reality. Progress should be measured by how fast you're learning." "We run a robust evaluation process that includes golden datasets, LLM as a judge, and real-time feedback. Here's how it works..." 📌 Watch now:

Peter Yang

59,720 次观看 • 1 年前

I don't think most PMs realize the PRD is becoming obsolete. For the last decade, the PM's core artifact was a qualitative spec. Clear requirements, user stories, acceptance criteria. The engineering team interpreted it, built something close, and the PM spent two weeks reconciling what shipped with what they wrote. The best AI companies replaced that entire loop with evals. A set of inputs your product needs to handle. A task that generates outputs. A scoring function that produces a number between 0 and 1. No ambiguity. No interpretation gap. Ankur Goyal built the eval platform behind Vercel, Replit, Ramp, Notion, and Airtable. An $800M company. He walked through building an eval from zero on this episode and the score went from 0 to 0.75 in under 20 minutes. That's a PM shipping a measurable quality bar before a single line of product code exists. Here's the part that changes the PM role permanently. When the product passes the eval and users still hate it, the eval is wrong. That's on the PM. Evals make PM judgment quantifiable in a way PRDs never did. You can't hide behind "the spec was ambiguous." There's a number now. Six months ago, PM interviews asked "how do you use AI in your workflow." The next wave of interviews is going to ask you to write an eval. The PMs who can encode user intent as a scoring function are building the one skill that survives every model change, every framework swap, every agent rewrite. Write the eval.
4:31

I don't think most PMs realize the PRD is becoming obsolete. For the last decade, the PM's core artifact was a qualitative spec. Clear requirements, user stories, acceptance criteria. The engineering team interpreted it, built something close, and the PM spent two weeks reconciling what shipped with what they wrote. The best AI companies replaced that entire loop with evals. A set of inputs your product needs to handle. A task that generates outputs. A scoring function that produces a number between 0 and 1. No ambiguity. No interpretation gap. Ankur Goyal built the eval platform behind Vercel, Replit, Ramp, Notion, and Airtable. An $800M company. He walked through building an eval from zero on this episode and the score went from 0 to 0.75 in under 20 minutes. That's a PM shipping a measurable quality bar before a single line of product code exists. Here's the part that changes the PM role permanently. When the product passes the eval and users still hate it, the eval is wrong. That's on the PM. Evals make PM judgment quantifiable in a way PRDs never did. You can't hide behind "the spec was ambiguous." There's a number now. Six months ago, PM interviews asked "how do you use AI in your workflow." The next wave of interviews is going to ask you to write an eval. The PMs who can encode user intent as a scoring function are building the one skill that survives every model change, every framework swap, every agent rewrite. Write the eval.

Aakash Gupta

78,275 次观看 • 4 个月前