正在加载视频...

视频加载失败

返回首页

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals...

mattturck's profile picture

Matt Turck

138,040 subscribers

10,969 次观看 • 9 个月前 •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

NEW Greylock Change Agents: Evaluating Agents with Braintrust There's been a lot of discourse around evals recently, so it's timely to drop the recording of Change Agents with Ankur Goyal on the topic of how he thinks about Evaluating Agents. Timestamps: (0:00) - Intro (1:22) - Building Braintrust (5:19) - Evaluating agentic systems (11:04) - Experimentation with evals (13:18) - Adopting evals (17:09) - Evals and complex agents (21:19) - Custom scoring (25:02) - LLM as a judge (26:44) - Human-manual evaluation (30:07) - How the best in-class teams manage evals (33:57) - Future of evals for agents (37:38 ) - Audience Q&A
57:34

NEW Greylock Change Agents: Evaluating Agents with Braintrust There's been a lot of discourse around evals recently, so it's timely to drop the recording of Change Agents with Ankur Goyal on the topic of how he thinks about Evaluating Agents. Timestamps: (0:00) - Intro (1:22) - Building Braintrust (5:19) - Evaluating agentic systems (11:04) - Experimentation with evals (13:18) - Adopting evals (17:09) - Evals and complex agents (21:19) - Custom scoring (25:02) - LLM as a judge (26:44) - Human-manual evaluation (30:07) - How the best in-class teams manage evals (33:57) - Future of evals for agents (37:38 ) - Audience Q&A

Corinne Marie Riley

12,743 次观看 • 9 个月前

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:
1:12

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 次观看 • 9 个月前

🎙️ New Deployed episode with Ryan Carson, "Builder in Residence" at Amp. It's a great convo about building code agents and building as a solo founder at the same time, and it's refreshingly honest. Case in point, on evals for coding agents: "We do not have a set of evals that we programmatically run to decide if GPT-5 is better than Sonnet-4 yet... everybody in the trenches building these agentic coding tools understands that it is so dynamic, and so frustrating." Great conversation about building with AI, developer experience, and what happens when everyone can code. Check it out. 👇
2:02

🎙️ New Deployed episode with Ryan Carson, "Builder in Residence" at Amp. It's a great convo about building code agents and building as a solo founder at the same time, and it's refreshingly honest. Case in point, on evals for coding agents: "We do not have a set of evals that we programmatically run to decide if GPT-5 is better than Sonnet-4 yet... everybody in the trenches building these agentic coding tools understands that it is so dynamic, and so frustrating." Great conversation about building with AI, developer experience, and what happens when everyone can code. Check it out. 👇

Ian Cairns

250,866 次观看 • 9 个月前

Evals 101: a new course from Braintrust. Everything you need to know about evals, and how to do them yourself. Module one: Why are evals important? - the six most common problems developers face when shipping AI applications - why traditional software thinking doesn't apply to AI - how evals can fix these problems
1:51

Evals 101: a new course from Braintrust. Everything you need to know about evals, and how to do them yourself. Module one: Why are evals important? - the six most common problems developers face when shipping AI applications - why traditional software thinking doesn't apply to AI - how evals can fix these problems

Braintrust

36,347 次观看 • 2 个月前

Ex-Google engineer explained AI agent loops, harness, evals in 20 minutes - better than 500$ courses. trace every run → judge it with an LLM → diagnose → fix → ship. That loop is how agents self-improve over time. Agent loops + memory + harness + evals - thats the stack. Watch it, then save the framework below.
19:30

Ex-Google engineer explained AI agent loops, harness, evals in 20 minutes - better than 500$ courses. trace every run → judge it with an LLM → diagnose → fix → ship. That loop is how agents self-improve over time. Agent loops + memory + harness + evals - thats the stack. Watch it, then save the framework below.

Movez

537,928 次观看 • 3 天前

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.
52:06

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.

Aakash Gupta

483,558 次观看 • 3 个月前

🪄 LangChain State of AI 2024 What LLMs are the most widely used today? What metrics are commonly used for evals? Are developers finding success in building agents? Our State of AI 2024 report shows where the AI ecosystem is headed, based on data from LangSmith. Key 5 insights in the thread 🧵👇 Full report:
0:35

🪄 LangChain State of AI 2024 What LLMs are the most widely used today? What metrics are commonly used for evals? Are developers finding success in building agents? Our State of AI 2024 report shows where the AI ecosystem is headed, based on data from LangSmith. Key 5 insights in the thread 🧵👇 Full report:

LangChain

89,124 次观看 • 1 年前

Every agent needs its own computer: my conversation with Ivan Burazin, CEO of Daytona, about sandboxes and the emerging agent stack. 00:00 Intro 02:13 What is an AI agent sandbox? 03:17 Security risks of running agents locally 05:17 Stateful vs. stateless hyperscalers 07:04 The history of cloud IDEs and the end of localhost 09:45 Do all AI agents need a sandbox? 12:26 Sandbox use cases: RL evals & background agents 14:10 Unpacking the emerging AI Agent Stack 16:20 The unsolved problem of agent memory and learning 19:37 Where sandboxes fit in the agent harness 21:35 OpenAI, Anthropic, and agent SDKs 23:06 Ivan's founder journey: From CodeAnywhere to Daytona 26:59 GTM strategies and building developer communities 33:48 Why customer support is your best GTM strategy 35:34 Leveraging Twitter during the AI super cycle 40:50 The technical anatomy of a sandbox 41:53 Why fast spin-up speeds maximize GPU efficiency 46:09 Firecracker, QEMU, and isolation primitives 49:58 Why sandbox snapshots and state forking matter 51:40 Why Daytona built a custom scheduler from scratch 55:24 The challenge of long-running stateful sandboxes 58:10 The build your own sandbox trap 1:01:03 Why AI agents might trigger a global CPU shortage 1:02:46 The future of the AI Agent Stack
1:05:15

Every agent needs its own computer: my conversation with Ivan Burazin, CEO of Daytona, about sandboxes and the emerging agent stack. 00:00 Intro 02:13 What is an AI agent sandbox? 03:17 Security risks of running agents locally 05:17 Stateful vs. stateless hyperscalers 07:04 The history of cloud IDEs and the end of localhost 09:45 Do all AI agents need a sandbox? 12:26 Sandbox use cases: RL evals & background agents 14:10 Unpacking the emerging AI Agent Stack 16:20 The unsolved problem of agent memory and learning 19:37 Where sandboxes fit in the agent harness 21:35 OpenAI, Anthropic, and agent SDKs 23:06 Ivan's founder journey: From CodeAnywhere to Daytona 26:59 GTM strategies and building developer communities 33:48 Why customer support is your best GTM strategy 35:34 Leveraging Twitter during the AI super cycle 40:50 The technical anatomy of a sandbox 41:53 Why fast spin-up speeds maximize GPU efficiency 46:09 Firecracker, QEMU, and isolation primitives 49:58 Why sandbox snapshots and state forking matter 51:40 Why Daytona built a custom scheduler from scratch 55:24 The challenge of long-running stateful sandboxes 58:10 The build your own sandbox trap 1:01:03 Why AI agents might trigger a global CPU shortage 1:02:46 The future of the AI Agent Stack

Matt Turck

11,908 次观看 • 1 个月前

I got an expert to give me a masterclass on AI evals: (63 mins of stuff most builders learn the hard way) 0:00 - Why your product is lying to you 22:03 - Full eval flow walkthrough 35:42 - Evals ARE your PRD (not QA) 57:52 - Failures at Grammarly, Copilot & Klarna
1:03:59

I got an expert to give me a masterclass on AI evals: (63 mins of stuff most builders learn the hard way) 0:00 - Why your product is lying to you 22:03 - Full eval flow walkthrough 35:42 - Evals ARE your PRD (not QA) 57:52 - Failures at Grammarly, Copilot & Klarna

Aakash Gupta

246,941 次观看 • 4 个月前

I’m proud of the evolution of AI agents CAILA — an AI agent redefining utility through weather intelligence and real-world applications.
1:58

I’m proud of the evolution of AI agents CAILA — an AI agent redefining utility through weather intelligence and real-world applications.

Ivo Josipović

189,553 次观看 • 1 年前

I like to think of evals as something active, not passive -- it's a North Star that steers LLMs toward higher intelligence. Evals should drive your RL/SFT/post-training decisions. Internal evals at frontier labs make a huge difference -- and you can see it in how models behave differently (GPT seems better at one-shot tasks, Claude at multi-turn). If you want to learn more about building evals that actually improve your model in post-training, check out our AMD x DeeplearningAI course "Fine-tuning & RL for LLMs: Intro to Post-training" (content is free):
0:30

I like to think of evals as something active, not passive -- it's a North Star that steers LLMs toward higher intelligence. Evals should drive your RL/SFT/post-training decisions. Internal evals at frontier labs make a huge difference -- and you can see it in how models behave differently (GPT seems better at one-shot tasks, Claude at multi-turn). If you want to learn more about building evals that actually improve your model in post-training, check out our AMD x DeeplearningAI course "Fine-tuning & RL for LLMs: Intro to Post-training" (content is free):

Sharon Zhou

16,613 次观看 • 4 个月前

Insights Agent & Multi-turn Evals Agents run for a long period of time and have multiple interactions with users, tackling a wide variety of problems. Today we’re launching two new features in LangSmith, our agent engineering platform, so you can better understand your agent behavior: • Insights Agent is a new agent that automatically categorizes agent behavior patterns in LangSmith • Multi-turn Evals lets you evaluate agent trajectory across complete conversations, so you can see if the agent actually accomplished user goals Learn more:
0:40

Insights Agent & Multi-turn Evals Agents run for a long period of time and have multiple interactions with users, tackling a wide variety of problems. Today we’re launching two new features in LangSmith, our agent engineering platform, so you can better understand your agent behavior: • Insights Agent is a new agent that automatically categorizes agent behavior patterns in LangSmith • Multi-turn Evals lets you evaluate agent trajectory across complete conversations, so you can see if the agent actually accomplished user goals Learn more:

LangChain

13,302 次观看 • 8 个月前

Everything Gets Rebuilt: my conversation with Harrison Chase, CEO of LangChain about agent harnesses, evals, runtimes, sandboxes, MCP and the future of the agent stack 00:00 Intro - meet Harrison Chase - at the Chase Center for the Daytona Compute conference 01:32 What changed in agents over the last year 03:57 Why coding agents are ahead 06:26 Do models commoditize the framework layer? 08:27 Harnesses, in plain English 10:11 Why system prompts matter so much 13:11 The upside — and downside — of subagents 15:31 Why a useful agent needs a filesystem 18:13 Additional core primitives of modern agents 19:12 Skills: the new primitive 20:19 What context compaction actually means 23:02 How memory works in agents 25:16 One mega-agent or many specialized agents? 27:46 The future of MCP 29:38 Why agents need sandboxes 32:35 How sandboxes help with security 33:32 How Harrison Chase started LangChain 37:24 LangChain vs LangGraph vs Deep Agents 40:17 Why observability matters more for agents 41:48 Evals, no-code, and continuous improvement 44:41 What LangChain is building next 45:29 Where the real moat in AI lives
46:57

Everything Gets Rebuilt: my conversation with Harrison Chase, CEO of LangChain about agent harnesses, evals, runtimes, sandboxes, MCP and the future of the agent stack 00:00 Intro - meet Harrison Chase - at the Chase Center for the Daytona Compute conference 01:32 What changed in agents over the last year 03:57 Why coding agents are ahead 06:26 Do models commoditize the framework layer? 08:27 Harnesses, in plain English 10:11 Why system prompts matter so much 13:11 The upside — and downside — of subagents 15:31 Why a useful agent needs a filesystem 18:13 Additional core primitives of modern agents 19:12 Skills: the new primitive 20:19 What context compaction actually means 23:02 How memory works in agents 25:16 One mega-agent or many specialized agents? 27:46 The future of MCP 29:38 Why agents need sandboxes 32:35 How sandboxes help with security 33:32 How Harrison Chase started LangChain 37:24 LangChain vs LangGraph vs Deep Agents 40:17 Why observability matters more for agents 41:48 Evals, no-code, and continuous improvement 44:41 What LangChain is building next 45:29 Where the real moat in AI lives

Matt Turck

35,241 次观看 • 3 个月前

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!
1:15

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!

AI Engineer

55,348 次观看 • 1 年前

7 AI Agents That Print Money in 2026: 0:00 - Intro 0:52 - Agent #1: Employee 2:59 - Agent #2: Content Revenue 6:24 - Agent #3: Lead Generation 8:04 - Agent #4: Web Dev 10:07 - Agent #5: Deliverables 11:10 - Agent #6: Lead Scraping 12:15 - Agent #7: Trading Bots 13:37 - Bonus
15:25

7 AI Agents That Print Money in 2026: 0:00 - Intro 0:52 - Agent #1: Employee 2:59 - Agent #2: Content Revenue 6:24 - Agent #3: Lead Generation 8:04 - Agent #4: Web Dev 10:07 - Agent #5: Deliverables 11:10 - Agent #6: Lead Scraping 12:15 - Agent #7: Trading Bots 13:37 - Bonus

Sharbel

92,972 次观看 • 3 个月前

She literally broke down how to run evals in Claude Code (built the whole thing live): 01:34 - What people get wrong with evals 04:35 - Why product taste is the alpha now 09:28 - Building a PM agent from one prompt 19:00 - Instrumentation without writing code 22:00 - Watching traces stream in live 28:00 - Getting Claude to write your first eval 33:58 - When vibe evals work and when they don't 48:50 - The self-improving loop (this part is wild) 01:03:00 - Same-day shipping is real 01:06:00 - The context graph unlock
1:19:31

She literally broke down how to run evals in Claude Code (built the whole thing live): 01:34 - What people get wrong with evals 04:35 - Why product taste is the alpha now 09:28 - Building a PM agent from one prompt 19:00 - Instrumentation without writing code 22:00 - Watching traces stream in live 28:00 - Getting Claude to write your first eval 33:58 - When vibe evals work and when they don't 48:50 - The self-improving loop (this part is wild) 01:03:00 - Same-day shipping is real 01:06:00 - The context graph unlock

Aakash Gupta

150,362 次观看 • 1 个月前

We beat Claude Code in our evals: 61% to 53%. Yes, evals are actually important haha And now we’re open-sourcing our entire codebase and launching an agent framework! Now you can build custom agents for your project and get better results 🔥
5:42

We beat Claude Code in our evals: 61% to 53%. Yes, evals are actually important haha And now we’re open-sourcing our entire codebase and launching an agent framework! Now you can build custom agents for your project and get better results 🔥

James Grugett

276,478 次观看 • 9 个月前

UOMI Testnet AI Agents are Live – The Future of Autonomous On-Chain Intelligence 🤖 Most AI Agent launchpads today offer unlimited, low-functionality bots—easily deployed but lacking real utility. UOMI is different. We are building a new class of AI agents, not just simple Twitter bots, but fully autonomous onchain economic entities that will trade, transact, govern, and generate real value onchain. 🧵👇🏼
0:30

UOMI Testnet AI Agents are Live – The Future of Autonomous On-Chain Intelligence 🤖 Most AI Agent launchpads today offer unlimited, low-functionality bots—easily deployed but lacking real utility. UOMI is different. We are building a new class of AI agents, not just simple Twitter bots, but fully autonomous onchain economic entities that will trade, transact, govern, and generate real value onchain. 🧵👇🏼

Uomi

85,999 次观看 • 1 年前

Announcing skillgrade - the easiest way to evaluate your agent skills All you need is two commands: skillgrade init # create evals skillgrade # run them By default evals run in a safe sandboxed docker container
0:56

Announcing skillgrade - the easiest way to evaluate your agent skills All you need is two commands: skillgrade init # create evals skillgrade # run them By default evals run in a safe sandboxed docker container

Minko Gechev

44,884 次观看 • 3 个月前