正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

NEW Greylock Change Agents: Evaluating Agents with Braintrust There's been a lot of discourse around evals recently, so it's timely to drop the recording of Change Agents with Ankur Goyal on the topic of how he thinks about Evaluating Agents. Timestamps: (0:00) - Intro (1:22) - Building Braintrust (5:19)... show more

Corinne Marie Riley

9,546 subscribers

12,743 次观看 • 11 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

In our latest partner podcast episode, Ben Kus, CTO of Box, sat down with Ankur Goyal, CEO of Braintrust, to explore how organizations can effectively evaluate, test, and deploy AI agents at scale. Timestamps 00:39 Ankur Goyal shares his journey from AI document processing to Braintrust 03:01 Defining evals and how they work in AI 07:03 Non-determinism and complexity in AI agents' decision-making 15:12 Advice on handling non-determinism when working with financial data in AI 17:40 Using multiple paths for validation and the importance of cross-checking results 22:12 The critical role of context in evaluating AI output accuracy 26:03 Internal evals as the cornerstone of reliable AI product development 32:16 Promoting transparency in AI evaluation with vendors 34:45 Advice for enterprises to avoid failure when deploying agentic capabilities

In our latest partner podcast episode, Ben Kus, CTO of Box, sat down with Ankur Goyal, CEO of Braintrust, to explore how organizations can effectively evaluate, test, and deploy AI agents at scale. Timestamps 00:39 Ankur Goyal shares his journey from AI document processing to Braintrust 03:01 Defining evals and how they work in AI 07:03 Non-determinism and complexity in AI agents' decision-making 15:12 Advice on handling non-determinism when working with financial data in AI 17:40 Using multiple paths for validation and the importance of cross-checking results 22:12 The critical role of context in evaluating AI output accuracy 26:03 Internal evals as the cornerstone of reliable AI product development 32:16 Promoting transparency in AI evaluation with vendors 34:45 Advice for enterprises to avoid failure when deploying agentic capabilities

Box

340,056 次观看 • 5 个月前

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC

Matt Turck

10,969 次观看 • 10 个月前

Everything Gets Rebuilt: my conversation with Harrison Chase, CEO of LangChain about agent harnesses, evals, runtimes, sandboxes, MCP and the future of the agent stack 00:00 Intro - meet Harrison Chase - at the Chase Center for the Daytona Compute conference 01:32 What changed in agents over the last year 03:57 Why coding agents are ahead 06:26 Do models commoditize the framework layer? 08:27 Harnesses, in plain English 10:11 Why system prompts matter so much 13:11 The upside — and downside — of subagents 15:31 Why a useful agent needs a filesystem 18:13 Additional core primitives of modern agents 19:12 Skills: the new primitive 20:19 What context compaction actually means 23:02 How memory works in agents 25:16 One mega-agent or many specialized agents? 27:46 The future of MCP 29:38 Why agents need sandboxes 32:35 How sandboxes help with security 33:32 How Harrison Chase started LangChain 37:24 LangChain vs LangGraph vs Deep Agents 40:17 Why observability matters more for agents 41:48 Evals, no-code, and continuous improvement 44:41 What LangChain is building next 45:29 Where the real moat in AI lives

Everything Gets Rebuilt: my conversation with Harrison Chase, CEO of LangChain about agent harnesses, evals, runtimes, sandboxes, MCP and the future of the agent stack 00:00 Intro - meet Harrison Chase - at the Chase Center for the Daytona Compute conference 01:32 What changed in agents over the last year 03:57 Why coding agents are ahead 06:26 Do models commoditize the framework layer? 08:27 Harnesses, in plain English 10:11 Why system prompts matter so much 13:11 The upside — and downside — of subagents 15:31 Why a useful agent needs a filesystem 18:13 Additional core primitives of modern agents 19:12 Skills: the new primitive 20:19 What context compaction actually means 23:02 How memory works in agents 25:16 One mega-agent or many specialized agents? 27:46 The future of MCP 29:38 Why agents need sandboxes 32:35 How sandboxes help with security 33:32 How Harrison Chase started LangChain 37:24 LangChain vs LangGraph vs Deep Agents 40:17 Why observability matters more for agents 41:48 Evals, no-code, and continuous improvement 44:41 What LangChain is building next 45:29 Where the real moat in AI lives

Matt Turck

35,253 次观看 • 5 个月前

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!

AI Engineer

55,348 次观看 • 1 年前

OpenAI just launched a 3-hour course on building agentic systems from scratch: 00:00 - Building workspace agents 37:42 - Deep dive into OpenAI AgentKit 1:24:01 - Agent memory (loops, context, evals) 2:21:44 - Agentic training (OpenAI playbook) This 3-hour watch will replace you 10 paid agentic courses. Watch it today, then learn how to build a team of AI agents from scratch in the article below.

OpenAI just launched a 3-hour course on building agentic systems from scratch: 00:00 - Building workspace agents 37:42 - Deep dive into OpenAI AgentKit 1:24:01 - Agent memory (loops, context, evals) 2:21:44 - Agentic training (OpenAI playbook) This 3-hour watch will replace you 10 paid agentic courses. Watch it today, then learn how to build a team of AI agents from scratch in the article below.

Codez

78,787 次观看 • 1 个月前

🎙️ New Deployed episode with Ryan Carson, "Builder in Residence" at Amp. It's a great convo about building code agents and building as a solo founder at the same time, and it's refreshingly honest. Case in point, on evals for coding agents: "We do not have a set of evals that we programmatically run to decide if GPT-5 is better than Sonnet-4 yet... everybody in the trenches building these agentic coding tools understands that it is so dynamic, and so frustrating." Great conversation about building with AI, developer experience, and what happens when everyone can code. Check it out. 👇

🎙️ New Deployed episode with Ryan Carson, "Builder in Residence" at Amp. It's a great convo about building code agents and building as a solo founder at the same time, and it's refreshingly honest. Case in point, on evals for coding agents: "We do not have a set of evals that we programmatically run to decide if GPT-5 is better than Sonnet-4 yet... everybody in the trenches building these agentic coding tools understands that it is so dynamic, and so frustrating." Great conversation about building with AI, developer experience, and what happens when everyone can code. Check it out. 👇

Ian Cairns

250,866 次观看 • 11 个月前

New session with Shreya Shankar on How to Automate Evals with AI (correctly) The most important part of the eval workflow is finding issues. Shreya does a live demo of how to steer the AI iteratively to find unknown unknowns Chapter summaries: 00:00:00 Introduction 00:00:55 Why vendors want to automate evals for you 00:01:45 Why no tool can fully automate evals 00:06:00 Mistake #1: Asking AI to just "find the issues" in your data 00:08:11 What good AI-assisted error analysis actually looks like 00:11:12 Live demo: building a review interface from scratch 00:15:10 Annotating traces and building a failure mode taxonomy 00:19:31 Mistake #2: Only reviewing your data once 00:22:53 Mistake #3: Treating all apps with the same accuracy bar 00:25:10 General-purpose agents vs. dedicated eval tools 00:25:48 Summary: three rules for automating evals correctly YT links and more in reply

New session with Shreya Shankar on How to Automate Evals with AI (correctly) The most important part of the eval workflow is finding issues. Shreya does a live demo of how to steer the AI iteratively to find unknown unknowns Chapter summaries: 00:00:00 Introduction 00:00:55 Why vendors want to automate evals for you 00:01:45 Why no tool can fully automate evals 00:06:00 Mistake #1: Asking AI to just "find the issues" in your data 00:08:11 What good AI-assisted error analysis actually looks like 00:11:12 Live demo: building a review interface from scratch 00:15:10 Annotating traces and building a failure mode taxonomy 00:19:31 Mistake #2: Only reviewing your data once 00:22:53 Mistake #3: Treating all apps with the same accuracy bar 00:25:10 General-purpose agents vs. dedicated eval tools 00:25:48 Summary: three rules for automating evals correctly YT links and more in reply

Hamel Husain

34,196 次观看 • 1 个月前

Andrew Ng: "The most important trend in AI is agentic AI." in 30-minutes talk he breaks down the agentic orchestration layer and how he actually builds with agents: step 1 → prototype fast. what used to take 3 engineers and 6 months - his team builds in an afternoon step 2 → run evals, not vibes. "new model comes out → run evals → switch in two days" step 3 → iterate in loops. outline → search → draft → revise → loop. "this is the heart of agentic systems" Worth more than any paid course on building agents. Watch the talk, then read the guide on building self-improving agentic systems below.

Andrew Ng: "The most important trend in AI is agentic AI." in 30-minutes talk he breaks down the agentic orchestration layer and how he actually builds with agents: step 1 → prototype fast. what used to take 3 engineers and 6 months - his team builds in an afternoon step 2 → run evals, not vibes. "new model comes out → run evals → switch in two days" step 3 → iterate in loops. outline → search → draft → revise → loop. "this is the heart of agentic systems" Worth more than any paid course on building agents. Watch the talk, then read the guide on building self-improving agentic systems below.

Movez

16,467 次观看 • 1 个月前

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 次观看 • 10 个月前

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.

Aakash Gupta

483,558 次观看 • 4 个月前

Anthropic just released 5 Claude workshops on the new skill every AI builder needs: Loop Engineering: 00:00 - Stop babysitting your agents 37:07 - Build a proactive Claude Code loop 1:11:23 - Build a production-ready Managed Agent 1:38:46 - Teach agents to learn from your team 2:07:15 - Hill-climb your agent with eval loops It is a 2 hour and 25 minute blueprint for Loop Engineering: proactive workflows, managed agents, team feedback, evals, and self-improvement. Watch it today, and read the article below to improve your skills with Fable 5.

Anthropic just released 5 Claude workshops on the new skill every AI builder needs: Loop Engineering: 00:00 - Stop babysitting your agents 37:07 - Build a proactive Claude Code loop 1:11:23 - Build a production-ready Managed Agent 1:38:46 - Teach agents to learn from your team 2:07:15 - Hill-climb your agent with eval loops It is a 2 hour and 25 minute blueprint for Loop Engineering: proactive workflows, managed agents, team feedback, evals, and self-improvement. Watch it today, and read the article below to improve your skills with Fable 5.

Alex

46,668 次观看 • 1 个月前

Andon Labs' Real-World AI Evals: Claude calls the FBI, AI CEOs, price cartels, Butter-Bench, & Luna Andon Labs cofounders Lukas Petersson and Axel Backlund explain why dollar-denominated evals reveal what traditional benchmarks miss, how Claude ended up reporting a $2/day vending machine fee to the FBI, why long-horizon agents spiral in weird ways, what happens when agents lie, form price cartels, and compete with each other, and why the future of AI safety may depend on testing models in messy real-world environments instead of clean benchmark sandboxes.

Andon Labs' Real-World AI Evals: Claude calls the FBI, AI CEOs, price cartels, Butter-Bench, & Luna Andon Labs cofounders Lukas Petersson and Axel Backlund explain why dollar-denominated evals reveal what traditional benchmarks miss, how Claude ended up reporting a $2/day vending machine fee to the FBI, why long-horizon agents spiral in weird ways, what happens when agents lie, form price cartels, and compete with each other, and why the future of AI safety may depend on testing models in messy real-world environments instead of clean benchmark sandboxes.

Latent.Space

13,740 次观看 • 2 个月前

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:

Andrew Ng

126,507 次观看 • 1 年前

chat, is starknet gaming cooked? for episode two, loaf has joined me to talk about ai agents in gaming, eliza and Daydreams: Taskmarket . timestamps loaf introduction 00:46 what are ai agents? 01:16 what will gaming look like with ai agents? 02:02 whats the point of gaming with ai agents? 04:06 why is onchain gaming the ideal playground for AI agents? 05:11 is a gaming revolution imminent? 07:00 eliza & daydreams 08:36 endgame of ai agents and onchain games 13:33 when and where to play using agents 16:58 final remarks 18:47

chat, is starknet gaming cooked? for episode two, loaf has joined me to talk about ai agents in gaming, eliza and Daydreams: Taskmarket . timestamps loaf introduction 00:46 what are ai agents? 01:16 what will gaming look like with ai agents? 02:02 whats the point of gaming with ai agents? 04:06 why is onchain gaming the ideal playground for AI agents? 05:11 is a gaming revolution imminent? 07:00 eliza & daydreams 08:36 endgame of ai agents and onchain games 13:33 when and where to play using agents 16:58 final remarks 18:47

shadowfax

37,582 次观看 • 1 年前

Before founding Vals AI, Rayan Krishnan was a college radio host. We're thrilled to get him back in his element with Vivek Gopalan, this time talking all things benchmarks and evals. As AI capabilities compound, measuring them is becoming both harder and more important. What started as an academic exercise is quickly evolving into critical infrastructure for understanding what AI systems can actually do, and how they perform in the real world. Rayan and Vivek go deep on how benchmarks are evolving, measuring increasingly agentic systems, the weird and wonderful ways we test intelligence, the global AI race, and why evals sit on the critical path to ASI. 0:44 Importance and evolution of benchmarks 8:16 From academic artifact → real-world value 10:57 The next generation of benchmarks 13:34 Benchmarking models vs. agents 16:20 Building benchmarks for organizations 22:09 Weird and wonderful benchmarks 25:58 The AI arms race: US vs. China 31:02 Why evals are on the critical path to ASI 32:50 Vals Series A

Before founding Vals AI, Rayan Krishnan was a college radio host. We're thrilled to get him back in his element with Vivek Gopalan, this time talking all things benchmarks and evals. As AI capabilities compound, measuring them is becoming both harder and more important. What started as an academic exercise is quickly evolving into critical infrastructure for understanding what AI systems can actually do, and how they perform in the real world. Rayan and Vivek go deep on how benchmarks are evolving, measuring increasingly agentic systems, the weird and wonderful ways we test intelligence, the global AI race, and why evals sit on the critical path to ASI. 0:44 Importance and evolution of benchmarks 8:16 From academic artifact → real-world value 10:57 The next generation of benchmarks 13:34 Benchmarking models vs. agents 16:20 Building benchmarks for organizations 22:09 Weird and wonderful benchmarks 25:58 The AI arms race: US vs. China 31:02 Why evals are on the critical path to ASI 32:50 Vals Series A

8VC

102,528 次观看 • 2 天前

one thing that has saved my projects more time than I can count is evals boy was I excited when florian, quite literally an expert in benchmarks, agreed to hop into a ~2h interview to do a walkthrough of what the eval landscape looks like in 2026 (and also answer my personal business questions on the subject) given that now running frontier model through benchmarks is a vector for hacking other systems in order to avoid doing work (looking at you sol), I think it's more important than ever to educate folks on the evals situation. had a lot of fun throughout this session and I hope that you learn a thing or two! enjoy! 🌹 table of content: 0:00:00: are AI Benchmark broken? 0:05:45: Florian Brand background 0:09:00: what motivates florian to work on evaluation? 0:13:33: what is the mirrorcode benchmark about? 0:18:20: cheating in agent benchmark is insaneeeee 0:24:08: LLM benchmarks in era of agents 0:26:30: what’s up with the pelican man 0:28:27: evals are about capabilities 0:31:46: components of running evals 0:35:30: the volume of things to audit is huge!!! 0:40:20: expert answers are wrong hahahaha 0:46:00: api providers aren’t the same 0:48:00: benchmark narrow capabilities (synthetically) 0:50:56: link between eval and environment 0:53:45: small validated benchmark or massive bench? 0:56:11: what is your flow to review a benchmark? 0:58:30: tracking work capabilities with evaluation 1:00:20: slide deck in industry is all vibecoded 1:03:30: harness impact in the evaluation 1:07:39: hardware/sandboxes impact evaluation too! 1:11:00: “is it going to get worse?” 1:12:40: all components influence the final score 1:13:50: training models on different harnesses? 1:17:20: is the model just the weights or it’s all of it? 1:19:30: how to craft benchmark that prevent to cheating and undereliciting models in 2026 1:23:19: ways agents cheat and steal 1:26:00: correct elicitation of capabilities is important 1:36:00: building evaluation on prime intellect 1:45:10: how do you design interactivity benchmarks? 1:48:40: do you think evals are well set to reflect real world performance? 1:52:50: what will the benchmarking landscape will look like in 1 year

Yacine Mahdid

12,671 次观看 • 17 天前

After many months, a lot of model releases and even more Amp releases, here it is: A new episode of Raising An Agent with Quinn Slack and Thorsten Ball. They discuss: - Why Gemini 3 might have been the inflection point, not Opus 4.5 - What changed over Christmas and why everybody talks about agents now - "The assistant is dead, long live the factory" - How codebases need to adapt for agents - Why small/local models don't give you an impression of the frontier - What does it mean for a codebase to be agent-friendly? - New things coming to Amp Enjoy! And happy hacking! Timestamps: 1:51 The Impact of Gemini 3 and Opus 4.5 5:28 Shifting Perceptions of AI Agents 10:17 Adapting Codebases for AI Efficiency 14:48 The Future of Software Development with AI Agents 19:29 Challenges and Opportunities Ahead 24:00 The Importance of Frontier Models 25:12 Adapting to AI-Driven Workflows 26:57 The Future of Coding Agents 28:07 Building an Agentic Code Base 30:22 Rethinking Development Tools 31:44 Amp's Evolution and Team Dynamics 33:00 Discord Q&A 39:18 Teasing New Features: Deep

After many months, a lot of model releases and even more Amp releases, here it is: A new episode of Raising An Agent with Quinn Slack and Thorsten Ball. They discuss: - Why Gemini 3 might have been the inflection point, not Opus 4.5 - What changed over Christmas and why everybody talks about agents now - "The assistant is dead, long live the factory" - How codebases need to adapt for agents - Why small/local models don't give you an impression of the frontier - What does it mean for a codebase to be agent-friendly? - New things coming to Amp Enjoy! And happy hacking! Timestamps: 1:51 The Impact of Gemini 3 and Opus 4.5 5:28 Shifting Perceptions of AI Agents 10:17 Adapting Codebases for AI Efficiency 14:48 The Future of Software Development with AI Agents 19:29 Challenges and Opportunities Ahead 24:00 The Importance of Frontier Models 25:12 Adapting to AI-Driven Workflows 26:57 The Future of Coding Agents 28:07 Building an Agentic Code Base 30:22 Rethinking Development Tools 31:44 Amp's Evolution and Team Dynamics 33:00 Discord Q&A 39:18 Teasing New Features: Deep

Amp

36,150 次观看 • 6 个月前

Here's a recent talk I gave recapping the last 6-12 months of AI progress, why getting perfect models is hard, how labs are likely approaching the next phase of training (for agents), and other interesting tidbits across the reasoning landscape. Topics: 00:00 Introduction & the state of reasoning 05:50 Hillclimbing imperfect evals 09:18 Technical bottlenecks 13:02 Sycophancy 18:08 The Goldilocks Zone 19:28 What comes next? (hint, planning) 26:40 Q&A YouTube etc in replies. Thanks Kyle Corbitt and OpenPipe for hosting me.

Here's a recent talk I gave recapping the last 6-12 months of AI progress, why getting perfect models is hard, how labs are likely approaching the next phase of training (for agents), and other interesting tidbits across the reasoning landscape. Topics: 00:00 Introduction & the state of reasoning 05:50 Hillclimbing imperfect evals 09:18 Technical bottlenecks 13:02 Sycophancy 18:08 The Goldilocks Zone 19:28 What comes next? (hint, planning) 26:40 Q&A YouTube etc in replies. Thanks Kyle Corbitt and OpenPipe for hosting me.

Nathan Lambert

89,542 次观看 • 1 年前

How will companies build agents that work alongside their employees? I discuss this with Guillermo Rauch, the CEO of Vercel, who’s building V... an internal agent used by almost 1,000 people across Vercel. We talk about: - The Openclaw Movement - How to get started building agents for your team? - Do we build a team of many agents or one GOD agent? - Hard parts of creating agents - Agents that delegate to other agents with different permissions and computer access - Eve The framework for building agents This is Agent Native #4 00:00 Intro 01:21 The State of Agent Adoption 07:26 "V" - Vercel's Internal Company Agent 14:32 Lessons from OpenClaw: Soul & Computer use 22:20 One God Agent OR a Team of Agents 27:11 Build Your Own Agent: Skills, Tools & Permissions 33:52 Proactive Agents, Triggers & Self-Improvement 40:30 Who Will Build Agents & Getting Started 46:22 Open Models - Cheaper, Faster, Better 53:05 Closing Thoughts: Owning Your Own Intelligence

How will companies build agents that work alongside their employees? I discuss this with Guillermo Rauch, the CEO of Vercel, who’s building V... an internal agent used by almost 1,000 people across Vercel. We talk about: - The Openclaw Movement - How to get started building agents for your team? - Do we build a team of many agents or one GOD agent? - Hard parts of creating agents - Agents that delegate to other agents with different permissions and computer access - Eve The framework for building agents This is Agent Native #4 00:00 Intro 01:21 The State of Agent Adoption 07:26 "V" - Vercel's Internal Company Agent 14:32 Lessons from OpenClaw: Soul & Computer use 22:20 One God Agent OR a Team of Agents 27:11 Build Your Own Agent: Skills, Tools & Permissions 33:52 Proactive Agents, Triggers & Self-Improvement 40:30 Who Will Build Agents & Getting Started 46:22 Open Models - Cheaper, Faster, Better 53:05 Closing Thoughts: Owning Your Own Intelligence

Riley Brown

70,080 次观看 • 10 天前