Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Why are evals on production agents so much more difficult than evals on coding agents?

ResolveAI

2,664 subscribers

15,248 views • 3 months ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

1 Comments

Zahran 🕹️🤖3 months ago

Because production agents operate in open, messy environments. Coding agents have clearer success criteria. Production agents deal with changing state, real users, side effects, and failures that only appear after deployment.

Related Videos

🎙️ New Deployed episode with Ryan Carson, "Builder in Residence" at Amp. It's a great convo about building code agents and building as a solo founder at the same time, and it's refreshingly honest. Case in point, on evals for coding agents: "We do not have a set of evals that we programmatically run to decide if GPT-5 is better than Sonnet-4 yet... everybody in the trenches building these agentic coding tools understands that it is so dynamic, and so frustrating." Great conversation about building with AI, developer experience, and what happens when everyone can code. Check it out. 👇

🎙️ New Deployed episode with Ryan Carson, "Builder in Residence" at Amp. It's a great convo about building code agents and building as a solo founder at the same time, and it's refreshingly honest. Case in point, on evals for coding agents: "We do not have a set of evals that we programmatically run to decide if GPT-5 is better than Sonnet-4 yet... everybody in the trenches building these agentic coding tools understands that it is so dynamic, and so frustrating." Great conversation about building with AI, developer experience, and what happens when everyone can code. Check it out. 👇

Ian Cairns

250,886 views • 1 year ago

Today we’re shipping new capabilities that make Resolve AI the platform where engineering teams run and fix production software with AI agents. New capabilities include background agents that run operational tasks, new agent architecture that delivers 2x investigation quality, new agent capabilities like governed actions, and new ways to work with agents in UI or terminal. With Resolve AI engineering teams can: - Delegate on-call to agents - Co-work with agents to resolve incidents - Run operational tasks with background agents.

Today we’re shipping new capabilities that make Resolve AI the platform where engineering teams run and fix production software with AI agents. New capabilities include background agents that run operational tasks, new agent architecture that delivers 2x investigation quality, new agent capabilities like governed actions, and new ways to work with agents in UI or terminal. With Resolve AI engineering teams can: - Delegate on-call to agents - Co-work with agents to resolve incidents - Run operational tasks with background agents.

Resolve AI

313,949 views • 5 months ago

AGNOST AI (YC S26) JUST LAUNCHED ON PRODUCT HUNT! AI agents don’t fail like traditional software. They return 200. They pass evals. And they still quietly frustrate & churn users. Agnost AI finds those silent failures that evals miss & only show up in production conversations. Then we turn those failures into harness improvements & better models.

AGNOST AI (YC S26) JUST LAUNCHED ON PRODUCT HUNT! AI agents don’t fail like traditional software. They return 200. They pass evals. And they still quietly frustrate & churn users. Agnost AI finds those silent failures that evals miss & only show up in production conversations. Then we turn those failures into harness improvements & better models.

shubham

198,998 views • 2 months ago

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 views • 1 year ago

An entire AI production house inside your laptop. invideo autonomously takes 500+ creative calls and evals on script, voiceover, sound, and edit to bring ANY idea you have, to life. Created with ❤️ entirely using 'Agents & Models' in invideo. More details coming soon.

An entire AI production house inside your laptop. invideo autonomously takes 500+ creative calls and evals on script, voiceover, sound, and edit to bring ANY idea you have, to life. Created with ❤️ entirely using 'Agents & Models' in invideo. More details coming soon.

Sanket Shah

1,169,522 views • 1 year ago

Evals 101: a new course from Braintrust. Everything you need to know about evals, and how to do them yourself. Module one: Why are evals important? - the six most common problems developers face when shipping AI applications - why traditional software thinking doesn't apply to AI - how evals can fix these problems

Evals 101: a new course from Braintrust. Everything you need to know about evals, and how to do them yourself. Module one: Why are evals important? - the six most common problems developers face when shipping AI applications - why traditional software thinking doesn't apply to AI - how evals can fix these problems

Braintrust

37,043 views • 6 months ago

We beat Claude Code in our evals: 61% to 53%. Yes, evals are actually important haha And now we’re open-sourcing our entire codebase and launching an agent framework! Now you can build custom agents for your project and get better results 🔥

We beat Claude Code in our evals: 61% to 53%. Yes, evals are actually important haha And now we’re open-sourcing our entire codebase and launching an agent framework! Now you can build custom agents for your project and get better results 🔥

James Grugett

277,025 views • 1 year ago

Introducing Respan Evals 2.0: Automation. Most AI production issues don’t show up as errors. They show up as answers that look valid but are wrong, incomplete, unsafe, or off-brand. Respan now runs automated online evals on live production logs to catch silent quality failures in real time.

Respan

110,832 views • 4 months ago

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC

Matt Turck

11,155 views • 1 year ago

You should not spend more then 2 Days on Evals if you know what you are doing 👍

You should not spend more then 2 Days on Evals if you know what you are doing 👍

TimNQ

44,637 views • 4 months ago

The Agent SDK is now available. Define agents as code, with prompts in Markdown and tools in TypeScript. Deploy them on our hosted infrastructure, or run them on your own. Built-in evals let you measure every change and make your agents better over time.

The Agent SDK is now available. Define agents as code, with prompts in Markdown and tools in TypeScript. Deploy them on our hosted infrastructure, or run them on your own. Built-in evals let you measure every change and make your agents better over time.

Cursor

413,714 views • 13 days ago

Everything Gets Rebuilt: my conversation with Harrison Chase, CEO of LangChain about agent harnesses, evals, runtimes, sandboxes, MCP and the future of the agent stack 00:00 Intro - meet Harrison Chase - at the Chase Center for the Daytona Compute conference 01:32 What changed in agents over the last year 03:57 Why coding agents are ahead 06:26 Do models commoditize the framework layer? 08:27 Harnesses, in plain English 10:11 Why system prompts matter so much 13:11 The upside — and downside — of subagents 15:31 Why a useful agent needs a filesystem 18:13 Additional core primitives of modern agents 19:12 Skills: the new primitive 20:19 What context compaction actually means 23:02 How memory works in agents 25:16 One mega-agent or many specialized agents? 27:46 The future of MCP 29:38 Why agents need sandboxes 32:35 How sandboxes help with security 33:32 How Harrison Chase started LangChain 37:24 LangChain vs LangGraph vs Deep Agents 40:17 Why observability matters more for agents 41:48 Evals, no-code, and continuous improvement 44:41 What LangChain is building next 45:29 Where the real moat in AI lives

Everything Gets Rebuilt: my conversation with Harrison Chase, CEO of LangChain about agent harnesses, evals, runtimes, sandboxes, MCP and the future of the agent stack 00:00 Intro - meet Harrison Chase - at the Chase Center for the Daytona Compute conference 01:32 What changed in agents over the last year 03:57 Why coding agents are ahead 06:26 Do models commoditize the framework layer? 08:27 Harnesses, in plain English 10:11 Why system prompts matter so much 13:11 The upside — and downside — of subagents 15:31 Why a useful agent needs a filesystem 18:13 Additional core primitives of modern agents 19:12 Skills: the new primitive 20:19 What context compaction actually means 23:02 How memory works in agents 25:16 One mega-agent or many specialized agents? 27:46 The future of MCP 29:38 Why agents need sandboxes 32:35 How sandboxes help with security 33:32 How Harrison Chase started LangChain 37:24 LangChain vs LangGraph vs Deep Agents 40:17 Why observability matters more for agents 41:48 Evals, no-code, and continuous improvement 44:41 What LangChain is building next 45:29 Where the real moat in AI lives

Matt Turck

35,333 views • 8 months ago

In 13 minutes, Jeff Barg, Vyshu Khota, and Soroush Khadem walk through how Clay scaled agent evals agents at 300M+ runs a month. Topics covered: ✅ Their four quadrant eval framework ✅ Why closing the production-to-eval loop is the hardest part ✅ How a data lake and long context changed what agents can do with data

In 13 minutes, Jeff Barg, Vyshu Khota, and Soroush Khadem walk through how Clay scaled agent evals agents at 300M+ runs a month. Topics covered: ✅ Their four quadrant eval framework ✅ Why closing the production-to-eval loop is the hardest part ✅ How a data lake and long context changed what agents can do with data

LangChain

11,325 views • 2 months ago

Andon Labs' Real-World AI Evals: Claude calls the FBI, AI CEOs, price cartels, Butter-Bench, & Luna Andon Labs cofounders Lukas Petersson and Axel Backlund explain why dollar-denominated evals reveal what traditional benchmarks miss, how Claude ended up reporting a $2/day vending machine fee to the FBI, why long-horizon agents spiral in weird ways, what happens when agents lie, form price cartels, and compete with each other, and why the future of AI safety may depend on testing models in messy real-world environments instead of clean benchmark sandboxes.

Andon Labs' Real-World AI Evals: Claude calls the FBI, AI CEOs, price cartels, Butter-Bench, & Luna Andon Labs cofounders Lukas Petersson and Axel Backlund explain why dollar-denominated evals reveal what traditional benchmarks miss, how Claude ended up reporting a $2/day vending machine fee to the FBI, why long-horizon agents spiral in weird ways, what happens when agents lie, form price cartels, and compete with each other, and why the future of AI safety may depend on testing models in messy real-world environments instead of clean benchmark sandboxes.

Latent.Space

13,810 views • 5 months ago

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!

AI Engineer

55,348 views • 1 year ago

Ex-Google engineer explained AI agent loops, harness, evals in 20 minutes - better than 500$ courses. trace every run → judge it with an LLM → diagnose → fix → ship. That loop is how agents self-improve over time. Agent loops + memory + harness + evals - thats the stack. Watch it, then save the framework below.

Ex-Google engineer explained AI agent loops, harness, evals in 20 minutes - better than 500$ courses. trace every run → judge it with an LLM → diagnose → fix → ship. That loop is how agents self-improve over time. Agent loops + memory + harness + evals - thats the stack. Watch it, then save the framework below.

Movez

568,462 views • 4 months ago