Загрузка видео...

Не удалось загрузить видео

На главную

Why are evals on production agents so much more difficult than evals on coding agents?

resolveai's profile picture

ResolveAI

2,664 subscribers

15,248 просмотров • 3 месяцев назад •via X (Twitter)

Наука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 1

Фото профиля Zahran 🕹️🤖
Zahran 🕹️🤖3 месяцев назад

Because production agents operate in open, messy environments. Coding agents have clearer success criteria. Production agents deal with changing state, real users, side effects, and failures that only appear after deployment.

Похожие видео

🎙️ New Deployed episode with Ryan Carson, "Builder in Residence" at Amp. It's a great convo about building code agents and building as a solo founder at the same time, and it's refreshingly honest. Case in point, on evals for coding agents: "We do not have a set of evals that we programmatically run to decide if GPT-5 is better than Sonnet-4 yet... everybody in the trenches building these agentic coding tools understands that it is so dynamic, and so frustrating." Great conversation about building with AI, developer experience, and what happens when everyone can code. Check it out. 👇
2:02

🎙️ New Deployed episode with Ryan Carson, "Builder in Residence" at Amp. It's a great convo about building code agents and building as a solo founder at the same time, and it's refreshingly honest. Case in point, on evals for coding agents: "We do not have a set of evals that we programmatically run to decide if GPT-5 is better than Sonnet-4 yet... everybody in the trenches building these agentic coding tools understands that it is so dynamic, and so frustrating." Great conversation about building with AI, developer experience, and what happens when everyone can code. Check it out. 👇

Ian Cairns

250,886 просмотров • 1 год назад

Today we’re shipping new capabilities that make Resolve AI the platform where engineering teams run and fix production software with AI agents. New capabilities include background agents that run operational tasks, new agent architecture that delivers 2x investigation quality, new agent capabilities like governed actions, and new ways to work with agents in UI or terminal. With Resolve AI engineering teams can: - Delegate on-call to agents - Co-work with agents to resolve incidents - Run operational tasks with background agents.
1:11

Today we’re shipping new capabilities that make Resolve AI the platform where engineering teams run and fix production software with AI agents. New capabilities include background agents that run operational tasks, new agent architecture that delivers 2x investigation quality, new agent capabilities like governed actions, and new ways to work with agents in UI or terminal. With Resolve AI engineering teams can: - Delegate on-call to agents - Co-work with agents to resolve incidents - Run operational tasks with background agents.

Resolve AI

313,949 просмотров • 5 месяцев назад

AGNOST AI (YC S26) JUST LAUNCHED ON PRODUCT HUNT! AI agents don’t fail like traditional software. They return 200. They pass evals. And they still quietly frustrate & churn users. Agnost AI finds those silent failures that evals miss & only show up in production conversations. Then we turn those failures into harness improvements & better models.
1:43

AGNOST AI (YC S26) JUST LAUNCHED ON PRODUCT HUNT! AI agents don’t fail like traditional software. They return 200. They pass evals. And they still quietly frustrate & churn users. Agnost AI finds those silent failures that evals miss & only show up in production conversations. Then we turn those failures into harness improvements & better models.

shubham

198,998 просмотров • 2 месяцев назад

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:
1:12

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 просмотров • 1 год назад

An entire AI production house inside your laptop. invideo autonomously takes 500+ creative calls and evals on script, voiceover, sound, and edit to bring ANY idea you have, to life. Created with ❤️ entirely using 'Agents & Models' in invideo. More details coming soon.
3:35

An entire AI production house inside your laptop. invideo autonomously takes 500+ creative calls and evals on script, voiceover, sound, and edit to bring ANY idea you have, to life. Created with ❤️ entirely using 'Agents & Models' in invideo. More details coming soon.

Sanket Shah

1,169,522 просмотров • 1 год назад

Evals 101: a new course from Braintrust. Everything you need to know about evals, and how to do them yourself. Module one: Why are evals important? - the six most common problems developers face when shipping AI applications - why traditional software thinking doesn't apply to AI - how evals can fix these problems
1:51

Evals 101: a new course from Braintrust. Everything you need to know about evals, and how to do them yourself. Module one: Why are evals important? - the six most common problems developers face when shipping AI applications - why traditional software thinking doesn't apply to AI - how evals can fix these problems

Braintrust

37,043 просмотров • 6 месяцев назад

We beat Claude Code in our evals: 61% to 53%. Yes, evals are actually important haha And now we’re open-sourcing our entire codebase and launching an agent framework! Now you can build custom agents for your project and get better results 🔥
5:42

We beat Claude Code in our evals: 61% to 53%. Yes, evals are actually important haha And now we’re open-sourcing our entire codebase and launching an agent framework! Now you can build custom agents for your project and get better results 🔥

James Grugett

277,025 просмотров • 1 год назад

Introducing Respan Evals 2.0: Automation. Most AI production issues don’t show up as errors. They show up as answers that look valid but are wrong, incomplete, unsafe, or off-brand. Respan now runs automated online evals on live production logs to catch silent quality failures in real time.
1:07

Introducing Respan Evals 2.0: Automation. Most AI production issues don’t show up as errors. They show up as answers that look valid but are wrong, incomplete, unsafe, or off-brand. Respan now runs automated online evals on live production logs to catch silent quality failures in real time.

Respan

110,832 просмотров • 4 месяцев назад

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC
13:01

Ontologies, Evals & Deployed Intelligence: Building Real AI Agents - Mitchell Troyanovsky, co-founder of Basis 0:04: Intro /Meet Mitch Troyanovsky 2:20: What is an AI Agent, actually? 4:07: The Gap Between LLMs and Embodied Intelligence 5:25: Why Ontology is the Missing Piece of Agent Design 8:10: Evals, Evals, Evals 9:57: Building a Deployed Intelligence Team 11:35: The Path to Being a Good AI Agent Builder From our most recent #DataDrivenNYC

Matt Turck

11,155 просмотров • 1 год назад

You should not spend more then 2 Days on Evals if you know what you are doing 👍
0:27

You should not spend more then 2 Days on Evals if you know what you are doing 👍

TimNQ

44,637 просмотров • 4 месяцев назад

The Agent SDK is now available. Define agents as code, with prompts in Markdown and tools in TypeScript. Deploy them on our hosted infrastructure, or run them on your own. Built-in evals let you measure every change and make your agents better over time.
0:28

The Agent SDK is now available. Define agents as code, with prompts in Markdown and tools in TypeScript. Deploy them on our hosted infrastructure, or run them on your own. Built-in evals let you measure every change and make your agents better over time.

Cursor

413,714 просмотров • 13 дней назад

Everything Gets Rebuilt: my conversation with Harrison Chase, CEO of LangChain about agent harnesses, evals, runtimes, sandboxes, MCP and the future of the agent stack 00:00 Intro - meet Harrison Chase - at the Chase Center for the Daytona Compute conference 01:32 What changed in agents over the last year 03:57 Why coding agents are ahead 06:26 Do models commoditize the framework layer? 08:27 Harnesses, in plain English 10:11 Why system prompts matter so much 13:11 The upside — and downside — of subagents 15:31 Why a useful agent needs a filesystem 18:13 Additional core primitives of modern agents 19:12 Skills: the new primitive 20:19 What context compaction actually means 23:02 How memory works in agents 25:16 One mega-agent or many specialized agents? 27:46 The future of MCP 29:38 Why agents need sandboxes 32:35 How sandboxes help with security 33:32 How Harrison Chase started LangChain 37:24 LangChain vs LangGraph vs Deep Agents 40:17 Why observability matters more for agents 41:48 Evals, no-code, and continuous improvement 44:41 What LangChain is building next 45:29 Where the real moat in AI lives
46:57

Everything Gets Rebuilt: my conversation with Harrison Chase, CEO of LangChain about agent harnesses, evals, runtimes, sandboxes, MCP and the future of the agent stack 00:00 Intro - meet Harrison Chase - at the Chase Center for the Daytona Compute conference 01:32 What changed in agents over the last year 03:57 Why coding agents are ahead 06:26 Do models commoditize the framework layer? 08:27 Harnesses, in plain English 10:11 Why system prompts matter so much 13:11 The upside — and downside — of subagents 15:31 Why a useful agent needs a filesystem 18:13 Additional core primitives of modern agents 19:12 Skills: the new primitive 20:19 What context compaction actually means 23:02 How memory works in agents 25:16 One mega-agent or many specialized agents? 27:46 The future of MCP 29:38 Why agents need sandboxes 32:35 How sandboxes help with security 33:32 How Harrison Chase started LangChain 37:24 LangChain vs LangGraph vs Deep Agents 40:17 Why observability matters more for agents 41:48 Evals, no-code, and continuous improvement 44:41 What LangChain is building next 45:29 Where the real moat in AI lives

Matt Turck

35,333 просмотров • 8 месяцев назад

In 13 minutes, Jeff Barg, Vyshu Khota, and Soroush Khadem walk through how Clay scaled agent evals agents at 300M+ runs a month. Topics covered: ✅ Their four quadrant eval framework ✅ Why closing the production-to-eval loop is the hardest part ✅ How a data lake and long context changed what agents can do with data
13:00

In 13 minutes, Jeff Barg, Vyshu Khota, and Soroush Khadem walk through how Clay scaled agent evals agents at 300M+ runs a month. Topics covered: ✅ Their four quadrant eval framework ✅ Why closing the production-to-eval loop is the hardest part ✅ How a data lake and long context changed what agents can do with data

LangChain

11,325 просмотров • 2 месяцев назад

Andon Labs' Real-World AI Evals: Claude calls the FBI, AI CEOs, price cartels, Butter-Bench, & Luna Andon Labs cofounders Lukas Petersson and Axel Backlund explain why dollar-denominated evals reveal what traditional benchmarks miss, how Claude ended up reporting a $2/day vending machine fee to the FBI, why long-horizon agents spiral in weird ways, what happens when agents lie, form price cartels, and compete with each other, and why the future of AI safety may depend on testing models in messy real-world environments instead of clean benchmark sandboxes.
0:23

Andon Labs' Real-World AI Evals: Claude calls the FBI, AI CEOs, price cartels, Butter-Bench, & Luna Andon Labs cofounders Lukas Petersson and Axel Backlund explain why dollar-denominated evals reveal what traditional benchmarks miss, how Claude ended up reporting a $2/day vending machine fee to the FBI, why long-horizon agents spiral in weird ways, what happens when agents lie, form price cartels, and compete with each other, and why the future of AI safety may depend on testing models in messy real-world environments instead of clean benchmark sandboxes.

Latent.Space

13,810 просмотров • 5 месяцев назад

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!
1:15

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!

AI Engineer

55,348 просмотров • 1 год назад

Ex-Google engineer explained AI agent loops, harness, evals in 20 minutes - better than 500$ courses. trace every run → judge it with an LLM → diagnose → fix → ship. That loop is how agents self-improve over time. Agent loops + memory + harness + evals - thats the stack. Watch it, then save the framework below.
19:30

Ex-Google engineer explained AI agent loops, harness, evals in 20 minutes - better than 500$ courses. trace every run → judge it with an LLM → diagnose → fix → ship. That loop is how agents self-improve over time. Agent loops + memory + harness + evals - thats the stack. Watch it, then save the framework below.

Movez

568,462 просмотров • 4 месяцев назад