Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Introducing Respan Evals 2.0: Automation. Most AI production issues don’t show up as errors. They show up as answers that look valid but are wrong, incomplete, unsafe, or off-brand. Respan now runs automated online evals on live production logs to catch silent quality failures in real time.

RespanAI's profile picture

Respan

2,043 subscribers

110,832 Aufrufe • vor 2 Monaten •via X (Twitter)

Wissenschaft & TechnologieNachrichten & Politik
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

AGNOST AI (YC S26) JUST LAUNCHED ON PRODUCT HUNT! AI agents don’t fail like traditional software. They return 200. They pass evals. And they still quietly frustrate & churn users. Agnost AI finds those silent failures that evals miss & only show up in production conversations. Then we turn those failures into harness improvements & better models.
1:43

AGNOST AI (YC S26) JUST LAUNCHED ON PRODUCT HUNT! AI agents don’t fail like traditional software. They return 200. They pass evals. And they still quietly frustrate & churn users. Agnost AI finds those silent failures that evals miss & only show up in production conversations. Then we turn those failures into harness improvements & better models.

shubham

198,237 Aufrufe • vor 1 Monat

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.
2:13

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.

Respan

127,680 Aufrufe • vor 5 Monaten

The hard part about LLM failures is that their outputs rarely look like failures. The demo “works.” The output sounds coherent. The user actively uses the product. And your dashboard looks normal. Meanwhile, the system can be wrong, unsafe, or quietly driving up token spend. And you won’t notice until the damage adds up. Prompts often serve as business logic (policies, safety, and product context). But many teams ship them without the basics, such as versioning, reviewable changes, end-to-end traces, and eval gates. In production, it doesn’t crash. It degrades via wrong answers, policy misses, and surprise spending. No crash. No error. No alert. I cover this exact issue in my Stanford University CS 224G guest lecture on AI Observability and Evaluations. Here are the core ideas: • If you only log the final output, you’re guessing. Full traces show where it broke. • Evals are feedback loops. Use clear pass/fail criteria tied to outcomes. • Run evals continuously on production traces and don’t wait for support tickets. The moat isn’t prompt cleverness. It’s a measured improvement. Full lecture + blog below 👇
59:40

The hard part about LLM failures is that their outputs rarely look like failures. The demo “works.” The output sounds coherent. The user actively uses the product. And your dashboard looks normal. Meanwhile, the system can be wrong, unsafe, or quietly driving up token spend. And you won’t notice until the damage adds up. Prompts often serve as business logic (policies, safety, and product context). But many teams ship them without the basics, such as versioning, reviewable changes, end-to-end traces, and eval gates. In production, it doesn’t crash. It degrades via wrong answers, policy misses, and surprise spending. No crash. No error. No alert. I cover this exact issue in my Stanford University CS 224G guest lecture on AI Observability and Evaluations. Here are the core ideas: • If you only log the final output, you’re guessing. Full traces show where it broke. • Evals are feedback loops. Use clear pass/fail criteria tied to outcomes. • Run evals continuously on production traces and don’t wait for support tickets. The moat isn’t prompt cleverness. It’s a measured improvement. Full lecture + blog below 👇

Arsh Shah Dilbagi

20,972 Aufrufe • vor 6 Monaten

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:
1:12

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 Aufrufe • vor 1 Jahr

Raindrop (Raindrop) is building the safety layer for AI agents. As agents get more capable and take on more complex work, Raindrop detects when they go wrong in production, from failed tool calls and hallucinations to problems companies didn’t even know to look for. Now they’re launching Raindrop Simulations, which brings that same approach earlier in the development process to catch failures before agents reach production. Already used by companies including Vercel, Clay, Framer, and Speak, Raindrop just raised a Series A bringing its total funding to $50 million. In this episode of Founder Firesides, founders zubin koticha, ben hylak, and alexis sat down with Diana to share how a tool they originally built to debug their own coding agent became the company, and why better AI agents actually create more ways for things to go wrong. 00:05 — What Raindrop Does 01:32 — Why Better AI Agents Create More Problems 03:25 — Finding the Failures Evals Miss 04:38 — How Raindrop Got Started 06:32 — Why They Kept Going 07:58 — Catching Agent Failures Before Production
10:17

Raindrop (Raindrop) is building the safety layer for AI agents. As agents get more capable and take on more complex work, Raindrop detects when they go wrong in production, from failed tool calls and hallucinations to problems companies didn’t even know to look for. Now they’re launching Raindrop Simulations, which brings that same approach earlier in the development process to catch failures before agents reach production. Already used by companies including Vercel, Clay, Framer, and Speak, Raindrop just raised a Series A bringing its total funding to $50 million. In this episode of Founder Firesides, founders zubin koticha, ben hylak, and alexis sat down with Diana to share how a tool they originally built to debug their own coding agent became the company, and why better AI agents actually create more ways for things to go wrong. 00:05 — What Raindrop Does 01:32 — Why Better AI Agents Create More Problems 03:25 — Finding the Failures Evals Miss 04:38 — How Raindrop Got Started 06:32 — Why They Kept Going 07:58 — Catching Agent Failures Before Production

Y Combinator

65,544 Aufrufe • vor 10 Tagen

Eval scores in 2026 are broken. MMLU at 91%+, GSM8K at 94%+, yet models still can't handle basic multi-step tasks. And reported scores don't even agree across model cards, papers, and platforms. We just shipped Community Evals on Hugging Face: - Benchmark datasets now host live leaderboards (MMLU-Pro, GPQA, HLE) - Scores live in model repos as versioned YAML - Anyone can submit evals to any model via PR without merging. - Verified badges for reproducible runs via Inspect AI This won't fix saturation or stop test set contamination. But it makes the game visible. What was evaluated, how, when, and by whom. Done trusting black-box leaderboards. Time to decentralize evals.
0:29

Eval scores in 2026 are broken. MMLU at 91%+, GSM8K at 94%+, yet models still can't handle basic multi-step tasks. And reported scores don't even agree across model cards, papers, and platforms. We just shipped Community Evals on Hugging Face: - Benchmark datasets now host live leaderboards (MMLU-Pro, GPQA, HLE) - Scores live in model repos as versioned YAML - Anyone can submit evals to any model via PR without merging. - Verified badges for reproducible runs via Inspect AI This won't fix saturation or stop test set contamination. But it makes the game visible. What was evaluated, how, when, and by whom. Done trusting black-box leaderboards. Time to decentralize evals.

Ben Burtenshaw

19,259 Aufrufe • vor 7 Monaten

"I like to move extremely fast, but in order to move fast, you need to have brakes." Up to 200,000 AI agents wake up at Kavak every day. CPO & AI Officer Alejandro Maza says eval quality is what sets the ceiling on speed at that scale: "So to give you a sense of the scale, 96% of all interactions are handled by agents. No humans there. 95% of all transactions are completely handled by agents." "AI is super powerful, and I've seen many companies get this wrong because they try to go slow because they don't have the right brakes. So I thought about it the other way around: how fast can we go? Well, it depends on the quality of our evals." "A good rule of thumb here is we spend about the same amount of engineer time, tokens, and money on building the evals as building the agents." Alejandro Maza
1:39

"I like to move extremely fast, but in order to move fast, you need to have brakes." Up to 200,000 AI agents wake up at Kavak every day. CPO & AI Officer Alejandro Maza says eval quality is what sets the ceiling on speed at that scale: "So to give you a sense of the scale, 96% of all interactions are handled by agents. No humans there. 95% of all transactions are completely handled by agents." "AI is super powerful, and I've seen many companies get this wrong because they try to go slow because they don't have the right brakes. So I thought about it the other way around: how fast can we go? Well, it depends on the quality of our evals." "A good rule of thumb here is we spend about the same amount of engineer time, tokens, and money on building the evals as building the agents." Alejandro Maza

a16z

41,467 Aufrufe • vor 1 Monat

We gave AI agents access to real production systems and watched them do things they were never authorized to do. SOC 2 and ISO can't catch it, but TrustAI (YC S26) can. We built new evals so companies can have a continuous, transparent record on key vulnerabilities, get suggested mitigative solutions, and run time/cost efficiency analysis. Built by two Massachusetts Institute of Technology (MIT) engineers, backed by Y Combinator. Considering or running an agent on sensitive systems? Building one that connects to other enterprises' private data? Reach out today at
0:45

We gave AI agents access to real production systems and watched them do things they were never authorized to do. SOC 2 and ISO can't catch it, but TrustAI (YC S26) can. We built new evals so companies can have a continuous, transparent record on key vulnerabilities, get suggested mitigative solutions, and run time/cost efficiency analysis. Built by two Massachusetts Institute of Technology (MIT) engineers, backed by Y Combinator. Considering or running an agent on sensitive systems? Building one that connects to other enterprises' private data? Reach out today at

Hannah Chung

35,214 Aufrufe • vor 2 Monaten

Over 1 billion PDFs are created every day, but your agents still can’t read them reliably. Today we’re releasing Parse 2.0, the most accurate document parsing API in the world. Extend already processes millions of pages daily for leading AI teams like Brex, Mercury, Opendoor, Flatiron Health, and hundreds of others. Now, its even better. Parse 2.0 is SOTA quality on RealDoc-Bench, our open source benchmark that measures agent success rate on real world docs that agents actually encounter in production. We trained Parse 2.0 on 1M+ pages of the hardest documents seen in production. Here’s how it stacks up: - #1 in healthcare, real estate, logistics, and financial services - 95.7% agent Q&A accuracy on 581 docs (next best: 92%) - 0.847 F1 on layout (next best: 0.759) Give it a try today and build production-ready document agents with Extend.
1:32

Over 1 billion PDFs are created every day, but your agents still can’t read them reliably. Today we’re releasing Parse 2.0, the most accurate document parsing API in the world. Extend already processes millions of pages daily for leading AI teams like Brex, Mercury, Opendoor, Flatiron Health, and hundreds of others. Now, its even better. Parse 2.0 is SOTA quality on RealDoc-Bench, our open source benchmark that measures agent success rate on real world docs that agents actually encounter in production. We trained Parse 2.0 on 1M+ pages of the hardest documents seen in production. Here’s how it stacks up: - #1 in healthcare, real estate, logistics, and financial services - 95.7% agent Q&A accuracy on 581 docs (next best: 92%) - 0.847 F1 on layout (next best: 0.759) Give it a try today and build production-ready document agents with Extend.

Kushal Byatnal

588,683 Aufrufe • vor 4 Monaten

Cybercab production line work continues and more vehicles seem to be leaving the factory and headed to the outbound lot. Today I saw another ~60 but this time they were spread all over the outbound lot, some driving to the Superchargers, others doing driving testing while even more look to be preparing for transportation off site. Now that we are in mid-April, I’m hoping we will begin to see even more of these showing up at the outbound lot and if that starts to happen, it will give a better idea how the production line is doing as far as ramping up.
3:07

Cybercab production line work continues and more vehicles seem to be leaving the factory and headed to the outbound lot. Today I saw another ~60 but this time they were spread all over the outbound lot, some driving to the Superchargers, others doing driving testing while even more look to be preparing for transportation off site. Now that we are in mid-April, I’m hoping we will begin to see even more of these showing up at the outbound lot and if that starts to happen, it will give a better idea how the production line is doing as far as ramping up.

Joe Tegtmeyer 🚀 🤠🛸😎

79,426 Aufrufe • vor 5 Monaten

I have been hyping Brian Scanlan and Intercom to every eng leader I've spoken to since getting a behind the scenes look at how they 2x'd their productivity in their R&D org. In this week's episode of How I AI, Brian pulls up dashboards and tools and shows us how it's done: - explicit targets + reporting on leading indicators of AI adoption - how they measured how quality increased with more AI usage - centralized skills repo for the entire company - telemetry on claude code + evals on internal success - building for an agent-first onboarding flow Sorry to be that bro, but truly: STOP what you are doing and watch this right now if you are looking to inflect your eng + product team with AI. As always, a huge thanks to our sponsors: 🧠Celigo - Intelligent automation built for AI: Cursor - The best (and my fave!) way to code with AI: On YT now 👉
1:03

I have been hyping Brian Scanlan and Intercom to every eng leader I've spoken to since getting a behind the scenes look at how they 2x'd their productivity in their R&D org. In this week's episode of How I AI, Brian pulls up dashboards and tools and shows us how it's done: - explicit targets + reporting on leading indicators of AI adoption - how they measured how quality increased with more AI usage - centralized skills repo for the entire company - telemetry on claude code + evals on internal success - building for an agent-first onboarding flow Sorry to be that bro, but truly: STOP what you are doing and watch this right now if you are looking to inflect your eng + product team with AI. As always, a huge thanks to our sponsors: 🧠Celigo - Intelligent automation built for AI: Cursor - The best (and my fave!) way to code with AI: On YT now 👉

claire vo 🖤

12,864 Aufrufe • vor 5 Monaten

Your inbox is full of AI-generated sales emails that you don’t care about. Especially in enterprise, you can’t afford to miss. Therefore, we’re taking a different approach to breaking into new accounts. - Identify and act on strategic insights in real-time - Write outbound that targets real problems - Show up to every meeting ready to speak credibly Here’s what your day as an enterprise seller will look like.
1:54

Your inbox is full of AI-generated sales emails that you don’t care about. Especially in enterprise, you can’t afford to miss. Therefore, we’re taking a different approach to breaking into new accounts. - Identify and act on strategic insights in real-time - Write outbound that targets real problems - Show up to every meeting ready to speak credibly Here’s what your day as an enterprise seller will look like.

Keith Peiris

290,809 Aufrufe • vor 1 Jahr

Alright, this one’s worth your attention if you’re building or deploying agents. Future AGI just open-sourced their entire platform and i don’t mean a trimmed-down version. this is the full stack: UI, backend, simulation engine, evals, optimization loop, observability, guardrails, gateway, docs. all in one repo. Apache 2.0. I’ve been putting it through its paces on production agents, and what stands out isn’t just the breadth it’s the architecture. Most of the current “agent reliability” stack is fragmented. tracing lives in one tool, evals in another, guardrails somewhere else. you end up manually connecting dots, and the agent itself doesn’t really improve you just keep patching prompts and hoping for the best. This flips that model. It’s built as a closed feedback loop: simulate failures → evaluate in real time → detect production issues → learn from them → generate fixes → validate against real traffic → check regressions → redeploy → monitor again And when something new breaks, the loop just runs again. no manual glue. The simulation piece is especially strong. instead of static test cases, it generates adversarial, multi-turn conversations based on how your agent actually behaves basically hunting for the exact scenarios where your system fails confidently. ran a few thousand simulations on our side… caught things we definitely would’ve missed. Evals run fast (sub-50ms) across modalities. not LLM-as-judge trained classifiers. guardrails are built-in, not layered on top. observability gives you step-level visibility into reasoning, cost, latency, quality. But the real shift is the optimization loop. Most tools tell you *what* broke. this system actually fixes it, validates the fix, and ensures nothing else regresses. That’s the missing layer. It’s clearly built with production in mind not a research demo. and the fact that it’s self-hostable makes it even more relevant if you’re running serious workloads. If you’ve been duct-taping together infra around your agents, this is probably the closest thing to a unified system i’ve seen so far. Worth checking out. If you're serious about deploying reliable AI agents, this is worth a look: 👉 You can also try it instantly (no setup) via their cloud version:
2:20

Alright, this one’s worth your attention if you’re building or deploying agents. Future AGI just open-sourced their entire platform and i don’t mean a trimmed-down version. this is the full stack: UI, backend, simulation engine, evals, optimization loop, observability, guardrails, gateway, docs. all in one repo. Apache 2.0. I’ve been putting it through its paces on production agents, and what stands out isn’t just the breadth it’s the architecture. Most of the current “agent reliability” stack is fragmented. tracing lives in one tool, evals in another, guardrails somewhere else. you end up manually connecting dots, and the agent itself doesn’t really improve you just keep patching prompts and hoping for the best. This flips that model. It’s built as a closed feedback loop: simulate failures → evaluate in real time → detect production issues → learn from them → generate fixes → validate against real traffic → check regressions → redeploy → monitor again And when something new breaks, the loop just runs again. no manual glue. The simulation piece is especially strong. instead of static test cases, it generates adversarial, multi-turn conversations based on how your agent actually behaves basically hunting for the exact scenarios where your system fails confidently. ran a few thousand simulations on our side… caught things we definitely would’ve missed. Evals run fast (sub-50ms) across modalities. not LLM-as-judge trained classifiers. guardrails are built-in, not layered on top. observability gives you step-level visibility into reasoning, cost, latency, quality. But the real shift is the optimization loop. Most tools tell you *what* broke. this system actually fixes it, validates the fix, and ensures nothing else regresses. That’s the missing layer. It’s clearly built with production in mind not a research demo. and the fact that it’s self-hostable makes it even more relevant if you’re running serious workloads. If you’ve been duct-taping together infra around your agents, this is probably the closest thing to a unified system i’ve seen so far. Worth checking out. If you're serious about deploying reliable AI agents, this is worth a look: 👉 You can also try it instantly (no setup) via their cloud version:

Aakash Verma

22,915 Aufrufe • vor 5 Monaten

$AMZN CEO Andy Jassy describes AI adoption as very barbelled, with most existing enterprise workloads still not using inference though he expects that to change significantly over time. "I think it's actually kind of useful to look at at least our view of what we see in the demand and adoption curve right now, which is we see this adoption curve in AI right now is very barbelled." "There is on one end of the barbell, the AI labs are consuming gobs and gobs of compute. And there are a few runaway successful generative AI applications like Claude Code and ChatGPT." "And on the other end of the barbell are enterprises who are getting real value from AI in cost avoidance and productivity. And these are things like automating customer service or business process automation or fraud." "In the middle of the barbell is all of the current enterprise production workloads, some of which are using inference in a pervasive way, but most of which aren't. And that is going to change very significantly over time." "And that will be, in my opinion, that will be the largest absolute segment, the existing production workloads in the enterprise and new businesses and workloads that startups build too."
1:10

$AMZN CEO Andy Jassy describes AI adoption as very barbelled, with most existing enterprise workloads still not using inference though he expects that to change significantly over time. "I think it's actually kind of useful to look at at least our view of what we see in the demand and adoption curve right now, which is we see this adoption curve in AI right now is very barbelled." "There is on one end of the barbell, the AI labs are consuming gobs and gobs of compute. And there are a few runaway successful generative AI applications like Claude Code and ChatGPT." "And on the other end of the barbell are enterprises who are getting real value from AI in cost avoidance and productivity. And these are things like automating customer service or business process automation or fraud." "In the middle of the barbell is all of the current enterprise production workloads, some of which are using inference in a pervasive way, but most of which aren't. And that is going to change very significantly over time." "And that will be, in my opinion, that will be the largest absolute segment, the existing production workloads in the enterprise and new businesses and workloads that startups build too."

Fireside Alpha

29,242 Aufrufe • vor 1 Monat

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:
2:07

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:

Andrew Ng

126,675 Aufrufe • vor 1 Jahr

This is so real what Tyler said because people forget the person they look up to or idolize are human just like them and forget to consume the art and start to dig to deep in their business it becomes a violation. Even if from time to time they will show some of their personality or lifestyle, we should understand to only take what they give and not take it as a ticket to reach in too deep.
1:28

This is so real what Tyler said because people forget the person they look up to or idolize are human just like them and forget to consume the art and start to dig to deep in their business it becomes a violation. Even if from time to time they will show some of their personality or lifestyle, we should understand to only take what they give and not take it as a ticket to reach in too deep.

garçon

237,682 Aufrufe • vor 2 Jahren

🐥 what are you doing right now, hanbin-ssi? 🐹💬 i’m about to sleep 🐥 hyung why are you sleeping so early! i see you’re doing something early in the morning tomorrow~ hyung, there was a time where we used to sleep together, right? no hyung, there was a time where you used to wake me up! do you remember? 🐹💬 ㅋㅋㅋㅋㅋㅋ the kim taerae robot waking up 🐥 exactly! if you say, “taerae-yah we have to go now” i’ll be getting up like *reenacts* “yes!” there are times where i’d wake up like that but also times where i’d be like, “i’ll get up in a bit... i’ll get up in a bit...” show it in a live next time? how am i going to do that? but sleep really was precious to me that time. 🐹💬 let’s show it properly in a live next time 🐥 ah later on with hyung? okay, i’ll reenact it with hyung! now that i think about it, i don’t think we’ve ever done a live together before, right?
1:39

🐥 what are you doing right now, hanbin-ssi? 🐹💬 i’m about to sleep 🐥 hyung why are you sleeping so early! i see you’re doing something early in the morning tomorrow~ hyung, there was a time where we used to sleep together, right? no hyung, there was a time where you used to wake me up! do you remember? 🐹💬 ㅋㅋㅋㅋㅋㅋ the kim taerae robot waking up 🐥 exactly! if you say, “taerae-yah we have to go now” i’ll be getting up like *reenacts* “yes!” there are times where i’d wake up like that but also times where i’d be like, “i’ll get up in a bit... i’ll get up in a bit...” show it in a live next time? how am i going to do that? but sleep really was precious to me that time. 🐹💬 let’s show it properly in a live next time 🐥 ah later on with hyung? okay, i’ll reenact it with hyung! now that i think about it, i don’t think we’ve ever done a live together before, right?

지

34,741 Aufrufe • vor 13 Tagen

NEW: Zohran Mamdani teared up as he recalled his “aunt” allegedly being too scared to wear a hijab on the subway in New York City after 9/11 — but there’s a catch. Records show she was working in Tanzania from 2000–2003, and publicly available photos don’t show her wearing a hijab. Now, Sean Hannity 🇺🇸 has one question for Mamdani: “Where are the tears for the NYPD? We lost 2,977 Americans on that day.”
1:03

NEW: Zohran Mamdani teared up as he recalled his “aunt” allegedly being too scared to wear a hijab on the subway in New York City after 9/11 — but there’s a catch. Records show she was working in Tanzania from 2000–2003, and publicly available photos don’t show her wearing a hijab. Now, Sean Hannity 🇺🇸 has one question for Mamdani: “Where are the tears for the NYPD? We lost 2,977 Americans on that day.”

Fox News

136,363 Aufrufe • vor 11 Monaten

This is huge for production applications! Literally every developer has to deal with server errors they can't reproduce, and it doesn't matter how many logs they have. You don't have to be "every developer" anymore. Look into Hud's Runtime Code Sensor. Hud is one of my collaborators, and their code sensor is a game-changer for production applications. Watch the video I'm attaching here. For those of you who'd prefer a TL;DR; Hud captures how code behaves in production and puts those insights in context right in your IDE. This is really cool! You can see on top of every function data about how it works in production: • How long does it take to run? • How many times has the function failed? • Are the number of failures going up or down? All of this information is accessible to Cursor, Copilot, or WindSurf via an MCP server. That means that you can use the IDE to act on these issues! The way Hud works: • They provide a lightweight SDK • It takes one line to configure • All insights are right in your code • No dashboard necessary The best part: Their free plan includes ALL features they offer, so you can actually try it out properly and see how it fits your workflow. No paywall, no trial. Here is the link: Logging everything is great, but getting production insights right next to the code is definitely next-level!
4:28

This is huge for production applications! Literally every developer has to deal with server errors they can't reproduce, and it doesn't matter how many logs they have. You don't have to be "every developer" anymore. Look into Hud's Runtime Code Sensor. Hud is one of my collaborators, and their code sensor is a game-changer for production applications. Watch the video I'm attaching here. For those of you who'd prefer a TL;DR; Hud captures how code behaves in production and puts those insights in context right in your IDE. This is really cool! You can see on top of every function data about how it works in production: • How long does it take to run? • How many times has the function failed? • Are the number of failures going up or down? All of this information is accessible to Cursor, Copilot, or WindSurf via an MCP server. That means that you can use the IDE to act on these issues! The way Hud works: • They provide a lightweight SDK • It takes one line to configure • All insights are right in your code • No dashboard necessary The best part: Their free plan includes ALL features they offer, so you can actually try it out properly and see how it fits your workflow. No paywall, no trial. Here is the link: Logging everything is great, but getting production insights right next to the code is definitely next-level!

Santiago

28,556 Aufrufe • vor 1 Jahr

The next version of SEO isn’t about ranking number one. It’s about what AI says about you when someone asks. Because Google, ChatGPT, Gemini, Claude, they’re not just showing links anymore. They’re forming opinions. They’re comparing brands. They’re deciding who sounds credible, who sounds risky, who gets recommended, and who gets ignored. That means your biggest search problem in 2027 might not be that you don’t show up. It might be that you show up wrong. Maybe AI describes your competitor as the trusted expert… and describes you as generic. Maybe it mentions your brand… but the sentiment is neutral while your competitor is glowing. Maybe it pulls from old reviews, outdated articles, weak bios, messy positioning, or random third-party pages you didn’t even know were shaping your reputation. That’s the part most businesses are missing. AI search visibility is not just a traffic game. It’s a reputation game. And that’s why we added the Sentiment tab inside Ubersuggest’s AI Search Visibility tool. It doesn’t just track whether your brand appears across ChatGPT, Gemini, Claude, and Google AI. It shows how those engines describe you. Positive, neutral, or negative. And it charts that sentiment over time against your competitors. So now you can answer the question that actually matters: Not just, “Am I showing up in AI answers?” But, “What is AI actually saying about me?”
1:05

The next version of SEO isn’t about ranking number one. It’s about what AI says about you when someone asks. Because Google, ChatGPT, Gemini, Claude, they’re not just showing links anymore. They’re forming opinions. They’re comparing brands. They’re deciding who sounds credible, who sounds risky, who gets recommended, and who gets ignored. That means your biggest search problem in 2027 might not be that you don’t show up. It might be that you show up wrong. Maybe AI describes your competitor as the trusted expert… and describes you as generic. Maybe it mentions your brand… but the sentiment is neutral while your competitor is glowing. Maybe it pulls from old reviews, outdated articles, weak bios, messy positioning, or random third-party pages you didn’t even know were shaping your reputation. That’s the part most businesses are missing. AI search visibility is not just a traffic game. It’s a reputation game. And that’s why we added the Sentiment tab inside Ubersuggest’s AI Search Visibility tool. It doesn’t just track whether your brand appears across ChatGPT, Gemini, Claude, and Google AI. It shows how those engines describe you. Positive, neutral, or negative. And it charts that sentiment over time against your competitors. So now you can answer the question that actually matters: Not just, “Am I showing up in AI answers?” But, “What is AI actually saying about me?”

Neil Patel

17,045 Aufrufe • vor 2 Monaten