Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

The hard part about LLM failures is that their outputs rarely look like failures. The demo “works.” The output sounds coherent. The user actively uses the product. And your dashboard looks normal. Meanwhile, the system can be wrong, unsafe, or quietly driving up token spend. And you won’t notice...

arshdilbagi's profile picture

Arsh Shah Dilbagi

2,580 subscribers

20,951 görüntüleme • 5 ay önce •via X (Twitter)

EğitimHaberler & PolitikaBilim & Teknoloji
LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.
2:13

Respan Launch Week, Day 2: Evals Evals are one of the hardest parts of building AI applications. It is not because teams cannot run them. It is because they are hard to structure, hard to compare, and even harder to improve over time. So teams end up guessing. Did this prompt actually get better? Is this model really an improvement? We built Evals in Respan to make this systematic. You define: - what you want to test, such as prompts, models, or configs - the dataset, from production logs or test cases - the evaluators, whether LLM, code, human, or a mix Then you run experiments and compare results side by side. Same inputs. Same evaluation. Clear answers. No more guessing. Start running your first eval.

Respan

127,680 görüntüleme • 4 ay önce

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:
2:07

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:

Andrew Ng

126,559 görüntüleme • 1 yıl önce

your eval measured one system. production ran another. they start from the same idea and end somewhere else, which is why the number was right and the outcome was not. the code is the same until someone rewrites it for prod, because the eval lived in a notebook and prod is a service. then the data diverges. clean fixtures in the test, late and incomplete data in the run. then execution diverges. the eval assumed every call lands. production has retries, timeouts and rate limits. by the end the two runs share a name and nothing else. > why nobody catches it both sides are internally correct. the eval measured what it ran. production ran what it was given. neither one is wrong, and neither one is responsible for checking they were the same artifact. so the gap widens quietly. someone patches prod without touching the eval. someone adds a fixture without touching prod. no test anywhere fails. > what to actually do ship one artifact. the code that was measured is the code that runs. model the friction. whatever the real environment charges you, charge it in the eval too. easiest way to see the gap on your own idea, without writing the harness: (only first 100 registrated) it runs the tested code live, unchanged. that is the whole point. and treat the eval as a prior, not a result. it tells you what to expect. only the live run tells you what happened. your eval was accurate. it was accurate about something you never shipped.
0:49

your eval measured one system. production ran another. they start from the same idea and end somewhere else, which is why the number was right and the outcome was not. the code is the same until someone rewrites it for prod, because the eval lived in a notebook and prod is a service. then the data diverges. clean fixtures in the test, late and incomplete data in the run. then execution diverges. the eval assumed every call lands. production has retries, timeouts and rate limits. by the end the two runs share a name and nothing else. > why nobody catches it both sides are internally correct. the eval measured what it ran. production ran what it was given. neither one is wrong, and neither one is responsible for checking they were the same artifact. so the gap widens quietly. someone patches prod without touching the eval. someone adds a fixture without touching prod. no test anywhere fails. > what to actually do ship one artifact. the code that was measured is the code that runs. model the friction. whatever the real environment charges you, charge it in the eval too. easiest way to see the gap on your own idea, without writing the harness: (only first 100 registrated) it runs the tested code live, unchanged. that is the whole point. and treat the eval as a prior, not a result. it tells you what to expect. only the live run tells you what happened. your eval was accurate. it was accurate about something you never shipped.

Hanako

45,239 görüntüleme • 21 gün önce

Alright, this one’s worth your attention if you’re building or deploying agents. Future AGI just open-sourced their entire platform and i don’t mean a trimmed-down version. this is the full stack: UI, backend, simulation engine, evals, optimization loop, observability, guardrails, gateway, docs. all in one repo. Apache 2.0. I’ve been putting it through its paces on production agents, and what stands out isn’t just the breadth it’s the architecture. Most of the current “agent reliability” stack is fragmented. tracing lives in one tool, evals in another, guardrails somewhere else. you end up manually connecting dots, and the agent itself doesn’t really improve you just keep patching prompts and hoping for the best. This flips that model. It’s built as a closed feedback loop: simulate failures → evaluate in real time → detect production issues → learn from them → generate fixes → validate against real traffic → check regressions → redeploy → monitor again And when something new breaks, the loop just runs again. no manual glue. The simulation piece is especially strong. instead of static test cases, it generates adversarial, multi-turn conversations based on how your agent actually behaves basically hunting for the exact scenarios where your system fails confidently. ran a few thousand simulations on our side… caught things we definitely would’ve missed. Evals run fast (sub-50ms) across modalities. not LLM-as-judge trained classifiers. guardrails are built-in, not layered on top. observability gives you step-level visibility into reasoning, cost, latency, quality. But the real shift is the optimization loop. Most tools tell you *what* broke. this system actually fixes it, validates the fix, and ensures nothing else regresses. That’s the missing layer. It’s clearly built with production in mind not a research demo. and the fact that it’s self-hostable makes it even more relevant if you’re running serious workloads. If you’ve been duct-taping together infra around your agents, this is probably the closest thing to a unified system i’ve seen so far. Worth checking out. If you're serious about deploying reliable AI agents, this is worth a look: 👉 You can also try it instantly (no setup) via their cloud version:
2:20

Alright, this one’s worth your attention if you’re building or deploying agents. Future AGI just open-sourced their entire platform and i don’t mean a trimmed-down version. this is the full stack: UI, backend, simulation engine, evals, optimization loop, observability, guardrails, gateway, docs. all in one repo. Apache 2.0. I’ve been putting it through its paces on production agents, and what stands out isn’t just the breadth it’s the architecture. Most of the current “agent reliability” stack is fragmented. tracing lives in one tool, evals in another, guardrails somewhere else. you end up manually connecting dots, and the agent itself doesn’t really improve you just keep patching prompts and hoping for the best. This flips that model. It’s built as a closed feedback loop: simulate failures → evaluate in real time → detect production issues → learn from them → generate fixes → validate against real traffic → check regressions → redeploy → monitor again And when something new breaks, the loop just runs again. no manual glue. The simulation piece is especially strong. instead of static test cases, it generates adversarial, multi-turn conversations based on how your agent actually behaves basically hunting for the exact scenarios where your system fails confidently. ran a few thousand simulations on our side… caught things we definitely would’ve missed. Evals run fast (sub-50ms) across modalities. not LLM-as-judge trained classifiers. guardrails are built-in, not layered on top. observability gives you step-level visibility into reasoning, cost, latency, quality. But the real shift is the optimization loop. Most tools tell you *what* broke. this system actually fixes it, validates the fix, and ensures nothing else regresses. That’s the missing layer. It’s clearly built with production in mind not a research demo. and the fact that it’s self-hostable makes it even more relevant if you’re running serious workloads. If you’ve been duct-taping together infra around your agents, this is probably the closest thing to a unified system i’ve seen so far. Worth checking out. If you're serious about deploying reliable AI agents, this is worth a look: 👉 You can also try it instantly (no setup) via their cloud version:

Aakash Verma

22,910 görüntüleme • 4 ay önce

I asked Garry Tan how to use meta prompting to get better at AI: "My partners at YC Jared Friedman and Pete Koomen showed me how to do this. You can take almost anything that you do all the time and just drop it into a context window. And then say, “Here’s a bunch of inputs and outputs." And maybe you also add a bunch of notes. And then you tell it, “Write me a prompt that can act as an agent that takes this input and makes this output over here.” You can do this for almost any type of knowledge work. And you can even introspect. "What are things you notice that I did to convert this from the input to the output?”. And then you can just start using the prompt. Initially, it’s going to suck. Because it’s just not that smart yet. But what’s funny is now, I also use it to Iterate my writing. You can be very direct, "I would never say that", "Don’t say it like this", or "Oh, you used the long word there, use the short word". Just speak to it conversationally. And then when you're happy with the output, you can use that new output to make a new prompt. "Based on this conversation, give me a better initial prompt that incorporates all the things we talked about." And you can do this with literally everything. And in theory, there’s so much it applies to that people do day-to-day. You could use it for tweets. You could use it for editing podcasts. You can use it for pretty much everything. I have a folder of prompts that I use all the time. My YouTube prompt is on v27 or something. I'll go through this process with all the different max models. I'll use GPT 5.2 Pro. I’ll use Grok. I'll use Claude. Then, I’ll take all the outputs from all the models and put them into Claude and say "Here’s my prompt, here’s the output from four LLMs, including yourself. Rate each response and tell me what the pros and cons of each approach are." And I usually say "give it to me in numbered form". And then you can agree with one, disagree with two, tell it three is this or that. And then after that, you say given all of this, synthesize it."
4:16

I asked Garry Tan how to use meta prompting to get better at AI: "My partners at YC Jared Friedman and Pete Koomen showed me how to do this. You can take almost anything that you do all the time and just drop it into a context window. And then say, “Here’s a bunch of inputs and outputs." And maybe you also add a bunch of notes. And then you tell it, “Write me a prompt that can act as an agent that takes this input and makes this output over here.” You can do this for almost any type of knowledge work. And you can even introspect. "What are things you notice that I did to convert this from the input to the output?”. And then you can just start using the prompt. Initially, it’s going to suck. Because it’s just not that smart yet. But what’s funny is now, I also use it to Iterate my writing. You can be very direct, "I would never say that", "Don’t say it like this", or "Oh, you used the long word there, use the short word". Just speak to it conversationally. And then when you're happy with the output, you can use that new output to make a new prompt. "Based on this conversation, give me a better initial prompt that incorporates all the things we talked about." And you can do this with literally everything. And in theory, there’s so much it applies to that people do day-to-day. You could use it for tweets. You could use it for editing podcasts. You can use it for pretty much everything. I have a folder of prompts that I use all the time. My YouTube prompt is on v27 or something. I'll go through this process with all the different max models. I'll use GPT 5.2 Pro. I’ll use Grok. I'll use Claude. Then, I’ll take all the outputs from all the models and put them into Claude and say "Here’s my prompt, here’s the output from four LLMs, including yourself. Rate each response and tell me what the pros and cons of each approach are." And I usually say "give it to me in numbered form". And then you can agree with one, disagree with two, tell it three is this or that. And then after that, you say given all of this, synthesize it."

The Peel

51,632 görüntüleme • 6 ay önce

McKinsey surveyed 2,000 companies in 2025. 51% said AI backfired on them. Top reason? Inaccuracy. From what I can tell, most of these systems weren't broken. They were unreliable. And unreliable is wayyyy worse because you can't predict when it fails. So I got Ash Tilawat (the Mr. Miyagi of teaching AI) from Gauntlet AI to walk me through the solution. Here's his 2026 framework for evaluating if your AI is trustworthy, reliable, and production-ready: 1. build your golden set Identify 30–50 core requests your AI must handle correctly. The stuff that, if broken, makes the whole system useless. And sit with the person whose job this AI is doing/automating/replacing/helping with. 2. test the weird stuff Your golden set covers common requests. But in production, users don't only ask common requests. So build a matrix of categories (topic x complexity) and fill the gaps. Every gap is a corner where failures can hide behind. 3. build a replay harness Record the exact state of every interaction so you can test prompt changes without burning API calls. Think of it like game film... you don't put players back on the field just to review the play. 4. create your rubric Use an LLM to grade outputs on accuracy, completeness, and tone. But calibrate it first -> run 50–100 examples through human and LLM scoring, find disagreements, fix the rubric, repeat until they match. 5. run experiments New model? Prompt rewrite? Run your eval suite against both versions. Ship if the golden set passes, no regressions, and the cost is acceptable. The teams still running production AI on vibes will be f***** in 2026. But the teams building eval libraries are compounding an advantage that gets harder to catch every month. Competitors can copy your product. They can't copy your test cases. h/t Austen Allred for helping put this together. Full playbook + vid below 👇
1:05:36

McKinsey surveyed 2,000 companies in 2025. 51% said AI backfired on them. Top reason? Inaccuracy. From what I can tell, most of these systems weren't broken. They were unreliable. And unreliable is wayyyy worse because you can't predict when it fails. So I got Ash Tilawat (the Mr. Miyagi of teaching AI) from Gauntlet AI to walk me through the solution. Here's his 2026 framework for evaluating if your AI is trustworthy, reliable, and production-ready: 1. build your golden set Identify 30–50 core requests your AI must handle correctly. The stuff that, if broken, makes the whole system useless. And sit with the person whose job this AI is doing/automating/replacing/helping with. 2. test the weird stuff Your golden set covers common requests. But in production, users don't only ask common requests. So build a matrix of categories (topic x complexity) and fill the gaps. Every gap is a corner where failures can hide behind. 3. build a replay harness Record the exact state of every interaction so you can test prompt changes without burning API calls. Think of it like game film... you don't put players back on the field just to review the play. 4. create your rubric Use an LLM to grade outputs on accuracy, completeness, and tone. But calibrate it first -> run 50–100 examples through human and LLM scoring, find disagreements, fix the rubric, repeat until they match. 5. run experiments New model? Prompt rewrite? Run your eval suite against both versions. Ship if the golden set passes, no regressions, and the cost is acceptable. The teams still running production AI on vibes will be f***** in 2026. But the teams building eval libraries are compounding an advantage that gets harder to catch every month. Competitors can copy your product. They can't copy your test cases. h/t Austen Allred for helping put this together. Full playbook + vid below 👇

Alex Lieberman

84,725 görüntüleme • 7 ay önce

Most AI video tools still feel like traditional editors with a few AI features added in. You still end up adjusting timelines, fixing frames, and spending time editing. But Hailuo AI felt different to me. I gave it a simple prompt and one image, and it generated a full video on its own. No complicated workflow, no manual editing just idea to video in minutes. The text-to-video results are surprisingly good, image-to-video transitions look smooth, and the overall output feels cinematic. What I liked most is the speed. You can quickly test ideas without getting stuck in editing. Try it here: #Hailuo
0:23

Most AI video tools still feel like traditional editors with a few AI features added in. You still end up adjusting timelines, fixing frames, and spending time editing. But Hailuo AI felt different to me. I gave it a simple prompt and one image, and it generated a full video on its own. No complicated workflow, no manual editing just idea to video in minutes. The text-to-video results are surprisingly good, image-to-video transitions look smooth, and the overall output feels cinematic. What I liked most is the speed. You can quickly test ideas without getting stuck in editing. Try it here: #Hailuo

Markandey Sharma

32,326 görüntüleme • 3 ay önce

I’m thrilled to announce we’ve raised $44M to build a new home for product design. Meet Noon. No workflow is more broken and fragmented in 2026 than the product designers’. The very same people who care most about building software don’t have software purpose built for them. Kushagra Sinha and I have lived this problem first hand as designers ourselves. That’s why we built Noon. The first product design tool that works entirely on your product code, so you can design not only how a product looks, but also how it works. With AI at its core that works in seconds, not minutes. For the first time, you can create, iterate, build, test and ship. All in one canvas. No translations or roundtrips to the codebase and back. Comment “Get Noon” and we’ll get you on the list for early access.
0:41

I’m thrilled to announce we’ve raised $44M to build a new home for product design. Meet Noon. No workflow is more broken and fragmented in 2026 than the product designers’. The very same people who care most about building software don’t have software purpose built for them. Kushagra Sinha and I have lived this problem first hand as designers ourselves. That’s why we built Noon. The first product design tool that works entirely on your product code, so you can design not only how a product looks, but also how it works. With AI at its core that works in seconds, not minutes. For the first time, you can create, iterate, build, test and ship. All in one canvas. No translations or roundtrips to the codebase and back. Comment “Get Noon” and we’ll get you on the list for early access.

Aditya Bandi

684,458 görüntüleme • 5 ay önce

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. ​ This is the application we all deserve (at least if you have macOS.) ​ Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. ​ ​ Here is the idea: ​ You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. ​ It sees what you see. It works right where you are. ​ That's pretty cool. ​ Here are a couple of cool examples: ​ • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack ​ AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: ​ ​ Thanks to the Omnipilot team for collaborating with me on this post!
0:29

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. ​ This is the application we all deserve (at least if you have macOS.) ​ Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. ​ ​ Here is the idea: ​ You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. ​ It sees what you see. It works right where you are. ​ That's pretty cool. ​ Here are a couple of cool examples: ​ • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack ​ AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: ​ ​ Thanks to the Omnipilot team for collaborating with me on this post!

Santiago

72,306 görüntüleme • 2 yıl önce

I don't think most PMs realize the PRD is becoming obsolete. For the last decade, the PM's core artifact was a qualitative spec. Clear requirements, user stories, acceptance criteria. The engineering team interpreted it, built something close, and the PM spent two weeks reconciling what shipped with what they wrote. The best AI companies replaced that entire loop with evals. A set of inputs your product needs to handle. A task that generates outputs. A scoring function that produces a number between 0 and 1. No ambiguity. No interpretation gap. Ankur Goyal built the eval platform behind Vercel, Replit, Ramp, Notion, and Airtable. An $800M company. He walked through building an eval from zero on this episode and the score went from 0 to 0.75 in under 20 minutes. That's a PM shipping a measurable quality bar before a single line of product code exists. Here's the part that changes the PM role permanently. When the product passes the eval and users still hate it, the eval is wrong. That's on the PM. Evals make PM judgment quantifiable in a way PRDs never did. You can't hide behind "the spec was ambiguous." There's a number now. Six months ago, PM interviews asked "how do you use AI in your workflow." The next wave of interviews is going to ask you to write an eval. The PMs who can encode user intent as a scoring function are building the one skill that survives every model change, every framework swap, every agent rewrite. Write the eval.
4:31

I don't think most PMs realize the PRD is becoming obsolete. For the last decade, the PM's core artifact was a qualitative spec. Clear requirements, user stories, acceptance criteria. The engineering team interpreted it, built something close, and the PM spent two weeks reconciling what shipped with what they wrote. The best AI companies replaced that entire loop with evals. A set of inputs your product needs to handle. A task that generates outputs. A scoring function that produces a number between 0 and 1. No ambiguity. No interpretation gap. Ankur Goyal built the eval platform behind Vercel, Replit, Ramp, Notion, and Airtable. An $800M company. He walked through building an eval from zero on this episode and the score went from 0 to 0.75 in under 20 minutes. That's a PM shipping a measurable quality bar before a single line of product code exists. Here's the part that changes the PM role permanently. When the product passes the eval and users still hate it, the eval is wrong. That's on the PM. Evals make PM judgment quantifiable in a way PRDs never did. You can't hide behind "the spec was ambiguous." There's a number now. Six months ago, PM interviews asked "how do you use AI in your workflow." The next wave of interviews is going to ask you to write an eval. The PMs who can encode user intent as a scoring function are building the one skill that survives every model change, every framework swap, every agent rewrite. Write the eval.

Aakash Gupta

78,275 görüntüleme • 5 ay önce

10 agent evals for AI engineers: (explained with usage) 1) golden set → a fixed set of cases you never edit, run on every single change. → use as the baseline that tells you whether anything moved at all. 2) llm as judge → a second model scores the output against a written rubric. → use when the answer is open-ended and there is no string to match against. 3) rubric scoring → one number per dimension: correctness, tone, safety, cost. → use when a single score hides which part actually got worse. 4) trajectory eval → grade the path the agent took, not only the answer it landed on. → use when the right answer for the wrong reason is going to bite you later. 5) tool unit tests → test each tool on its own, with fixtures, no model in the loop. → use always. most agent bugs are tool bugs wearing a costume. 6) regression suite → replay past runs against the new prompt or model and diff the results. → use before every prompt change, because prompts have no type system. 7) a/b in prod → split live traffic between two versions and compare outcomes, not vibes. → use when offline scores stopped predicting what users actually do. 8) human review → sample a slice of runs and have a person grade them honestly. → use to calibrate your judge, because a judge nobody checks quietly drifts. 9) shadow run → the candidate runs on real traffic in parallel and its output is shown to nobody. → use before a risky rollout, when one bad answer would be expensive. 10) red team → deliberately attack it: jailbreaks, injection, exfil, tool abuse. → use before anyone external can reach it, not after. offline evals tell you it works. online evals tell you it still works. both sides matter, but not all ten do. run the two that would have caught your last outage. save this. then read the full breakdown on loop engineering below.
0:40

10 agent evals for AI engineers: (explained with usage) 1) golden set → a fixed set of cases you never edit, run on every single change. → use as the baseline that tells you whether anything moved at all. 2) llm as judge → a second model scores the output against a written rubric. → use when the answer is open-ended and there is no string to match against. 3) rubric scoring → one number per dimension: correctness, tone, safety, cost. → use when a single score hides which part actually got worse. 4) trajectory eval → grade the path the agent took, not only the answer it landed on. → use when the right answer for the wrong reason is going to bite you later. 5) tool unit tests → test each tool on its own, with fixtures, no model in the loop. → use always. most agent bugs are tool bugs wearing a costume. 6) regression suite → replay past runs against the new prompt or model and diff the results. → use before every prompt change, because prompts have no type system. 7) a/b in prod → split live traffic between two versions and compare outcomes, not vibes. → use when offline scores stopped predicting what users actually do. 8) human review → sample a slice of runs and have a person grade them honestly. → use to calibrate your judge, because a judge nobody checks quietly drifts. 9) shadow run → the candidate runs on real traffic in parallel and its output is shown to nobody. → use before a risky rollout, when one bad answer would be expensive. 10) red team → deliberately attack it: jailbreaks, injection, exfil, tool abuse. → use before anyone external can reach it, not after. offline evals tell you it works. online evals tell you it still works. both sides matter, but not all ten do. run the two that would have caught your last outage. save this. then read the full breakdown on loop engineering below.

Hanako

150,676 görüntüleme • 1 ay önce

we just raised $0M to change the way traditional demos are made. demoboy started as a vibecon project and has finally come to life. our goal is to turn product launches and demos into interactive experiences, meaning you don’t just watch a video (like you currently do), but you can also pause and interact with the product without signing up or even leaving the demo. i mean, imagine a feed filled with interactive demo videos where people can not only discover your product but actually experience it. anyway, none of this will matter if you don’t watch the video and try the demo yourself: either this changes the way we make demos, or this gets like 3 likes lol. p.s. 1 we recommend opening the link on a laptop for the best experience. p.s. 2 we absolutely need your genuine feedback no matter if it’s good or bad.
3:01

we just raised $0M to change the way traditional demos are made. demoboy started as a vibecon project and has finally come to life. our goal is to turn product launches and demos into interactive experiences, meaning you don’t just watch a video (like you currently do), but you can also pause and interact with the product without signing up or even leaving the demo. i mean, imagine a feed filled with interactive demo videos where people can not only discover your product but actually experience it. anyway, none of this will matter if you don’t watch the video and try the demo yourself: either this changes the way we make demos, or this gets like 3 likes lol. p.s. 1 we recommend opening the link on a laptop for the best experience. p.s. 2 we absolutely need your genuine feedback no matter if it’s good or bad.

annie

40,377 görüntüleme • 2 ay önce

This is huge for production applications! Literally every developer has to deal with server errors they can't reproduce, and it doesn't matter how many logs they have. You don't have to be "every developer" anymore. Look into Hud's Runtime Code Sensor. Hud is one of my collaborators, and their code sensor is a game-changer for production applications. Watch the video I'm attaching here. For those of you who'd prefer a TL;DR; Hud captures how code behaves in production and puts those insights in context right in your IDE. This is really cool! You can see on top of every function data about how it works in production: • How long does it take to run? • How many times has the function failed? • Are the number of failures going up or down? All of this information is accessible to Cursor, Copilot, or WindSurf via an MCP server. That means that you can use the IDE to act on these issues! The way Hud works: • They provide a lightweight SDK • It takes one line to configure • All insights are right in your code • No dashboard necessary The best part: Their free plan includes ALL features they offer, so you can actually try it out properly and see how it fits your workflow. No paywall, no trial. Here is the link: Logging everything is great, but getting production insights right next to the code is definitely next-level!
4:28

This is huge for production applications! Literally every developer has to deal with server errors they can't reproduce, and it doesn't matter how many logs they have. You don't have to be "every developer" anymore. Look into Hud's Runtime Code Sensor. Hud is one of my collaborators, and their code sensor is a game-changer for production applications. Watch the video I'm attaching here. For those of you who'd prefer a TL;DR; Hud captures how code behaves in production and puts those insights in context right in your IDE. This is really cool! You can see on top of every function data about how it works in production: • How long does it take to run? • How many times has the function failed? • Are the number of failures going up or down? All of this information is accessible to Cursor, Copilot, or WindSurf via an MCP server. That means that you can use the IDE to act on these issues! The way Hud works: • They provide a lightweight SDK • It takes one line to configure • All insights are right in your code • No dashboard necessary The best part: Their free plan includes ALL features they offer, so you can actually try it out properly and see how it fits your workflow. No paywall, no trial. Here is the link: Logging everything is great, but getting production insights right next to the code is definitely next-level!

Santiago

28,556 görüntüleme • 1 yıl önce

HOW TO USE AI LOOPS TO RUN YOUR BUSINESS 24/7 A lot has been written about loop engineering for building products. Almost nothing about using loops to run the business itself. That's the bigger idea. A loop is when you give an agent a goal, a way to check its own work, and permission to keep trying until it hits that goal. Build. Verify. Repeat. Stop when the condition is met. Here's what it looks like in practice: 1/SEO loop You're position 30 for a term you want. The loop runs once a month, makes changes, checks where you rank, and keeps pushing until you're on page one. This is running in production right now on Inbox Zero. 2/Ads loop You're spending $100 a day and losing money. The loop tests creative, checks profitability, kills what fails, and keeps going until the account is in the black. 3/Eval loop Your AI feature is only 88% accurate. The loop keeps adjusting the prompt and swapping the model until it passes 90%. 4/LLM visibility loop People search in ChatGPT now, not just Google. Same loop, new scoreboard. Are we the answer or not? The whole thing hinges on one thing: a metric that comes back black and white. Where do I rank? Did it hit profitability? Did the evals pass? Give an agent that scoreboard and it runs for months. Loops used to run for 30 minutes. These run for a year. Take a step, sleep, wake up next month, take another one. You're basically hiring an agency that never sleeps, gets paid in tokens instead of invoices, and undoes its own mistakes when the number goes down. Full episode on The Startup Ideas Podcast (SIP) 🧃 watch
39:43

HOW TO USE AI LOOPS TO RUN YOUR BUSINESS 24/7 A lot has been written about loop engineering for building products. Almost nothing about using loops to run the business itself. That's the bigger idea. A loop is when you give an agent a goal, a way to check its own work, and permission to keep trying until it hits that goal. Build. Verify. Repeat. Stop when the condition is met. Here's what it looks like in practice: 1/SEO loop You're position 30 for a term you want. The loop runs once a month, makes changes, checks where you rank, and keeps pushing until you're on page one. This is running in production right now on Inbox Zero. 2/Ads loop You're spending $100 a day and losing money. The loop tests creative, checks profitability, kills what fails, and keeps going until the account is in the black. 3/Eval loop Your AI feature is only 88% accurate. The loop keeps adjusting the prompt and swapping the model until it passes 90%. 4/LLM visibility loop People search in ChatGPT now, not just Google. Same loop, new scoreboard. Are we the answer or not? The whole thing hinges on one thing: a metric that comes back black and white. Where do I rank? Did it hit profitability? Did the evals pass? Give an agent that scoreboard and it runs for months. Loops used to run for 30 minutes. These run for a year. Take a step, sleep, wake up next month, take another one. You're basically hiring an agency that never sleeps, gets paid in tokens instead of invoices, and undoes its own mistakes when the number goes down. Full episode on The Startup Ideas Podcast (SIP) 🧃 watch

GREG ISENBERG

82,960 görüntüleme • 1 ay önce

.Governor JB Pritzker, you cannot blame the federal government for your failures. You’ve chosen to limit cooperation with federal immigration enforcement and advance sanctuary policies, but you haven’t put equally strong safeguards in place to protect your own citizens and uphold public safety. You favor illegal immigrants and provide special privileges and consideration which citizens are not entitled to. You cannot have it both ways. You don’t get to deflect blame when the consequences surface. If you choose to sideline federal frameworks, the responsibility doesn’t disappear, it shifts squarely onto your shoulders. That means owning the outcomes, not pointing fingers when things go wrong. Right now, what’s missing isn’t just policy clarity, it’s accountability. Leadership isn’t about ideology; it’s about responsibility. If your policies create gaps, you answer for those gaps. If your approach introduces risk, you confront it directly. Silence, deflection, and blame-shifting aren’t leadership, they’re avoidance. So you’re either the governor of Illinois, or you’re not. Leadership means owning the outcomes of your decisions, not shifting responsibility when those decisions are questioned. Either you take responsibility for the system you’ve built, or you admit it isn’t working. But refusing to do either erodes trust and leaves the public without clear answers from the person ultimately in charge. And silence on cases like Katie’s only deepens the sense that responsibility is being avoided. Say her name: Katie Abraham.
0:50

.Governor JB Pritzker, you cannot blame the federal government for your failures. You’ve chosen to limit cooperation with federal immigration enforcement and advance sanctuary policies, but you haven’t put equally strong safeguards in place to protect your own citizens and uphold public safety. You favor illegal immigrants and provide special privileges and consideration which citizens are not entitled to. You cannot have it both ways. You don’t get to deflect blame when the consequences surface. If you choose to sideline federal frameworks, the responsibility doesn’t disappear, it shifts squarely onto your shoulders. That means owning the outcomes, not pointing fingers when things go wrong. Right now, what’s missing isn’t just policy clarity, it’s accountability. Leadership isn’t about ideology; it’s about responsibility. If your policies create gaps, you answer for those gaps. If your approach introduces risk, you confront it directly. Silence, deflection, and blame-shifting aren’t leadership, they’re avoidance. So you’re either the governor of Illinois, or you’re not. Leadership means owning the outcomes of your decisions, not shifting responsibility when those decisions are questioned. Either you take responsibility for the system you’ve built, or you admit it isn’t working. But refusing to do either erodes trust and leaves the public without clear answers from the person ultimately in charge. And silence on cases like Katie’s only deepens the sense that responsibility is being avoided. Say her name: Katie Abraham.

Joe Abraham

32,431 görüntüleme • 5 ay önce

Most underrated skill today: Observability. I feel you can build a career just on this and have 2 decades of guaranteed money if you learn the skill well, and I'll tell you how below. Reality: Most teams I help look at me with dead eyes when I ask them about observability. That's sad, but it's the current state of the industry: • No tracing • No structured data • No way to understand what's happening when things break Outside of the most sophisticated companies, that's still where most teams are. AI is now writing a ton of code, and people are shipping this code to production. However important observability was before, it is 10x more important today. Believe me, you don't want to be the one pushing slop into production and then guessing how it works, with no way to understand what happens. Here is your first step: Honeycomb is running a FREE, 3-day virtual event focused entirely on observability. The event is running May 12-14. Did I mention it's completely free? Everything will revolve around observability for the agent era. Register here: Thanks to the Honeycomb.io System Status Updates team for partnering with me on this post.
1:33

Most underrated skill today: Observability. I feel you can build a career just on this and have 2 decades of guaranteed money if you learn the skill well, and I'll tell you how below. Reality: Most teams I help look at me with dead eyes when I ask them about observability. That's sad, but it's the current state of the industry: • No tracing • No structured data • No way to understand what's happening when things break Outside of the most sophisticated companies, that's still where most teams are. AI is now writing a ton of code, and people are shipping this code to production. However important observability was before, it is 10x more important today. Believe me, you don't want to be the one pushing slop into production and then guessing how it works, with no way to understand what happens. Here is your first step: Honeycomb is running a FREE, 3-day virtual event focused entirely on observability. The event is running May 12-14. Did I mention it's completely free? Everything will revolve around observability for the agent era. Register here: Thanks to the Honeycomb.io System Status Updates team for partnering with me on this post.

Santiago

19,983 görüntüleme • 3 ay önce

What is the “dark factory” that there’s so much talk about, in relation to software factories? dex, founder of HumanLayer explains: “You can imagine a car factory where it’s all robots building the cars, and they don’t have lights. There’s no humans. You walk in there, there’s no lights, there’s not even light switches. And in software, you can have mini loops that are dark in your thing: hey, if the code review agent comes back with a problem, you loop that back to the builder agent, it fixes it and comes back, and that’s a dark factory. You don’t need a human loop for that. But the full dark factory where you don’t read any code: yeah, it’s a good way to maximize your token utilization. If your belief is that ‘my job is to extract as much intelligence out of the machine as I can’, then you go token harder. And my take is token smarter: how do I get as much value out of AI as I can without having to turn the lights off, while still maintaining control and taste and judgment and understanding the system architecture?”
1:14

What is the “dark factory” that there’s so much talk about, in relation to software factories? dex, founder of HumanLayer explains: “You can imagine a car factory where it’s all robots building the cars, and they don’t have lights. There’s no humans. You walk in there, there’s no lights, there’s not even light switches. And in software, you can have mini loops that are dark in your thing: hey, if the code review agent comes back with a problem, you loop that back to the builder agent, it fixes it and comes back, and that’s a dark factory. You don’t need a human loop for that. But the full dark factory where you don’t read any code: yeah, it’s a good way to maximize your token utilization. If your belief is that ‘my job is to extract as much intelligence out of the machine as I can’, then you go token harder. And my take is token smarter: how do I get as much value out of AI as I can without having to turn the lights off, while still maintaining control and taste and judgment and understanding the system architecture?”

The Pragmatic Engineer

37,636 görüntüleme • 1 ay önce

The most expensive mistake in SaaS isn’t bad ads. It’s bad onboarding. If a user doesn’t experience value in the first 2 minutes after signing up, they won’t come back. And trust me, they’ll go to your competitor without hesitation. As a product marketer, this is the part that keeps me up at night. You can spend months on campaigns, content, and growth tactics… but if the first product experience fails, everything else is wasted. That’s why we rebuilt Chatbase onboarding from scratch. From just one link, you get a fully branded AI agent, trained on your site, with your logo, your colors, and your prompt, in under a minute. That “wow” moment matters more than any campaign I’ll ever run. Because the instant a user feels the product working for them, they trust it.
0:42

The most expensive mistake in SaaS isn’t bad ads. It’s bad onboarding. If a user doesn’t experience value in the first 2 minutes after signing up, they won’t come back. And trust me, they’ll go to your competitor without hesitation. As a product marketer, this is the part that keeps me up at night. You can spend months on campaigns, content, and growth tactics… but if the first product experience fails, everything else is wasted. That’s why we rebuilt Chatbase onboarding from scratch. From just one link, you get a fully branded AI agent, trained on your site, with your logo, your colors, and your prompt, in under a minute. That “wow” moment matters more than any campaign I’ll ever run. Because the instant a user feels the product working for them, they trust it.

sandra djajic

12,310 görüntüleme • 11 ay önce

Today, we're making Error Tracking by Better Stack generally available. Sentry-compatible. AI-native. At 1/6th the price. Here's why we built it, and how to get the most out of it. What's wrong with error tracking today? Most teams use Sentry. It's solid! But at scale, the bills get brutal. Just 100M exceptions with 90 day lookback? ~$30,000 on Sentry. We charge ~$5,000 for the exact same thing. The math isn't subtle. And so most teams still end up sampling. Which means missing the exact exception that caused the outage. The bigger problem: errors are orphaned data. Your exception lands in Sentry. Your logs are in Datadog. Your traces are somewhere else. Root cause analysis becomes a multi-tab archaeology project at 3 am. We built error tracking natively inside Better Stack: the same platform where your logs, traces, metrics, uptime checks, and on-call schedules already live. Errors are just another signal. They belong together. The part that changes how your team works: Our AI SRE doesn't just surface errors. It fixes them. See a new exception? One click. The AI SRE analyzes the full context, from stack traces, environment variables, browser sessions, related logs and recent deploys, and opens a pull request. Not a ticket. Not a summary. A pull request with the fix. This is what happens when error tracking is fully integrated with the rest of your observability stack instead of bolted on separately. The AI has everything it needs to actually act. The migration is trivial: 1. Keep your existing Sentry SDK. Don't touch a single line of instrumentation code. 2. Point the DSN at Better Stack. 3. Done. Errors flow in. Your dashboards work. Your alerts work. 4. New exception appears. Click "Fix with AI SRE." Pull request lands in your repo. 5. Review, merge, close. That's the whole workflow. The AI angle is real, not a marketing badge. LLMs are genuinely good at fixing bugs if they have full context. The reason AI coding assistants sometimes frustrate engineers is incomplete information, not the model. We solve that by giving the AI SRE your entire telemetry stack as context. Stack traces, logs, traces, service maps, previous incidents and much more. All of it, in one place, at the moment it matters. Observability tools are only useful if you actually ingest all your data. At current prices of other tools, most teams can't afford to. Now you can, and your AI SRE can actually do something about it.
0:20

Today, we're making Error Tracking by Better Stack generally available. Sentry-compatible. AI-native. At 1/6th the price. Here's why we built it, and how to get the most out of it. What's wrong with error tracking today? Most teams use Sentry. It's solid! But at scale, the bills get brutal. Just 100M exceptions with 90 day lookback? ~$30,000 on Sentry. We charge ~$5,000 for the exact same thing. The math isn't subtle. And so most teams still end up sampling. Which means missing the exact exception that caused the outage. The bigger problem: errors are orphaned data. Your exception lands in Sentry. Your logs are in Datadog. Your traces are somewhere else. Root cause analysis becomes a multi-tab archaeology project at 3 am. We built error tracking natively inside Better Stack: the same platform where your logs, traces, metrics, uptime checks, and on-call schedules already live. Errors are just another signal. They belong together. The part that changes how your team works: Our AI SRE doesn't just surface errors. It fixes them. See a new exception? One click. The AI SRE analyzes the full context, from stack traces, environment variables, browser sessions, related logs and recent deploys, and opens a pull request. Not a ticket. Not a summary. A pull request with the fix. This is what happens when error tracking is fully integrated with the rest of your observability stack instead of bolted on separately. The AI has everything it needs to actually act. The migration is trivial: 1. Keep your existing Sentry SDK. Don't touch a single line of instrumentation code. 2. Point the DSN at Better Stack. 3. Done. Errors flow in. Your dashboards work. Your alerts work. 4. New exception appears. Click "Fix with AI SRE." Pull request lands in your repo. 5. Review, merge, close. That's the whole workflow. The AI angle is real, not a marketing badge. LLMs are genuinely good at fixing bugs if they have full context. The reason AI coding assistants sometimes frustrate engineers is incomplete information, not the model. We solve that by giving the AI SRE your entire telemetry stack as context. Stack traces, logs, traces, service maps, previous incidents and much more. All of it, in one place, at the moment it matters. Observability tools are only useful if you actually ingest all your data. At current prices of other tools, most teams can't afford to. Now you can, and your AI SRE can actually do something about it.

Juraj Masar

15,063 görüntüleme • 5 ay önce

New Short Course: Getting Structured LLM Output! Learn how to get structured outputs from your LLM applications in this course, built in partnership with .txt, and taught by Will Kurt, a Founding Engineer, and , Developer Relations Engineer. It's challenging for software to automatically parse through an LLM's freeform text outputs. Structured outputs—like JSON—solve this by converting natural language into consistent, clear, data that a machine can read and process. This course teaches you how to generate structured outputs while building several use cases, including a social media analysis agent. You’ll learn about structured outputs and efficient ways to generate outputs in your defined schema or format. You’ll begin by using structured output APIs, then use re-prompting libraries like “instructor” to generate structured output. Finally, you’ll learn how constrained decoding works; this is a very clever technique in which constraints are applied on each subsequent token generated, blocking any tokens that don’t fit your defined schema. In detail, you’ll: - Learn why structured outputs are important, how they allow for scalable software development, and the different approaches to generate them, including vendor-provided APIs, re-prompting libraries, and structured generation. - Build a simple social media agent using OpenAI’s structured output API, learn how to define a model's desired structured output using Pydantic, and perform basic programming with your outputs, such as importing structured data into a data frame using pandas. - Learn how to use the open-source library "instructor," which checks the structured output of the model and re-prompts the model until it validates the desired output, and explore the limitations of this approach. - Understand how structured generation by the “outlines” library works by modifying LLM logits, on a per-generated-token basis based on the desired format, to give a particular output structure. - Learn how regular expressions, which outlines works with, are represented as finite-state machines, and how they can be used to develop a range of structured outputs beyond JSON. By the end of this course, you’ll have broadened your knowledge of the approaches you can use to get structured outputs from your LLM applications. Please sign up here:
2:05

New Short Course: Getting Structured LLM Output! Learn how to get structured outputs from your LLM applications in this course, built in partnership with .txt, and taught by Will Kurt, a Founding Engineer, and , Developer Relations Engineer. It's challenging for software to automatically parse through an LLM's freeform text outputs. Structured outputs—like JSON—solve this by converting natural language into consistent, clear, data that a machine can read and process. This course teaches you how to generate structured outputs while building several use cases, including a social media analysis agent. You’ll learn about structured outputs and efficient ways to generate outputs in your defined schema or format. You’ll begin by using structured output APIs, then use re-prompting libraries like “instructor” to generate structured output. Finally, you’ll learn how constrained decoding works; this is a very clever technique in which constraints are applied on each subsequent token generated, blocking any tokens that don’t fit your defined schema. In detail, you’ll: - Learn why structured outputs are important, how they allow for scalable software development, and the different approaches to generate them, including vendor-provided APIs, re-prompting libraries, and structured generation. - Build a simple social media agent using OpenAI’s structured output API, learn how to define a model's desired structured output using Pydantic, and perform basic programming with your outputs, such as importing structured data into a data frame using pandas. - Learn how to use the open-source library "instructor," which checks the structured output of the model and re-prompts the model until it validates the desired output, and explore the limitations of this approach. - Understand how structured generation by the “outlines” library works by modifying LLM logits, on a per-generated-token basis based on the desired format, to give a particular output structure. - Learn how regular expressions, which outlines works with, are represented as finite-state machines, and how they can be used to develop a range of structured outputs beyond JSON. By the end of this course, you’ll have broadened your knowledge of the approaches you can use to get structured outputs from your LLM applications. Please sign up here:

Andrew Ng

89,792 görüntüleme • 1 yıl önce