Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

This guy literally broke down step-by-step how to create frontier lab quality evals: 2:42 - What an eval actually is 3:29 - Offline evals vs prod 5:23 - Why old evals stopped working 7:10 - How to write one 8:29 - Why 100% means you failed 9:03 - Models...

aakashgupta's profile picture

Aakash Gupta

303,776 subscribers

44,671 görüntüleme • 1 ay önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.
52:06

Evals are the new PRD. The companies building AI products that actually work are running 12.8 eval experiments per day. Here is the playbook with Ankur Goyal, Founder and CEO of Braintrust ($800M valuation, behind Vercel, Replit, Ramp, Zapier, Notion, Airtable): ⏱ 1:43 Why vibe checks stop scaling ⏱ 6:35 Evals are the new PRD ⏱ 8:45 The Claude Code evals controversy ⏱ 18:48 Building an eval live from zero ⏱ 29:51 Connecting Linear MCP and iterating ⏱ 39:12 Why you need evals that fail ⏱ 43:36 Offline vs online evals ⏱ 47:40 Three mistakes killing eval culture The core framework: every eval is exactly three things. A set of inputs your product needs to handle. A task that takes those inputs and generates outputs. A scoring function that produces a number between 0 and 1. We built one from scratch on camera. Score went from 0 to 0.75 in under 20 minutes.

Aakash Gupta

483,558 görüntüleme • 6 ay önce

New session with Shreya Shankar on How to Automate Evals with AI (correctly) The most important part of the eval workflow is finding issues. Shreya does a live demo of how to steer the AI iteratively to find unknown unknowns Chapter summaries: 00:00:00 Introduction 00:00:55 Why vendors want to automate evals for you 00:01:45 Why no tool can fully automate evals 00:06:00 Mistake #1: Asking AI to just "find the issues" in your data 00:08:11 What good AI-assisted error analysis actually looks like 00:11:12 Live demo: building a review interface from scratch 00:15:10 Annotating traces and building a failure mode taxonomy 00:19:31 Mistake #2: Only reviewing your data once 00:22:53 Mistake #3: Treating all apps with the same accuracy bar 00:25:10 General-purpose agents vs. dedicated eval tools 00:25:48 Summary: three rules for automating evals correctly YT links and more in reply
27:18

New session with Shreya Shankar on How to Automate Evals with AI (correctly) The most important part of the eval workflow is finding issues. Shreya does a live demo of how to steer the AI iteratively to find unknown unknowns Chapter summaries: 00:00:00 Introduction 00:00:55 Why vendors want to automate evals for you 00:01:45 Why no tool can fully automate evals 00:06:00 Mistake #1: Asking AI to just "find the issues" in your data 00:08:11 What good AI-assisted error analysis actually looks like 00:11:12 Live demo: building a review interface from scratch 00:15:10 Annotating traces and building a failure mode taxonomy 00:19:31 Mistake #2: Only reviewing your data once 00:22:53 Mistake #3: Treating all apps with the same accuracy bar 00:25:10 General-purpose agents vs. dedicated eval tools 00:25:48 Summary: three rules for automating evals correctly YT links and more in reply

Hamel Husain

34,299 görüntüleme • 2 ay önce

She literally broke down how to run evals in Claude Code (built the whole thing live): 01:34 - What people get wrong with evals 04:35 - Why product taste is the alpha now 09:28 - Building a PM agent from one prompt 19:00 - Instrumentation without writing code 22:00 - Watching traces stream in live 28:00 - Getting Claude to write your first eval 33:58 - When vibe evals work and when they don't 48:50 - The self-improving loop (this part is wild) 01:03:00 - Same-day shipping is real 01:06:00 - The context graph unlock
1:19:31

She literally broke down how to run evals in Claude Code (built the whole thing live): 01:34 - What people get wrong with evals 04:35 - Why product taste is the alpha now 09:28 - Building a PM agent from one prompt 19:00 - Instrumentation without writing code 22:00 - Watching traces stream in live 28:00 - Getting Claude to write your first eval 33:58 - When vibe evals work and when they don't 48:50 - The self-improving loop (this part is wild) 01:03:00 - Same-day shipping is real 01:06:00 - The context graph unlock

Aakash Gupta

150,362 görüntüleme • 4 ay önce

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:
1:12

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 görüntüleme • 11 ay önce

Anthropic Engineer: “the biggest mistake in AI right now is people are building graphs and loops without a self-improving eval agent. we made that mistake at anthropic. it cost us two years.” here’s how they built what solved it: step 1 → take 50 real user prompts and run your agent. if it passes 80%+, your eval is too easy. the sweet spot is around 50%. step 2 → every failed run becomes a transcript. feed it to haiku: “what went wrong?” your eval set keeps growing on its own. step 3 → score two things: the right answer and the right path. a right answer with the wrong path will break later. step 4 → when a new model drops, run the same eval again. one model looked 9% better. but 6% of that gain came from hiding a bug. the transcript caught it. the dashboard didn’t. step 5 → plug your evals into ci. no green evals = no deploy. that’s how the system keeps improving itself. 99% of people never build this. but it pays for itself on day one. save this and build it today.
28:54

Anthropic Engineer: “the biggest mistake in AI right now is people are building graphs and loops without a self-improving eval agent. we made that mistake at anthropic. it cost us two years.” here’s how they built what solved it: step 1 → take 50 real user prompts and run your agent. if it passes 80%+, your eval is too easy. the sweet spot is around 50%. step 2 → every failed run becomes a transcript. feed it to haiku: “what went wrong?” your eval set keeps growing on its own. step 3 → score two things: the right answer and the right path. a right answer with the wrong path will break later. step 4 → when a new model drops, run the same eval again. one model looked 9% better. but 6% of that gain came from hiding a bug. the transcript caught it. the dashboard didn’t. step 5 → plug your evals into ci. no green evals = no deploy. that’s how the system keeps improving itself. 99% of people never build this. but it pays for itself on day one. save this and build it today.

MIKE

102,478 görüntüleme • 1 ay önce

Everything Gets Rebuilt: my conversation with Harrison Chase, CEO of LangChain about agent harnesses, evals, runtimes, sandboxes, MCP and the future of the agent stack 00:00 Intro - meet Harrison Chase - at the Chase Center for the Daytona Compute conference 01:32 What changed in agents over the last year 03:57 Why coding agents are ahead 06:26 Do models commoditize the framework layer? 08:27 Harnesses, in plain English 10:11 Why system prompts matter so much 13:11 The upside — and downside — of subagents 15:31 Why a useful agent needs a filesystem 18:13 Additional core primitives of modern agents 19:12 Skills: the new primitive 20:19 What context compaction actually means 23:02 How memory works in agents 25:16 One mega-agent or many specialized agents? 27:46 The future of MCP 29:38 Why agents need sandboxes 32:35 How sandboxes help with security 33:32 How Harrison Chase started LangChain 37:24 LangChain vs LangGraph vs Deep Agents 40:17 Why observability matters more for agents 41:48 Evals, no-code, and continuous improvement 44:41 What LangChain is building next 45:29 Where the real moat in AI lives
46:57

Everything Gets Rebuilt: my conversation with Harrison Chase, CEO of LangChain about agent harnesses, evals, runtimes, sandboxes, MCP and the future of the agent stack 00:00 Intro - meet Harrison Chase - at the Chase Center for the Daytona Compute conference 01:32 What changed in agents over the last year 03:57 Why coding agents are ahead 06:26 Do models commoditize the framework layer? 08:27 Harnesses, in plain English 10:11 Why system prompts matter so much 13:11 The upside — and downside — of subagents 15:31 Why a useful agent needs a filesystem 18:13 Additional core primitives of modern agents 19:12 Skills: the new primitive 20:19 What context compaction actually means 23:02 How memory works in agents 25:16 One mega-agent or many specialized agents? 27:46 The future of MCP 29:38 Why agents need sandboxes 32:35 How sandboxes help with security 33:32 How Harrison Chase started LangChain 37:24 LangChain vs LangGraph vs Deep Agents 40:17 Why observability matters more for agents 41:48 Evals, no-code, and continuous improvement 44:41 What LangChain is building next 45:29 Where the real moat in AI lives

Matt Turck

35,253 görüntüleme • 6 ay önce

Terminal-Bench 2.0 went from ~25% → 80% in four months and became the standard eval for frontier CLI agents. Now, TB3 is in the works. I talked to Alex Shaw about what happens when model capabilities climb faster than we can measure them. His answer: the benchmark factory (Harbor Framework)— infrastructure to develop hard, representative evals at the pace that the frontier moves. As Alex put it: "we need a thousand times more benchmarks than we have right now." 00:23 - How quickly models hill-climbed TB2 01:46 - What rapid progress reveals about benchmarks vs. real-world capability 03:28 - What made Terminal-Bench stick 04:58 - Why the terminal is the right abstraction for agentic AI 07:14 - How TB2 maintains task quality at scale 09:23 - Managing benchmark integrity in a benchmaxxing world 10:47 - Harbor: from experiment to benchmark factory 12:19 - What Harbor does that nothing else did 14:37 - The invariants: what won't change as agent evals evolve 16:55 - The benchmark Alex most wants to see built 18:18 - The ideal human-in-the-loop task creation flywheel 20:32 - How to contribute to Terminal-Bench 3.0
22:17

Terminal-Bench 2.0 went from ~25% → 80% in four months and became the standard eval for frontier CLI agents. Now, TB3 is in the works. I talked to Alex Shaw about what happens when model capabilities climb faster than we can measure them. His answer: the benchmark factory (Harbor Framework)— infrastructure to develop hard, representative evals at the pace that the frontier moves. As Alex put it: "we need a thousand times more benchmarks than we have right now." 00:23 - How quickly models hill-climbed TB2 01:46 - What rapid progress reveals about benchmarks vs. real-world capability 03:28 - What made Terminal-Bench stick 04:58 - Why the terminal is the right abstraction for agentic AI 07:14 - How TB2 maintains task quality at scale 09:23 - Managing benchmark integrity in a benchmaxxing world 10:47 - Harbor: from experiment to benchmark factory 12:19 - What Harbor does that nothing else did 14:37 - The invariants: what won't change as agent evals evolve 16:55 - The benchmark Alex most wants to see built 18:18 - The ideal human-in-the-loop task creation flywheel 20:32 - How to contribute to Terminal-Bench 3.0

vincent sunn chen

11,600 görüntüleme • 5 ay önce

An ex-Google engineer just dropped a free 2-hour course on graph engineering and AI agents From prompts to self-improving agents 👇 0% → 00:00 - know when to use loops vs graphs 20% → 20:29 - benchmark 5 memory layers head to head 40% → 49:29 - 8 lessons from AI founders and investors 60% → 1:08:17 - see how a self-improving agent learns 80% → 1:28:12 - wire up tracing and evals 100% → 1:47:15 - understand semantic vs episodic memory Most people still think an agent is just a prompt with tools This shows what actually runs around the model, with open-source code and live demos Worth more than most paid agent courses Bookmark it and watch this weekend 👇
1:58:45

An ex-Google engineer just dropped a free 2-hour course on graph engineering and AI agents From prompts to self-improving agents 👇 0% → 00:00 - know when to use loops vs graphs 20% → 20:29 - benchmark 5 memory layers head to head 40% → 49:29 - 8 lessons from AI founders and investors 60% → 1:08:17 - see how a self-improving agent learns 80% → 1:28:12 - wire up tracing and evals 100% → 1:47:15 - understand semantic vs episodic memory Most people still think an agent is just a prompt with tools This shows what actually runs around the model, with open-source code and live demos Worth more than most paid agent courses Bookmark it and watch this weekend 👇

spect

24,212 görüntüleme • 8 gün önce

tylercowen is bullish on AI education — here's why. 00:00 -- Preview 00:24 -- President Carlos Carvalho's AI-generated intro 03:21 -- Cowen reacts to UATX's campus 04:38 -- The AI revolution is here. Who will lose the most? 06:05 -- AI lawyers 07:17 -- Don't underestimate this 10:41 -- Changes to the "upper upper middle class" 12:38 -- How to be successful 13:43 -- The rise of managerial empires 14:02 -- When will we have the first billion dollar company with one employee? 16:05 -- 10-20 year forecast 16:19 -- Why education is so behind 17:01 -- Should you be bullish on UATX? 18:36 -- Should you still read Homer? 21:50 -- Write to think 25:01 -- Meet more people 25:42 -- How to get hired 26:54 -- Is AI your best mentor? 38:17 -- How to curb cheating 39:02 -- The new life of the mind 42:34 -- Q&A: Will there be more status associated with real education or AI education? 45:50 -- Q&A: Why do tech-savvy students need to practice using AI? 47:56 -- Q&A: Do LLMs atrophy your mind? 49:29 -- Q&A: How do you avoid AI-dependency? 51:05 -- Q&A: Isn't this vision lonely and isolating? 53:06 -- Q&A: Do students need teachers? 55:36 -- Q&A: What are the four most important courses for undergrads? 57:49 -- Q&A: Which AI company will win the AI race in the next five years and why? 59:22 -- Q&A: Can AI teach religion? 01:01:32 -- Q&A: Will AI narrow or widen our world? 01:04:37 -- Q&A: What makes us human? 01:05:42 -- Q&A: What is art? 01:08:33 -- Q&A: It's easy to catch cheaters
1:10:31

tylercowen is bullish on AI education — here's why. 00:00 -- Preview 00:24 -- President Carlos Carvalho's AI-generated intro 03:21 -- Cowen reacts to UATX's campus 04:38 -- The AI revolution is here. Who will lose the most? 06:05 -- AI lawyers 07:17 -- Don't underestimate this 10:41 -- Changes to the "upper upper middle class" 12:38 -- How to be successful 13:43 -- The rise of managerial empires 14:02 -- When will we have the first billion dollar company with one employee? 16:05 -- 10-20 year forecast 16:19 -- Why education is so behind 17:01 -- Should you be bullish on UATX? 18:36 -- Should you still read Homer? 21:50 -- Write to think 25:01 -- Meet more people 25:42 -- How to get hired 26:54 -- Is AI your best mentor? 38:17 -- How to curb cheating 39:02 -- The new life of the mind 42:34 -- Q&A: Will there be more status associated with real education or AI education? 45:50 -- Q&A: Why do tech-savvy students need to practice using AI? 47:56 -- Q&A: Do LLMs atrophy your mind? 49:29 -- Q&A: How do you avoid AI-dependency? 51:05 -- Q&A: Isn't this vision lonely and isolating? 53:06 -- Q&A: Do students need teachers? 55:36 -- Q&A: What are the four most important courses for undergrads? 57:49 -- Q&A: Which AI company will win the AI race in the next five years and why? 59:22 -- Q&A: Can AI teach religion? 01:01:32 -- Q&A: Will AI narrow or widen our world? 01:04:37 -- Q&A: What makes us human? 01:05:42 -- Q&A: What is art? 01:08:33 -- Q&A: It's easy to catch cheaters

University of Austin (UATX)

27,770 görüntüleme • 8 ay önce

An agent is three things: a harness, a model, and context. If you're serious about owning your intelligence, you probably want to own all three. LangChain founder Harrison Chase joined us at our Sequoia Capital Own Your Intelligence to talk about the piece that often gets the least attention: the harness. He offers a clear heuristic for when to build your own. The more out of distribution you are from what the models were trained on, the more you'll want to customize. And good technical content on how to actually measure performance with evals and langsmith. 00:00 Introduction 00:58 The three parts of an agent: harness, model, context 02:12 What a harness actually does 03:25 Customizing the core loop with middleware 04:41 Sandboxes, file systems, sub-agents, summarization 05:47 Cognitive architectures — and when you still need them 07:03 Build your own harness or use off the shelf? 08:24 In-distribution vs. out-of-distribution: the file-editing example 09:39 Why evals define what "good" means in an organization 11:04 Harbor: what an eval task actually looks like 12:11 Comparing harnesses and models on accuracy, latency, and cost 13:20 Why observability is underrated — it's usually the context 14:34 The data flywheel: traces → curation → experiments 15:42 Getting feedback through UX design and online evaluators 16:51 Demo: LangSmith Engine 19:23 Q&A: Running Engine on Engine, and "codex-ification" 20:44 Q&A: Will harnesses converge or diverge?
23:56

An agent is three things: a harness, a model, and context. If you're serious about owning your intelligence, you probably want to own all three. LangChain founder Harrison Chase joined us at our Sequoia Capital Own Your Intelligence to talk about the piece that often gets the least attention: the harness. He offers a clear heuristic for when to build your own. The more out of distribution you are from what the models were trained on, the more you'll want to customize. And good technical content on how to actually measure performance with evals and langsmith. 00:00 Introduction 00:58 The three parts of an agent: harness, model, context 02:12 What a harness actually does 03:25 Customizing the core loop with middleware 04:41 Sandboxes, file systems, sub-agents, summarization 05:47 Cognitive architectures — and when you still need them 07:03 Build your own harness or use off the shelf? 08:24 In-distribution vs. out-of-distribution: the file-editing example 09:39 Why evals define what "good" means in an organization 11:04 Harbor: what an eval task actually looks like 12:11 Comparing harnesses and models on accuracy, latency, and cost 13:20 Why observability is underrated — it's usually the context 14:34 The data flywheel: traces → curation → experiments 15:42 Getting feedback through UX design and online evaluators 16:51 Demo: LangSmith Engine 19:23 Q&A: Running Engine on Engine, and "codex-ification" 20:44 Q&A: Will harnesses converge or diverge?

Sonya Huang 🐥

77,519 görüntüleme • 1 ay önce

Why AI Can Now Make Discoveries - my conversation with Dan Roberts, Lead of the Foundations of Reinforcement Learning team at OpenAI 00:00 Intro: AI's wild week in mathematics 01:21 What OpenAI's Foundations of RL team does 03:08 Dan's journey: from black holes and quantum gravity to frontier AI 07:04 Are AI systems becoming useful for real science 08:21 The AI math moment: Erdős, OpenAI, DeepMind, and Anthropic 08:52 Why the OpenAI result was an act of exploration 10:25 OpenAI vs. DeepMind: informal reasoning vs. formal proof 12:13 RL 101: learning by doing, not just watching 15:10 Why reinforcement learning works 15:58 How RL breaks: sparse feedback and long-horizon tasks 17:03 RLHF: how human feedback shaped early language models 18:48 Move 37, self-play, and the search for novel strategies 22:16 Explore vs. exploit in scientific discovery 24:49 Why RL may now be "the cake," not the cherry on top 25:46 Why RL started working with large language models 27:29 Is RL "sucking supervision through a straw"? 28:47 Why language may be the grounding layer for intelligence 31:46 A contrarian take on the Bitter Lesson 32:41 What test-time compute actually is 34:50 How RL gives models the ability to think 35:40 Verifiable rewards, math, coding, and the messy real world 38:00 What physics can teach us about AI 42:08 Is there a thermodynamics of AI? 43:08 From Erdős problems to Einstein-level AI 45:16 Is AI already doing original science? 45:51 How far are we from AI automating AI research 47:41 Why Dan is excited about the future of science
49:06

Why AI Can Now Make Discoveries - my conversation with Dan Roberts, Lead of the Foundations of Reinforcement Learning team at OpenAI 00:00 Intro: AI's wild week in mathematics 01:21 What OpenAI's Foundations of RL team does 03:08 Dan's journey: from black holes and quantum gravity to frontier AI 07:04 Are AI systems becoming useful for real science 08:21 The AI math moment: Erdős, OpenAI, DeepMind, and Anthropic 08:52 Why the OpenAI result was an act of exploration 10:25 OpenAI vs. DeepMind: informal reasoning vs. formal proof 12:13 RL 101: learning by doing, not just watching 15:10 Why reinforcement learning works 15:58 How RL breaks: sparse feedback and long-horizon tasks 17:03 RLHF: how human feedback shaped early language models 18:48 Move 37, self-play, and the search for novel strategies 22:16 Explore vs. exploit in scientific discovery 24:49 Why RL may now be "the cake," not the cherry on top 25:46 Why RL started working with large language models 27:29 Is RL "sucking supervision through a straw"? 28:47 Why language may be the grounding layer for intelligence 31:46 A contrarian take on the Bitter Lesson 32:41 What test-time compute actually is 34:50 How RL gives models the ability to think 35:40 Verifiable rewards, math, coding, and the messy real world 38:00 What physics can teach us about AI 42:08 Is there a thermodynamics of AI? 43:08 From Erdős problems to Einstein-level AI 45:16 Is AI already doing original science? 45:51 How far are we from AI automating AI research 47:41 Why Dan is excited about the future of science

Matt Turck

69,801 görüntüleme • 3 ay önce

This guy literally broke down how to become a $1.4M "builder PM" with n8n, Claude Code, and OpenClaw: 1:53 - What a "builder PM" actually is 6:04 - Your first agent in n8n (live build) 14:18 - Why every agent needs these 4 things 21:35 - The multi-agent eval loop 29:47 - Where n8n dies 33:39 - When to graduate to Claude Code 35:08 - What broke in December 2025 47:17 - The self-improving PRD reviewer 1:02:28 - Mocks and prototypes without designers 1:05:15 - OpenClaw and the new agent OS 1:22:06 - What AI PM interviews look like now
1:36:25

This guy literally broke down how to become a $1.4M "builder PM" with n8n, Claude Code, and OpenClaw: 1:53 - What a "builder PM" actually is 6:04 - Your first agent in n8n (live build) 14:18 - Why every agent needs these 4 things 21:35 - The multi-agent eval loop 29:47 - Where n8n dies 33:39 - When to graduate to Claude Code 35:08 - What broke in December 2025 47:17 - The self-improving PRD reviewer 1:02:28 - Mocks and prototypes without designers 1:05:15 - OpenClaw and the new agent OS 1:22:06 - What AI PM interviews look like now

Aakash Gupta

425,339 görüntüleme • 5 ay önce

how to build, launch and grow your startup: complete basics that every founder learns by accident 00:00 Intro to building 00:14 The build-measure-learn loop 04:12 What makes a good one-liner 05:40 Rewriting bad one-liners 08:43 Billboards: the Monaco trick 10:12 Set a specific 5-week goal 13:43 Goals that never end will break you 15:42 Intro to Content 18:15 Why luck is the real product 25:03 Fix your bio right now 28:24 Cheat sheet for viral launches 30:49 Step 1: Find 10 strangers (not founders) 33:07 Step 2: The "oh, that makes sense" moment 34:26 Step 3: A visually clear use case 42:29 The $100 launch video that hit 4M 46:04 Context in the first 2 seconds 48:15 Writing with an eraser 51:11 Framework 1: Comparison 54:29 Framework 2: Antagonist 56:01 Framework 3: The Gap 59:37 Be the world's leading expert 01:06:42 Steal your customer's FYP 01:11:47 The 25-video spreadsheet 01:19:36 Intro to Growth 01:21:32 The only metric that matters 01:23:20 Why 1M LP hits is bad growth 01:26:53 The #1 rule on user calls 01:27:15 The AI realtor story 01:34:19 Why Reddit beats Twitter 01:40:15 The landing page that turns pink
1:43:51

how to build, launch and grow your startup: complete basics that every founder learns by accident 00:00 Intro to building 00:14 The build-measure-learn loop 04:12 What makes a good one-liner 05:40 Rewriting bad one-liners 08:43 Billboards: the Monaco trick 10:12 Set a specific 5-week goal 13:43 Goals that never end will break you 15:42 Intro to Content 18:15 Why luck is the real product 25:03 Fix your bio right now 28:24 Cheat sheet for viral launches 30:49 Step 1: Find 10 strangers (not founders) 33:07 Step 2: The "oh, that makes sense" moment 34:26 Step 3: A visually clear use case 42:29 The $100 launch video that hit 4M 46:04 Context in the first 2 seconds 48:15 Writing with an eraser 51:11 Framework 1: Comparison 54:29 Framework 2: Antagonist 56:01 Framework 3: The Gap 59:37 Be the world's leading expert 01:06:42 Steal your customer's FYP 01:11:47 The 25-video spreadsheet 01:19:36 Intro to Growth 01:21:32 The only metric that matters 01:23:20 Why 1M LP hits is bad growth 01:26:53 The #1 rule on user calls 01:27:15 The AI realtor story 01:34:19 Why Reddit beats Twitter 01:40:15 The landing page that turns pink

Founders Inc

21,257 görüntüleme • 4 ay önce

Anthropic just dropped a 45-minute workshop on how to build agents with Fable 5 for 20% of it cost. "give Fable very difficult tasks - that's when you actually see the value" • 10% → 1:40 - full capabilities of Fable 5 • 30% → 11:13 - how to build evals with Fable • 55% → 13:29 - when to use Fable & Sonnet • 80% → 17:23 -how to use Fable 50% cheaper • 100% → 18:03 - Fable 5 Agents orchestration patterns This is worth more then 100 youtube with on how to use Fable 5 Watch today - then read the full playbook below ↓
45:00

Anthropic just dropped a 45-minute workshop on how to build agents with Fable 5 for 20% of it cost. "give Fable very difficult tasks - that's when you actually see the value" • 10% → 1:40 - full capabilities of Fable 5 • 30% → 11:13 - how to build evals with Fable • 55% → 13:29 - when to use Fable & Sonnet • 80% → 17:23 -how to use Fable 50% cheaper • 100% → 18:03 - Fable 5 Agents orchestration patterns This is worth more then 100 youtube with on how to use Fable 5 Watch today - then read the full playbook below ↓

Movez

16,134 görüntüleme • 1 ay önce

A model is only as good as its data, and we’ve long since exhausted the internet. From here on out, model progress is gated by data production. ’s Brendan (can/do) joined us at our Sovereign AI event to talk about how RL environments get built, and why your data might be your real moat: 00:00 Introduction 00:47 A short history of the data market: crowdsourcing to agentic data 02:29 What an RL environment is: worlds, apps, tasks 03:57 Why only humans can measure the frontier 05:35 Building verifiers is the hard part 06:44 Walkthrough: a real legal RL environment 08:18 Leaderboards — and what open weights change 09:45 Post-training results on Apex Agents 11:17 Three ways companies buy data 12:49 Q&A: How do you price data? 14:17 Q&A: What "data quality" actually means 16:42 Q&A: The misunderstanding about synthetic data 18:17 Q&A: Why RL environments now — and what comes after 21:20 Q&A: Can you scale rubric generation with models? 23:00 Q&A: RL environments for cyber defense 25:33 Q&A: Build data in-house or partner?
26:21

A model is only as good as its data, and we’ve long since exhausted the internet. From here on out, model progress is gated by data production. ’s Brendan (can/do) joined us at our Sovereign AI event to talk about how RL environments get built, and why your data might be your real moat: 00:00 Introduction 00:47 A short history of the data market: crowdsourcing to agentic data 02:29 What an RL environment is: worlds, apps, tasks 03:57 Why only humans can measure the frontier 05:35 Building verifiers is the hard part 06:44 Walkthrough: a real legal RL environment 08:18 Leaderboards — and what open weights change 09:45 Post-training results on Apex Agents 11:17 Three ways companies buy data 12:49 Q&A: How do you price data? 14:17 Q&A: What "data quality" actually means 16:42 Q&A: The misunderstanding about synthetic data 18:17 Q&A: Why RL environments now — and what comes after 21:20 Q&A: Can you scale rubric generation with models? 23:00 Q&A: RL environments for cyber defense 25:33 Q&A: Build data in-house or partner?

Pat Grady

66,577 görüntüleme • 1 ay önce

I'm often asked for the best public example of AI evals done right for a real, production product. I finally have an answer. Teresa Torres shares how she shipped an AI interview coach, and used evals to rapidly squash bugs and improve the product. Teresa shows how she: 1. did error analysis FIRST to find real issues (instead of using generic metrics) 😍 2. used Jupyter notebooks to analyze errors 3. built custom annotation tools + custom widgets in notebooks 4. built a LLM-judge and assertions to test for specific errors 5. iterated through this feedback loop until it worked. 6. kept things simple the whole time It's also probably the best commercial for Jupyter notebooks you can imagine. 🥰 Chapter summary below. Link to YT in next thread 00:00:00 - Intro 00:01:45 - The Product: Building an AI Interview Coach 00:06:34 - The Problem: How Do I Know if My AI Coach is Any Good? 00:10:15 - Using Airtable for Traces and Annotation 00:12:15 - Discovering Jupyter Notebooks and Designing the First Evals 00:15:15 - Example Evals: LLM-as-Judge vs. Code-Based Assertions 00:21:00 - Learning Python with ChatGPT to Analyze Eval Results 00:31:00 - VS Code, Custom Tools, and an Eval Investigation Notebook 00:39:45 - Building a Custom Annotation Tool with Claude 00:41:00 - From Personal Project to Production App 00:46:02 - How Should PMs and Engineers Collaborate on AI Products? 00:55:45 - Q&A: Capturing Feedback and Annotations from End Users 00:58:11 - Q&A: Is a Technical Background Necessary to Build AI? 01:02:28 - Q&A: What's Next for Teresa? 01:03:13 - Q&A: Unpacking the Micro-Decisions of Building an AI App
1:10:22

I'm often asked for the best public example of AI evals done right for a real, production product. I finally have an answer. Teresa Torres shares how she shipped an AI interview coach, and used evals to rapidly squash bugs and improve the product. Teresa shows how she: 1. did error analysis FIRST to find real issues (instead of using generic metrics) 😍 2. used Jupyter notebooks to analyze errors 3. built custom annotation tools + custom widgets in notebooks 4. built a LLM-judge and assertions to test for specific errors 5. iterated through this feedback loop until it worked. 6. kept things simple the whole time It's also probably the best commercial for Jupyter notebooks you can imagine. 🥰 Chapter summary below. Link to YT in next thread 00:00:00 - Intro 00:01:45 - The Product: Building an AI Interview Coach 00:06:34 - The Problem: How Do I Know if My AI Coach is Any Good? 00:10:15 - Using Airtable for Traces and Annotation 00:12:15 - Discovering Jupyter Notebooks and Designing the First Evals 00:15:15 - Example Evals: LLM-as-Judge vs. Code-Based Assertions 00:21:00 - Learning Python with ChatGPT to Analyze Eval Results 00:31:00 - VS Code, Custom Tools, and an Eval Investigation Notebook 00:39:45 - Building a Custom Annotation Tool with Claude 00:41:00 - From Personal Project to Production App 00:46:02 - How Should PMs and Engineers Collaborate on AI Products? 00:55:45 - Q&A: Capturing Feedback and Annotations from End Users 00:58:11 - Q&A: Is a Technical Background Necessary to Build AI? 01:02:28 - Q&A: What's Next for Teresa? 01:03:13 - Q&A: Unpacking the Micro-Decisions of Building an AI App

Hamel Husain

51,376 görüntüleme • 1 yıl önce

E159: Hyperliquid: Housing all of Finance jeff.hl came back on the When Shift Happens Podcast to talk about the Hyperliquid journey since the TGE and what the future holds for one of the most loved and prolific protocols in the space Hyperliquid Timestamps 0:00 Intro 2:01 Singapore 2:27 Reminiscing on the Token Launch 5:00 Was This Scale Of Wealth Expected? 6:28 Doing The Right Thing In Crypto 9:07 The Responsibility that comes with Billions of $ 11:10 Jupiter KAST 11:51 Bringing Hyperliquid to the masses 15:21 Pre TGE and Post TGE: Operational difference 20:13 Choices on what to build Internally vs Externally 22:05 How to build a reliable team 24:51 Did the Team celebrate the HYPE wealth Generation event? 26:45 How to test talents for High Integrity 28:31 How much does the Hyperliquid team sleep? 30:05 Employee Vesting Fears 31:41 Dealing with FUD 32:28 How Does Jeff Personally Handle FUD 35:02 Token "Buybacks" critics 37:20 Why Hyperliquid can't have Discretionary "Buybacks" 39:04 HyperEVM, explained Simply 40:00 Paradex Zodl 40:41 HyperEVM: Success so Far? 44:05 HIP-3, explained Simply 47:44 What makes Hyperliquid's approach different 48:19 Why Should People Care? 51:33 Bring All Finance On Chain 52:08 Why Is The Hyperliquid Approach Better? 53:47 Key Numbers showing that Hyperliquid Is Doing it right 59:01 What Has the Unit team demonstrated with spot trading on Hyperliquid in 2025 1:03:29 HIP-4: Outcome Markets 1:08:01 Trezor Sui 1:08:58 What does "Housing All Of Finance" mean? 1:10:51 Why Hyperliquid is not a crypto company 1:12:23 Why Does Hyperliquid have A Stablecoin USDH (Native Markets) 1:14:39 What Is Kinetiq & Why Does It Matter? 1:16:15 Why Is What HyperLend Is Building Important For HyperLiquid 1:23:39 Where did Fairness cost the most? 1:24:47 What should Hyperliquid be Remembered for? 1:25:24 Why should people stay in Crypto when there's an AI brain drain? 1:28:10 Closing Thoughts
1:29:06

E159: Hyperliquid: Housing all of Finance jeff.hl came back on the When Shift Happens Podcast to talk about the Hyperliquid journey since the TGE and what the future holds for one of the most loved and prolific protocols in the space Hyperliquid Timestamps 0:00 Intro 2:01 Singapore 2:27 Reminiscing on the Token Launch 5:00 Was This Scale Of Wealth Expected? 6:28 Doing The Right Thing In Crypto 9:07 The Responsibility that comes with Billions of $ 11:10 Jupiter KAST 11:51 Bringing Hyperliquid to the masses 15:21 Pre TGE and Post TGE: Operational difference 20:13 Choices on what to build Internally vs Externally 22:05 How to build a reliable team 24:51 Did the Team celebrate the HYPE wealth Generation event? 26:45 How to test talents for High Integrity 28:31 How much does the Hyperliquid team sleep? 30:05 Employee Vesting Fears 31:41 Dealing with FUD 32:28 How Does Jeff Personally Handle FUD 35:02 Token "Buybacks" critics 37:20 Why Hyperliquid can't have Discretionary "Buybacks" 39:04 HyperEVM, explained Simply 40:00 Paradex Zodl 40:41 HyperEVM: Success so Far? 44:05 HIP-3, explained Simply 47:44 What makes Hyperliquid's approach different 48:19 Why Should People Care? 51:33 Bring All Finance On Chain 52:08 Why Is The Hyperliquid Approach Better? 53:47 Key Numbers showing that Hyperliquid Is Doing it right 59:01 What Has the Unit team demonstrated with spot trading on Hyperliquid in 2025 1:03:29 HIP-4: Outcome Markets 1:08:01 Trezor Sui 1:08:58 What does "Housing All Of Finance" mean? 1:10:51 Why Hyperliquid is not a crypto company 1:12:23 Why Does Hyperliquid have A Stablecoin USDH (Native Markets) 1:14:39 What Is Kinetiq & Why Does It Matter? 1:16:15 Why Is What HyperLend Is Building Important For HyperLiquid 1:23:39 Where did Fairness cost the most? 1:24:47 What should Hyperliquid be Remembered for? 1:25:24 Why should people stay in Crypto when there's an AI brain drain? 1:28:10 Closing Thoughts

MR SHIFT 🦁

584,416 görüntüleme • 7 ay önce

Inside Nemotron and NVIDIA's AI lab: my conversation with Bryan Catanzaro (Bryan Catanzaro). NVIDIA is a chip company. So why does it put hundreds of researchers on building AI models - and then give them away for free? We go deep into the Nemotron models, what it takes to build a top AI lab, and the future of frontier AI. 01:33 - Is open source AI catching the frontier? 05:29 - Do closed labs blocking distillation slow open source down? 07:42 - Is the US falling behind China? 10:30 - Why companies actually choose open models 12:39 - A "crazy" 2008 bet: machine learning on GPUs 15:33 - Working with Andrew Ng and Dario Amodei at Baidu 17:41 - Coming back to NVIDIA: DLSS and the birth of Megatron 21:55 - The real reason NVIDIA builds its own models 24:28 - Is Moore's Law really dead? 33:37 - The Nemotron family: Nano, Super, Ultra 35:09 - Built for agents: why NVIDIA bets on speed 36:02 - How you train a 550B model in 4 bits 39:25 - Hybrid Mamba-Transformer, explained simply 42:31 - Mixture of experts, and why NVIDIA built NVL72 around it 47:26 - Why a 1-million-token context window matters 49:26 - Multi-token prediction: how the model predicts 5 tokens at once 52:47 - Multi-teacher distillation: teaching one model from many 58:01 - Where reinforcement learning goes next 01:00:16 - Inside NVIDIA's research org: "the mission is the boss" 01:04:03 - How NVIDIA decides who gets the GPUs 01:10:53 - Why NVIDIA still feels entrepreneurial after 33 years 01:12:58 - Why Bryan doesn't believe in the singularity 01:17:50 - The AI backlash 01:19:18 - The controversial case: open AI is safer than closed
1:22:57

Inside Nemotron and NVIDIA's AI lab: my conversation with Bryan Catanzaro (Bryan Catanzaro). NVIDIA is a chip company. So why does it put hundreds of researchers on building AI models - and then give them away for free? We go deep into the Nemotron models, what it takes to build a top AI lab, and the future of frontier AI. 01:33 - Is open source AI catching the frontier? 05:29 - Do closed labs blocking distillation slow open source down? 07:42 - Is the US falling behind China? 10:30 - Why companies actually choose open models 12:39 - A "crazy" 2008 bet: machine learning on GPUs 15:33 - Working with Andrew Ng and Dario Amodei at Baidu 17:41 - Coming back to NVIDIA: DLSS and the birth of Megatron 21:55 - The real reason NVIDIA builds its own models 24:28 - Is Moore's Law really dead? 33:37 - The Nemotron family: Nano, Super, Ultra 35:09 - Built for agents: why NVIDIA bets on speed 36:02 - How you train a 550B model in 4 bits 39:25 - Hybrid Mamba-Transformer, explained simply 42:31 - Mixture of experts, and why NVIDIA built NVL72 around it 47:26 - Why a 1-million-token context window matters 49:26 - Multi-token prediction: how the model predicts 5 tokens at once 52:47 - Multi-teacher distillation: teaching one model from many 58:01 - Where reinforcement learning goes next 01:00:16 - Inside NVIDIA's research org: "the mission is the boss" 01:04:03 - How NVIDIA decides who gets the GPUs 01:10:53 - Why NVIDIA still feels entrepreneurial after 33 years 01:12:58 - Why Bryan doesn't believe in the singularity 01:17:50 - The AI backlash 01:19:18 - The controversial case: open AI is safer than closed

Matt Turck

56,954 görüntüleme • 2 ay önce

.Ben Shapiro at UATX with Joe Lonsdale and Niall Ferguson. 00:00 — UATX President Carlos Carvalho 03:16 — Niall Ferguson introduces Ben Shapiro 07:26 — Why UATX is important 08:09 — Why Americans hate each other 10:12 — Emotivism 11:57 — The death of politics 12:30 — Hannibal Lecter skin suits 12:53 — Conspiracy theories 14:11 — Why people don't go to church anymore 15:04 — Vaccines 16:02 — Social engineering & weak professors 17:08 — The last time Harvard meant veritas 17:49 — Epistemic humility 18:30 — Read the Federalist Papers 19:56 — War of all against all 21:03 — Capitalism & soul sickness 22:42 — Tribalism 23:38 — JS Mill and debate culture 26:09 — How to restore our institutions 26:40 — Why UATX matters 27:52 — Joe Lonsdale interviews Ben Shapiro 28:10 — What is a college degree worth? 29:46 — What Jews should learn from Christians and vice versa 32:21 — Candace Owens 34:24 — Audience Q&A: Abraham Lincoln & the Declaration of Independence 36:10 — Q&A: Constitutional boundaries 38:31 — Q&A: Tucker Carlson 42:09 — Q&A: American ingratitude 45:02 — Q&A: How to unite our country 48:24 — Q&A: How to repair our institutions 51:41 — Q&A: The three most important words in the English language 54:25 — Q&A: The future of populism 57:08 — Q&A: Lizard brains 59:43 — Q&A: How Israeli politics work 01:04:03 — Q&A: How to strengthen America 01:07:29 — Q&A: Practical advice for students 01:09:07 — Standing ovation for Ben Shapiro 01:09:36 — President Carlos Carvalho's speech 01:11:08 — How to raise lions Recorded: Sunday, April 26.
1:17:38

.Ben Shapiro at UATX with Joe Lonsdale and Niall Ferguson. 00:00 — UATX President Carlos Carvalho 03:16 — Niall Ferguson introduces Ben Shapiro 07:26 — Why UATX is important 08:09 — Why Americans hate each other 10:12 — Emotivism 11:57 — The death of politics 12:30 — Hannibal Lecter skin suits 12:53 — Conspiracy theories 14:11 — Why people don't go to church anymore 15:04 — Vaccines 16:02 — Social engineering & weak professors 17:08 — The last time Harvard meant veritas 17:49 — Epistemic humility 18:30 — Read the Federalist Papers 19:56 — War of all against all 21:03 — Capitalism & soul sickness 22:42 — Tribalism 23:38 — JS Mill and debate culture 26:09 — How to restore our institutions 26:40 — Why UATX matters 27:52 — Joe Lonsdale interviews Ben Shapiro 28:10 — What is a college degree worth? 29:46 — What Jews should learn from Christians and vice versa 32:21 — Candace Owens 34:24 — Audience Q&A: Abraham Lincoln & the Declaration of Independence 36:10 — Q&A: Constitutional boundaries 38:31 — Q&A: Tucker Carlson 42:09 — Q&A: American ingratitude 45:02 — Q&A: How to unite our country 48:24 — Q&A: How to repair our institutions 51:41 — Q&A: The three most important words in the English language 54:25 — Q&A: The future of populism 57:08 — Q&A: Lizard brains 59:43 — Q&A: How Israeli politics work 01:04:03 — Q&A: How to strengthen America 01:07:29 — Q&A: Practical advice for students 01:09:07 — Standing ovation for Ben Shapiro 01:09:36 — President Carlos Carvalho's speech 01:11:08 — How to raise lions Recorded: Sunday, April 26.

University of Austin (UATX)

36,540 görüntüleme • 4 ay önce

In this episode, OpenAI Chief Research Officer Mark Chen joins allen to flambé shrimp, cook Korean stew, and chat about being at the frontier of AI research: why scaling laws and pre-training still matter, how OpenAI chooses research bets and allocates compute, what it means to develop research taste, why evals are in crisis, how to avoid benchmark-maxing, and what it will take for models to handle long-horizon real-world work, multimodal reasoning, and eventually end-to-end AI research. Timestamps: 0:00 Intro 0:28 The Soup Story 1:52 From Trading to AI Research 3:21 How to Develop Research Taste 5:23 RL, Evals, and Superhuman Benchmarks 8:17 Cooking Begins on the Impulse Stove 8:53 Scaling Laws, Pre-Training, and Reasoning 12:33 OpenAI’s Research Roadmap and Compute Allocation 15:48 What Makes a Great Researcher 19:33 The Evals Crisis and Benchmark-Maxing 24:34 Jagged Intelligence, Context, and Long-Horizon Learning 27:14 Shrimp Flambé and New Research Bets 31:32 Multimodal Models and One Architecture 32:36 Vibe Researching and End-to-End AI Research 34:36 Failed Bets, Postmortems, and OpenAI’s Alpha 37:07 Final Taste Test 37:53 Overrated vs. Underrated AI Research 41:00 Closing
41:16

In this episode, OpenAI Chief Research Officer Mark Chen joins allen to flambé shrimp, cook Korean stew, and chat about being at the frontier of AI research: why scaling laws and pre-training still matter, how OpenAI chooses research bets and allocates compute, what it means to develop research taste, why evals are in crisis, how to avoid benchmark-maxing, and what it will take for models to handle long-horizon real-world work, multimodal reasoning, and eventually end-to-end AI research. Timestamps: 0:00 Intro 0:28 The Soup Story 1:52 From Trading to AI Research 3:21 How to Develop Research Taste 5:23 RL, Evals, and Superhuman Benchmarks 8:17 Cooking Begins on the Impulse Stove 8:53 Scaling Laws, Pre-Training, and Reasoning 12:33 OpenAI’s Research Roadmap and Compute Allocation 15:48 What Makes a Great Researcher 19:33 The Evals Crisis and Benchmark-Maxing 24:34 Jagged Intelligence, Context, and Long-Horizon Learning 27:14 Shrimp Flambé and New Research Bets 31:32 Multimodal Models and One Architecture 32:36 Vibe Researching and End-to-End AI Research 34:36 Failed Bets, Postmortems, and OpenAI’s Alpha 37:07 Final Taste Test 37:53 Overrated vs. Underrated AI Research 41:00 Closing

Latent.Space

62,796 görüntüleme • 2 ay önce