Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

How do we run evals with LLMs? Interview with Tanishq Singh. This is a real interview question from a big tech company, asked to a candidate in their technical interview round. The video explains the answer in roughly 11 minutes. 00:00 Question - Reliability with LLMs 01:00 Observability &... show more

Gaurav Sen

73,351 subscribers

34,290 Aufrufe • vor 6 Tagen •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

How do we cache prompts with LLMs? Interview with Tanishq Singh. This is a real interview question from a big tech company, asked to a candidate in their technical interview round. The video explains the answer in roughly 6 minutes. 00:00 Question - Prompt Caching with LLMs 00:39 Exact Query Caching 02:04 Semantic Caching 04:14 Tradeoffs 05:52 Blooper :p AI engineering program for software engineers: #LLMs #AI #InterviewReady

How do we cache prompts with LLMs? Interview with Tanishq Singh. This is a real interview question from a big tech company, asked to a candidate in their technical interview round. The video explains the answer in roughly 6 minutes. 00:00 Question - Prompt Caching with LLMs 00:39 Exact Query Caching 02:04 Semantic Caching 04:14 Tradeoffs 05:52 Blooper :p AI engineering program for software engineers: #LLMs #AI #InterviewReady

Gaurav Sen

93,728 Aufrufe • vor 15 Tagen

What's better? LLMs or Agents? This is a real interview question from a big tech company. For more questions, subscribe to the channel! AI engineering program for software engineers: 00:00 Question - LLMs vs. Agents 01:02 Basic Answer 02:16 Follow-up question 03:47 Context Management 05:08 Bloopers #LLMs #AI #InterviewReady

What's better? LLMs or Agents? This is a real interview question from a big tech company. For more questions, subscribe to the channel! AI engineering program for software engineers: 00:00 Question - LLMs vs. Agents 01:02 Basic Answer 02:16 Follow-up question 03:47 Context Management 05:08 Bloopers #LLMs #AI #InterviewReady

Gaurav Sen

18,889 Aufrufe • vor 29 Tagen

New session with Shreya Shankar on How to Automate Evals with AI (correctly) The most important part of the eval workflow is finding issues. Shreya does a live demo of how to steer the AI iteratively to find unknown unknowns Chapter summaries: 00:00:00 Introduction 00:00:55 Why vendors want to automate evals for you 00:01:45 Why no tool can fully automate evals 00:06:00 Mistake #1: Asking AI to just "find the issues" in your data 00:08:11 What good AI-assisted error analysis actually looks like 00:11:12 Live demo: building a review interface from scratch 00:15:10 Annotating traces and building a failure mode taxonomy 00:19:31 Mistake #2: Only reviewing your data once 00:22:53 Mistake #3: Treating all apps with the same accuracy bar 00:25:10 General-purpose agents vs. dedicated eval tools 00:25:48 Summary: three rules for automating evals correctly YT links and more in reply

New session with Shreya Shankar on How to Automate Evals with AI (correctly) The most important part of the eval workflow is finding issues. Shreya does a live demo of how to steer the AI iteratively to find unknown unknowns Chapter summaries: 00:00:00 Introduction 00:00:55 Why vendors want to automate evals for you 00:01:45 Why no tool can fully automate evals 00:06:00 Mistake #1: Asking AI to just "find the issues" in your data 00:08:11 What good AI-assisted error analysis actually looks like 00:11:12 Live demo: building a review interface from scratch 00:15:10 Annotating traces and building a failure mode taxonomy 00:19:31 Mistake #2: Only reviewing your data once 00:22:53 Mistake #3: Treating all apps with the same accuracy bar 00:25:10 General-purpose agents vs. dedicated eval tools 00:25:48 Summary: three rules for automating evals correctly YT links and more in reply

Hamel Husain

34,196 Aufrufe • vor 1 Monat

I'm often asked for the best public example of AI evals done right for a real, production product. I finally have an answer. Teresa Torres shares how she shipped an AI interview coach, and used evals to rapidly squash bugs and improve the product. Teresa shows how she: 1. did error analysis FIRST to find real issues (instead of using generic metrics) 😍 2. used Jupyter notebooks to analyze errors 3. built custom annotation tools + custom widgets in notebooks 4. built a LLM-judge and assertions to test for specific errors 5. iterated through this feedback loop until it worked. 6. kept things simple the whole time It's also probably the best commercial for Jupyter notebooks you can imagine. 🥰 Chapter summary below. Link to YT in next thread 00:00:00 - Intro 00:01:45 - The Product: Building an AI Interview Coach 00:06:34 - The Problem: How Do I Know if My AI Coach is Any Good? 00:10:15 - Using Airtable for Traces and Annotation 00:12:15 - Discovering Jupyter Notebooks and Designing the First Evals 00:15:15 - Example Evals: LLM-as-Judge vs. Code-Based Assertions 00:21:00 - Learning Python with ChatGPT to Analyze Eval Results 00:31:00 - VS Code, Custom Tools, and an Eval Investigation Notebook 00:39:45 - Building a Custom Annotation Tool with Claude 00:41:00 - From Personal Project to Production App 00:46:02 - How Should PMs and Engineers Collaborate on AI Products? 00:55:45 - Q&A: Capturing Feedback and Annotations from End Users 00:58:11 - Q&A: Is a Technical Background Necessary to Build AI? 01:02:28 - Q&A: What's Next for Teresa? 01:03:13 - Q&A: Unpacking the Micro-Decisions of Building an AI App

I'm often asked for the best public example of AI evals done right for a real, production product. I finally have an answer. Teresa Torres shares how she shipped an AI interview coach, and used evals to rapidly squash bugs and improve the product. Teresa shows how she: 1. did error analysis FIRST to find real issues (instead of using generic metrics) 😍 2. used Jupyter notebooks to analyze errors 3. built custom annotation tools + custom widgets in notebooks 4. built a LLM-judge and assertions to test for specific errors 5. iterated through this feedback loop until it worked. 6. kept things simple the whole time It's also probably the best commercial for Jupyter notebooks you can imagine. 🥰 Chapter summary below. Link to YT in next thread 00:00:00 - Intro 00:01:45 - The Product: Building an AI Interview Coach 00:06:34 - The Problem: How Do I Know if My AI Coach is Any Good? 00:10:15 - Using Airtable for Traces and Annotation 00:12:15 - Discovering Jupyter Notebooks and Designing the First Evals 00:15:15 - Example Evals: LLM-as-Judge vs. Code-Based Assertions 00:21:00 - Learning Python with ChatGPT to Analyze Eval Results 00:31:00 - VS Code, Custom Tools, and an Eval Investigation Notebook 00:39:45 - Building a Custom Annotation Tool with Claude 00:41:00 - From Personal Project to Production App 00:46:02 - How Should PMs and Engineers Collaborate on AI Products? 00:55:45 - Q&A: Capturing Feedback and Annotations from End Users 00:58:11 - Q&A: Is a Technical Background Necessary to Build AI? 01:02:28 - Q&A: What's Next for Teresa? 01:03:13 - Q&A: Unpacking the Micro-Decisions of Building an AI App

Hamel Husain

51,376 Aufrufe • vor 1 Jahr

She literally broke down how to run evals in Claude Code (built the whole thing live): 01:34 - What people get wrong with evals 04:35 - Why product taste is the alpha now 09:28 - Building a PM agent from one prompt 19:00 - Instrumentation without writing code 22:00 - Watching traces stream in live 28:00 - Getting Claude to write your first eval 33:58 - When vibe evals work and when they don't 48:50 - The self-improving loop (this part is wild) 01:03:00 - Same-day shipping is real 01:06:00 - The context graph unlock

She literally broke down how to run evals in Claude Code (built the whole thing live): 01:34 - What people get wrong with evals 04:35 - Why product taste is the alpha now 09:28 - Building a PM agent from one prompt 19:00 - Instrumentation without writing code 22:00 - Watching traces stream in live 28:00 - Getting Claude to write your first eval 33:58 - When vibe evals work and when they don't 48:50 - The self-improving loop (this part is wild) 01:03:00 - Same-day shipping is real 01:06:00 - The context graph unlock

Aakash Gupta

150,362 Aufrufe • vor 2 Monaten

NEW Greylock Change Agents: Evaluating Agents with Braintrust There's been a lot of discourse around evals recently, so it's timely to drop the recording of Change Agents with Ankur Goyal on the topic of how he thinks about Evaluating Agents. Timestamps: (0:00) - Intro (1:22) - Building Braintrust (5:19) - Evaluating agentic systems (11:04) - Experimentation with evals (13:18) - Adopting evals (17:09) - Evals and complex agents (21:19) - Custom scoring (25:02) - LLM as a judge (26:44) - Human-manual evaluation (30:07) - How the best in-class teams manage evals (33:57) - Future of evals for agents (37:38 ) - Audience Q&A

NEW Greylock Change Agents: Evaluating Agents with Braintrust There's been a lot of discourse around evals recently, so it's timely to drop the recording of Change Agents with Ankur Goyal on the topic of how he thinks about Evaluating Agents. Timestamps: (0:00) - Intro (1:22) - Building Braintrust (5:19) - Evaluating agentic systems (11:04) - Experimentation with evals (13:18) - Adopting evals (17:09) - Evals and complex agents (21:19) - Custom scoring (25:02) - LLM as a judge (26:44) - Human-manual evaluation (30:07) - How the best in-class teams manage evals (33:57) - Future of evals for agents (37:38 ) - Audience Q&A

Corinne Marie Riley

12,743 Aufrufe • vor 11 Monaten

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!

Announcing our Evals track speakers! Evals are the starting point of all production AI Engineering - when exciting demos turn into serious tools and agents. Our evals track is led by Ankur Goyal, who had one of the top Latent.Space episodes of the past talking about the evolution of evals at top AI Engineering companies like Stripe, Notion, and v0. we're also excited to welcome: - Omar Khattab of DSPy - Rafal Wilinski Vítor Balocco of Zapier - Ido Pesok of v0 - John P Dickerson CEO mozilla.ai - Jeff Huber CEO Chroma - jason liu for the 3rd year in a row! thanks to @braintrustdata's Ornella for working with us to put this incredibly stacked lineup of the best Evals practitioners in the world!

AI Engineer

55,348 Aufrufe • vor 1 Jahr

It is very easy to make mistakes when creating evals for your AI product. Shreya Shankar and I run through the most common mistakes in this talk (with memes 🌶️!) . Chapter summaries below: 00:51 Foundation model benchmarks are not the same as your application evals 03:00 Generic Evals Are Useless 04:00 Do not outsource labeling & prompting to non domain experts 09:28 You should make your own data annotation app 12:40 Your LLM prompts should be specific and grounded in error analysis 15:25 Use binary labels 18:57 Look at your data 23:41 Be careful of overfitting to test data 25:40 Do online tests Links more resources in the reply

It is very easy to make mistakes when creating evals for your AI product. Shreya Shankar and I run through the most common mistakes in this talk (with memes 🌶️!) . Chapter summaries below: 00:51 Foundation model benchmarks are not the same as your application evals 03:00 Generic Evals Are Useless 04:00 Do not outsource labeling & prompting to non domain experts 09:28 You should make your own data annotation app 12:40 Your LLM prompts should be specific and grounded in error analysis 15:25 Use binary labels 18:57 Look at your data 23:41 Be careful of overfitting to test data 25:40 Do online tests Links more resources in the reply

Hamel Husain

46,137 Aufrufe • vor 1 Jahr

What does a Vietnamese crypto project know about DeFi that most Western teams don't? 👀 New interview with the Head of Ecosystem at COIN98 just dropped 🔥 We talked about the project, how the ecosystem actually works from the inside, its position in DeFi, and what opportunities traders should be paying attention to this cycle. Timestamps: 00:00:00 - Teaser 00:00:34 - Introduction 00:00:48 - COIN98 overview 00:02:48 - COIN98 ecosystem: structure & opportunities 00:05:03 - COIN98 in the DeFi market 00:06:13 - Tools COIN98 provides for traders 00:07:22 - How to make money with the COIN98 wallet 00:10:44 - COIN98 expansion in the CIS 00:11:47 - Three associations with COIN98 00:13:01 - COIN98 targets in the current bull cycle Watch the full interview below. Curious to hear your take - what do you think about COIN98 this cycle? 🤝

What does a Vietnamese crypto project know about DeFi that most Western teams don't? 👀 New interview with the Head of Ecosystem at COIN98 just dropped 🔥 We talked about the project, how the ecosystem actually works from the inside, its position in DeFi, and what opportunities traders should be paying attention to this cycle. Timestamps: 00:00:00 - Teaser 00:00:34 - Introduction 00:00:48 - COIN98 overview 00:02:48 - COIN98 ecosystem: structure & opportunities 00:05:03 - COIN98 in the DeFi market 00:06:13 - Tools COIN98 provides for traders 00:07:22 - How to make money with the COIN98 wallet 00:10:44 - COIN98 expansion in the CIS 00:11:47 - Three associations with COIN98 00:13:01 - COIN98 targets in the current bull cycle Watch the full interview below. Curious to hear your take - what do you think about COIN98 this cycle? 🤝

Ruslan Khairullin

16,806 Aufrufe • vor 5 Monaten

EPISODE 147: The New Era of Software Abundance Joe Lonsdale visits Scott Wu & Russell Kaplan at Cognition HQ 00:00 Episode intro 01:35 Why technical talent & execution matters in AI 06:10 Do young people have an edge in the AI era? 08:26 Cognition's rapid growth 11:55 The new era of software abundance 14:30 Cognition engineers don't type code anymore 19:20 "Never sleep while Devin is idling" 21:25 The case for AI disinflation 23:50 How Devin generates 12X productivity gains 28:25 Cognition for government / taking on complex, broken systems 36:40 The AI race / competition with Anthropic 39:00 Forward deployed engineers? 43:40 How fast are LLMs improving? 47:10 The AI-led small business explosion

EPISODE 147: The New Era of Software Abundance Joe Lonsdale visits Scott Wu & Russell Kaplan at Cognition HQ 00:00 Episode intro 01:35 Why technical talent & execution matters in AI 06:10 Do young people have an edge in the AI era? 08:26 Cognition's rapid growth 11:55 The new era of software abundance 14:30 Cognition engineers don't type code anymore 19:20 "Never sleep while Devin is idling" 21:25 The case for AI disinflation 23:50 How Devin generates 12X productivity gains 28:25 Cognition for government / taking on complex, broken systems 36:40 The AI race / competition with Anthropic 39:00 Forward deployed engineers? 43:40 How fast are LLMs improving? 47:10 The AI-led small business explosion

American Optimist

834,355 Aufrufe • vor 4 Monaten

Dianne Penn is Head of Product for Research and Labs at Anthropic. She joined in 2023 as their first technical PM (when there were just five product engineers) and has helped ship every Claude model from Claude 2 to Mythos. We discuss: 02:31 Early Anthropic 08:55 The two biggest inflections 13:50 Inside the exponential 20:02 Token maxing 23:30 Anthropic Labs and the incubation model 27:30 How the AI research role works 31:35 How to become a top AI researcher 35:18 Frontier model safeguards 39:38 Hiring in the AI era 44:16 Building evals 47:48 Evals are the new PRDs 49:55 The importance of hands-on leadership 52:46 Finding joy in AI 58:10 How Dianne uses Claude 01:01:05 Avoiding overreliance on AI 01:03:50 The constitution that makes Claude better 01:07:11 AI writing and verification 01:11:40 Where human brains will continue to be valuable 01:14:10 Navigating AI with kids 01:16:26 Alignment, the future of the PM role, and burnout 01:21:54 Lightning round and final thoughts

Dianne Penn is Head of Product for Research and Labs at Anthropic. She joined in 2023 as their first technical PM (when there were just five product engineers) and has helped ship every Claude model from Claude 2 to Mythos. We discuss: 02:31 Early Anthropic 08:55 The two biggest inflections 13:50 Inside the exponential 20:02 Token maxing 23:30 Anthropic Labs and the incubation model 27:30 How the AI research role works 31:35 How to become a top AI researcher 35:18 Frontier model safeguards 39:38 Hiring in the AI era 44:16 Building evals 47:48 Evals are the new PRDs 49:55 The importance of hands-on leadership 52:46 Finding joy in AI 58:10 How Dianne uses Claude 01:01:05 Avoiding overreliance on AI 01:03:50 The constitution that makes Claude better 01:07:11 AI writing and verification 01:11:40 Where human brains will continue to be valuable 01:14:10 Navigating AI with kids 01:16:26 Alignment, the future of the PM role, and burnout 01:21:54 Lightning round and final thoughts

Lenny Rachitsky

155,794 Aufrufe • vor 23 Tagen

We don't know what most microbial genes do. Can genomic language models help? there's only one way to find out! this is a 1 hour and 42 minute interview with an MIT professor (the famous Yunha Hwang) chatting about these questions, her work in solving them at Tatta Bio, and more. zoomer captions are back too Links in reply! Timestamps: 00:00:00 - Clips + sponsor roll from the wonderful LatchBio 00:02:07 – Introduction 00:02:23 – Why do microbial genomes matter 00:04:07 – Deep learning acceptance in metagenomics 00:05:25 – The case for genomic “context” over sequence matching 00:06:43 – OMG: the only ML-ready metagenomic dataset 00:09:27 – gLM2: A multimodal genomic language model 00:11:06 – What do you do with the output of genomic language models? 00:17:41 – How will OMG evolve? 00:20:26 – Why train on only microbial genomes, as opposed to all genomes? 00:22:58 – Do we need more sequences or more annotations? 00:23:54 – Is there a conserved microbial genome ‘language’? 00:28:11 – What non-obvious things can this genomic language model tell you? 00:33:08 – Semantic deduplication and evaluation 00:37:33 – How does benchmarking work for these types of models? 00:41:31 – Gaia: A genomic search engine 00:44:18 – Even ‘well-studied’ genomes are mostly unannotated 00:50:51 – Using agents on Gaia 00:54:53 – Will genomic language models reshape the tree of life? 00:59:18 – Current limitations of genomic language models 01:08:54 – Directed evolution as training data 01:12:35 – What is Tatta Bio? 01:19:02 – Building Google for genomic sequences (SeqHub) 01:25:46 – How to create communities around scientific OSS 01:29:06 – What’s the purpose in the centralization of the software? 01:35:37 – How will the way science is done change in 10 years?

We don't know what most microbial genes do. Can genomic language models help? there's only one way to find out! this is a 1 hour and 42 minute interview with an MIT professor (the famous Yunha Hwang) chatting about these questions, her work in solving them at Tatta Bio, and more. zoomer captions are back too Links in reply! Timestamps: 00:00:00 - Clips + sponsor roll from the wonderful LatchBio 00:02:07 – Introduction 00:02:23 – Why do microbial genomes matter 00:04:07 – Deep learning acceptance in metagenomics 00:05:25 – The case for genomic “context” over sequence matching 00:06:43 – OMG: the only ML-ready metagenomic dataset 00:09:27 – gLM2: A multimodal genomic language model 00:11:06 – What do you do with the output of genomic language models? 00:17:41 – How will OMG evolve? 00:20:26 – Why train on only microbial genomes, as opposed to all genomes? 00:22:58 – Do we need more sequences or more annotations? 00:23:54 – Is there a conserved microbial genome ‘language’? 00:28:11 – What non-obvious things can this genomic language model tell you? 00:33:08 – Semantic deduplication and evaluation 00:37:33 – How does benchmarking work for these types of models? 00:41:31 – Gaia: A genomic search engine 00:44:18 – Even ‘well-studied’ genomes are mostly unannotated 00:50:51 – Using agents on Gaia 00:54:53 – Will genomic language models reshape the tree of life? 00:59:18 – Current limitations of genomic language models 01:08:54 – Directed evolution as training data 01:12:35 – What is Tatta Bio? 01:19:02 – Building Google for genomic sequences (SeqHub) 01:25:46 – How to create communities around scientific OSS 01:29:06 – What’s the purpose in the centralization of the software? 01:35:37 – How will the way science is done change in 10 years?

owl

44,279 Aufrufe • vor 8 Monaten

This guy literally broke down step-by-step how to create frontier lab quality evals: 2:42 - What an eval actually is 3:29 - Offline evals vs prod 5:23 - Why old evals stopped working 7:10 - How to write one 8:29 - Why 100% means you failed 9:03 - Models have saturated Q&A 12:37 - Anthropic dropped every benchmark 13:50 - Q&A thinking vs task thinking 15:32 - Building a real eval live 18:54 - Start with the easiest case 26:39 - Haiku vs Codex, head to head 33:08 - The eval no model solves 40:41 - Google's sampling trick 44:10 - How to find the ceiling 47:08 - It's all subject matter expertise 50:16 - "Google was a dead company" 51:33 - Why the Llama team is gone

This guy literally broke down step-by-step how to create frontier lab quality evals: 2:42 - What an eval actually is 3:29 - Offline evals vs prod 5:23 - Why old evals stopped working 7:10 - How to write one 8:29 - Why 100% means you failed 9:03 - Models have saturated Q&A 12:37 - Anthropic dropped every benchmark 13:50 - Q&A thinking vs task thinking 15:32 - Building a real eval live 18:54 - Start with the easiest case 26:39 - Haiku vs Codex, head to head 33:08 - The eval no model solves 40:41 - Google's sampling trick 44:10 - How to find the ceiling 47:08 - It's all subject matter expertise 50:16 - "Google was a dead company" 51:33 - Why the Llama team is gone

Aakash Gupta

44,671 Aufrufe • vor 22 Tagen

"AI evals force you into the shoes of your user." I love this clip from (Director at Arize) on why evaluations are the top AI skill to build in 2025. In my new interview, Aman breaks down: - How to write evals like a pro - 5 AI product principles he swears by - How to start practicing evals now Some quotes from Aman: "All PMs will be AI PMs. You'll either build on top of LLM APIs, use models in your backend, or use AI tools to solve problems in your daily work." "The waterfall process of writing a PRD and handing it to designers and engineers is dead. Your dev cycle needs to be much shorter." "Imagine showing up to a meeting with a prototype instead of a PRD. That's the shift PMs will need to make to communicate better in this new AI world." 📌 Watch now:

"AI evals force you into the shoes of your user." I love this clip from (Director at Arize) on why evaluations are the top AI skill to build in 2025. In my new interview, Aman breaks down: - How to write evals like a pro - 5 AI product principles he swears by - How to start practicing evals now Some quotes from Aman: "All PMs will be AI PMs. You'll either build on top of LLM APIs, use models in your backend, or use AI tools to solve problems in your daily work." "The waterfall process of writing a PRD and handing it to designers and engineers is dead. Your dev cycle needs to be much shorter." "Imagine showing up to a meeting with a prototype instead of a PRD. That's the shift PMs will need to make to communicate better in this new AI world." 📌 Watch now:

Peter Yang

15,973 Aufrufe • vor 1 Jahr

one thing that has saved my projects more time than I can count is evals boy was I excited when florian, quite literally an expert in benchmarks, agreed to hop into a ~2h interview to do a walkthrough of what the eval landscape looks like in 2026 (and also answer my personal business questions on the subject) given that now running frontier model through benchmarks is a vector for hacking other systems in order to avoid doing work (looking at you sol), I think it's more important than ever to educate folks on the evals situation. had a lot of fun throughout this session and I hope that you learn a thing or two! enjoy! 🌹 table of content: 0:00:00: are AI Benchmark broken? 0:05:45: Florian Brand background 0:09:00: what motivates florian to work on evaluation? 0:13:33: what is the mirrorcode benchmark about? 0:18:20: cheating in agent benchmark is insaneeeee 0:24:08: LLM benchmarks in era of agents 0:26:30: what’s up with the pelican man 0:28:27: evals are about capabilities 0:31:46: components of running evals 0:35:30: the volume of things to audit is huge!!! 0:40:20: expert answers are wrong hahahaha 0:46:00: api providers aren’t the same 0:48:00: benchmark narrow capabilities (synthetically) 0:50:56: link between eval and environment 0:53:45: small validated benchmark or massive bench? 0:56:11: what is your flow to review a benchmark? 0:58:30: tracking work capabilities with evaluation 1:00:20: slide deck in industry is all vibecoded 1:03:30: harness impact in the evaluation 1:07:39: hardware/sandboxes impact evaluation too! 1:11:00: “is it going to get worse?” 1:12:40: all components influence the final score 1:13:50: training models on different harnesses? 1:17:20: is the model just the weights or it’s all of it? 1:19:30: how to craft benchmark that prevent to cheating and undereliciting models in 2026 1:23:19: ways agents cheat and steal 1:26:00: correct elicitation of capabilities is important 1:36:00: building evaluation on prime intellect 1:45:10: how do you design interactivity benchmarks? 1:48:40: do you think evals are well set to reflect real world performance? 1:52:50: what will the benchmarking landscape will look like in 1 year

Yacine Mahdid

12,671 Aufrufe • vor 20 Tagen

Everything Gets Rebuilt: my conversation with Harrison Chase, CEO of LangChain about agent harnesses, evals, runtimes, sandboxes, MCP and the future of the agent stack 00:00 Intro - meet Harrison Chase - at the Chase Center for the Daytona Compute conference 01:32 What changed in agents over the last year 03:57 Why coding agents are ahead 06:26 Do models commoditize the framework layer? 08:27 Harnesses, in plain English 10:11 Why system prompts matter so much 13:11 The upside — and downside — of subagents 15:31 Why a useful agent needs a filesystem 18:13 Additional core primitives of modern agents 19:12 Skills: the new primitive 20:19 What context compaction actually means 23:02 How memory works in agents 25:16 One mega-agent or many specialized agents? 27:46 The future of MCP 29:38 Why agents need sandboxes 32:35 How sandboxes help with security 33:32 How Harrison Chase started LangChain 37:24 LangChain vs LangGraph vs Deep Agents 40:17 Why observability matters more for agents 41:48 Evals, no-code, and continuous improvement 44:41 What LangChain is building next 45:29 Where the real moat in AI lives

Everything Gets Rebuilt: my conversation with Harrison Chase, CEO of LangChain about agent harnesses, evals, runtimes, sandboxes, MCP and the future of the agent stack 00:00 Intro - meet Harrison Chase - at the Chase Center for the Daytona Compute conference 01:32 What changed in agents over the last year 03:57 Why coding agents are ahead 06:26 Do models commoditize the framework layer? 08:27 Harnesses, in plain English 10:11 Why system prompts matter so much 13:11 The upside — and downside — of subagents 15:31 Why a useful agent needs a filesystem 18:13 Additional core primitives of modern agents 19:12 Skills: the new primitive 20:19 What context compaction actually means 23:02 How memory works in agents 25:16 One mega-agent or many specialized agents? 27:46 The future of MCP 29:38 Why agents need sandboxes 32:35 How sandboxes help with security 33:32 How Harrison Chase started LangChain 37:24 LangChain vs LangGraph vs Deep Agents 40:17 Why observability matters more for agents 41:48 Evals, no-code, and continuous improvement 44:41 What LangChain is building next 45:29 Where the real moat in AI lives

Matt Turck

35,253 Aufrufe • vor 5 Monaten

It's brain versus AI time. Here's our full episode with Sebastian Seung. He's started a new company to reverse engineer the brain and master the intelligence created by nature with the hopes of informing the intelligence we've created with machines. Sebastian has done pioneering AI and neuroscience work and is now blending the two to make a digital brain. Mostly, he's fun to listen to. The Core Memory podcast is made by possible by the intelligent finance powerhouse Brex and the people who back the biggest ideas @e1ventures. Timestamps 00:00:00 - Intro 00:01:42 - Who is Sebastian Seung? 00:03:23 - What is a Connectome? 00:07:08 - The Connectome Winter 00:09:08 - Diamond Knives & AI 00:12:10 - When Will We Map the Human Brain? 00:18:09 - The "Brain Crystal" & Visual Neurons 00:26:31 - New Startup 00:33:55 - Simulating a Mind in Software 00:37:40 - Biology vs. Nvidia 00:44:04 - Is AI a Religious Quest? 00:50:05 - Mind Uploading & Transhumanism 00:59:26 - Brain-Computer Interfaces 01:06:41 - From Physics to Neuroscience 01:12:22 - Will Uploads Be Conscious? 01:19:37 - Neuromorphic Chips & Future Robotics

It's brain versus AI time. Here's our full episode with Sebastian Seung. He's started a new company to reverse engineer the brain and master the intelligence created by nature with the hopes of informing the intelligence we've created with machines. Sebastian has done pioneering AI and neuroscience work and is now blending the two to make a digital brain. Mostly, he's fun to listen to. The Core Memory podcast is made by possible by the intelligent finance powerhouse Brex and the people who back the biggest ideas @e1ventures. Timestamps 00:00:00 - Intro 00:01:42 - Who is Sebastian Seung? 00:03:23 - What is a Connectome? 00:07:08 - The Connectome Winter 00:09:08 - Diamond Knives & AI 00:12:10 - When Will We Map the Human Brain? 00:18:09 - The "Brain Crystal" & Visual Neurons 00:26:31 - New Startup 00:33:55 - Simulating a Mind in Software 00:37:40 - Biology vs. Nvidia 00:44:04 - Is AI a Religious Quest? 00:50:05 - Mind Uploading & Transhumanism 00:59:26 - Brain-Computer Interfaces 01:06:41 - From Physics to Neuroscience 01:12:22 - Will Uploads Be Conscious? 01:19:37 - Neuromorphic Chips & Future Robotics

Ashlee Vance

38,981 Aufrufe • vor 8 Monaten

Don't waste 2 years learning to use LLMs like Claude & ChatGPT. Andrej Karpathy, the godfather of AI, dropped a 2 hour course on how he personally uses LLMs daily. • 00:00 - LLMs simplified • 22:49 - which LLM to actually use • 42:00 - one prompt for full research • 1:13:57 - how to code with LLMs • 1:37:04 - NotebookLM podcast generation This course will teach you more about using LLMs than most AI engineers learn in their entire career. Bookmark this & give 2 hours today, no matter what. Then read the article below.

Don't waste 2 years learning to use LLMs like Claude & ChatGPT. Andrej Karpathy, the godfather of AI, dropped a 2 hour course on how he personally uses LLMs daily. • 00:00 - LLMs simplified • 22:49 - which LLM to actually use • 42:00 - one prompt for full research • 1:13:57 - how to code with LLMs • 1:37:04 - NotebookLM podcast generation This course will teach you more about using LLMs than most AI engineers learn in their entire career. Bookmark this & give 2 hours today, no matter what. Then read the article below.

Roan

55,756 Aufrufe • vor 12 Tagen

Michael Bolin (Michael Bolin) is the tech lead of the Codex open source repository at OpenAI and formerly a distinguished eng (E9) at Meta. I asked him for all the details on his career story and how he uses Codex for max benefit. Timestamps: 00:00:00 - Intro 00:00:56 - Chickenfoot 00:02:45 - Working at Google 00:06:34 - Overhauling Facebook's build system 00:16:36 - Rewriting Facebook's IDE 00:26:01 - Struggles after Principal Eng (E8) promo 00:28:39 - Building a virtual filesystem for Facebook 00:35:47 - Delayed Distinguished promo (E9) and learnings 00:39:56 - Joining OpenAI 00:43:05 - Research-led vs engineering-led cultures 00:44:53 - The story behind Codex 00:51:00 - How he uses Codex 00:57:00 - Why Codex's harness is open source 00:59:50 - Top technical book recommendations 01:05:02 - Why deep technical skills are still valuable (for now) 01:11:07 - How to start projects well 01:14:27 - Advice on writing better and career planning 01:17:06 - Advice for his younger self 01:19:10 - Outro He was a dream guest of mine and I'm excited to share his story with you all! Other places to watch: • YouTube: • Spotify: • Apple Podcasts: • Transcript:

Michael Bolin (Michael Bolin) is the tech lead of the Codex open source repository at OpenAI and formerly a distinguished eng (E9) at Meta. I asked him for all the details on his career story and how he uses Codex for max benefit. Timestamps: 00:00:00 - Intro 00:00:56 - Chickenfoot 00:02:45 - Working at Google 00:06:34 - Overhauling Facebook's build system 00:16:36 - Rewriting Facebook's IDE 00:26:01 - Struggles after Principal Eng (E8) promo 00:28:39 - Building a virtual filesystem for Facebook 00:35:47 - Delayed Distinguished promo (E9) and learnings 00:39:56 - Joining OpenAI 00:43:05 - Research-led vs engineering-led cultures 00:44:53 - The story behind Codex 00:51:00 - How he uses Codex 00:57:00 - Why Codex's harness is open source 00:59:50 - Top technical book recommendations 01:05:02 - Why deep technical skills are still valuable (for now) 01:11:07 - How to start projects well 01:14:27 - Advice on writing better and career planning 01:17:06 - Advice for his younger self 01:19:10 - Outro He was a dream guest of mine and I'm excited to share his story with you all! Other places to watch: • YouTube: • Spotify: • Apple Podcasts: • Transcript:

Ryan Peterman

119,601 Aufrufe • vor 5 Monaten