正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Are AI benchmarks doomed? Greg Burnham and Tom Adamczewski join Anson Ho to push back on benchmark pessimism and dig into what the next generation of AI benchmarks could look like. (0:00:00) - Preview (0:00:36) - Intro: Are AI benchmarks doomed? (0:03:13) - The costs and benefits of benchmark... show more

Epoch AI

45,633 subscribers

22,037 次观看 • 4 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

one thing that has saved my projects more time than I can count is evals boy was I excited when florian, quite literally an expert in benchmarks, agreed to hop into a ~2h interview to do a walkthrough of what the eval landscape looks like in 2026 (and also answer my personal business questions on the subject) given that now running frontier model through benchmarks is a vector for hacking other systems in order to avoid doing work (looking at you sol), I think it's more important than ever to educate folks on the evals situation. had a lot of fun throughout this session and I hope that you learn a thing or two! enjoy! 🌹 table of content: 0:00:00: are AI Benchmark broken? 0:05:45: Florian Brand background 0:09:00: what motivates florian to work on evaluation? 0:13:33: what is the mirrorcode benchmark about? 0:18:20: cheating in agent benchmark is insaneeeee 0:24:08: LLM benchmarks in era of agents 0:26:30: what’s up with the pelican man 0:28:27: evals are about capabilities 0:31:46: components of running evals 0:35:30: the volume of things to audit is huge!!! 0:40:20: expert answers are wrong hahahaha 0:46:00: api providers aren’t the same 0:48:00: benchmark narrow capabilities (synthetically) 0:50:56: link between eval and environment 0:53:45: small validated benchmark or massive bench? 0:56:11: what is your flow to review a benchmark? 0:58:30: tracking work capabilities with evaluation 1:00:20: slide deck in industry is all vibecoded 1:03:30: harness impact in the evaluation 1:07:39: hardware/sandboxes impact evaluation too! 1:11:00: “is it going to get worse?” 1:12:40: all components influence the final score 1:13:50: training models on different harnesses? 1:17:20: is the model just the weights or it’s all of it? 1:19:30: how to craft benchmark that prevent to cheating and undereliciting models in 2026 1:23:19: ways agents cheat and steal 1:26:00: correct elicitation of capabilities is important 1:36:00: building evaluation on prime intellect 1:45:10: how do you design interactivity benchmarks? 1:48:40: do you think evals are well set to reflect real world performance? 1:52:50: what will the benchmarking landscape will look like in 1 year

Yacine Mahdid

12,923 次观看 • 1 个月前

We are excited to share that Logan Kilpatrick joined us on The Bench to discuss Google's new Gemini 3.5 Flash: why it's deliberately more persistent and capable than previous Flash models, how it hit #1 on our FinanceAgent Benchmark taking 82 steps where competitors stopped at 13, and what justifies the price increase. We also get into why AI benchmarks need a paradigm shift, the trade-off of building everything vs staying focused, the Pope, and why Omni might kill the Subway Surfers content era. 0:11:00 – Flash is being rebased for the agent era, not just a cheaper model anymore 0:14:03 – Persistence by design: 82 tool calls vs competitors' 13 0:17:52 – Why pricing went up and how Google thinks about value per token 0:22:55 – Coding performance: from 20th to 10th place in one generation 0:28:28 – Why benchmarks have historically been misleading and what the new era of evaluation looks like 0:29:28 Logan on why Google has the best researchers in the world 0:36:16 – The cost of being Google 0:39:07 – The Pope’s encyclical on AI and whether most people see frontier intelligence as a good thing 0:51:12 – Why Omni is the thing that recently clicked

We are excited to share that Logan Kilpatrick joined us on The Bench to discuss Google's new Gemini 3.5 Flash: why it's deliberately more persistent and capable than previous Flash models, how it hit #1 on our FinanceAgent Benchmark taking 82 steps where competitors stopped at 13, and what justifies the price increase. We also get into why AI benchmarks need a paradigm shift, the trade-off of building everything vs staying focused, the Pope, and why Omni might kill the Subway Surfers content era. 0:11:00 – Flash is being rebased for the agent era, not just a cheaper model anymore 0:14:03 – Persistence by design: 82 tool calls vs competitors' 13 0:17:52 – Why pricing went up and how Google thinks about value per token 0:22:55 – Coding performance: from 20th to 10th place in one generation 0:28:28 – Why benchmarks have historically been misleading and what the new era of evaluation looks like 0:29:28 Logan on why Google has the best researchers in the world 0:36:16 – The cost of being Google 0:39:07 – The Pope’s encyclical on AI and whether most people see frontier intelligence as a good thing 0:51:12 – Why Omni is the thing that recently clicked

Vals AI

16,486 次观看 • 3 个月前

The Delta of Intelligence is Human Data! Curtis G. Northcutt is Director of AI Research at Handshake AI. In this episode, Curtis shares his unique perspectives on human data, RL envs, four quadrants of the AI market, journey of Handshake and much more. 0:00:00 - INTRO 0:02:08 - The Snowman Effect: From Rural Kentucky to 7 Years at MIT 0:09:50 - Confident Learning, Two Systems of Intelligence, Classical ML Has to Prevail 0:20:37 - Origin of CleanLab and Handshake acquisition 0:28:00 - How does GPT-5 become GPT-6? 0:31:20 - The Fastest growing Data Lab: What makes Handshake different? 0:35:32 - Difference between Good Data and Bad Data 0:38:17 - New kind of Data, IA and AI 0:42:04 - Scaling Coding Benchmarks and Efficiency in Long Horizon Tasks 0:49:12 - On Pre-Training, Misconception around Human Data 0:57:13 - Open Question: Taste and Personality, Quick Fire 01:05:15 - Advice to 20yo : Skill and Obsession

The Delta of Intelligence is Human Data! Curtis G. Northcutt is Director of AI Research at Handshake AI. In this episode, Curtis shares his unique perspectives on human data, RL envs, four quadrants of the AI market, journey of Handshake and much more. 0:00:00 - INTRO 0:02:08 - The Snowman Effect: From Rural Kentucky to 7 Years at MIT 0:09:50 - Confident Learning, Two Systems of Intelligence, Classical ML Has to Prevail 0:20:37 - Origin of CleanLab and Handshake acquisition 0:28:00 - How does GPT-5 become GPT-6? 0:31:20 - The Fastest growing Data Lab: What makes Handshake different? 0:35:32 - Difference between Good Data and Bad Data 0:38:17 - New kind of Data, IA and AI 0:42:04 - Scaling Coding Benchmarks and Efficiency in Long Horizon Tasks 0:49:12 - On Pre-Training, Misconception around Human Data 0:57:13 - Open Question: Taste and Personality, Quick Fire 01:05:15 - Advice to 20yo : Skill and Obsession

himanshu

22,264 次观看 • 3 个月前

How persuadable are humans, really? Sam Altman returns to discuss accidental AI takeover, why GPT-6 might crack real science, when we'll see AI CEOs running the show, and the question he asked someone to pose to the Dalai Lama. Timestamps 0:00:00 – Sam’s basic trick for productivity 0:01:22 – Hiring hardware vs. software people 0:05:46 – How long before an AI CEO 0:10:48 – Government backstops for AI companies 0:13:26 – Monetizing AI services 0:18:04 – Negotiating deals with Saudia Arabia and the UAE 0:22:00 – How good GPT-6 will be at poetry 0:25:15 – Chip-building and where the energy will come from 0:29:04 – Sam’s changing health habits 0:30:20 – UAP’s and conspiracy theories 0:31:43 – Revitalizing St. Louis 0:33:12 – Regulating AI agents 0:34:39 – New ways to interface with AI 0:38:12 – How normies will learn AI 0:44:36 – The trajectory of healthcare and housing prices 0:46:57 – Reexamining freedom of speech 0:50:08 – Humanity’s persuadability Recorded live with tylercowen and Sam Altman at the 2025 Progress Conference, hosted by Roots of Progress Institute. Thanks to Big Think for the video production.

How persuadable are humans, really? Sam Altman returns to discuss accidental AI takeover, why GPT-6 might crack real science, when we'll see AI CEOs running the show, and the question he asked someone to pose to the Dalai Lama. Timestamps 0:00:00 – Sam’s basic trick for productivity 0:01:22 – Hiring hardware vs. software people 0:05:46 – How long before an AI CEO 0:10:48 – Government backstops for AI companies 0:13:26 – Monetizing AI services 0:18:04 – Negotiating deals with Saudia Arabia and the UAE 0:22:00 – How good GPT-6 will be at poetry 0:25:15 – Chip-building and where the energy will come from 0:29:04 – Sam’s changing health habits 0:30:20 – UAP’s and conspiracy theories 0:31:43 – Revitalizing St. Louis 0:33:12 – Regulating AI agents 0:34:39 – New ways to interface with AI 0:38:12 – How normies will learn AI 0:44:36 – The trajectory of healthcare and housing prices 0:46:57 – Reexamining freedom of speech 0:50:08 – Humanity’s persuadability Recorded live with tylercowen and Sam Altman at the 2025 Progress Conference, hosted by Roots of Progress Institute. Thanks to Big Think for the video production.

Conversations with Tyler

74,177 次观看 • 10 个月前

How does math research change when the cost of trying your first dumb idea goes to zero? Daniel Litt joins Greg Burnham and Anson Ho to discuss what today’s models can and can’t do in math, and how far they are from doing high-quality research. 0:00:00 What's the hardest math problem AI can solve today? 00:16:08 How helpful are today’s AI models for math research? 00:23:36 Junk papers, LLM-generated proofs, and the refereeing crisis 00:27:21 AI enables searching through problems at scale 00:33:49 When will AI be good enough to publish in top math journals? 00:42:15 What are the returns to intelligence? 00:59:50 Will AI solve Millennium problems? 01:11:54 Is math full of low-hanging fruit? 01:18:47 How Daniel has adapted his professional life to AI progress 01:25:28 What do AI math benchmarks actually measure? 01:33:05 Designing the Open Problems benchmark 01:56:35 Do mathematicians believe heuristic arguments about conjectures? 02:01:24 What if FrontierMath: Open Problems gets solved? 02:06:53 Is AI on the cusp of accelerating math progress?

How does math research change when the cost of trying your first dumb idea goes to zero? Daniel Litt joins Greg Burnham and Anson Ho to discuss what today’s models can and can’t do in math, and how far they are from doing high-quality research. 0:00:00 What's the hardest math problem AI can solve today? 00:16:08 How helpful are today’s AI models for math research? 00:23:36 Junk papers, LLM-generated proofs, and the refereeing crisis 00:27:21 AI enables searching through problems at scale 00:33:49 When will AI be good enough to publish in top math journals? 00:42:15 What are the returns to intelligence? 00:59:50 Will AI solve Millennium problems? 01:11:54 Is math full of low-hanging fruit? 01:18:47 How Daniel has adapted his professional life to AI progress 01:25:28 What do AI math benchmarks actually measure? 01:33:05 Designing the Open Problems benchmark 01:56:35 Do mathematicians believe heuristic arguments about conjectures? 02:01:24 What if FrontierMath: Open Problems gets solved? 02:06:53 Is AI on the cusp of accelerating math progress?

Epoch AI

178,703 次观看 • 7 个月前

The vik episode on Ground Zero! [youtube link in replies] 0:00:00 - Vik's story and Moondream 0:18:50 - The core thesis of small and scale 0:28:35 - The data problem 0:33:35 - Deciding the training architecture 0:43:20 - Post-Training and RL perf 0:46:06 - Post-Training recipes of Moondream 3 0:47:40 - Open Source and VLM development priorities 0:52:05 - AI War : America and China 0:55:07 - Moondream acquisition and Future 1:04:08 - Community Questions 1:16:40 - Trivia, Lores and Opinions on state of AI 1:28:07 - Advice to 20yo

The vik episode on Ground Zero! [youtube link in replies] 0:00:00 - Vik's story and Moondream 0:18:50 - The core thesis of small and scale 0:28:35 - The data problem 0:33:35 - Deciding the training architecture 0:43:20 - Post-Training and RL perf 0:46:06 - Post-Training recipes of Moondream 3 0:47:40 - Open Source and VLM development priorities 0:52:05 - AI War : America and China 0:55:07 - Moondream acquisition and Future 1:04:08 - Community Questions 1:16:40 - Trivia, Lores and Opinions on state of AI 1:28:07 - Advice to 20yo

himanshu

43,246 次观看 • 8 个月前

What are current economic models missing about AGI? How would we know if we were approaching explosive growth? Stanford economist Phil Trammell has been rigorously thinking about the intersection of economic theory and AI (incl. AGI) for over five years, long before the recent surge of interest in large language models. In this episode of Epoch After Hours, Philip Trammell and Epoch AI researcher Anson Ho discuss what economic theory really has to say about the development and impacts of AGI: what current economic models get wrong, the odds of explosive economic growth, what “real GDP” actually measures, and much more! -- Timestamps -- 0:00:00 - Problems with existing work on the economics of AI 0:10:18 - Declining returns to R&D 0:18:28 - What real GDP misses 0:26:57 - Task-based models & AI automation 0:49:32 - The limits of economic theory 1:09:11 - How to detect an economic singularity 1:23:32 - Increasing returns to scale

What are current economic models missing about AGI? How would we know if we were approaching explosive growth? Stanford economist Phil Trammell has been rigorously thinking about the intersection of economic theory and AI (incl. AGI) for over five years, long before the recent surge of interest in large language models. In this episode of Epoch After Hours, Philip Trammell and Epoch AI researcher Anson Ho discuss what economic theory really has to say about the development and impacts of AGI: what current economic models get wrong, the odds of explosive economic growth, what “real GDP” actually measures, and much more! -- Timestamps -- 0:00:00 - Problems with existing work on the economics of AI 0:10:18 - Declining returns to R&D 0:18:28 - What real GDP misses 0:26:57 - Task-based models & AI automation 0:49:32 - The limits of economic theory 1:09:11 - How to detect an economic singularity 1:23:32 - Increasing returns to scale

Epoch AI

103,973 次观看 • 11 个月前

My interview with Amazon Web Services CEO Matt Garman. 0:00 Intro 0:57 White Collar Jobs 8:51 How much of AWS's code is written by AI? 12:54 How to have a career in the AI era 15:43 AI Bottlenecks 18:06 Inference vs Training Growth 20:05 AWS Custom Silicon 25:50 Annapurna Acquisition 27:53 AI Models 33:35 Open vs Closed Models 41:28 Benchmarks 47:13 Agents

My interview with Amazon Web Services CEO Matt Garman. 0:00 Intro 0:57 White Collar Jobs 8:51 How much of AWS's code is written by AI? 12:54 How to have a career in the AI era 15:43 AI Bottlenecks 18:06 Inference vs Training Growth 20:05 AWS Custom Silicon 25:50 Annapurna Acquisition 27:53 AI Models 33:35 Open vs Closed Models 41:28 Benchmarks 47:13 Agents

Matthew Berman

49,150 次观看 • 1 年前

François Chollet (François Chollet) on the ARC Prize and how we get to AGI. At AI Startup School in San Francisco. 00:00 - The Falling Cost of Compute 00:57 - Deep-Learning’s Scaling Era & Benchmarks 01:59 - The ARC Benchmark 03:02 - The 2024 Shift to Test-Time Adaptation 05:01 - What Is Intelligence? 07:12 - Why Benchmarks Matter (and Mislead) 08:57 - ARC 1 Exposes Scaling Limits 10:58 - ARC 2: Compositional Reasoning Arrives 12:55 - Humans vs. Models on ARC2 14:58 - Previewing ARC3 & Interactive Agency 17:00 - Kaleidoscopic Hypothesis and Abstractions 22:00 - Type 1 vs. Type 2 Abstractions 26:00 - Discrete Program Search & Inventive AI 29:00 - Fusing Intuition with Symbolic Reasoning 32:00 - Building AGI Through Meta-Learning Systems

François Chollet (François Chollet) on the ARC Prize and how we get to AGI. At AI Startup School in San Francisco. 00:00 - The Falling Cost of Compute 00:57 - Deep-Learning’s Scaling Era & Benchmarks 01:59 - The ARC Benchmark 03:02 - The 2024 Shift to Test-Time Adaptation 05:01 - What Is Intelligence? 07:12 - Why Benchmarks Matter (and Mislead) 08:57 - ARC 1 Exposes Scaling Limits 10:58 - ARC 2: Compositional Reasoning Arrives 12:55 - Humans vs. Models on ARC2 14:58 - Previewing ARC3 & Interactive Agency 17:00 - Kaleidoscopic Hypothesis and Abstractions 22:00 - Type 1 vs. Type 2 Abstractions 26:00 - Discrete Program Search & Inventive AI 29:00 - Fusing Intuition with Symbolic Reasoning 32:00 - Building AGI Through Meta-Learning Systems

Y Combinator

231,796 次观看 • 1 年前

Will AI drive Europe towards a “high interest rates, no growth” future? How will automation impact entry-level workers and the structure of firms? What role does Europe play as the US and China continue to push AI progress? On this episode of Epoch After Hours, Luis Garicano 🇪🇺🇺🇦 joins Epoch AI researchers Anson Ho and Andrei Potlogea to discuss these questions. -- Timestamps -- 0:00:00 – Will AI trigger explosive growth? 0:06:26 – Short-run macroeconomic effects 0:11:29 – The decline of junior jobs 0:20:21 – The missing training ladder 0:39:31 – Europe’s AI regulation problem 0:52:46 – Who captures AI value? 01:08:17 – AI, interest rates & fiscal future

Will AI drive Europe towards a “high interest rates, no growth” future? How will automation impact entry-level workers and the structure of firms? What role does Europe play as the US and China continue to push AI progress? On this episode of Epoch After Hours, Luis Garicano 🇪🇺🇺🇦 joins Epoch AI researchers Anson Ho and Andrei Potlogea to discuss these questions. -- Timestamps -- 0:00:00 – Will AI trigger explosive growth? 0:06:26 – Short-run macroeconomic effects 0:11:29 – The decline of junior jobs 0:20:21 – The missing training ladder 0:39:31 – Europe’s AI regulation problem 0:52:46 – Who captures AI value? 01:08:17 – AI, interest rates & fiscal future

Epoch AI

33,102 次观看 • 8 个月前

NEW EPISODE: “It’s not great that students use AI to cheat, but it’s a wake-up call to higher education: if AI can cheat so effectively, you’re not teaching a useful skill. We’ve done very little in response other than complain. We need to get our act together.”tylercowen 0:00:00 Tyler Cowen 0:00:24 AI, Cybersecurity & the Pace of Change 0:05:57 How Tyler Sees AI 0:10:03 School, Social Media & Sacred Cows 0:15:03 AI in Education 0:25:13 Agentic AI, Open Source & Startups 0:33:34 Finding Talent in an AI World 0:52:47 AI, Intuition & Machine Intelligence 1:01:46 AI Doomerism, Safety & Censorship 1:28:06 Data Centers, Crypto & the EU 2:01:24 Jobs, Aesthetics, Hollywood & What Makes Nations Happy

NEW EPISODE: “It’s not great that students use AI to cheat, but it’s a wake-up call to higher education: if AI can cheat so effectively, you’re not teaching a useful skill. We’ve done very little in response other than complain. We need to get our act together.”tylercowen 0:00:00 Tyler Cowen 0:00:24 AI, Cybersecurity & the Pace of Change 0:05:57 How Tyler Sees AI 0:10:03 School, Social Media & Sacred Cows 0:15:03 AI in Education 0:25:13 Agentic AI, Open Source & Startups 0:33:34 Finding Talent in an AI World 0:52:47 AI, Intuition & Machine Intelligence 1:01:46 AI Doomerism, Safety & Censorship 1:28:06 Data Centers, Crypto & the EU 2:01:24 Jobs, Aesthetics, Hollywood & What Makes Nations Happy

Tetragrammaton

71,985 次观看 • 16 天前

.Satya Nadella on: - why he doesn’t believe in AGI but does believe in 10% economic growth - Microsoft’s new topological qubit breakthrough and gaming world models - whether Office commoditizes LLMs or the other way around Links below. Enjoy! Timestamps 0:00:00 - Intro 0:05:48 - AI won't be winner-take-all 0:16:02 - World economy growing by 10% 0:22:23 - Decreasing price of intelligence 0:31:03 - Microsoft's Quantum breakthrough 0:43:35 - Microsoft's gaming world model 0:50:35 - Legal barriers to AI 0:56:30 - Getting AGI safety right 1:05:43 - 34 years at Microsoft 1:11:31 - Does Satya Nadella believe in AGI?

.Satya Nadella on: - why he doesn’t believe in AGI but does believe in 10% economic growth - Microsoft’s new topological qubit breakthrough and gaming world models - whether Office commoditizes LLMs or the other way around Links below. Enjoy! Timestamps 0:00:00 - Intro 0:05:48 - AI won't be winner-take-all 0:16:02 - World economy growing by 10% 0:22:23 - Decreasing price of intelligence 0:31:03 - Microsoft's Quantum breakthrough 0:43:35 - Microsoft's gaming world model 0:50:35 - Legal barriers to AI 0:56:30 - Getting AGI safety right 1:05:43 - 34 years at Microsoft 1:11:31 - Does Satya Nadella believe in AGI?

Dwarkesh Patel

2,264,372 次观看 • 1 年前

.Jeremy Boreing on the Rise of Grift and Conspiracy Media 0:00 Jeremy’s 10-month Sabbath 0:01:15 How the Right learned to win online 0:02:40 Algorithms reward engagement over truth 0:04:42 Losing cultural literacy: no shared story, no shared reality 0:06:26 Conservatives and new tech: social media to AI panic 0:23:51 “Online Christianity” as tribal politics 0:26:00 Conspiracies that erase trust 0:36:43 Grift mechanics: influencers, fame, and “human algorithm” behavior 0:46:16 The post-Trump fight: power, populism, and revolutionary instincts 1:00:36 The antidote: agency, responsibility, family—and rebuilding meaning

.Jeremy Boreing on the Rise of Grift and Conspiracy Media 0:00 Jeremy’s 10-month Sabbath 0:01:15 How the Right learned to win online 0:02:40 Algorithms reward engagement over truth 0:04:42 Losing cultural literacy: no shared story, no shared reality 0:06:26 Conservatives and new tech: social media to AI panic 0:23:51 “Online Christianity” as tribal politics 0:26:00 Conspiracies that erase trust 0:36:43 Grift mechanics: influencers, fame, and “human algorithm” behavior 0:46:16 The post-Trump fight: power, populism, and revolutionary instincts 1:00:36 The antidote: agency, responsibility, family—and rebuilding meaning

Marissa Streit

61,203 次观看 • 6 个月前

Episode w Dylan Patel and Asianometry out! A bonanza on how the semiconductor industry actually works. & what Xi could do if he becomes scaling pilled, how we can train models with 10,000x GPT-4's compute by end of decade, & much more. This one is very fun. Enjoy! Links below. 0:00:00 – Xi’s path to AGI 0:05:05 – Liang Mong Song 0:09:10 – How semiconductors get better 0:12:01 – China can centralize compute 0:19:35 – Export controls & sanctions 0:33:45 – Huawei’s intense culture 0:39:45 – Why the semiconductor industry is so stratified 0:41:50 – N2 should not exist 0:46:47 – Taiwan invasion hypothetical 0:50:15 – Mind-boggling complexity of semiconductors 1:00:06 – Chip architecture design 1:05:29 – Architectures lead to different AI models? China vs. US 1:11:06 – Being head of compute at an AI lab 1:17:17 – Scaling costs and power demand 1:37:59 – Are we financing an AI bubble?

Episode w Dylan Patel and Asianometry out! A bonanza on how the semiconductor industry actually works. & what Xi could do if he becomes scaling pilled, how we can train models with 10,000x GPT-4's compute by end of decade, & much more. This one is very fun. Enjoy! Links below. 0:00:00 – Xi’s path to AGI 0:05:05 – Liang Mong Song 0:09:10 – How semiconductors get better 0:12:01 – China can centralize compute 0:19:35 – Export controls & sanctions 0:33:45 – Huawei’s intense culture 0:39:45 – Why the semiconductor industry is so stratified 0:41:50 – N2 should not exist 0:46:47 – Taiwan invasion hypothetical 0:50:15 – Mind-boggling complexity of semiconductors 1:00:06 – Chip architecture design 1:05:29 – Architectures lead to different AI models? China vs. US 1:11:06 – Being head of compute at an AI lab 1:17:17 – Scaling costs and power demand 1:37:59 – Are we financing an AI bubble?

Dwarkesh Patel

261,054 次观看 • 1 年前

.Satya Nadella gave me and Dylan Patel an exclusive tour of Fairwater 2, the most powerful AI datacenter in the world. We then chatted through Satya's vision for Microsoft in a world with AGI. 0:00:00 - Fairwater 2 0:04:15 - Business models for AGI 0:13:42 - Copilot 0:20:56 - Whose margins will expand most? 0:37:12 - MAI 0:48:42 - The hyperscale business 1:03:39 - In-house chip & OpenAI partnership 1:10:30 - The CAPEX explosion 1:16:01 - Will the world trust US companies to lead AI? Look up Dwarkesh Podcast on Youtube, Apple Podcasts or Spotify to tune in.

.Satya Nadella gave me and Dylan Patel an exclusive tour of Fairwater 2, the most powerful AI datacenter in the world. We then chatted through Satya's vision for Microsoft in a world with AGI. 0:00:00 - Fairwater 2 0:04:15 - Business models for AGI 0:13:42 - Copilot 0:20:56 - Whose margins will expand most? 0:37:12 - MAI 0:48:42 - The hyperscale business 1:03:39 - In-house chip & OpenAI partnership 1:10:30 - The CAPEX explosion 1:16:01 - Will the world trust US companies to lead AI? Look up Dwarkesh Podcast on Youtube, Apple Podcasts or Spotify to tune in.

Dwarkesh Patel

1,895,952 次观看 • 9 个月前

🌄 Beyond SWE: The Future of Long Horizon Environments A discussion with our founders about KellyBench, and the need for new environments that require agents to adapt over time and act under uncertainty. 0:00:17 What is KellyBench? 0:02:10 Openendedness, non-stationarity and continual learning 0:03:40 Analytical versus operational capabilities 0:04:13 Why are models bad at KellyBench? 0:05:39 Situational awareness in dynamic environments 0:06:37 Feature stability and real-world non-stationarity 0:07:07 The power of context 0:07:34 "The first principle is that you must not fool yourself" 0:08:20 Machiavelli, fortuna and the ability to adapt to change 0:09:23 How can models improve on evals like KellyBench? 0:10:12 Limitations of KellyBench: data availability and market odds timing 0:11:44 Implications beyond quant finance / sports betting 0:13:26 Civilisations as the ultimate time horizon 0:14:05 Would a mega prompt / better elicitation do much better on the benchmark? 0:14:52 What new types of capability is GR excited about? 0:17:48 Taste and the ability to pursue long-term goals even if they aren't immediately rewarding 0:18:56 Deep learning as an example of a method that took a long time to bear fruit 0:19:25 Optimism about the future of AI

🌄 Beyond SWE: The Future of Long Horizon Environments A discussion with our founders about KellyBench, and the need for new environments that require agents to adapt over time and act under uncertainty. 0:00:17 What is KellyBench? 0:02:10 Openendedness, non-stationarity and continual learning 0:03:40 Analytical versus operational capabilities 0:04:13 Why are models bad at KellyBench? 0:05:39 Situational awareness in dynamic environments 0:06:37 Feature stability and real-world non-stationarity 0:07:07 The power of context 0:07:34 "The first principle is that you must not fool yourself" 0:08:20 Machiavelli, fortuna and the ability to adapt to change 0:09:23 How can models improve on evals like KellyBench? 0:10:12 Limitations of KellyBench: data availability and market odds timing 0:11:44 Implications beyond quant finance / sports betting 0:13:26 Civilisations as the ultimate time horizon 0:14:05 Would a mega prompt / better elicitation do much better on the benchmark? 0:14:52 What new types of capability is GR excited about? 0:17:48 Taste and the ability to pursue long-term goals even if they aren't immediately rewarding 0:18:56 Deep learning as an example of a method that took a long time to bear fruit 0:19:25 Optimism about the future of AI

General Reasoning

15,585 次观看 • 4 个月前

Australia's Productivity Commission Chair Danielle Wood on: - why AI probably won't cause a jobpocalypse, - how Australia can get a piece of the AI action, - why profits may not flow to the model layer of the stack, - and why Australia shouldn't follow the EU in legislating an AI Act -- even if AI turns out to be an "abnormal" technology. Watch below, or wherever you get your podcasts. Enjoy! Timestamps: (0:00:00) – Introduction. (0:01:43) – Is AI as big as the Industrial Revolution? (0:06:35) – The labour market implications of AI (0:21:18) – Industry policy for AI: when to intervene, when to step back (0:31:27) – Where will the profits flow? Chips, models, or applications? (0:52:37) – Data centres: should Australia support them? (1:15:07) – AI risks, and why Australia shouldn't pass an AI Act (1:23:27) – Measuring progress in the age of AI: beyond GDP and labour productivity

Australia's Productivity Commission Chair Danielle Wood on: - why AI probably won't cause a jobpocalypse, - how Australia can get a piece of the AI action, - why profits may not flow to the model layer of the stack, - and why Australia shouldn't follow the EU in legislating an AI Act -- even if AI turns out to be an "abnormal" technology. Watch below, or wherever you get your podcasts. Enjoy! Timestamps: (0:00:00) – Introduction. (0:01:43) – Is AI as big as the Industrial Revolution? (0:06:35) – The labour market implications of AI (0:21:18) – Industry policy for AI: when to intervene, when to step back (0:31:27) – Where will the profits flow? Chips, models, or applications? (0:52:37) – Data centres: should Australia support them? (1:15:07) – AI risks, and why Australia shouldn't pass an AI Act (1:23:27) – Measuring progress in the age of AI: beyond GDP and labour productivity

Joseph Noel Walker

12,138 次观看 • 3 个月前

The Dario Amodei interview. 0:00:00 - What exactly are we scaling? 0:12:36 - Is diffusion cope? 0:29:42 - Is continual learning necessary? 0:46:20 - If AGI is imminent, why not buy more compute? 0:58:49 - How will AI labs actually make profit? 1:31:19 - Will regulations destroy the boons of AGI? 1:47:41 - Why can’t China and America both have a country of geniuses in a datacenter? Look up Dwarkesh Podcast on Youtube, Spotify, Apple Podcasts, etc.

The Dario Amodei interview. 0:00:00 - What exactly are we scaling? 0:12:36 - Is diffusion cope? 0:29:42 - Is continual learning necessary? 0:46:20 - If AGI is imminent, why not buy more compute? 0:58:49 - How will AI labs actually make profit? 1:31:19 - Will regulations destroy the boons of AGI? 1:47:41 - Why can’t China and America both have a country of geniuses in a datacenter? Look up Dwarkesh Podcast on Youtube, Spotify, Apple Podcasts, etc.

Dwarkesh Patel

2,062,832 次观看 • 6 个月前

In this episode, OpenAI Chief Research Officer Mark Chen joins allen to flambé shrimp, cook Korean stew, and chat about being at the frontier of AI research: why scaling laws and pre-training still matter, how OpenAI chooses research bets and allocates compute, what it means to develop research taste, why evals are in crisis, how to avoid benchmark-maxing, and what it will take for models to handle long-horizon real-world work, multimodal reasoning, and eventually end-to-end AI research. Timestamps: 0:00 Intro 0:28 The Soup Story 1:52 From Trading to AI Research 3:21 How to Develop Research Taste 5:23 RL, Evals, and Superhuman Benchmarks 8:17 Cooking Begins on the Impulse Stove 8:53 Scaling Laws, Pre-Training, and Reasoning 12:33 OpenAI’s Research Roadmap and Compute Allocation 15:48 What Makes a Great Researcher 19:33 The Evals Crisis and Benchmark-Maxing 24:34 Jagged Intelligence, Context, and Long-Horizon Learning 27:14 Shrimp Flambé and New Research Bets 31:32 Multimodal Models and One Architecture 32:36 Vibe Researching and End-to-End AI Research 34:36 Failed Bets, Postmortems, and OpenAI’s Alpha 37:07 Final Taste Test 37:53 Overrated vs. Underrated AI Research 41:00 Closing

In this episode, OpenAI Chief Research Officer Mark Chen joins allen to flambé shrimp, cook Korean stew, and chat about being at the frontier of AI research: why scaling laws and pre-training still matter, how OpenAI chooses research bets and allocates compute, what it means to develop research taste, why evals are in crisis, how to avoid benchmark-maxing, and what it will take for models to handle long-horizon real-world work, multimodal reasoning, and eventually end-to-end AI research. Timestamps: 0:00 Intro 0:28 The Soup Story 1:52 From Trading to AI Research 3:21 How to Develop Research Taste 5:23 RL, Evals, and Superhuman Benchmarks 8:17 Cooking Begins on the Impulse Stove 8:53 Scaling Laws, Pre-Training, and Reasoning 12:33 OpenAI’s Research Roadmap and Compute Allocation 15:48 What Makes a Great Researcher 19:33 The Evals Crisis and Benchmark-Maxing 24:34 Jagged Intelligence, Context, and Long-Horizon Learning 27:14 Shrimp Flambé and New Research Bets 31:32 Multimodal Models and One Architecture 32:36 Vibe Researching and End-to-End AI Research 34:36 Failed Bets, Postmortems, and OpenAI’s Alpha 37:07 Final Taste Test 37:53 Overrated vs. Underrated AI Research 41:00 Closing

Latent.Space

62,796 次观看 • 2 个月前