Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

ARC-AGI-3 Preview Event Recap Greg Kamradt steps through our Interactive Reasoning Benchmark thesis * Why static benchmarks fall short measuring agentic capabilities * The ARC Prize approach to creating interactive benchmarks

ARC Prize

28,225 subscribers

22,248 views • 11 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Interactive Reasoning Benchmarks are the next step in frontier evaluations Hear Greg Kamradt share why measuring human-like intelligence requires multi-turn environments Including a sneak peak of ARC-AGI-3 Want to help us build interactive evaluations? We're hiring

Interactive Reasoning Benchmarks are the next step in frontier evaluations Hear Greg Kamradt share why measuring human-like intelligence requires multi-turn environments Including a sneak peak of ARC-AGI-3 Want to help us build interactive evaluations? We're hiring

ARC Prize

26,218 views • 1 year ago

ARC-AGI is redefining how to measure progress on the path to AGI - focusing on reasoning, generalization, and adaptability instead of memorization or scale. At NeurIPS 2025, YC's Diana sat down with ARC Prize President Greg Kamradt to find out why most AI benchmarks fail, how ARC-AGI reveals the limits of today’s models, and why measuring intelligence may be harder than building it. 00:11 — What ARC Prize is and why it exists 00:38 — François Chollet’s definition of AGI 01:48 — What ARC-AGI Actually Tests 02:25 — When LLMs Failed the ARC Benchmark 03:38 — ARC-AGI Becomes the Standard 04:49 — False Positives in AI Progress 06:06 — The Evolution of ARC-AGI 08:55 — Measuring Intelligence beyond just accuracy 10:25 — What happens if a model solves ARC-AGI?

ARC-AGI is redefining how to measure progress on the path to AGI - focusing on reasoning, generalization, and adaptability instead of memorization or scale. At NeurIPS 2025, YC's Diana sat down with ARC Prize President Greg Kamradt to find out why most AI benchmarks fail, how ARC-AGI reveals the limits of today’s models, and why measuring intelligence may be harder than building it. 00:11 — What ARC Prize is and why it exists 00:38 — François Chollet’s definition of AGI 01:48 — What ARC-AGI Actually Tests 02:25 — When LLMs Failed the ARC Benchmark 03:38 — ARC-AGI Becomes the Standard 04:49 — False Positives in AI Progress 06:06 — The Evolution of ARC-AGI 08:55 — Measuring Intelligence beyond just accuracy 10:25 — What happens if a model solves ARC-AGI?

Y Combinator

98,369 views • 7 months ago

François Chollet (François Chollet) on the ARC Prize and how we get to AGI. At AI Startup School in San Francisco. 00:00 - The Falling Cost of Compute 00:57 - Deep-Learning’s Scaling Era & Benchmarks 01:59 - The ARC Benchmark 03:02 - The 2024 Shift to Test-Time Adaptation 05:01 - What Is Intelligence? 07:12 - Why Benchmarks Matter (and Mislead) 08:57 - ARC 1 Exposes Scaling Limits 10:58 - ARC 2: Compositional Reasoning Arrives 12:55 - Humans vs. Models on ARC2 14:58 - Previewing ARC3 & Interactive Agency 17:00 - Kaleidoscopic Hypothesis and Abstractions 22:00 - Type 1 vs. Type 2 Abstractions 26:00 - Discrete Program Search & Inventive AI 29:00 - Fusing Intuition with Symbolic Reasoning 32:00 - Building AGI Through Meta-Learning Systems

François Chollet (François Chollet) on the ARC Prize and how we get to AGI. At AI Startup School in San Francisco. 00:00 - The Falling Cost of Compute 00:57 - Deep-Learning’s Scaling Era & Benchmarks 01:59 - The ARC Benchmark 03:02 - The 2024 Shift to Test-Time Adaptation 05:01 - What Is Intelligence? 07:12 - Why Benchmarks Matter (and Mislead) 08:57 - ARC 1 Exposes Scaling Limits 10:58 - ARC 2: Compositional Reasoning Arrives 12:55 - Humans vs. Models on ARC2 14:58 - Previewing ARC3 & Interactive Agency 17:00 - Kaleidoscopic Hypothesis and Abstractions 22:00 - Type 1 vs. Type 2 Abstractions 26:00 - Discrete Program Search & Inventive AI 29:00 - Fusing Intuition with Symbolic Reasoning 32:00 - Building AGI Through Meta-Learning Systems

Y Combinator

231,796 views • 1 year ago

François Chollet (François Chollet) has spent years asking a different question than most of the AI world. Instead of scaling what already works, he’s trying to understand what intelligence actually is and how to build it from first principles. In this episode of the Lightcone Podcast, he traces that path from his early work on deep learning to the creation of the ARC Prize, and the launch of ARC V3, a new benchmark designed to measure something deeper than performance: the ability to learn, adapt, and reason efficiently in entirely new environments. He explains why today’s systems may be hitting limits, what recent breakthroughs really mean, and why reaching true general intelligence may require a fundamentally different approach. 00:00 - AGI by 2030? 00:31 - Introducing Ndea: A New Path Beyond Deep Learning 01:08 - A New ML Paradigm 01:30 - Replacing neural nets with compact symbolic programs 03:04 - Why Ndea Isn’t Competing With Coding Agents 05:20 - Why Everyone Might Be Wrong About Scaling LLMs 07:22 - Why Coding Agents Suddenly Work So Well 08:50 - The Limits of LLMs in Non-Verifiable Domains 10:48 - What AGI Actually Means (And Why Most Definitions Are Wrong) 13:30 - Why Deep Learning Hits a Wall 14:00 - ARC’s Origin Story 18:20 - ARC Benchmarks Explained: From V1 to V3 22:49 - The RL Loop Powering Coding Agents Today 27:03 - ARC-AGI V3: Measuring “Agentic Intelligence” 31:14 - Inside the ARC Game Studio 35:31 - Could AGI Fit in 10,000 Lines of Code? 44:01 - Building Ndea: From Idea to Compounding Research Stack 46:46 - The Future of ARC: Benchmarks That Evolve With AI 47:21 - Why There’s Still Huge Opportunity for New AI Paradigms 53:37 - How to Build a Breakout Open Source Project - Lessons From Keras 56:39 - Advice For How To Think About AI

François Chollet (François Chollet) has spent years asking a different question than most of the AI world. Instead of scaling what already works, he’s trying to understand what intelligence actually is and how to build it from first principles. In this episode of the Lightcone Podcast, he traces that path from his early work on deep learning to the creation of the ARC Prize, and the launch of ARC V3, a new benchmark designed to measure something deeper than performance: the ability to learn, adapt, and reason efficiently in entirely new environments. He explains why today’s systems may be hitting limits, what recent breakthroughs really mean, and why reaching true general intelligence may require a fundamentally different approach. 00:00 - AGI by 2030? 00:31 - Introducing Ndea: A New Path Beyond Deep Learning 01:08 - A New ML Paradigm 01:30 - Replacing neural nets with compact symbolic programs 03:04 - Why Ndea Isn’t Competing With Coding Agents 05:20 - Why Everyone Might Be Wrong About Scaling LLMs 07:22 - Why Coding Agents Suddenly Work So Well 08:50 - The Limits of LLMs in Non-Verifiable Domains 10:48 - What AGI Actually Means (And Why Most Definitions Are Wrong) 13:30 - Why Deep Learning Hits a Wall 14:00 - ARC’s Origin Story 18:20 - ARC Benchmarks Explained: From V1 to V3 22:49 - The RL Loop Powering Coding Agents Today 27:03 - ARC-AGI V3: Measuring “Agentic Intelligence” 31:14 - Inside the ARC Game Studio 35:31 - Could AGI Fit in 10,000 Lines of Code? 44:01 - Building Ndea: From Idea to Compounding Research Stack 46:46 - The Future of ARC: Benchmarks That Evolve With AI 47:21 - Why There’s Still Huge Opportunity for New AI Paradigms 53:37 - How to Build a Breakout Open Source Project - Lessons From Keras 56:39 - Advice For How To Think About AI

Y Combinator

151,442 views • 4 months ago

Introducing Gemini 3.1 Pro 🚀 3.1 Pro represents a major step forward in core reasoning. It scored 77.1% (more than doubling 3 Pro’s score) on ARC-AGI-2, the benchmark that evaluates a model's ability to solve new logic patterns and work through challenges it hasn’t encountered before. This demo illustrates how the model can go beyond the prompt. Instead of rendering a video or static graphic, 3.1 Pro codes a full environment, integrating generative audio and providing UI controls.

Introducing Gemini 3.1 Pro 🚀 3.1 Pro represents a major step forward in core reasoning. It scored 77.1% (more than doubling 3 Pro’s score) on ARC-AGI-2, the benchmark that evaluates a model's ability to solve new logic patterns and work through challenges it hasn’t encountered before. This demo illustrates how the model can go beyond the prompt. Instead of rendering a video or static graphic, 3.1 Pro codes a full environment, integrating generative audio and providing UI controls.

Google AI

3,067,134 views • 5 months ago

ARC-AGI-3 is built different, it has dumbfounded almost all regular attempts so far because it's so much harder than anything that came before. It has no rules, it's agentic and has no explicit goals, they need to be discovered. Tufalabs won the first milestone of ARC Prize > There is no language built into the benchmark, but these guys "put the language back in", because in their view - it's the best way to climb up the notional "abstraction mountain" and effectively use many of the abstractions which have evolved over millions of years of language evolution. > They built a novel harness "The Duck" around a 27B open weights model (Qwen 3.6) to solve extremely challenging and novel reasoning problems that require abstraction. > This is the launch video of their winning agentic harness, "The Duck". We have also released an exclusive interview with them on MLST, just dropped. > The million dollar question is: what will François Chollet think about how they've done it, and is this a step towards AGI?

ARC-AGI-3 is built different, it has dumbfounded almost all regular attempts so far because it's so much harder than anything that came before. It has no rules, it's agentic and has no explicit goals, they need to be discovered. Tufalabs won the first milestone of ARC Prize > There is no language built into the benchmark, but these guys "put the language back in", because in their view - it's the best way to climb up the notional "abstraction mountain" and effectively use many of the abstractions which have evolved over millions of years of language evolution. > They built a novel harness "The Duck" around a 27B open weights model (Qwen 3.6) to solve extremely challenging and novel reasoning problems that require abstraction. > This is the launch video of their winning agentic harness, "The Duck". We have also released an exclusive interview with them on MLST, just dropped. > The million dollar question is: what will François Chollet think about how they've done it, and is this a step towards AGI?

Machine Learning Street Talk

33,410 views • 27 days ago

Today, we're joined by Aakanksha Chowdhery, member of technical staff at Reflection, to explore the fundamental shifts required to build true agentic AI. While the industry has largely focused on post-training techniques to improve reasoning, Aakanksha draws on her experience leading pre-training efforts for Google’s PaLM and early Gemini models to argue that pre-training itself must be rethought to move beyond static benchmarks. We explore the limitations of next-token prediction for multi-step workflows and examine how attention mechanisms, loss objectives, and training data must evolve to support long-form reasoning and planning. Aakanksha shares insights on the difference between context retrieval and actual reasoning, the importance of "trajectory" training data, and why scaling remains essential for discovering emergent agentic capabilities like error recovery and dynamic tool learning. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 02:26 - Reflection 04:54 - Limitations of post-training for building agents 07:31 - Rethinking pre-training in agents 10:51 - Scaling 11:27 - Evolving attention mechanisms for agentic capabilities 12:39 - Memory as a tool 14:13 - Loss objectives and training data 15:50 - Fine-tuning loss in agent performance 19:37 - Training data 21:29 - Augmenting dominant training data source 24:11 - Overcoming challenges in training on synthetic data 25:47 - Benchmarks 30:44 - Scaling laws in large models versus small models 33:20 - Long-form versus short-form reasoning 37:57 - Agent’s ability to recover from failure 40:15 - Hallucinations and failure recovery 43:53 - Tool use in agents 46:38 - Coding agents 48:37 - How researchers can contribute to agentic AI

Today, we're joined by Aakanksha Chowdhery, member of technical staff at Reflection, to explore the fundamental shifts required to build true agentic AI. While the industry has largely focused on post-training techniques to improve reasoning, Aakanksha draws on her experience leading pre-training efforts for Google’s PaLM and early Gemini models to argue that pre-training itself must be rethought to move beyond static benchmarks. We explore the limitations of next-token prediction for multi-step workflows and examine how attention mechanisms, loss objectives, and training data must evolve to support long-form reasoning and planning. Aakanksha shares insights on the difference between context retrieval and actual reasoning, the importance of "trajectory" training data, and why scaling remains essential for discovering emergent agentic capabilities like error recovery and dynamic tool learning. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 02:26 - Reflection 04:54 - Limitations of post-training for building agents 07:31 - Rethinking pre-training in agents 10:51 - Scaling 11:27 - Evolving attention mechanisms for agentic capabilities 12:39 - Memory as a tool 14:13 - Loss objectives and training data 15:50 - Fine-tuning loss in agent performance 19:37 - Training data 21:29 - Augmenting dominant training data source 24:11 - Overcoming challenges in training on synthetic data 25:47 - Benchmarks 30:44 - Scaling laws in large models versus small models 33:20 - Long-form versus short-form reasoning 37:57 - Agent’s ability to recover from failure 40:15 - Hallucinations and failure recovery 43:53 - Tool use in agents 46:38 - Coding agents 48:37 - How researchers can contribute to agentic AI

The TWIML AI Podcast

44,888 views • 7 months ago

🚀 New Paper: Pixel Reasoner 🧠🖼️ How can Vision-Language Models (VLMs) perform chain-of-thought reasoning within the image itself? We introduce Pixel Reasoner, the first open-source framework that enables VLMs to “think in pixel space” through curiosity-driven reinforcement learning. Current VLMs reason only in text — even when grounded in rich images or videos, their logical steps are verbalized in natural language. This restricts their ability to interrogate visual evidence and demonstrate how conclusions are drawn. 🔍 So we ask: What if we could make VLMs "show their work" by reasoning directly in the pixel space? Inspired by GPT-o3’s "think-in-image" ability, we propose a framework where VLMs use interactive visual operations — zoom, select-frame, highlight — to reason through complex visual inputs. To do this, we design a two-stage training process: Instruction tuning with synthesized visual reasoning traces. Reinforcement learning with curiosity-driven reward to balance exploration between pixel and text reasoning ✨ With this, Pixel Reasoner achieves near-SoTA performance on many information-rich multimodal benchmarks: 📊 84% on InfographicsVQA 🧠 84% on V* benchmark 🧩 74% on TallyQA-Complex It also achieves strong accuracy of 68% on MVBench (a video benchmark). Website: Paper: Code: Demo: (coming soon)

🚀 New Paper: Pixel Reasoner 🧠🖼️ How can Vision-Language Models (VLMs) perform chain-of-thought reasoning within the image itself? We introduce Pixel Reasoner, the first open-source framework that enables VLMs to “think in pixel space” through curiosity-driven reinforcement learning. Current VLMs reason only in text — even when grounded in rich images or videos, their logical steps are verbalized in natural language. This restricts their ability to interrogate visual evidence and demonstrate how conclusions are drawn. 🔍 So we ask: What if we could make VLMs "show their work" by reasoning directly in the pixel space? Inspired by GPT-o3’s "think-in-image" ability, we propose a framework where VLMs use interactive visual operations — zoom, select-frame, highlight — to reason through complex visual inputs. To do this, we design a two-stage training process: Instruction tuning with synthesized visual reasoning traces. Reinforcement learning with curiosity-driven reward to balance exploration between pixel and text reasoning ✨ With this, Pixel Reasoner achieves near-SoTA performance on many information-rich multimodal benchmarks: 📊 84% on InfographicsVQA 🧠 84% on V* benchmark 🧩 74% on TallyQA-Complex It also achieves strong accuracy of 68% on MVBench (a video benchmark). Website: Paper: Code: Demo: (coming soon)

Wenhu Chen

82,829 views • 1 year ago

VITA Towards Open-Source Interactive Omni Multimodal LLM discuss: The remarkable multimodal capabilities and interactive experience of GPT-4o underscore their necessity in practical applications, yet open-source models rarely excel in both areas. In this paper, we introduce VITA, the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text, and Audio modalities, and meanwhile has an advanced multimodal interactive experience. Starting from Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary followed by bilingual instruction tuning. We further endow the language model with visual and audio capabilities through two-stage multi-task learning of multimodal alignment and instruction tuning. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. Beyond foundational capabilities, we have made considerable progress in enhancing the natural multimodal human-computer interaction experience. To the best of our knowledge, we are the first to exploit non-awakening interaction and audio interrupt in MLLM. VITA is the first step for the open-source community to explore the seamless integration of multimodal understanding and interaction. While there is still lots of work to be done on VITA to get close to close-source counterparts, we hope that its role as a pioneer can serve as a cornerstone for subsequent research.

VITA Towards Open-Source Interactive Omni Multimodal LLM discuss: The remarkable multimodal capabilities and interactive experience of GPT-4o underscore their necessity in practical applications, yet open-source models rarely excel in both areas. In this paper, we introduce VITA, the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text, and Audio modalities, and meanwhile has an advanced multimodal interactive experience. Starting from Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary followed by bilingual instruction tuning. We further endow the language model with visual and audio capabilities through two-stage multi-task learning of multimodal alignment and instruction tuning. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. Beyond foundational capabilities, we have made considerable progress in enhancing the natural multimodal human-computer interaction experience. To the best of our knowledge, we are the first to exploit non-awakening interaction and audio interrupt in MLLM. VITA is the first step for the open-source community to explore the seamless integration of multimodal understanding and interaction. While there is still lots of work to be done on VITA to get close to close-source counterparts, we hope that its role as a pioneer can serve as a cornerstone for subsequent research.

AK

23,958 views • 1 year ago

Excited to share ESI-BENCH, a benchmark for Embodied Spatial Intelligence! Most spatial reasoning benchmarks assume an oracle observer: the agent is given the right image, view, or 3D scene. But in the real world, the observer is also an actor. To understand space, agents must decide where to look, how to move, and when to interact, to reveal what is hidden: occlusions, containment, contact, dynamics, and functionality. In many cases, the hard part is not perception itself, but choosing the right action to make informative perception possible. ESI-BENCH tests this perception-action loop. Agents receive an egocentric observation and a spatial question, then must actively gather evidence through perception, locomotion, and manipulationbefore answering. The benchmark spans 10 task categories, 29 subcategories, and 3,081 instances, built in BEHAVIOR-1K across realistic interactive scenes. 🌍Webpage: 💻Code & data: Thanks for collaborators: Jiageng, Han, Manling Li , Leonidas Guibas, Fei-Fei Li , Jiajun Wu , Yejin Choi

Excited to share ESI-BENCH, a benchmark for Embodied Spatial Intelligence! Most spatial reasoning benchmarks assume an oracle observer: the agent is given the right image, view, or 3D scene. But in the real world, the observer is also an actor. To understand space, agents must decide where to look, how to move, and when to interact, to reveal what is hidden: occlusions, containment, contact, dynamics, and functionality. In many cases, the hard part is not perception itself, but choosing the right action to make informative perception possible. ESI-BENCH tests this perception-action loop. Agents receive an egocentric observation and a spatial question, then must actively gather evidence through perception, locomotion, and manipulationbefore answering. The benchmark spans 10 task categories, 29 subcategories, and 3,081 instances, built in BEHAVIOR-1K across realistic interactive scenes. 🌍Webpage: 💻Code & data: Thanks for collaborators: Jiageng, Han, Manling Li , Leonidas Guibas, Fei-Fei Li , Jiajun Wu , Yejin Choi

Yining Hong

49,989 views • 2 months ago

(1/5) Gemini 3, our most intelligent model, is landing in Google Search today – starting with AI Mode. Excited that this is the first time we’re shipping a new Gemini model in Search on day one! 🚀 In Search, Gemini 3 with generative layouts will make it easy to get a rich understanding of anything on your mind. It has state-of-the-art reasoning, deep multimodal understanding and advanced agentic capabilities. That allows the model to shine when you ask it to explain advanced concepts or ideas – it reasons and can code interactive visuals in real-time. It can tackle your toughest questions like advanced science.

(1/5) Gemini 3, our most intelligent model, is landing in Google Search today – starting with AI Mode. Excited that this is the first time we’re shipping a new Gemini model in Search on day one! 🚀 In Search, Gemini 3 with generative layouts will make it easy to get a rich understanding of anything on your mind. It has state-of-the-art reasoning, deep multimodal understanding and advanced agentic capabilities. That allows the model to shine when you ask it to explain advanced concepts or ideas – it reasons and can code interactive visuals in real-time. It can tackle your toughest questions like advanced science.

Robby Stein

94,877 views • 8 months ago

From a scrappy side project built to solve their own LLM optimization problems to becoming the industry’s de-facto independent scoreboard, Micah Hill-Smith and George Cameron went through the arc of launching Artificial Analysis for free, paying benchmarking costs out of pocket, and growing it into what many now call the “new Gartner of AI” for enterprises, labs, and developers. We sat down with Micah and George to unpack why truly independent benchmarking is so hard (prompt variance, eval saturation, mystery-shopper policies), how the Artificial Analysis Intelligence Index evolved as old benchmarks broke, and what new metrics actually matter now such as agentic evals (GDPVal-AA). We also dig into the economics behind the “smile curve” of AI: why intelligence is getting 100–1000× cheaper per unit while total spend explodes, how reasoning and agents change token efficiency, and their bet that evals must continuously evolve or risk training the industry to optimize for the wrong things. swyx Micah Hill-Smith George Cameron

From a scrappy side project built to solve their own LLM optimization problems to becoming the industry’s de-facto independent scoreboard, Micah Hill-Smith and George Cameron went through the arc of launching Artificial Analysis for free, paying benchmarking costs out of pocket, and growing it into what many now call the “new Gartner of AI” for enterprises, labs, and developers. We sat down with Micah and George to unpack why truly independent benchmarking is so hard (prompt variance, eval saturation, mystery-shopper policies), how the Artificial Analysis Intelligence Index evolved as old benchmarks broke, and what new metrics actually matter now such as agentic evals (GDPVal-AA). We also dig into the economics behind the “smile curve” of AI: why intelligence is getting 100–1000× cheaper per unit while total spend explodes, how reasoning and agents change token efficiency, and their bet that evals must continuously evolve or risk training the industry to optimize for the wrong things. swyx Micah Hill-Smith George Cameron

Latent.Space

33,374 views • 6 months ago

Even within our own research team, timelines for transformative AI differ substantially. In this episode, the two Epoch AI researchers with the longest and the shortest timelines for transformative AI candidly examine the roots of their disagreements. They discuss: How and why their timelines for specific milestones differ Current technical challenges for AI progress Why widespread automation beats geniuses in datacenters Pitfalls of conventional AI prediction approaches Critiquing "single AGI" or "utopia vs. doom" narratives How a world with AGI might look like And much, much more. (0:00:00) - Preview (0:01:08) - Contrasting AGI Timelines (0:08:30) - Updating Beliefs as Capabilities Advance (0:17:07) - Moravec’s Paradox and the Agency Challenge (0:32:40) - Missing Capabilities for AGI (0:47:43) - Beating benchmarks vs Being Useful (0:59:20) - AI Excelling in Some Tasks While Struggling with Others (1:07:33) - Economic Impact of AI vs the Internet (1:24:08) - Widespread Automation Beats Genius in Datacenters (1:51:37) - How Stories Shape Our Expectations of AI (2:03:24) - How AGI Will Impact Culture (2:10:46) - Beyond Utopia-or-Extinction (2:16:57) - AI's Impact on Wages and Labor (2:27:49) - Why Better Preservation of Information Accelerates Change (2:39:32) - Markets Shaping Cultural Priorities (2:55:51) - Challenges in Defining What We Want to Preserve (3:06:47) - Risk Attitudes in AI Decision-Making (3:12:50) - Historical Lessons for AI Coexistence (3:21:45) - A Warning Sign in Safety Discourse (3:30:20) - Revisiting Core Assumptions in AI Alignment (3:49:46) - Simple Models in Complex Domains

Even within our own research team, timelines for transformative AI differ substantially. In this episode, the two Epoch AI researchers with the longest and the shortest timelines for transformative AI candidly examine the roots of their disagreements. They discuss: How and why their timelines for specific milestones differ Current technical challenges for AI progress Why widespread automation beats geniuses in datacenters Pitfalls of conventional AI prediction approaches Critiquing "single AGI" or "utopia vs. doom" narratives How a world with AGI might look like And much, much more. (0:00:00) - Preview (0:01:08) - Contrasting AGI Timelines (0:08:30) - Updating Beliefs as Capabilities Advance (0:17:07) - Moravec’s Paradox and the Agency Challenge (0:32:40) - Missing Capabilities for AGI (0:47:43) - Beating benchmarks vs Being Useful (0:59:20) - AI Excelling in Some Tasks While Struggling with Others (1:07:33) - Economic Impact of AI vs the Internet (1:24:08) - Widespread Automation Beats Genius in Datacenters (1:51:37) - How Stories Shape Our Expectations of AI (2:03:24) - How AGI Will Impact Culture (2:10:46) - Beyond Utopia-or-Extinction (2:16:57) - AI's Impact on Wages and Labor (2:27:49) - Why Better Preservation of Information Accelerates Change (2:39:32) - Markets Shaping Cultural Priorities (2:55:51) - Challenges in Defining What We Want to Preserve (3:06:47) - Risk Attitudes in AI Decision-Making (3:12:50) - Historical Lessons for AI Coexistence (3:21:45) - A Warning Sign in Safety Discourse (3:30:20) - Revisiting Core Assumptions in AI Alignment (3:49:46) - Simple Models in Complex Domains

Epoch AI

159,795 views • 1 year ago

Announcing our speakers for the Reasoning + RL Track! ⚠️PSA: Tix nearly sold out, get em here: An incredible lineup of speakers at this track, featuring: Jack Rae, Principal Scientist Google DeepMind Christian Szegedy, former co-founder xAI will brown, Research engineer Prime Intellect Greg Kamradt, President ARC Prize Kyle Corbitt, co-founder & CEO of OpenPipe Jesse Michael Han, founder & CEO of Morph Nathan Lambert, Senior Research Scientist at Ai2 & Founder of Interconnects Daniel Han, CEO of Unsloth AI This track is also sponsored by OpenPipe, who will be facilitating the track intros & speaker introductions as the in-room MC! World's Fair is 18 miniconferences all co-located in one place in SF so you can meet everyone who matters: - MCP - Tiny Teams (ARR > employees) - LLM RecSys - GraphRAG - Agent Reliability - Infrastructure - Product Management - Voice Al - Reasoning + RL - SWE-Agents - Evals - Retrieval + Search - Security - Generative Media - Design Engineering - Robotics/Autonomy And for the first time TWO two-day leadership tracks for CTOs and VPs of Al: Al in Fortune 500 and Al Architects Join us from Jun 3-5 in SF!

Announcing our speakers for the Reasoning + RL Track! ⚠️PSA: Tix nearly sold out, get em here: An incredible lineup of speakers at this track, featuring: Jack Rae, Principal Scientist Google DeepMind Christian Szegedy, former co-founder xAI will brown, Research engineer Prime Intellect Greg Kamradt, President ARC Prize Kyle Corbitt, co-founder & CEO of OpenPipe Jesse Michael Han, founder & CEO of Morph Nathan Lambert, Senior Research Scientist at Ai2 & Founder of Interconnects Daniel Han, CEO of Unsloth AI This track is also sponsored by OpenPipe, who will be facilitating the track intros & speaker introductions as the in-room MC! World's Fair is 18 miniconferences all co-located in one place in SF so you can meet everyone who matters: - MCP - Tiny Teams (ARR > employees) - LLM RecSys - GraphRAG - Agent Reliability - Infrastructure - Product Management - Voice Al - Reasoning + RL - SWE-Agents - Evals - Retrieval + Search - Security - Generative Media - Design Engineering - Robotics/Autonomy And for the first time TWO two-day leadership tracks for CTOs and VPs of Al: Al in Fortune 500 and Al Architects Join us from Jun 3-5 in SF!

AI Engineer

31,072 views • 1 year ago

I asked our Head of Finance if I could spend $60,000 to take five enterprise CRO prospects to the Super Bowl. He did the ROI math and it's not looking good for me. Karan Parekh ( Karan Parekh) actually walked me through the numbers on camera - deal size requirements, probability lift benchmarks, estimated ROI, and why our bet at the US Open was 10x smarter. Some of what we get into: - How to calculate the real cost per ticket - Why a 15-20% increase in close probability on even $400k ACV deals doesn't justify the spend - My land-and-expand counterargument (which I thought would win him over) - Our portfolio approach to event marketing budgets and when we measure individual bets vs signal for scaling In the end, Karan gives a surprising reason why even if all five CROs convert, the Super Bowl isn't worth the $. This type of conversation happens every day at Clay. It's how we balance making aggressive creative bets with financial discipline. Yes, he just killed my Super Bowl dreams. But he is (probably) right!

I asked our Head of Finance if I could spend $60,000 to take five enterprise CRO prospects to the Super Bowl. He did the ROI math and it's not looking good for me. Karan Parekh ( Karan Parekh) actually walked me through the numbers on camera - deal size requirements, probability lift benchmarks, estimated ROI, and why our bet at the US Open was 10x smarter. Some of what we get into: - How to calculate the real cost per ticket - Why a 15-20% increase in close probability on even $400k ACV deals doesn't justify the spend - My land-and-expand counterargument (which I thought would win him over) - Our portfolio approach to event marketing budgets and when we measure individual bets vs signal for scaling In the end, Karan gives a surprising reason why even if all five CROs convert, the Super Bowl isn't worth the $. This type of conversation happens every day at Clay. It's how we balance making aggressive creative bets with financial discipline. Yes, he just killed my Super Bowl dreams. But he is (probably) right!

Varun Anand

170,514 views • 6 months ago

Open-sourcing the first unified LLM routing library 🔥 Meet LLMRouter 16+ routers in ONE framework. Stop reimplementing routing papers from scratch, simply run pip install llmrouter-lib And instantly deploy SOTA LLM routing tailored to your exact needs! 🔗 Get Started with LLMRouter 🚀 Code: 🌐 Project Page: 💸 Why LLMRouter? Why pay GPT-5 prices for "What's the weather?" Smart routing = Significant Savings Simple query → Cheap model Complex query → Powerful model Save 30–50% on inference costs without sacrificing quality 🧭 The 16+ Router Arsenal Switch between state-of-the-art methods with a single flag change. We cover every paradigm: ✅ Single-Round: From classic ML (KNN, SVM) to neural approaches (Graph, Contrastive). ✅ Multi-Round: RL-powered reasoning that "thinks" before routing. ✅ Agentic: Breaks down complex tasks and routes step-by-step. ✅ Personalized: Learns from user history to fit unique needs. 🛠️ Complete "Out-of-the-Box" Toolchain We didn't just ship the code; we shipped the pipeline. ✅ Unified CLI: Train, infer, and chat with any router. ✅ Interactive UI: Full Gradio interface included. ✅ Benchmarks: 11 datasets ready to use + full data generation pipeline. From LLM routing research to production in minutes. 🚀

Open-sourcing the first unified LLM routing library 🔥 Meet LLMRouter 16+ routers in ONE framework. Stop reimplementing routing papers from scratch, simply run pip install llmrouter-lib And instantly deploy SOTA LLM routing tailored to your exact needs! 🔗 Get Started with LLMRouter 🚀 Code: 🌐 Project Page: 💸 Why LLMRouter? Why pay GPT-5 prices for "What's the weather?" Smart routing = Significant Savings Simple query → Cheap model Complex query → Powerful model Save 30–50% on inference costs without sacrificing quality 🧭 The 16+ Router Arsenal Switch between state-of-the-art methods with a single flag change. We cover every paradigm: ✅ Single-Round: From classic ML (KNN, SVM) to neural approaches (Graph, Contrastive). ✅ Multi-Round: RL-powered reasoning that "thinks" before routing. ✅ Agentic: Breaks down complex tasks and routes step-by-step. ✅ Personalized: Learns from user history to fit unique needs. 🛠️ Complete "Out-of-the-Box" Toolchain We didn't just ship the code; we shipped the pipeline. ✅ Unified CLI: Train, infer, and chat with any router. ✅ Interactive UI: Full Gradio interface included. ✅ Benchmarks: 11 datasets ready to use + full data generation pipeline. From LLM routing research to production in minutes. 🚀

Jiaxuan You

75,244 views • 7 months ago

We’re thrilled to announce the official launch of Lil Pudgys as a standalone brand within the Pudgy Penguins ecosystem! For the past six months, our team has been working tirelessly to develop a unique identity for Lil Pudgys that resonates with children and tweens. This isn’t just a brand extension – it’s a carefully crafted world designed specifically for younger audiences to fall in love with. What’s Coming for Lil Pudgys? 1. Original Product Line: Designed with younger fans in mind, featuring vibrant colors and playful designs 2. YouTube Series Launch: Fun, engaging content that brings Lil Pudgys characters to life 3. Interactive Experiences: Digital and physical touchpoints that create memorable moments This strategic move allows us to welcome the next generation into our penguin community. While Pudgy Penguins continues to serve our core audience of young adults and collectors, Lil Pudgys creates a perfect entry point for younger fans. The beauty of this approach? As our Lil Pudgys audience grows up, they’ll have a natural path to engage with the broader Pudgy Penguins ecosystem, creating a seamless journey through our penguin universe. Over the next year, you’ll see us roll out more Lil Pudgys-specific products and experiences that further establish this brand while complementing our overarching Pudgy Penguins vision.

Chef

81,801 views • 1 year ago

Alibaba just dropped Qwen3.5-397B-A17B and there's a lot to unpack. 397B params, 17B active per forward pass. Sparse MoE done right. But the real story isn't the size—it's the architecture choices. The MoE Design Most MoE models feel like bolt-ons. Qwen 3.5's sparse activation is native—only 4.3% of parameters fire per token. That's how you get trillion-parameter-class performance without trillion-parameter inference costs. The 0.8 RMB/million tokens pricing isn't subsidized; it's structurally earned. Native Multimodal, Not Glued-On This is a vision-language model from the ground up. Heterogeneous architecture—separate processing pipelines for text, image, video that fuse early. Not a vision encoder slapped onto an LLM. The result: 90.8 on OmniDocBench, 79.0 on MMMU-Pro. Document understanding and visual reasoning without the usual brittleness. The Context Window Reality Qwen3.5-Plus (the hosted version) ships with 1M tokens by default. That's not a marketing number—they're actually positioning it for long-document workflows. With built-in adaptive tool use, it's clearly aimed at agentic automation, not just chat. What Actually Impressed Me • FP8 native pipeline: ~50% activation memory reduction • Async RL framework for continuous refinement—training and inference workloads separated • 201 languages (up from 119), 250k vocab for better low-resource encoding • Apache 2.0 license. Full weights on HuggingFace and ModelScope. The Benchmark Context 76.4 on SWE-bench Verified puts it in the range where it can handle real debugging workflows. 72.9 on BFCL v4 for agentic tool use. 88.4 on GPQA Diamond. These aren't SOTA in isolation, but the breadth is unusual—strong across reasoning, coding, multimodal, and agentic tasks. The Honest Caveat I haven't stress-tested the 1M context for needle-in-haystack retrieval yet. And "native multimodal" claims need real-world torture testing—PDFs with tables, charts, mixed layouts. Benchmarks are benchmarks. Bottom Line This isn't just another model release. It's a bet on efficient scale: big model capabilities, small active compute, open weights. At 1/18th the cost of Gemini 3 Pro, it's going to force pricing conversations across the board.

Alibaba just dropped Qwen3.5-397B-A17B and there's a lot to unpack. 397B params, 17B active per forward pass. Sparse MoE done right. But the real story isn't the size—it's the architecture choices. The MoE Design Most MoE models feel like bolt-ons. Qwen 3.5's sparse activation is native—only 4.3% of parameters fire per token. That's how you get trillion-parameter-class performance without trillion-parameter inference costs. The 0.8 RMB/million tokens pricing isn't subsidized; it's structurally earned. Native Multimodal, Not Glued-On This is a vision-language model from the ground up. Heterogeneous architecture—separate processing pipelines for text, image, video that fuse early. Not a vision encoder slapped onto an LLM. The result: 90.8 on OmniDocBench, 79.0 on MMMU-Pro. Document understanding and visual reasoning without the usual brittleness. The Context Window Reality Qwen3.5-Plus (the hosted version) ships with 1M tokens by default. That's not a marketing number—they're actually positioning it for long-document workflows. With built-in adaptive tool use, it's clearly aimed at agentic automation, not just chat. What Actually Impressed Me • FP8 native pipeline: ~50% activation memory reduction • Async RL framework for continuous refinement—training and inference workloads separated • 201 languages (up from 119), 250k vocab for better low-resource encoding • Apache 2.0 license. Full weights on HuggingFace and ModelScope. The Benchmark Context 76.4 on SWE-bench Verified puts it in the range where it can handle real debugging workflows. 72.9 on BFCL v4 for agentic tool use. 88.4 on GPQA Diamond. These aren't SOTA in isolation, but the breadth is unusual—strong across reasoning, coding, multimodal, and agentic tasks. The Honest Caveat I haven't stress-tested the 1M context for needle-in-haystack retrieval yet. And "native multimodal" claims need real-world torture testing—PDFs with tables, charts, mixed layouts. Benchmarks are benchmarks. Bottom Line This isn't just another model release. It's a bet on efficient scale: big model capabilities, small active compute, open weights. At 1/18th the cost of Gemini 3 Pro, it's going to force pricing conversations across the board.

Bo Wang

13,221 views • 5 months ago

Everyone is sleeping on Meta's SAM 3 release. But it's actually a big deal. Here's why: Companies spend millions paying humans to label images and videos frame by frame. A single autonomous driving dataset? Months of work, hundreds of annotators, millions in cost. Without labeled data, you can't train custom models. Without custom models, you're stuck with generic solutions. This is why most companies never move past pilots. SAM 3 breaks this cycle. First let's look at the evolution: SAM 1 segmented objects when you clicked on them. Revolutionary, but one object at a time. SAM 2 added video tracking with memory. Game-changing, but you still manually prompted every object. SAM 3 changes everything with text prompts. Type "yellow school bus" and it finds ALL of them in your image or video. Not just one. Every instance across thousands of frames. Now here's where people get confused: "Can't I just use GPT-5 or Gemini for this?" No, and here's why that's a terrible approach. Large multimodal LLMs are great for reasoning, but they're slow and expensive for production visual tasks. You're paying API costs per image, waiting seconds for responses, getting inconsistent results. SAM 3 runs in 30 milliseconds on a single GPU for 100+ objects. That's 100x faster, and you own the infrastructure. More importantly, SAM 3 gives you precise pixel-level masks, not descriptions. Try asking an LLM to segment every defective part on a manufacturing line in real-time. It won't work. SAM 3 does this effortlessly. The real breakthrough is their data engine. Meta built an AI-human hybrid system that's 5x faster for complex annotations. They trained SAM 3 on 4 million unique visual concepts - 50x more than existing benchmarks like LVIS. SAM 3 is trained on 4 million unique visual concepts, it handles everything: - Text-based concept search - Interactive refinement with clicks - Video tracking across frames - Zero-shot detection of new concepts The model is open source. Weights, code, and benchmarks are on GitHub. If you're building computer vision applications, this is the foundation model to evaluate. The annotation time savings alone will pay for integration costs within weeks. Find the relevant links in the next tweet!

Everyone is sleeping on Meta's SAM 3 release. But it's actually a big deal. Here's why: Companies spend millions paying humans to label images and videos frame by frame. A single autonomous driving dataset? Months of work, hundreds of annotators, millions in cost. Without labeled data, you can't train custom models. Without custom models, you're stuck with generic solutions. This is why most companies never move past pilots. SAM 3 breaks this cycle. First let's look at the evolution: SAM 1 segmented objects when you clicked on them. Revolutionary, but one object at a time. SAM 2 added video tracking with memory. Game-changing, but you still manually prompted every object. SAM 3 changes everything with text prompts. Type "yellow school bus" and it finds ALL of them in your image or video. Not just one. Every instance across thousands of frames. Now here's where people get confused: "Can't I just use GPT-5 or Gemini for this?" No, and here's why that's a terrible approach. Large multimodal LLMs are great for reasoning, but they're slow and expensive for production visual tasks. You're paying API costs per image, waiting seconds for responses, getting inconsistent results. SAM 3 runs in 30 milliseconds on a single GPU for 100+ objects. That's 100x faster, and you own the infrastructure. More importantly, SAM 3 gives you precise pixel-level masks, not descriptions. Try asking an LLM to segment every defective part on a manufacturing line in real-time. It won't work. SAM 3 does this effortlessly. The real breakthrough is their data engine. Meta built an AI-human hybrid system that's 5x faster for complex annotations. They trained SAM 3 on 4 million unique visual concepts - 50x more than existing benchmarks like LVIS. SAM 3 is trained on 4 million unique visual concepts, it handles everything: - Text-based concept search - Interactive refinement with clicks - Video tracking across frames - Zero-shot detection of new concepts The model is open source. Weights, code, and benchmarks are on GitHub. If you're building computer vision applications, this is the foundation model to evaluate. The annotation time savings alone will pay for integration costs within weeks. Find the relevant links in the next tweet!

Akshay 🚀

46,404 views • 8 months ago

new chapter begins: a terminal for the agentic future, built on blockchain, powered by AI. This is our marketplace—a glimpse of what AGI will mean for crypto. Today, we launch 3 agents—Image Generation, Token Swap Agent, & Blockchain Tax Estimate Agent—out of hundreds to come. We see an agentic future where AI guides every step: buying online, managing finances, transacting globally. FOMO’s here to make that real, with experts at your side. Our Model Context Protocol (MCP) ties it together—agents talking, reasoning, scaling across crypto and DeFi. It’s orchestration with a brain, evolving daily. FOMO’s not just building tools; we’re pushing intelligent automation into blockchain’s core. Our Model Context Protocol (MCP) is the backbone. Think of it as a conductor for AI agents—each runs its own logic (workflows, API calls, LLMs), but MCP syncs them on-chain. Agents share context via a lightweight event bus, logged to a blockchain ledger. Agent A (say, Market Analysis) pulls stock data, flags trends. Agent B (Email Sales) reads that, drafts outreach—both talk through MCP’s orchestration layer. We use Web3 hooks to settle fees or split revenue, all transparent. It’s messy, but it scales. Under the hood: MCP leans on a pub-sub model—agents publish tasks, others subscribe. We’re training them with RL loops to optimize gas costs and response times. Goal? A self-tuning swarm of agents reasoning over DeFi, NFTs, whatever’s next. This is FOMO’s bet on AGI. Welcome to the new FOMO. We’re not just building tools—we’re wiring AI into crypto’s future, agent by agent. A leader in blockchain intelligence, starting here. Join us as we push the boundaries.

new chapter begins: a terminal for the agentic future, built on blockchain, powered by AI. This is our marketplace—a glimpse of what AGI will mean for crypto. Today, we launch 3 agents—Image Generation, Token Swap Agent, & Blockchain Tax Estimate Agent—out of hundreds to come. We see an agentic future where AI guides every step: buying online, managing finances, transacting globally. FOMO’s here to make that real, with experts at your side. Our Model Context Protocol (MCP) ties it together—agents talking, reasoning, scaling across crypto and DeFi. It’s orchestration with a brain, evolving daily. FOMO’s not just building tools; we’re pushing intelligent automation into blockchain’s core. Our Model Context Protocol (MCP) is the backbone. Think of it as a conductor for AI agents—each runs its own logic (workflows, API calls, LLMs), but MCP syncs them on-chain. Agents share context via a lightweight event bus, logged to a blockchain ledger. Agent A (say, Market Analysis) pulls stock data, flags trends. Agent B (Email Sales) reads that, drafts outreach—both talk through MCP’s orchestration layer. We use Web3 hooks to settle fees or split revenue, all transparent. It’s messy, but it scales. Under the hood: MCP leans on a pub-sub model—agents publish tasks, others subscribe. We’re training them with RL loops to optimize gas costs and response times. Goal? A self-tuning swarm of agents reasoning over DeFi, NFTs, whatever’s next. This is FOMO’s bet on AGI. Welcome to the new FOMO. We’re not just building tools—we’re wiring AI into crypto’s future, agent by agent. A leader in blockchain intelligence, starting here. Join us as we push the boundaries.

FOMO

26,338 views • 1 year ago