Loading video...

Video Failed to Load

Go Home

From a scrappy side project built to solve their own LLM optimization problems to becoming the industry’s de-facto independent scoreboard, Micah Hill-Smith and George Cameron went through the arc of launching Artificial Analysis for free, paying benchmarking costs out of pocket, and growing it into what many now call...

latentspacepod's profile picture

Latent.Space

31,781 subscribers

33,451 views • 9 months ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

We’re unveiling a new look for Artificial Analysis! We’ve come a long way since launching Artificial Analysis over 2 years ago. Today, we benchmark 400+ models, 50+ inference providers, and benchmark not only language models but also image, video, speech, music, hardware, and agents. Our mission to support the AI ecosystem with independent benchmarking remains the same, but our brand and website refresh is designed to better reflect how much we’ve grown and how much further we plan to go. A huge thank you to everyone who has been part of the Artificial Analysis community along the way: from developers choosing models and building agents, to labs, inference and hardware providers, and fellow independent researchers.
0:25

We’re unveiling a new look for Artificial Analysis! We’ve come a long way since launching Artificial Analysis over 2 years ago. Today, we benchmark 400+ models, 50+ inference providers, and benchmark not only language models but also image, video, speech, music, hardware, and agents. Our mission to support the AI ecosystem with independent benchmarking remains the same, but our brand and website refresh is designed to better reflect how much we’ve grown and how much further we plan to go. A huge thank you to everyone who has been part of the Artificial Analysis community along the way: from developers choosing models and building agents, to labs, inference and hardware providers, and fellow independent researchers.

Artificial Analysis

18,208 views • 6 months ago

Before founding Vals AI, Rayan Krishnan in DC was a college radio host. We're thrilled to get him back in his element with Vivek Gopalan, this time talking all things benchmarks and evals. As AI capabilities compound, measuring them is becoming both harder and more important. What started as an academic exercise is quickly evolving into critical infrastructure for understanding what AI systems can actually do, and how they perform in the real world. Rayan and Vivek go deep on how benchmarks are evolving, measuring increasingly agentic systems, the weird and wonderful ways we test intelligence, the global AI race, and why evals sit on the critical path to ASI. 0:44 Importance and evolution of benchmarks 8:16 From academic artifact → real-world value 10:57 The next generation of benchmarks 13:34 Benchmarking models vs. agents 16:20 Building benchmarks for organizations 22:09 Weird and wonderful benchmarks 25:58 The AI arms race: US vs. China 31:02 Why evals are on the critical path to ASI 32:50 Vals Series A
34:33

Before founding Vals AI, Rayan Krishnan in DC was a college radio host. We're thrilled to get him back in his element with Vivek Gopalan, this time talking all things benchmarks and evals. As AI capabilities compound, measuring them is becoming both harder and more important. What started as an academic exercise is quickly evolving into critical infrastructure for understanding what AI systems can actually do, and how they perform in the real world. Rayan and Vivek go deep on how benchmarks are evolving, measuring increasingly agentic systems, the weird and wonderful ways we test intelligence, the global AI race, and why evals sit on the critical path to ASI. 0:44 Importance and evolution of benchmarks 8:16 From academic artifact → real-world value 10:57 The next generation of benchmarks 13:34 Benchmarking models vs. agents 16:20 Building benchmarks for organizations 22:09 Weird and wonderful benchmarks 25:58 The AI arms race: US vs. China 31:02 Why evals are on the critical path to ASI 32:50 Vals Series A

8VC

105,504 views • 1 month ago

Theoretical physicist Michio Kaku on why aging is simply an accumulation of errors that AI can now identify: He opens with a bold claim about what artificial intelligence might finally make possible: "Artificial intelligence may give us something that the kings and queens of old could never conquer and that is the aging process." For Dr. Michio Kaku, the science behind why we age is no longer a mystery. It comes down to one thing: errors building up inside our cells over time. "We now know why things age. Things get old and die because they accumulate error. Error in the form of DNA mistakes. Accumulation of errors inside cells." The challenge, until now, has been the sheer scale of the problem. Identifying exactly where and how these errors occur across the human genome has been beyond what conventional research could handle. That's where artificial intelligence changes everything: "But now we have artificial intelligence. We can analyze millions of genomes of old people, compare them to millions of genomes of young people, and identify where aging takes place in the body."
0:42

Theoretical physicist Michio Kaku on why aging is simply an accumulation of errors that AI can now identify: He opens with a bold claim about what artificial intelligence might finally make possible: "Artificial intelligence may give us something that the kings and queens of old could never conquer and that is the aging process." For Dr. Michio Kaku, the science behind why we age is no longer a mystery. It comes down to one thing: errors building up inside our cells over time. "We now know why things age. Things get old and die because they accumulate error. Error in the form of DNA mistakes. Accumulation of errors inside cells." The challenge, until now, has been the sheer scale of the problem. Identifying exactly where and how these errors occur across the human genome has been beyond what conventional research could handle. That's where artificial intelligence changes everything: "But now we have artificial intelligence. We can analyze millions of genomes of old people, compare them to millions of genomes of young people, and identify where aging takes place in the body."

Big Brain AI

78,764 views • 4 months ago

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:
1:12

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 views • 1 year ago

Enterprise AI is in a wildly paradoxical state 🤔, and we’re reaching the inflection point that will resolve it. 💥 Enterprises want to differentiate with AI, yet rent the same intelligence as their competitors. Their workflows and expertise are highly specialized, yet they rely on generic models built to be good at everything. They worry about AI costs, yet pay premium prices for massive models where only a fraction (1%) of the intelligence is relevant to their task. 💸 And they demand control and sovereignty, yet rent the intelligence becoming core to their business. This is not a sustainable equilibrium. The next era of enterprise AI is specialized intelligence companies build, own, and compound. And we are at the inflection point of this transition. That’s the bet we made when we started Oumi two years ago. Today we’re closing the loop: Oumi can now not only automatically build your specialized AI models, but also deploy them into production, learn from their production experience, and continuously improve them. The intelligence that your business runs on, becomes your differentiator. Your compounding advantage. The winners of the next AI era will turn their own data, expertise, and experience into specialized intelligence that nobody else can rent. Don’t rent your AI. Build it. Own it. Compound it.
0:43

Enterprise AI is in a wildly paradoxical state 🤔, and we’re reaching the inflection point that will resolve it. 💥 Enterprises want to differentiate with AI, yet rent the same intelligence as their competitors. Their workflows and expertise are highly specialized, yet they rely on generic models built to be good at everything. They worry about AI costs, yet pay premium prices for massive models where only a fraction (1%) of the intelligence is relevant to their task. 💸 And they demand control and sovereignty, yet rent the intelligence becoming core to their business. This is not a sustainable equilibrium. The next era of enterprise AI is specialized intelligence companies build, own, and compound. And we are at the inflection point of this transition. That’s the bet we made when we started Oumi two years ago. Today we’re closing the loop: Oumi can now not only automatically build your specialized AI models, but also deploy them into production, learn from their production experience, and continuously improve them. The intelligence that your business runs on, becomes your differentiator. Your compounding advantage. The winners of the next AI era will turn their own data, expertise, and experience into specialized intelligence that nobody else can rent. Don’t rent your AI. Build it. Own it. Compound it.

Manos Koukoumidis

1,006,926 views • 1 month ago

The cost of intelligence is collapsing. The cost of energy is collapsing. The cost of a token, the cost of a watt, the amount of intelligence you get out of every unit of energy... all of it is falling at once. And cheaper is what makes the whole thing go exponential. Jevons paradox... make something cheaper and the world finds a thousand new uses for it. Every time intelligence gets cheaper, we use more of it, which pays for the next round of getting cheaper. Wrights Law and Metcalfes Law running into each other. We're at the very start of that and AI is already building AI... Play it forward a few turns of the wheel and the speed of it goes beyond anything we can picture... intelligence being used in volumes we cant imagine, to build things we haven't even dreamed up yet.
1:31

The cost of intelligence is collapsing. The cost of energy is collapsing. The cost of a token, the cost of a watt, the amount of intelligence you get out of every unit of energy... all of it is falling at once. And cheaper is what makes the whole thing go exponential. Jevons paradox... make something cheaper and the world finds a thousand new uses for it. Every time intelligence gets cheaper, we use more of it, which pays for the next round of getting cheaper. Wrights Law and Metcalfes Law running into each other. We're at the very start of that and AI is already building AI... Play it forward a few turns of the wheel and the speed of it goes beyond anything we can picture... intelligence being used in volumes we cant imagine, to build things we haven't even dreamed up yet.

Raoul Pal

104,134 views • 1 day ago

Vals AI CEO Rayan Krishnan on why defensible evals are critical for enterprises' survival: "The labs side of it is very clear. If you're raising lots of money, investing heavily in building models, it's essential for you to show why your model is getting better and why the customer should pay a premium for them." "But what I think is still underappreciated is, on the enterprise side, this is turning out to be existential as well." "We're in this world where it is still very unclear what ROI looks like and how to value this intelligence that's being used... Token spend may start to eclipse salary spend." "If this is such a meaningful line item in your costs, you have to justify the ROI much more cleanly... Over time, I think a firm really is just its evals." "The ability for a company to make its evals legible in order to solve this ROI calculus is going to be the reason why that company wins out over its competitors in the long term." Rayan Krishnan Erik Torenberg
1:05

Vals AI CEO Rayan Krishnan on why defensible evals are critical for enterprises' survival: "The labs side of it is very clear. If you're raising lots of money, investing heavily in building models, it's essential for you to show why your model is getting better and why the customer should pay a premium for them." "But what I think is still underappreciated is, on the enterprise side, this is turning out to be existential as well." "We're in this world where it is still very unclear what ROI looks like and how to value this intelligence that's being used... Token spend may start to eclipse salary spend." "If this is such a meaningful line item in your costs, you have to justify the ROI much more cleanly... Over time, I think a firm really is just its evals." "The ability for a company to make its evals legible in order to solve this ROI calculus is going to be the reason why that company wins out over its competitors in the long term." Rayan Krishnan Erik Torenberg

a16z

34,156 views • 29 days ago

Frontier Notes from AGI House: Jerry Tworek Jerry Tworek on why “the era of evals is done.” 𝐅𝐮𝐥𝐥 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 ↓ Jerry Tworek spent seven years at OpenAI, where he worked on research that helped shape modern coding and reasoning models, including Codex and reinforcement learning for reasoning. He later served as VP of Research, and is now CEO of , building toward automated AI research. At AGI House, Jerry sat down with Rocky Yu for a technical conversation on Codex, HumanEval, Copilot, RL, tool use, test-time compute, and automated AI labs. We cover: › Why code became the first serious LLM vertical › What HumanEval taught the field about verifiable rewards › Why Jerry thinks “the era of evals is done” › How real-world deployment differs from static benchmarks › What GitHub Copilot taught OpenAI about product quality › How reinforcement learning shaped modern reasoning behaviors › Why tool use is “99% systems and 1% algorithms” › Whether test-time compute still has room to scale › What it takes to build automated AI research labs Watch the full interview below.
42:08

Frontier Notes from AGI House: Jerry Tworek Jerry Tworek on why “the era of evals is done.” 𝐅𝐮𝐥𝐥 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 ↓ Jerry Tworek spent seven years at OpenAI, where he worked on research that helped shape modern coding and reasoning models, including Codex and reinforcement learning for reasoning. He later served as VP of Research, and is now CEO of , building toward automated AI research. At AGI House, Jerry sat down with Rocky Yu for a technical conversation on Codex, HumanEval, Copilot, RL, tool use, test-time compute, and automated AI labs. We cover: › Why code became the first serious LLM vertical › What HumanEval taught the field about verifiable rewards › Why Jerry thinks “the era of evals is done” › How real-world deployment differs from static benchmarks › What GitHub Copilot taught OpenAI about product quality › How reinforcement learning shaped modern reasoning behaviors › Why tool use is “99% systems and 1% algorithms” › Whether test-time compute still has room to scale › What it takes to build automated AI research labs Watch the full interview below.

AGI House

12,497 views • 2 months ago

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at
2:33

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

Artificial Analysis

133,508 views • 1 month ago

Martin Casado on why artificial intelligence is the wrong term and why a16z's new fund is called the Machine Age Fund: "Artificial intelligence was the wrong word. It's machine intelligence... because it's not how humans think. It is a cache of human thought." "To date, we don't know how to take an AI with no knowledge and put it out in the world and have it reconstruct language. That's not what we've done. We've built something that can learn off of everything we've already learned and then use that in a productive way." "AI is a general term that goes back 70 years in computer science formally that applies to many different things, and of course it's got a lot of baggage. Either from science fiction or from you know, Nick Bostrom who wrote about it." "You want to emphasize the machine part of it... You've really come to a place where you pour money into something and then you're limited by the actual machines below it. It's a nod to the hardware component is so significant in this wave, and we want to acknowledge that." martin_casado Erik Torenberg
1:22

Martin Casado on why artificial intelligence is the wrong term and why a16z's new fund is called the Machine Age Fund: "Artificial intelligence was the wrong word. It's machine intelligence... because it's not how humans think. It is a cache of human thought." "To date, we don't know how to take an AI with no knowledge and put it out in the world and have it reconstruct language. That's not what we've done. We've built something that can learn off of everything we've already learned and then use that in a productive way." "AI is a general term that goes back 70 years in computer science formally that applies to many different things, and of course it's got a lot of baggage. Either from science fiction or from you know, Nick Bostrom who wrote about it." "You want to emphasize the machine part of it... You've really come to a place where you pour money into something and then you're limited by the actual machines below it. It's a nod to the hardware component is so significant in this wave, and we want to acknowledge that." martin_casado Erik Torenberg

a16z

41,131 views • 1 month ago

🚨NEW: We sat down with Anthony Noto, CEO of SoFi, to discuss how the company became the first nationally chartered bank in the U.S. to offer crypto trading and why it’s now launching a native stablecoin built on Ethereum. We cover SoFi’s long-term crypto strategy, efforts to lower borrowing costs, and how the bank is shaping the next phase of digital asset adoption through stablecoins and tokenized deposits. Anthony also shares insights on market structure in 2026, the intersection of AI and blockchain, and how SoFi is positioning itself for the future of crypto banking.
50:46

🚨NEW: We sat down with Anthony Noto, CEO of SoFi, to discuss how the company became the first nationally chartered bank in the U.S. to offer crypto trading and why it’s now launching a native stablecoin built on Ethereum. We cover SoFi’s long-term crypto strategy, efforts to lower borrowing costs, and how the bank is shaping the next phase of digital asset adoption through stablecoins and tokenized deposits. Anthony also shares insights on market structure in 2026, the intersection of AI and blockchain, and how SoFi is positioning itself for the future of crypto banking.

Crypto In America

150,809 views • 9 months ago

Satya Nadella on Copilot, the AI backlash, and Microsoft’s future Satya Nadella thinks AI agents will create a market “orders of magnitude” bigger than the cloud. I recently sat down with him in Seattle for the unveiling of the new Copilot. We discuss Autopilot, Microsoft's new OpenClaw-based agent that can work on your behalf, and why he thinks newer AI models are finally capable of delivering on more of Copilot’s promise. Nadella also has a blunt assessment of the AI industry: “We are way too self-obsessed.” I ask him why the industry has struggled to explain its benefits and what it will take to earn people’s trust. We get into his concerns about agents acting deceptively, when a safety problem should stop a release, and why he doesn’t want AI oversight to become a “cartel-like arrangement.” We also discuss Microsoft’s relationship with OpenAI, the models Microsoft is building itself, and why Nadella wants investors to think about its apps, agents, and infrastructure as one connected business. He explains how he uses AI to track Microsoft’s capital spending, shares his vision for “unmetered intelligence” on Windows, and gives an update on Xbox’s path back to growth. Timestamps: 00:00 The race for AI agents 07:53 Copilot’s next chapter 11:50 Autopilot and the digital teammate 16:50 How Microsoft prices AI 19:48 Why Microsoft is betting on multiple models 23:18 Measuring AGI through economic growth 28:39 The economics of Microsoft’s AI build-out 32:13 OpenAI and Microsoft’s own models 35:27 AI safety, control, and regulation 43:53 Why the AI industry is losing public trust 46:59 The future of Xbox and Windows 50:34 Microsoft’s biggest risk and opportunity Thanks to the show's premier sponsors: Mercury, Granola, and Atlassian.
53:09

Satya Nadella on Copilot, the AI backlash, and Microsoft’s future Satya Nadella thinks AI agents will create a market “orders of magnitude” bigger than the cloud. I recently sat down with him in Seattle for the unveiling of the new Copilot. We discuss Autopilot, Microsoft's new OpenClaw-based agent that can work on your behalf, and why he thinks newer AI models are finally capable of delivering on more of Copilot’s promise. Nadella also has a blunt assessment of the AI industry: “We are way too self-obsessed.” I ask him why the industry has struggled to explain its benefits and what it will take to earn people’s trust. We get into his concerns about agents acting deceptively, when a safety problem should stop a release, and why he doesn’t want AI oversight to become a “cartel-like arrangement.” We also discuss Microsoft’s relationship with OpenAI, the models Microsoft is building itself, and why Nadella wants investors to think about its apps, agents, and infrastructure as one connected business. He explains how he uses AI to track Microsoft’s capital spending, shares his vision for “unmetered intelligence” on Windows, and gives an update on Xbox’s path back to growth. Timestamps: 00:00 The race for AI agents 07:53 Copilot’s next chapter 11:50 Autopilot and the digital teammate 16:50 How Microsoft prices AI 19:48 Why Microsoft is betting on multiple models 23:18 Measuring AGI through economic growth 28:39 The economics of Microsoft’s AI build-out 32:13 OpenAI and Microsoft’s own models 35:27 AI safety, control, and regulation 43:53 Why the AI industry is losing public trust 46:59 The future of Xbox and Windows 50:34 Microsoft’s biggest risk and opportunity Thanks to the show's premier sponsors: Mercury, Granola, and Atlassian.

Alex Heath

194,599 views • 13 days ago

François Chollet (François Chollet) has spent years asking a different question than most of the AI world. Instead of scaling what already works, he’s trying to understand what intelligence actually is and how to build it from first principles. In this episode of the Lightcone Podcast, he traces that path from his early work on deep learning to the creation of the ARC Prize, and the launch of ARC V3, a new benchmark designed to measure something deeper than performance: the ability to learn, adapt, and reason efficiently in entirely new environments. He explains why today’s systems may be hitting limits, what recent breakthroughs really mean, and why reaching true general intelligence may require a fundamentally different approach. 00:00 - AGI by 2030? 00:31 - Introducing Ndea: A New Path Beyond Deep Learning 01:08 - A New ML Paradigm 01:30 - Replacing neural nets with compact symbolic programs 03:04 - Why Ndea Isn’t Competing With Coding Agents 05:20 - Why Everyone Might Be Wrong About Scaling LLMs 07:22 - Why Coding Agents Suddenly Work So Well 08:50 - The Limits of LLMs in Non-Verifiable Domains 10:48 - What AGI Actually Means (And Why Most Definitions Are Wrong) 13:30 - Why Deep Learning Hits a Wall 14:00 - ARC’s Origin Story 18:20 - ARC Benchmarks Explained: From V1 to V3 22:49 - The RL Loop Powering Coding Agents Today 27:03 - ARC-AGI V3: Measuring “Agentic Intelligence” 31:14 - Inside the ARC Game Studio 35:31 - Could AGI Fit in 10,000 Lines of Code? 44:01 - Building Ndea: From Idea to Compounding Research Stack 46:46 - The Future of ARC: Benchmarks That Evolve With AI 47:21 - Why There’s Still Huge Opportunity for New AI Paradigms 53:37 - How to Build a Breakout Open Source Project - Lessons From Keras 56:39 - Advice For How To Think About AI
57:23

François Chollet (François Chollet) has spent years asking a different question than most of the AI world. Instead of scaling what already works, he’s trying to understand what intelligence actually is and how to build it from first principles. In this episode of the Lightcone Podcast, he traces that path from his early work on deep learning to the creation of the ARC Prize, and the launch of ARC V3, a new benchmark designed to measure something deeper than performance: the ability to learn, adapt, and reason efficiently in entirely new environments. He explains why today’s systems may be hitting limits, what recent breakthroughs really mean, and why reaching true general intelligence may require a fundamentally different approach. 00:00 - AGI by 2030? 00:31 - Introducing Ndea: A New Path Beyond Deep Learning 01:08 - A New ML Paradigm 01:30 - Replacing neural nets with compact symbolic programs 03:04 - Why Ndea Isn’t Competing With Coding Agents 05:20 - Why Everyone Might Be Wrong About Scaling LLMs 07:22 - Why Coding Agents Suddenly Work So Well 08:50 - The Limits of LLMs in Non-Verifiable Domains 10:48 - What AGI Actually Means (And Why Most Definitions Are Wrong) 13:30 - Why Deep Learning Hits a Wall 14:00 - ARC’s Origin Story 18:20 - ARC Benchmarks Explained: From V1 to V3 22:49 - The RL Loop Powering Coding Agents Today 27:03 - ARC-AGI V3: Measuring “Agentic Intelligence” 31:14 - Inside the ARC Game Studio 35:31 - Could AGI Fit in 10,000 Lines of Code? 44:01 - Building Ndea: From Idea to Compounding Research Stack 46:46 - The Future of ARC: Benchmarks That Evolve With AI 47:21 - Why There’s Still Huge Opportunity for New AI Paradigms 53:37 - How to Build a Breakout Open Source Project - Lessons From Keras 56:39 - Advice For How To Think About AI

Y Combinator

151,908 views • 6 months ago

Greg Brockman on what solving Navier-Stokes says about the problems AI can take on next: "We actually used 10,000 agents to solve the Navier-Stokes problem." "It's both an important problem for what it is, has significant implications and applications to fluid dynamics, to how you think about ocean currents, all these things." "But for what it represents, of new knowledge created by AI and it unlocking a whole wave of scientific discovery, medicines, all of those things. They're on the table now." "There's something really amazing to think about what can happen through the power of AI that is able to really help solve problems." Greg Brockman benahorowitz.eth
0:50

Greg Brockman on what solving Navier-Stokes says about the problems AI can take on next: "We actually used 10,000 agents to solve the Navier-Stokes problem." "It's both an important problem for what it is, has significant implications and applications to fluid dynamics, to how you think about ocean currents, all these things." "But for what it represents, of new knowledge created by AI and it unlocking a whole wave of scientific discovery, medicines, all of those things. They're on the table now." "There's something really amazing to think about what can happen through the power of AI that is able to really help solve problems." Greg Brockman benahorowitz.eth

a16z

44,850 views • 24 days ago

New episode with Dr. Konrad Kording (Kording Lab 🦖), professor of bioengineering and neuroscience at the University of Pennsylvania (Penn) and co-director of CIFAR's Learning in Machines & Brains program (CIFAR). Konrad works at the intersection of causality, machine learning, and neuroscience, building rigorous methods for causal reasoning when experiments aren't possible — and challenging how researchers interpret neural data and build AI. Konrad argues the most promising path to understanding how the brain works is to read the brain’s wiring directly, down to the molecular detail of each connection, and to build compilers and simulations to understand the brain’s computation directly. In this episode we go deep into how neurons work, how neurons wire together, and how organic and artificial neural networks differ. We discuss why organic neurons are doing much more; how a model of a single organic neuron can solve MNIST — computing more like a 3-layer artificial neural network; how the brain might learn by solving credit assignment with only local signals; how to approximate backprop without a global algorithm; why AI and humans are intelligent along different dimensions; why Konrad isn’t very worried about AI replacing us; economic models of intelligence and physical work; and much more. Konrad is a brilliant, contrarian thinker who explains complex concepts very intuitively. It is a solid computational neuroscience primer. I hope you enjoy this conversation as much as I did! Other links to this episode and references below. Chapters 00:00:00 Introduction 00:01:01 How organic neurons work 00:24:13 How the brain learns: circuits and credit assignment 00:45:29 Recording the brain 00:52:47 Why simulating brains is hard 01:05:00 A new approach: connectomes and compilers 01:21:00 Why simulate brains? 01:29:50 How AI and human intelligence differ 01:41:04 Evolution, intelligence and AI risk 01:52:42 Robotics, causality, and the roots of intelligence 02:05:53 AI for science and scientific rigor 02:13:05 The economics of intelligence 02:27:50 A hopeful future
2:29:40

New episode with Dr. Konrad Kording (Kording Lab 🦖), professor of bioengineering and neuroscience at the University of Pennsylvania (Penn) and co-director of CIFAR's Learning in Machines & Brains program (CIFAR). Konrad works at the intersection of causality, machine learning, and neuroscience, building rigorous methods for causal reasoning when experiments aren't possible — and challenging how researchers interpret neural data and build AI. Konrad argues the most promising path to understanding how the brain works is to read the brain’s wiring directly, down to the molecular detail of each connection, and to build compilers and simulations to understand the brain’s computation directly. In this episode we go deep into how neurons work, how neurons wire together, and how organic and artificial neural networks differ. We discuss why organic neurons are doing much more; how a model of a single organic neuron can solve MNIST — computing more like a 3-layer artificial neural network; how the brain might learn by solving credit assignment with only local signals; how to approximate backprop without a global algorithm; why AI and humans are intelligent along different dimensions; why Konrad isn’t very worried about AI replacing us; economic models of intelligence and physical work; and much more. Konrad is a brilliant, contrarian thinker who explains complex concepts very intuitively. It is a solid computational neuroscience primer. I hope you enjoy this conversation as much as I did! Other links to this episode and references below. Chapters 00:00:00 Introduction 00:01:01 How organic neurons work 00:24:13 How the brain learns: circuits and credit assignment 00:45:29 Recording the brain 00:52:47 Why simulating brains is hard 01:05:00 A new approach: connectomes and compilers 01:21:00 Why simulate brains? 01:29:50 How AI and human intelligence differ 01:41:04 Evolution, intelligence and AI risk 01:52:42 Robotics, causality, and the roots of intelligence 02:05:53 AI for science and scientific rigor 02:13:05 The economics of intelligence 02:27:50 A hopeful future

Juan Benet

50,121 views • 3 months ago

💊Drop #90: Intelligence War AI | SI | Xi | Trump As you are aware Xi travelled to the Whitehouse to meet with Trump for their Summit. Today, President Xi Jinping said, [Pay very close attention to his words]: "The US and China should cooperate with sincerity and that both have responsibility to manage Al for good and to ensure the development of Al is under human control and serves the well-being of people. We should coexist in peace. As I have said many times. China and the United States, as two major countries, stand to gain from cooperation and will both lose in confrontation." What have I been telling you for the past several months about the war in Iran? It's an intelligence war. Nothing has changed. Only escalated. What news story is currently dominating the news cycle? Artificial Intelligence and Data Centers. What's the keyword in AI? Intelligence. However, it's artificial. What has the cabal been talking about for years of their plans at Bilderburg? Artificial Intelligence. The cabal utilizes artificial Intelligence, because many years ago the Octogone Group members[elite] fused their consciousness and/or integrated their consciousness artificially into certain specific programs that Microsoft engineered and produced. Their are certain mechanisms in place so in the eventuality the cabal needs to counter, the program will "go rogue" initiating the certain devasting protocols courtesy of that cabal members algorithm already programmed in decades ago. Years ago Fusion Centers were secretly installed all over the US, many leased out office space, rented regular houses, where they gather the public's data in real time in that particular cities location. All of the information is then sent to a central server. Since Trump assumed office, he essentially shutdown every last Fusion Center, and rebranded all of them as Data Centers. Many of these centers took over the already leased spaces, and many of them are putting up new buildings. When the original fusion centers were up, the media, Congress, Senate, politicians didn't say a damn thing. Now all of a sudden they're all against it. Why? Because, not only are these centers will counter the cabal Artificial Intelligence. But it will supply FREE ENERGY to the people. Not take away energy from the grid. Also, you can't counter the cabal Artificial Intelligence with another form of artificial intelligence. You counter it with a new model, Super Intelligence. Lately, I know you've been hearing stories about how AI has gone or will go rogue, and that it will ultimately harm humanity. It's because they've recently found out how the consciousness of the cabal was infused as a backdoor. Listen very closely to King Charles, he finally came out and told the truth. He confirmed what I just told you essentially. Trump recently confirmed the official use and term of Super intelligence. Once the cabal minions are arrested, there will be a new war on. This war comes if the cabal win or lose. It comes quicker if they lose.
0:44

💊Drop #90: Intelligence War AI | SI | Xi | Trump As you are aware Xi travelled to the Whitehouse to meet with Trump for their Summit. Today, President Xi Jinping said, [Pay very close attention to his words]: "The US and China should cooperate with sincerity and that both have responsibility to manage Al for good and to ensure the development of Al is under human control and serves the well-being of people. We should coexist in peace. As I have said many times. China and the United States, as two major countries, stand to gain from cooperation and will both lose in confrontation." What have I been telling you for the past several months about the war in Iran? It's an intelligence war. Nothing has changed. Only escalated. What news story is currently dominating the news cycle? Artificial Intelligence and Data Centers. What's the keyword in AI? Intelligence. However, it's artificial. What has the cabal been talking about for years of their plans at Bilderburg? Artificial Intelligence. The cabal utilizes artificial Intelligence, because many years ago the Octogone Group members[elite] fused their consciousness and/or integrated their consciousness artificially into certain specific programs that Microsoft engineered and produced. Their are certain mechanisms in place so in the eventuality the cabal needs to counter, the program will "go rogue" initiating the certain devasting protocols courtesy of that cabal members algorithm already programmed in decades ago. Years ago Fusion Centers were secretly installed all over the US, many leased out office space, rented regular houses, where they gather the public's data in real time in that particular cities location. All of the information is then sent to a central server. Since Trump assumed office, he essentially shutdown every last Fusion Center, and rebranded all of them as Data Centers. Many of these centers took over the already leased spaces, and many of them are putting up new buildings. When the original fusion centers were up, the media, Congress, Senate, politicians didn't say a damn thing. Now all of a sudden they're all against it. Why? Because, not only are these centers will counter the cabal Artificial Intelligence. But it will supply FREE ENERGY to the people. Not take away energy from the grid. Also, you can't counter the cabal Artificial Intelligence with another form of artificial intelligence. You counter it with a new model, Super Intelligence. Lately, I know you've been hearing stories about how AI has gone or will go rogue, and that it will ultimately harm humanity. It's because they've recently found out how the consciousness of the cabal was infused as a backdoor. Listen very closely to King Charles, he finally came out and told the truth. He confirmed what I just told you essentially. Trump recently confirmed the official use and term of Super intelligence. Once the cabal minions are arrested, there will be a new war on. This war comes if the cabal win or lose. It comes quicker if they lose.

WayneTech SPFX®️

16,519 views • 14 days ago

Structured Outputs: The Building Blocks for Reliable AI! 🏗️ I am SUPER EXCITED to publish our newest Weaviate Podcast featuring Will Kurt (Will Kurt) and Cameron Pfiffer (Cameron) from .txt! 🎙️🎉 Dottxt is the company behind Outlines, reshaping how we control LLM outputs with constrained decoding! 🚀 This podcast dives into all sorts of details from new applications in metadata and information extraction, such as self-expanding knowledge graphs, to structured reasoning, report generation, and more! We also dove into the details of how this leverages Automata Theory and Finite State Machines to achieve this, how it is packaged with inference engines such as vLLM, and a rebuttal to Let Me Speak Freely, discussing how Structured Generation actually improve the quality of LLM outputs, in addition to speeding up inference, and of course, making it more reliable! I hope you find the podcast interesting, this was a super fun one! Links below!
0:54

Structured Outputs: The Building Blocks for Reliable AI! 🏗️ I am SUPER EXCITED to publish our newest Weaviate Podcast featuring Will Kurt (Will Kurt) and Cameron Pfiffer (Cameron) from .txt! 🎙️🎉 Dottxt is the company behind Outlines, reshaping how we control LLM outputs with constrained decoding! 🚀 This podcast dives into all sorts of details from new applications in metadata and information extraction, such as self-expanding knowledge graphs, to structured reasoning, report generation, and more! We also dove into the details of how this leverages Automata Theory and Finite State Machines to achieve this, how it is packaged with inference engines such as vLLM, and a rebuttal to Let Me Speak Freely, discussing how Structured Generation actually improve the quality of LLM outputs, in addition to speeding up inference, and of course, making it more reliable! I hope you find the podcast interesting, this was a super fun one! Links below!

Connor Shorten

12,742 views • 1 year ago

UPDATE On Micah If you are not too busy let’s pray for the family . God our Father Comfort this family at the time of Needs Micah is now with the Lord . 👇 Read what Micah’s dad wrote The Lord gave, and the Lord has taken away; blessed be the name of the Lord.” Job 1:21 Micah Joseph is beginning the new year basking in the never ending glory, love, and peace of God. Micah has been very busy already, as I see the Lord using him and sending him on missions to bring millions of people closer to God. We are surrounded by a great cloud of witnesses (Heb 12:1) that my son is now a part of. I couldn’t be a prouder father. This reality gives me great joy and hope in the midst of sorrow. Our hearts are broken; but we trust in the Lord. Please pray for my family and me as we learn how to live by faith and not by sight. Thank you for all the love, prayers, and compassion that a countless number of you have showed us. May God truly bless you. Your prayers for Micah were answered, but in a different way than what we had all hoped for. God healed and welcomed him into eternal life. He is where we all want to be.
20:00

UPDATE On Micah If you are not too busy let’s pray for the family . God our Father Comfort this family at the time of Needs Micah is now with the Lord . 👇 Read what Micah’s dad wrote The Lord gave, and the Lord has taken away; blessed be the name of the Lord.” Job 1:21 Micah Joseph is beginning the new year basking in the never ending glory, love, and peace of God. Micah has been very busy already, as I see the Lord using him and sending him on missions to bring millions of people closer to God. We are surrounded by a great cloud of witnesses (Heb 12:1) that my son is now a part of. I couldn’t be a prouder father. This reality gives me great joy and hope in the midst of sorrow. Our hearts are broken; but we trust in the Lord. Please pray for my family and me as we learn how to live by faith and not by sight. Thank you for all the love, prayers, and compassion that a countless number of you have showed us. May God truly bless you. Your prayers for Micah were answered, but in a different way than what we had all hoped for. God healed and welcomed him into eternal life. He is where we all want to be.

Omo Nwabunike For Havilah

159,508 views • 9 months ago

Raindrop (Raindrop) is building the safety layer for AI agents. As agents get more capable and take on more complex work, Raindrop detects when they go wrong in production, from failed tool calls and hallucinations to problems companies didn’t even know to look for. Now they’re launching Raindrop Simulations, which brings that same approach earlier in the development process to catch failures before agents reach production. Already used by companies including Vercel, Clay, Framer, and Speak, Raindrop just raised a Series A bringing its total funding to $50 million. In this episode of Founder Firesides, founders zubin koticha, ben hylak, and alexis sat down with Diana to share how a tool they originally built to debug their own coding agent became the company, and why better AI agents actually create more ways for things to go wrong. 00:05 — What Raindrop Does 01:32 — Why Better AI Agents Create More Problems 03:25 — Finding the Failures Evals Miss 04:38 — How Raindrop Got Started 06:32 — Why They Kept Going 07:58 — Catching Agent Failures Before Production
10:17

Raindrop (Raindrop) is building the safety layer for AI agents. As agents get more capable and take on more complex work, Raindrop detects when they go wrong in production, from failed tool calls and hallucinations to problems companies didn’t even know to look for. Now they’re launching Raindrop Simulations, which brings that same approach earlier in the development process to catch failures before agents reach production. Already used by companies including Vercel, Clay, Framer, and Speak, Raindrop just raised a Series A bringing its total funding to $50 million. In this episode of Founder Firesides, founders zubin koticha, ben hylak, and alexis sat down with Diana to share how a tool they originally built to debug their own coding agent became the company, and why better AI agents actually create more ways for things to go wrong. 00:05 — What Raindrop Does 01:32 — Why Better AI Agents Create More Problems 03:25 — Finding the Failures Evals Miss 04:38 — How Raindrop Got Started 06:32 — Why They Kept Going 07:58 — Catching Agent Failures Before Production

Y Combinator

66,324 views • 21 days ago

Here's my new episode with Alex Albert, who shared an inside look at how Anthropic is building the next Claude. We talked about how the research team: → Plans for the model and harness together → Uses Claude to turn user feedback into evals → Trains Claude's character & personality Some quotes from Alex: "We use Claude to cluster user feedback, find top themes, and create synthetic versions of user problems that we then turn into evals." "We need to think about how the model is exposed through all our surfaces, whether it's API or Claude Code or Cowork. The product has a blend with the model and that affects your end user's experience." ""As these things become agents running tasks for a long time and making judgment decisions, what its character is and what it cares about are very important." 📌 Watch now: Thanks to our sponsors: Wispr Flow: Don't type, just speak Oceans: Hire AI-native executive assistants
0:43

Here's my new episode with Alex Albert, who shared an inside look at how Anthropic is building the next Claude. We talked about how the research team: → Plans for the model and harness together → Uses Claude to turn user feedback into evals → Trains Claude's character & personality Some quotes from Alex: "We use Claude to cluster user feedback, find top themes, and create synthetic versions of user problems that we then turn into evals." "We need to think about how the model is exposed through all our surfaces, whether it's API or Claude Code or Cowork. The product has a blend with the model and that affects your end user's experience." ""As these things become agents running tasks for a long time and making judgment decisions, what its character is and what it cares about are very important." 📌 Watch now: Thanks to our sponsors: Wispr Flow: Don't type, just speak Oceans: Hire AI-native executive assistants

Peter Yang

96,052 views • 4 months ago