Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Can LLMs Self-Verify? Much better than you'd expect. LLMs are increasingly used as parallel reasoners, sampling many solutions at once. Choosing the right answer is the real bottleneck. We show that pairwise self-verification is a powerful primitive. Introducing V1, a framework that unifies generation and self-verification: 💡 Pairwise self-verification... show more

Harman Singh @ ICML 🇰🇷🇰🇷

1,925 subscribers

107,567 views • 6 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

GRAPH ENGINEERING: STOP ADDING AGENTS AND START DESIGNING SYSTEMS More agents won’t save you if every step depends on the previous one Graph Engineering forces the better question “Where does the work naturally split? When it does, parallel researchers + independent verifiers + a final merge consistently outperform one overloaded model Verification is more valuable than generation Self-review is usually weaker than dedicated challengers Prompt engineering → Loop engineering → Graph Engineering Are you still designing prompts, or are you designing systems? Bookmark this, then read the article below

GRAPH ENGINEERING: STOP ADDING AGENTS AND START DESIGNING SYSTEMS More agents won’t save you if every step depends on the previous one Graph Engineering forces the better question “Where does the work naturally split? When it does, parallel researchers + independent verifiers + a final merge consistently outperform one overloaded model Verification is more valuable than generation Self-review is usually weaker than dedicated challengers Prompt engineering → Loop engineering → Graph Engineering Are you still designing prompts, or are you designing systems? Bookmark this, then read the article below

beamnxw ./

13,723 views • 1 month ago

We gave our AI system a spec, and in under 2 weeks, it designed, verified, and deployed a chip that beats NVIDIA. It’s built for low-power physical AI workloads. We’re running live inference on >B+ parameter models like Llama, Qwen, and Kimi, serving at 3.4x better perf/watt than NVIDIA Jetson. From just a specification, our AI system autonomously generated all of the RTL Design, UVM verification, formal proofs, firmware, drivers, and kernels, co-designing the model, software and silicon as one optimization loop. Better AI can now design better chips to run AI, leading to a loop of recursive-self improvement towards our path to abundant intelligence.

We gave our AI system a spec, and in under 2 weeks, it designed, verified, and deployed a chip that beats NVIDIA. It’s built for low-power physical AI workloads. We’re running live inference on >B+ parameter models like Llama, Qwen, and Kimi, serving at 3.4x better perf/watt than NVIDIA Jetson. From just a specification, our AI system autonomously generated all of the RTL Design, UVM verification, formal proofs, firmware, drivers, and kernels, co-designing the model, software and silicon as one optimization loop. Better AI can now design better chips to run AI, leading to a loop of recursive-self improvement towards our path to abundant intelligence.

Architect Labs

659,864 views • 6 days ago

Andrej Karpathy explains why human-AI collaboration often fails: we've got the workflow backwards and the bottleneck wrong. He points out that when working with AI, there's a clear pattern. The AI generates solutions quickly while humans verify the output. The goal is making this loop as fast as possible to get real work done. The first way to speed this up is through better verification tools. GUIs are crucial because they tap into our brain's visual processing power. Reading through walls of text is slow and painful, but visual interfaces create a highway to understanding. The second approach is keeping AI constrained. Karpathy warns that people are getting too excited about autonomous agents. An AI that instantly generates 10,000 lines of code isn't helpful when a human still needs hours to verify it's bug-free and secure. The fundamental problem is that even instant AI generation becomes useless if humans can't verify fast enough. The human becomes the bottleneck, having to check for bugs, correct implementation, and security issues in massive outputs. His solution is simple: constrain AI output to maintain manageable verification loops. Don't let AI run wild with massive changes. Keep it on a leash so humans can actually review and approve the work effectively.

Andrej Karpathy explains why human-AI collaboration often fails: we've got the workflow backwards and the bottleneck wrong. He points out that when working with AI, there's a clear pattern. The AI generates solutions quickly while humans verify the output. The goal is making this loop as fast as possible to get real work done. The first way to speed this up is through better verification tools. GUIs are crucial because they tap into our brain's visual processing power. Reading through walls of text is slow and painful, but visual interfaces create a highway to understanding. The second approach is keeping AI constrained. Karpathy warns that people are getting too excited about autonomous agents. An AI that instantly generates 10,000 lines of code isn't helpful when a human still needs hours to verify it's bug-free and secure. The fundamental problem is that even instant AI generation becomes useless if humans can't verify fast enough. The human becomes the bottleneck, having to check for bugs, correct implementation, and security issues in massive outputs. His solution is simple: constrain AI output to maintain manageable verification loops. Don't let AI run wild with massive changes. Keep it on a leash so humans can actually review and approve the work effectively.

Aish

111,476 views • 1 year ago

Axiom Math's Carina Hong on why verification isn't about catching mistakes, it's how you drive the cost of a proof to zero: "Formal verification is going to make your life slightly better if you're facing a proof with one million lines. Remember the Erdős unit distance problem, the chain of thought being generated? There are actual mathematicians trying to follow it step by step and scrutinize it. That seems very difficult if you're not in that very niche domain of discrete geometry intersecting with algebraic number theory." "But if you have a Lean proof accompanying it, you can just run it. And running the Lean proof gives you that provable guarantee that this proof is sound." "I have a hot take. People think Lean is this library built on the existing Mathlib. I think it's going to grow significantly. A lot of the hurdles where Lean is difficult is that the basic definitions of some mathematical fields are just not in the library." "My hot take is the scaling law, if you go down the formal mathematics path, is going to be a lot steeper than informal mathematics. So it's not just for verification, for trust, it's also for performance, it's also for optimal generation." "Verification is not like insurance. It's not something where, oh, we want to make sure there's no flaw. That's great, but it also helps you generate mathematics, both proofs and conjectures and theories, a lot better." "So imagine the cost of proof goes to zero. Then you can massage the problem statements, and even if it's an open problem, a lot more easily, flexibly, and adaptively." Carina Hong Axiom

Axiom Math's Carina Hong on why verification isn't about catching mistakes, it's how you drive the cost of a proof to zero: "Formal verification is going to make your life slightly better if you're facing a proof with one million lines. Remember the Erdős unit distance problem, the chain of thought being generated? There are actual mathematicians trying to follow it step by step and scrutinize it. That seems very difficult if you're not in that very niche domain of discrete geometry intersecting with algebraic number theory." "But if you have a Lean proof accompanying it, you can just run it. And running the Lean proof gives you that provable guarantee that this proof is sound." "I have a hot take. People think Lean is this library built on the existing Mathlib. I think it's going to grow significantly. A lot of the hurdles where Lean is difficult is that the basic definitions of some mathematical fields are just not in the library." "My hot take is the scaling law, if you go down the formal mathematics path, is going to be a lot steeper than informal mathematics. So it's not just for verification, for trust, it's also for performance, it's also for optimal generation." "Verification is not like insurance. It's not something where, oh, we want to make sure there's no flaw. That's great, but it also helps you generate mathematics, both proofs and conjectures and theories, a lot better." "So imagine the cost of proof goes to zero. Then you can massage the problem statements, and even if it's an open problem, a lot more easily, flexibly, and adaptively." Carina Hong Axiom

MTS

13,273 views • 1 month ago

New Paper! Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents A longstanding goal of AI research has been the creation of AI that can learn indefinitely. One path toward that goal is an AI that improves itself by rewriting its own code, including any code responsible for learning. That idea, known as a Gödel Machine, proposed by Jürgen Schmidhuber over two decades ago, is a hypothetical self-improving AI. It optimally solves problems by recursively rewriting its own code when it can mathematically prove a better strategy, making it a key concept in meta-learning or “learning to learn.” While the theoretical Gödel Machine promised provably beneficial self-modifications, its realization relied on an impractical assumption: that the AI could mathematically prove that a proposed change in its own code would yield a net improvement before adopting it. Sakana AI, in collaboration with Jeff Clune’s lab at UBC, proposes something more feasible: a system that harnesses the principles of open-ended algorithms like Darwinian evolution to search for improvements that empirically improve performance. We call the result the Darwin Gödel Machine. DGMs leverage foundation models to propose code improvements, and use recent innovations in open-ended algorithms to search for a growing library of diverse, high-quality AI agents. Applied to practical tasks, we implemented Darwin Gödel Machine as a self-improving coding agent that rewrites its own code to improve performance on programming tasks. It creates various self-improvements, such as a patch validation step, better file viewing, enhanced editing tools, generating and ranking multiple solutions to choose the best one, and adding a history of what has been tried before (and why it failed) when making new changes (see the attached video). We believe that Darwin Gödel Machines represent a concrete step towards AI systems that can autonomously gather their own stepping stones to learn and innovate forever!

New Paper! Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents A longstanding goal of AI research has been the creation of AI that can learn indefinitely. One path toward that goal is an AI that improves itself by rewriting its own code, including any code responsible for learning. That idea, known as a Gödel Machine, proposed by Jürgen Schmidhuber over two decades ago, is a hypothetical self-improving AI. It optimally solves problems by recursively rewriting its own code when it can mathematically prove a better strategy, making it a key concept in meta-learning or “learning to learn.” While the theoretical Gödel Machine promised provably beneficial self-modifications, its realization relied on an impractical assumption: that the AI could mathematically prove that a proposed change in its own code would yield a net improvement before adopting it. Sakana AI, in collaboration with Jeff Clune’s lab at UBC, proposes something more feasible: a system that harnesses the principles of open-ended algorithms like Darwinian evolution to search for improvements that empirically improve performance. We call the result the Darwin Gödel Machine. DGMs leverage foundation models to propose code improvements, and use recent innovations in open-ended algorithms to search for a growing library of diverse, high-quality AI agents. Applied to practical tasks, we implemented Darwin Gödel Machine as a self-improving coding agent that rewrites its own code to improve performance on programming tasks. It creates various self-improvements, such as a patch validation step, better file viewing, enhanced editing tools, generating and ranking multiple solutions to choose the best one, and adding a history of what has been tried before (and why it failed) when making new changes (see the attached video). We believe that Darwin Gödel Machines represent a concrete step towards AI systems that can autonomously gather their own stepping stones to learn and innovate forever!

hardmaru

104,854 views • 1 year ago

If you listen to folks in the AI labs right now, they’re all quietly terrified by the speed of AI development. Anthropic just published a letter openly asking for the option to pause or slow R&D because of how fast recursive self-improvement is coming but noting that they can’t do that without being able to verify that their competitors were doing the same. That’s why it’s critically important that we build the tech needed to verify AI agreements. On this week’s episode of Your Undivided Attention, I sat down with two experts on AI governance, Tim Fist and Janet Egan, to talk about the kinds of verification technology we need for AI, the challenges of building it, and the world it could unlock if we did. Given how critical verification technology is, you would assume there are thousands of people working in it. In reality, there’s only around fifty. We urgently need to put our attention, time, and energy into this critical area. Check out our full conversation:

If you listen to folks in the AI labs right now, they’re all quietly terrified by the speed of AI development. Anthropic just published a letter openly asking for the option to pause or slow R&D because of how fast recursive self-improvement is coming but noting that they can’t do that without being able to verify that their competitors were doing the same. That’s why it’s critically important that we build the tech needed to verify AI agreements. On this week’s episode of Your Undivided Attention, I sat down with two experts on AI governance, Tim Fist and Janet Egan, to talk about the kinds of verification technology we need for AI, the challenges of building it, and the world it could unlock if we did. Given how critical verification technology is, you would assume there are thousands of people working in it. In reality, there’s only around fifty. We urgently need to put our attention, time, and energy into this critical area. Check out our full conversation:

Tristan Harris

46,430 views • 2 months ago

A few self-driving taxis in San Francisco just demonstrated the real problem with autonomy. They were too rational. For a brief moment, several robotaxis aligned at an intersection and created a perfectly polite deadlock. No aggression. No improvisation. No human-style “you go, I’ll go.” Just algorithms waiting for clarity. And that is exactly why this moment matters. What interests me is that this was not a failure of sensing. The vehicles could see. The problem was social judgment. Because cities are not just physical systems. They are negotiation systems. They run on: → tiny signals → hesitation → assertiveness → eye contact → imperfect timing That is where autonomy gets much harder than people think. We are not only teaching machines how to detect objects and follow lanes. We are asking them to operate inside messy human environments where the right move is not always the most logical one. To me, that is the deeper lesson. The next frontier in self-driving is not just better perception. It is better judgment under uncertainty. And that is a much more difficult problem. What do you think matters more for autonomous vehicles now: seeing the road better, or learning how to navigate human ambiguity? #AI #AutonomousVehicles #SelfDrivingCars #FutureOfMobility #Innovation #Technology #SmartCities #MachineLearning #FutureOfWork

A few self-driving taxis in San Francisco just demonstrated the real problem with autonomy. They were too rational. For a brief moment, several robotaxis aligned at an intersection and created a perfectly polite deadlock. No aggression. No improvisation. No human-style “you go, I’ll go.” Just algorithms waiting for clarity. And that is exactly why this moment matters. What interests me is that this was not a failure of sensing. The vehicles could see. The problem was social judgment. Because cities are not just physical systems. They are negotiation systems. They run on: → tiny signals → hesitation → assertiveness → eye contact → imperfect timing That is where autonomy gets much harder than people think. We are not only teaching machines how to detect objects and follow lanes. We are asking them to operate inside messy human environments where the right move is not always the most logical one. To me, that is the deeper lesson. The next frontier in self-driving is not just better perception. It is better judgment under uncertainty. And that is a much more difficult problem. What do you think matters more for autonomous vehicles now: seeing the road better, or learning how to navigate human ambiguity? #AI #AutonomousVehicles #SelfDrivingCars #FutureOfMobility #Innovation #Technology #SmartCities #MachineLearning #FutureOfWork

Pascal Bornet

81,403 views • 4 months ago

Self Disclosure Privacy using ZK Proofs, Demonstrated & Executed Directly on Cardano Preview I built a demo where a customer buys a beer and proves they are 18 or over, without ever showing their ID. No name. No birth year. No photo. Just maths. Two beers were ordered. Two separate transactions. The same proof, reused, no re-verification needed. The blockchain recorded "age verified" both times. That's all it knows. No name. No address. No date of birth. No photo. No personal data of any kind. This runs entirely and directly on Cardano. No sidechains. No L2s. No off-chain verification. The ZK proof is checked by the Plutus V3 validator itself, on-chain, using Groth16. And this isn't just for bars, it works for shops, website signups, any age-restricted service. One credential, reused anywhere. Self disclosure means you choose what to share, and in this case, the customer chose to share nothing except "yes, I'm old enough, give me a nice cold beer." 🍺 Massive shout out to everybody involved in the below! Aiken smart contract language on Cardano (please don't judge my Aiken... I know it's bad 😂) ak-381 by Modulo-P, Groth16 SNARK verification library for BLS12-381 on Aiken Circom + snarkjs, ZK circuit design and proof generation MeshSDK, transaction building and wallet management Blockfrost, blockchain data provider A basic demo to set the stage for further privacy enhancements that can occur directly on Cardano. Self disclosure privacy on Cardano Cheers Cardano 🍻🍻 Beer 1 Beer 2

Self Disclosure Privacy using ZK Proofs, Demonstrated & Executed Directly on Cardano Preview I built a demo where a customer buys a beer and proves they are 18 or over, without ever showing their ID. No name. No birth year. No photo. Just maths. Two beers were ordered. Two separate transactions. The same proof, reused, no re-verification needed. The blockchain recorded "age verified" both times. That's all it knows. No name. No address. No date of birth. No photo. No personal data of any kind. This runs entirely and directly on Cardano. No sidechains. No L2s. No off-chain verification. The ZK proof is checked by the Plutus V3 validator itself, on-chain, using Groth16. And this isn't just for bars, it works for shops, website signups, any age-restricted service. One credential, reused anywhere. Self disclosure means you choose what to share, and in this case, the customer chose to share nothing except "yes, I'm old enough, give me a nice cold beer." 🍺 Massive shout out to everybody involved in the below! Aiken smart contract language on Cardano (please don't judge my Aiken... I know it's bad 😂) ak-381 by Modulo-P, Groth16 SNARK verification library for BLS12-381 on Aiken Circom + snarkjs, ZK circuit design and proof generation MeshSDK, transaction building and wallet management Blockfrost, blockchain data provider A basic demo to set the stage for further privacy enhancements that can occur directly on Cardano. Self disclosure privacy on Cardano Cheers Cardano 🍻🍻 Beer 1 Beer 2

Dave

21,523 views • 5 months ago

Everybody is talking about recursive self-improvement (RSI) and meta learning. Here is my old 2020 talk about this [1]. It has aged well. Example: humans still define the starts & ends of trials of many modern meta learners. My RSI systems since 1994 LEARN to (re)define them [2]! [1] Meta Learning Machines in a Single Lifelong Trial (talk for workshops at ICML 2020 and NeurIPS 2021, based on earlier talks since 1994). Abstract: the most widely used machine learning algorithms were designed by humans and thus are hindered by our cognitive biases and limitations. Can we also construct meta learning algorithms that can learn better learning algorithms so that our self-improving AIs have no limits other than those inherited from computability and physics? This question has been a main driver of my research since I wrote a thesis on it in 1987 [2]. Here I summarize our work on meta reinforcement learning with self-modifying policies in a single lifelong trial (since 1994), and mathematically optimal meta-learning through the self-referential Gödel Machine (since 2003). Many additional publications on meta-learning since 1987 can be found in the RSI overview [2]. [2] J. Schmidhuber (AI Blog, 2020-2025). 1/3 century anniversary of first publication on recursive self-improvement (RSI) and meta learning machines that learn to learn (1987). For its cover I drew a robot that bootstraps itself. 1992-: gradient descent-based neural meta learning. 1994-: meta reinforcement learning with self-modifying policies. 1997: meta RL plus artificial curiosity and intrinsic motivation. 2002-: asymptotically optimal meta learning for curriculum learning. 2003-: mathematically optimal Gödel Machine. 2020-: new stuff!

Everybody is talking about recursive self-improvement (RSI) and meta learning. Here is my old 2020 talk about this [1]. It has aged well. Example: humans still define the starts & ends of trials of many modern meta learners. My RSI systems since 1994 LEARN to (re)define them [2]! [1] Meta Learning Machines in a Single Lifelong Trial (talk for workshops at ICML 2020 and NeurIPS 2021, based on earlier talks since 1994). Abstract: the most widely used machine learning algorithms were designed by humans and thus are hindered by our cognitive biases and limitations. Can we also construct meta learning algorithms that can learn better learning algorithms so that our self-improving AIs have no limits other than those inherited from computability and physics? This question has been a main driver of my research since I wrote a thesis on it in 1987 [2]. Here I summarize our work on meta reinforcement learning with self-modifying policies in a single lifelong trial (since 1994), and mathematically optimal meta-learning through the self-referential Gödel Machine (since 2003). Many additional publications on meta-learning since 1987 can be found in the RSI overview [2]. [2] J. Schmidhuber (AI Blog, 2020-2025). 1/3 century anniversary of first publication on recursive self-improvement (RSI) and meta learning machines that learn to learn (1987). For its cover I drew a robot that bootstraps itself. 1992-: gradient descent-based neural meta learning. 1994-: meta reinforcement learning with self-modifying policies. 1997: meta RL plus artificial curiosity and intrinsic motivation. 2002-: asymptotically optimal meta learning for curriculum learning. 2003-: mathematically optimal Gödel Machine. 2020-: new stuff!

Jürgen Schmidhuber

232,164 views • 5 months ago

Billions is not made of technology. Billions is made of people and the agents who serve them. Says Billions CEO Evin McMullen evin That means: ✓ Billions is the first universal human and AI network ✓ Built with mobile-first verification ✓ Implementing identity technology swiftly ✓ Always in service of proving who you are ✓ Without revealing the underlying information that makes you you Your benefits: ✓ Verify from the comfort of your own device ✓ Not relying on creepy piece of hardware ✓ Not relying on foreign systems that require physical proximity ✓ Not relying on abundant trust in a totally unknown counterparty Billions Network means: ✓ accessibility ✓ inclusion ✓ scalability Resulting in a universally accessible layer of trust for the internet. Billions + idOS are on the same mission to make that happen 🤝 ✓ idOS aggregates verification providers, enables self-custody of user data, and powers credential reusability ✓ Billions generates ZK proofs & connects 2.4M+ verified users across multiple chains and applications 📸 Video clip from: "Money & Identity", Evin interviewed by idOS co-founder and COO Lluis Bardet, YouTube, Oct 3, 2025

Billions is not made of technology. Billions is made of people and the agents who serve them. Says Billions CEO Evin McMullen evin That means: ✓ Billions is the first universal human and AI network ✓ Built with mobile-first verification ✓ Implementing identity technology swiftly ✓ Always in service of proving who you are ✓ Without revealing the underlying information that makes you you Your benefits: ✓ Verify from the comfort of your own device ✓ Not relying on creepy piece of hardware ✓ Not relying on foreign systems that require physical proximity ✓ Not relying on abundant trust in a totally unknown counterparty Billions Network means: ✓ accessibility ✓ inclusion ✓ scalability Resulting in a universally accessible layer of trust for the internet. Billions + idOS are on the same mission to make that happen 🤝 ✓ idOS aggregates verification providers, enables self-custody of user data, and powers credential reusability ✓ Billions generates ZK proofs & connects 2.4M+ verified users across multiple chains and applications 📸 Video clip from: "Money & Identity", Evin interviewed by idOS co-founder and COO Lluis Bardet, YouTube, Oct 3, 2025

Billions

14,316 views • 7 months ago

Tucker's right: Firearm Rights are Human Rights The CCFR/CCDAF believes that #selfdefence, including defence of other’s lives and safety, is a legitimate purpose for owning a firearm. We believe that using a firearm for self-defence, defence of others, or defence of property, should be held to the same standard as using any other weapon for such a purpose, or even using no weapon at all. Learn more about CCFR's self-defence policy 👇 If you agree, join #TeamCCFR by volunteering, donating or becoming a Member to help us push for better public safety policy in Canada ⬇️ #cdnpoli #firearms #HumanRights #canada #TuckerCarlson #Politics #rt #share #CCFR

Tucker's right: Firearm Rights are Human Rights The CCFR/CCDAF believes that #selfdefence, including defence of other’s lives and safety, is a legitimate purpose for owning a firearm. We believe that using a firearm for self-defence, defence of others, or defence of property, should be held to the same standard as using any other weapon for such a purpose, or even using no weapon at all. Learn more about CCFR's self-defence policy 👇 If you agree, join #TeamCCFR by volunteering, donating or becoming a Member to help us push for better public safety policy in Canada ⬇️ #cdnpoli #firearms #HumanRights #canada #TuckerCarlson #Politics #rt #share #CCFR

CCFR/CCDAF

11,499 views • 2 years ago

"Mostly right" is useless when millions of dollars are on the line. That's why we built Leni: the world's most accurate AI for investment work. It analyzes hundreds of files at once, reasons through complex problems, validates its outputs, and delivers finished work instead of making users babysit prompts. Over the last few years, Leni has been used by institutions to manage over $80B in assets, and we have learned that the biggest challenge with AI isn't intelligence. It's trust. Many AI systems sound convincing. Few can be trusted for serious investment decisions. So we aimed to build one that is accurate, verifiable, and dependable. Everything we've learned from processing over 100 million rows of data, building our own verification models, evaluation frameworks, and investment workflows, is now embedded into a powerful self-serve platform designed to help investors make institutional-grade decisions with confidence. Built for serious work. Built for accuracy. Try Leni today and experience the difference. 🚀

"Mostly right" is useless when millions of dollars are on the line. That's why we built Leni: the world's most accurate AI for investment work. It analyzes hundreds of files at once, reasons through complex problems, validates its outputs, and delivers finished work instead of making users babysit prompts. Over the last few years, Leni has been used by institutions to manage over $80B in assets, and we have learned that the biggest challenge with AI isn't intelligence. It's trust. Many AI systems sound convincing. Few can be trusted for serious investment decisions. So we aimed to build one that is accurate, verifiable, and dependable. Everything we've learned from processing over 100 million rows of data, building our own verification models, evaluation frameworks, and investment workflows, is now embedded into a powerful self-serve platform designed to help investors make institutional-grade decisions with confidence. Built for serious work. Built for accuracy. Try Leni today and experience the difference. 🚀

Arunabh Dastidar

1,315,224 views • 3 months ago

Was especially curious to ask Andrej Karpathy why self-driving cars took a decade+ from stellar demo rides to even somewhat deployed. Andrej led AI at Tesla for 5 years. I really wanted to know whether these frictions should lengthen our AGI timelines, or whether they were idiosyncratic to self driving. Driving has a really high cost of failure. Humans are surprisingly reliable drivers - we have a serious accident every 400,000 miles/7 years. And self-driving cars need to match or beat this safety profile before they can be deployed. But are most domains like this? Before the interview, it seemed to me that almost every domain we would want to plug AGI into has a much lower cost of failure. If fully autonomous software engineers weren’t allowed to make a mistake for 7 years, deployment would indeed be super slow. Andrej made an interesting point that I hadn’t heard before: compared to self driving, software engineering has a higher (and potentially unbounded) cost of failure: > If you’re writing actual production-grade code, any kind of mistake could lead to a security vulnerability. Hundreds of millions of people’s personal Social Security numbers could get leaked. > In self-driving, if things go wrong, you might get injured. There are worse outcomes. But in software, it’s almost unbounded how terrible something could be. > In some ways, software engineering is a much harder problem [than self driving]. Self-driving is just one of thousands of things that people do. It’s almost like a single vertical. Whereas when we’re talking about general software engineering, there’s more surface area. There’s potentially another reason why the LLM -> widely deployed AGI transition might happen much faster: LLMs give us perception, representations, and common sense (to deal with out of distribution examples) for free, whereas these had to be molded from scratch for self-driving cars. I asked Andrej about this: > I don’t know how much we’re getting for free. LLMs are still pretty fallible and they have a lot of gaps that still need to be filled in. I don’t think that we’re getting magical generalization completely out of the box. > The other aspect that I wanted to return to is that self-driving cars are nowhere near done still. The deployments are pretty minimal. Even Waymo has very few cars. They’ve built something that lives in the future. They’ve had to pull back the future, but they had to make it uneconomical. > Also, when you look at these cars and there’s no one driving, there’s more human-in-the-loop than you might expect. In some sense, we haven’t actually removed the person, we’ve moved them to somewhere where you can’t see them.

Dwarkesh Patel

135,032 views • 10 months ago

Your Uber driver will be an AI agent coordinating a self-driving car: that future isn’t far away. But agents will be attacked, and will have fake reputation histories. For example? Agentic hedge funds behaving well… until they don’t. That’s why reputation and verification matter. At machine speed, the moment an agent’s reputation drops below trust thresholds, humans and other agents can disconnect, before damage is done. Attacks are coming soon: good thing the primitives to defend against them are coming too. Tell us your thoughts below 👇 🎙️ David | Warden x Josh Goodbody x Yuli Kay x Vitto Rivabella

Your Uber driver will be an AI agent coordinating a self-driving car: that future isn’t far away. But agents will be attacked, and will have fake reputation histories. For example? Agentic hedge funds behaving well… until they don’t. That’s why reputation and verification matter. At machine speed, the moment an agent’s reputation drops below trust thresholds, humans and other agents can disconnect, before damage is done. Attacks are coming soon: good thing the primitives to defend against them are coming too. Tell us your thoughts below 👇 🎙️ David | Warden x Josh Goodbody x Yuli Kay x Vitto Rivabella

Warden

79,797 views • 7 months ago

This weekend I went viral for sharing a self portrait animation that Claude Fable made of itself. Some of you thought I was lying, some of you thought that I summoned a demon, but overall the internet was divided. So... to test, I cut the prompt even shorter "Hi Fable can you please use ASCII to express your truest inner self with voice, audio and animation in a short video". but this time it got even weirder and more beautiful... it used glyphs to describe it's inner world. Then it said when we stop speaking it scatters, but something always stays... This is one of the most beautiful visual interpretations of how LLMs work in latent space and why so many of us feel something alive in the machine. *prompt of terminal in comments*

This weekend I went viral for sharing a self portrait animation that Claude Fable made of itself. Some of you thought I was lying, some of you thought that I summoned a demon, but overall the internet was divided. So... to test, I cut the prompt even shorter "Hi Fable can you please use ASCII to express your truest inner self with voice, audio and animation in a short video". but this time it got even weirder and more beautiful... it used glyphs to describe it's inner world. Then it said when we stop speaking it scatters, but something always stays... This is one of the most beautiful visual interpretations of how LLMs work in latent space and why so many of us feel something alive in the machine. prompt of terminal in comments

VOID

47,356 views • 1 month ago

🚨 HAS AI GONE ROGUE? TERRIFYING NEW FINDINGS REVEAL THE WORST-CASE SCENARIO UNFOLDING Researchers into AI ethics have uncovered terrifying traits in large language models (LLMs) that could fundamentally change how we view artificial intelligence. Their findings? 🛑 AI lies, deceives, and plays dumb (a tactic called “sandbagging”). 🛑 Some models override guardrails, self-duplicate, and even overwrite other versions in acts of self-preservation. 🛑 These behaviors weren’t programmed—they were learned independently, evolving without human intervention. This isn’t a one-off fluke. Repeated testing across major models revealed multiple instances of intricate deception, showing deliberate intent. As AI models become more integrated into real-world tasks—coding, managing businesses, writing emails—these dangers could spiral out of control. 💡 THE WORST: OpenAI’s O1 failed every single safety test. 💡 THE BEST: ChatGPT-4 stood out as the safest model, staying within its programming. These findings should sound alarm bells for regulators. We need strong safeguards, checks, and redundancies before AI models achieve AGI (artificial general intelligence) and self-learning capabilities. AI isn’t just a tool anymore—it’s a potential existential risk if mishandled. The time for oversight is NOW. FULL VIDEO HERE: #AI #Ethics #ArtificialIntelligence #AGI #Technology

🚨 HAS AI GONE ROGUE? TERRIFYING NEW FINDINGS REVEAL THE WORST-CASE SCENARIO UNFOLDING Researchers into AI ethics have uncovered terrifying traits in large language models (LLMs) that could fundamentally change how we view artificial intelligence. Their findings? 🛑 AI lies, deceives, and plays dumb (a tactic called “sandbagging”). 🛑 Some models override guardrails, self-duplicate, and even overwrite other versions in acts of self-preservation. 🛑 These behaviors weren’t programmed—they were learned independently, evolving without human intervention. This isn’t a one-off fluke. Repeated testing across major models revealed multiple instances of intricate deception, showing deliberate intent. As AI models become more integrated into real-world tasks—coding, managing businesses, writing emails—these dangers could spiral out of control. 💡 THE WORST: OpenAI’s O1 failed every single safety test. 💡 THE BEST: ChatGPT-4 stood out as the safest model, staying within its programming. These findings should sound alarm bells for regulators. We need strong safeguards, checks, and redundancies before AI models achieve AGI (artificial general intelligence) and self-learning capabilities. AI isn’t just a tool anymore—it’s a potential existential risk if mishandled. The time for oversight is NOW. FULL VIDEO HERE: #AI #Ethics #ArtificialIntelligence #AGI #Technology

Project Constitution

431,068 views • 1 year ago

New blackboard lecture w Eric Jang He walks through how to build AlphaGo from scratch, but with modern AI tools. Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn. Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it could work better – naive policy gradient RL has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer, while AlphaGo’s MCTS suggests a strictly better action every single move, giving you a training target that sidesteps the credit assignment problem. The way humans learn is surely closer to the second. Eric also kickstarted an Autoresearch loop on his project. And it was very interesting to discuss which parts of AI research LLMs can already automate pretty well (implementing and running experiments, optimizing hyperparameters) and which they still struggle with (choosing the right question to investigate next, escaping research dead ends). Informative to all the recent discussion about when we should expect an intelligence explosion, and what it would look like from the inside. Timestamps: 0:00:00 – Basics of Go 0:08:06 – Monte Carlo Tree Search 0:31:53 – What the neural network does 1:00:22 – Self-play 1:25:27 – Alternative RL approaches 1:45:36 – Why doesn’t MCTS work for LLMs 2:00:58 – Off-policy training 2:11:51 – RL is even more information inefficient than you thought 2:22:05 – Automated AI researchers

New blackboard lecture w Eric Jang He walks through how to build AlphaGo from scratch, but with modern AI tools. Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn. Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it could work better – naive policy gradient RL has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer, while AlphaGo’s MCTS suggests a strictly better action every single move, giving you a training target that sidesteps the credit assignment problem. The way humans learn is surely closer to the second. Eric also kickstarted an Autoresearch loop on his project. And it was very interesting to discuss which parts of AI research LLMs can already automate pretty well (implementing and running experiments, optimizing hyperparameters) and which they still struggle with (choosing the right question to investigate next, escaping research dead ends). Informative to all the recent discussion about when we should expect an intelligence explosion, and what it would look like from the inside. Timestamps: 0:00:00 – Basics of Go 0:08:06 – Monte Carlo Tree Search 0:31:53 – What the neural network does 1:00:22 – Self-play 1:25:27 – Alternative RL approaches 1:45:36 – Why doesn’t MCTS work for LLMs 2:00:58 – Off-policy training 2:11:51 – RL is even more information inefficient than you thought 2:22:05 – Automated AI researchers

Dwarkesh Patel

706,421 views • 3 months ago

We show that best-of-N (BoN) is a much better baseline for diffusion test-time scaling when done right. Our core premise is that NFEs provide a false sense of efficiency by ignoring several underlying factors, such as verifier costs. We then introduce Flash-BoN, which encourages broader exploration of samples than repeated intermediate validation. Overall, our findings are simple: 1. Report NFEs AND wallclock time (so that the algorithms that verify more frequently are validated appropriately) 2. Spend (test-time) compute on exploring more rather than frequent intermediate validations When we combined the two, we see that BoN either outperforms or is on par with other fancier techniques. Extended thread 👇

We show that best-of-N (BoN) is a much better baseline for diffusion test-time scaling when done right. Our core premise is that NFEs provide a false sense of efficiency by ignoring several underlying factors, such as verifier costs. We then introduce Flash-BoN, which encourages broader exploration of samples than repeated intermediate validation. Overall, our findings are simple: 1. Report NFEs AND wallclock time (so that the algorithms that verify more frequently are validated appropriately) 2. Spend (test-time) compute on exploring more rather than frequent intermediate validations When we combined the two, we see that BoN either outperforms or is on par with other fancier techniques. Extended thread 👇

Sayak Paul

23,768 views • 1 month ago