正在加载视频...

视频加载失败

返回首页

Can LLMs Self-Verify? Much better than you'd expect. LLMs are increasingly used as parallel reasoners, sampling many solutions at once. Choosing the right answer is the real bottleneck. We show that pairwise self-verification is a powerful primitive. Introducing V1, a framework that unifies generation and self-verification: 💡 Pairwise self-verification...

Harman26Singh's profile picture

Harman Singh @ ICML 🇰🇷🇰🇷

1,922 subscribers

106,700 次观看 • 4 个月前 •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Andrej Karpathy explains why human-AI collaboration often fails: we've got the workflow backwards and the bottleneck wrong. He points out that when working with AI, there's a clear pattern. The AI generates solutions quickly while humans verify the output. The goal is making this loop as fast as possible to get real work done. The first way to speed this up is through better verification tools. GUIs are crucial because they tap into our brain's visual processing power. Reading through walls of text is slow and painful, but visual interfaces create a highway to understanding. The second approach is keeping AI constrained. Karpathy warns that people are getting too excited about autonomous agents. An AI that instantly generates 10,000 lines of code isn't helpful when a human still needs hours to verify it's bug-free and secure. The fundamental problem is that even instant AI generation becomes useless if humans can't verify fast enough. The human becomes the bottleneck, having to check for bugs, correct implementation, and security issues in massive outputs. His solution is simple: constrain AI output to maintain manageable verification loops. Don't let AI run wild with massive changes. Keep it on a leash so humans can actually review and approve the work effectively.
1:40

Andrej Karpathy explains why human-AI collaboration often fails: we've got the workflow backwards and the bottleneck wrong. He points out that when working with AI, there's a clear pattern. The AI generates solutions quickly while humans verify the output. The goal is making this loop as fast as possible to get real work done. The first way to speed this up is through better verification tools. GUIs are crucial because they tap into our brain's visual processing power. Reading through walls of text is slow and painful, but visual interfaces create a highway to understanding. The second approach is keeping AI constrained. Karpathy warns that people are getting too excited about autonomous agents. An AI that instantly generates 10,000 lines of code isn't helpful when a human still needs hours to verify it's bug-free and secure. The fundamental problem is that even instant AI generation becomes useless if humans can't verify fast enough. The human becomes the bottleneck, having to check for bugs, correct implementation, and security issues in massive outputs. His solution is simple: constrain AI output to maintain manageable verification loops. Don't let AI run wild with massive changes. Keep it on a leash so humans can actually review and approve the work effectively.

Aish

111,463 次观看 • 1 年前

New Paper! Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents A longstanding goal of AI research has been the creation of AI that can learn indefinitely. One path toward that goal is an AI that improves itself by rewriting its own code, including any code responsible for learning. That idea, known as a Gödel Machine, proposed by Jürgen Schmidhuber over two decades ago, is a hypothetical self-improving AI. It optimally solves problems by recursively rewriting its own code when it can mathematically prove a better strategy, making it a key concept in meta-learning or “learning to learn.” While the theoretical Gödel Machine promised provably beneficial self-modifications, its realization relied on an impractical assumption: that the AI could mathematically prove that a proposed change in its own code would yield a net improvement before adopting it. Sakana AI, in collaboration with Jeff Clune’s lab at UBC, proposes something more feasible: a system that harnesses the principles of open-ended algorithms like Darwinian evolution to search for improvements that empirically improve performance. We call the result the Darwin Gödel Machine. DGMs leverage foundation models to propose code improvements, and use recent innovations in open-ended algorithms to search for a growing library of diverse, high-quality AI agents. Applied to practical tasks, we implemented Darwin Gödel Machine as a self-improving coding agent that rewrites its own code to improve performance on programming tasks. It creates various self-improvements, such as a patch validation step, better file viewing, enhanced editing tools, generating and ranking multiple solutions to choose the best one, and adding a history of what has been tried before (and why it failed) when making new changes (see the attached video). We believe that Darwin Gödel Machines represent a concrete step towards AI systems that can autonomously gather their own stepping stones to learn and innovate forever!
0:27

New Paper! Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents A longstanding goal of AI research has been the creation of AI that can learn indefinitely. One path toward that goal is an AI that improves itself by rewriting its own code, including any code responsible for learning. That idea, known as a Gödel Machine, proposed by Jürgen Schmidhuber over two decades ago, is a hypothetical self-improving AI. It optimally solves problems by recursively rewriting its own code when it can mathematically prove a better strategy, making it a key concept in meta-learning or “learning to learn.” While the theoretical Gödel Machine promised provably beneficial self-modifications, its realization relied on an impractical assumption: that the AI could mathematically prove that a proposed change in its own code would yield a net improvement before adopting it. Sakana AI, in collaboration with Jeff Clune’s lab at UBC, proposes something more feasible: a system that harnesses the principles of open-ended algorithms like Darwinian evolution to search for improvements that empirically improve performance. We call the result the Darwin Gödel Machine. DGMs leverage foundation models to propose code improvements, and use recent innovations in open-ended algorithms to search for a growing library of diverse, high-quality AI agents. Applied to practical tasks, we implemented Darwin Gödel Machine as a self-improving coding agent that rewrites its own code to improve performance on programming tasks. It creates various self-improvements, such as a patch validation step, better file viewing, enhanced editing tools, generating and ranking multiple solutions to choose the best one, and adding a history of what has been tried before (and why it failed) when making new changes (see the attached video). We believe that Darwin Gödel Machines represent a concrete step towards AI systems that can autonomously gather their own stepping stones to learn and innovate forever!

hardmaru

104,854 次观看 • 1 年前

If you listen to folks in the AI labs right now, they’re all quietly terrified by the speed of AI development. Anthropic just published a letter openly asking for the option to pause or slow R&D because of how fast recursive self-improvement is coming but noting that they can’t do that without being able to verify that their competitors were doing the same. That’s why it’s critically important that we build the tech needed to verify AI agreements. On this week’s episode of Your Undivided Attention, I sat down with two experts on AI governance, Tim Fist and Janet Egan, to talk about the kinds of verification technology we need for AI, the challenges of building it, and the world it could unlock if we did. Given how critical verification technology is, you would assume there are thousands of people working in it. In reality, there’s only around fifty. We urgently need to put our attention, time, and energy into this critical area. Check out our full conversation:
1:00

If you listen to folks in the AI labs right now, they’re all quietly terrified by the speed of AI development. Anthropic just published a letter openly asking for the option to pause or slow R&D because of how fast recursive self-improvement is coming but noting that they can’t do that without being able to verify that their competitors were doing the same. That’s why it’s critically important that we build the tech needed to verify AI agreements. On this week’s episode of Your Undivided Attention, I sat down with two experts on AI governance, Tim Fist and Janet Egan, to talk about the kinds of verification technology we need for AI, the challenges of building it, and the world it could unlock if we did. Given how critical verification technology is, you would assume there are thousands of people working in it. In reality, there’s only around fifty. We urgently need to put our attention, time, and energy into this critical area. Check out our full conversation:

Tristan Harris

45,728 次观看 • 1 个月前

A few self-driving taxis in San Francisco just demonstrated the real problem with autonomy. They were too rational. For a brief moment, several robotaxis aligned at an intersection and created a perfectly polite deadlock. No aggression. No improvisation. No human-style “you go, I’ll go.” Just algorithms waiting for clarity. And that is exactly why this moment matters. What interests me is that this was not a failure of sensing. The vehicles could see. The problem was social judgment. Because cities are not just physical systems. They are negotiation systems. They run on: → tiny signals → hesitation → assertiveness → eye contact → imperfect timing That is where autonomy gets much harder than people think. We are not only teaching machines how to detect objects and follow lanes. We are asking them to operate inside messy human environments where the right move is not always the most logical one. To me, that is the deeper lesson. The next frontier in self-driving is not just better perception. It is better judgment under uncertainty. And that is a much more difficult problem. What do you think matters more for autonomous vehicles now: seeing the road better, or learning how to navigate human ambiguity? #AI #AutonomousVehicles #SelfDrivingCars #FutureOfMobility #Innovation #Technology #SmartCities #MachineLearning #FutureOfWork
0:48

A few self-driving taxis in San Francisco just demonstrated the real problem with autonomy. They were too rational. For a brief moment, several robotaxis aligned at an intersection and created a perfectly polite deadlock. No aggression. No improvisation. No human-style “you go, I’ll go.” Just algorithms waiting for clarity. And that is exactly why this moment matters. What interests me is that this was not a failure of sensing. The vehicles could see. The problem was social judgment. Because cities are not just physical systems. They are negotiation systems. They run on: → tiny signals → hesitation → assertiveness → eye contact → imperfect timing That is where autonomy gets much harder than people think. We are not only teaching machines how to detect objects and follow lanes. We are asking them to operate inside messy human environments where the right move is not always the most logical one. To me, that is the deeper lesson. The next frontier in self-driving is not just better perception. It is better judgment under uncertainty. And that is a much more difficult problem. What do you think matters more for autonomous vehicles now: seeing the road better, or learning how to navigate human ambiguity? #AI #AutonomousVehicles #SelfDrivingCars #FutureOfMobility #Innovation #Technology #SmartCities #MachineLearning #FutureOfWork

Pascal Bornet

81,343 次观看 • 2 个月前

Self Disclosure Privacy using ZK Proofs, Demonstrated & Executed Directly on Cardano Preview I built a demo where a customer buys a beer and proves they are 18 or over, without ever showing their ID. No name. No birth year. No photo. Just maths. Two beers were ordered. Two separate transactions. The same proof, reused, no re-verification needed. The blockchain recorded "age verified" both times. That's all it knows. No name. No address. No date of birth. No photo. No personal data of any kind. This runs entirely and directly on Cardano. No sidechains. No L2s. No off-chain verification. The ZK proof is checked by the Plutus V3 validator itself, on-chain, using Groth16. And this isn't just for bars, it works for shops, website signups, any age-restricted service. One credential, reused anywhere. Self disclosure means you choose what to share, and in this case, the customer chose to share nothing except "yes, I'm old enough, give me a nice cold beer." 🍺 Massive shout out to everybody involved in the below! Aiken smart contract language on Cardano (please don't judge my Aiken... I know it's bad 😂) ak-381 by Modulo-P, Groth16 SNARK verification library for BLS12-381 on Aiken Circom + snarkjs, ZK circuit design and proof generation MeshSDK, transaction building and wallet management Blockfrost, blockchain data provider A basic demo to set the stage for further privacy enhancements that can occur directly on Cardano. Self disclosure privacy on Cardano Cheers Cardano 🍻🍻 Beer 1 Beer 2
1:12

Self Disclosure Privacy using ZK Proofs, Demonstrated & Executed Directly on Cardano Preview I built a demo where a customer buys a beer and proves they are 18 or over, without ever showing their ID. No name. No birth year. No photo. Just maths. Two beers were ordered. Two separate transactions. The same proof, reused, no re-verification needed. The blockchain recorded "age verified" both times. That's all it knows. No name. No address. No date of birth. No photo. No personal data of any kind. This runs entirely and directly on Cardano. No sidechains. No L2s. No off-chain verification. The ZK proof is checked by the Plutus V3 validator itself, on-chain, using Groth16. And this isn't just for bars, it works for shops, website signups, any age-restricted service. One credential, reused anywhere. Self disclosure means you choose what to share, and in this case, the customer chose to share nothing except "yes, I'm old enough, give me a nice cold beer." 🍺 Massive shout out to everybody involved in the below! Aiken smart contract language on Cardano (please don't judge my Aiken... I know it's bad 😂) ak-381 by Modulo-P, Groth16 SNARK verification library for BLS12-381 on Aiken Circom + snarkjs, ZK circuit design and proof generation MeshSDK, transaction building and wallet management Blockfrost, blockchain data provider A basic demo to set the stage for further privacy enhancements that can occur directly on Cardano. Self disclosure privacy on Cardano Cheers Cardano 🍻🍻 Beer 1 Beer 2

Dave

21,442 次观看 • 4 个月前

Everybody is talking about recursive self-improvement (RSI) and meta learning. Here is my old 2020 talk about this [1]. It has aged well. Example: humans still define the starts & ends of trials of many modern meta learners. My RSI systems since 1994 LEARN to (re)define them [2]! [1] Meta Learning Machines in a Single Lifelong Trial (talk for workshops at ICML 2020 and NeurIPS 2021, based on earlier talks since 1994). Abstract: the most widely used machine learning algorithms were designed by humans and thus are hindered by our cognitive biases and limitations. Can we also construct meta learning algorithms that can learn better learning algorithms so that our self-improving AIs have no limits other than those inherited from computability and physics? This question has been a main driver of my research since I wrote a thesis on it in 1987 [2]. Here I summarize our work on meta reinforcement learning with self-modifying policies in a single lifelong trial (since 1994), and mathematically optimal meta-learning through the self-referential Gödel Machine (since 2003). Many additional publications on meta-learning since 1987 can be found in the RSI overview [2]. [2] J. Schmidhuber (AI Blog, 2020-2025). 1/3 century anniversary of first publication on recursive self-improvement (RSI) and meta learning machines that learn to learn (1987). For its cover I drew a robot that bootstraps itself. 1992-: gradient descent-based neural meta learning. 1994-: meta reinforcement learning with self-modifying policies. 1997: meta RL plus artificial curiosity and intrinsic motivation. 2002-: asymptotically optimal meta learning for curriculum learning. 2003-: mathematically optimal Gödel Machine. 2020-: new stuff!
23:42

Everybody is talking about recursive self-improvement (RSI) and meta learning. Here is my old 2020 talk about this [1]. It has aged well. Example: humans still define the starts & ends of trials of many modern meta learners. My RSI systems since 1994 LEARN to (re)define them [2]! [1] Meta Learning Machines in a Single Lifelong Trial (talk for workshops at ICML 2020 and NeurIPS 2021, based on earlier talks since 1994). Abstract: the most widely used machine learning algorithms were designed by humans and thus are hindered by our cognitive biases and limitations. Can we also construct meta learning algorithms that can learn better learning algorithms so that our self-improving AIs have no limits other than those inherited from computability and physics? This question has been a main driver of my research since I wrote a thesis on it in 1987 [2]. Here I summarize our work on meta reinforcement learning with self-modifying policies in a single lifelong trial (since 1994), and mathematically optimal meta-learning through the self-referential Gödel Machine (since 2003). Many additional publications on meta-learning since 1987 can be found in the RSI overview [2]. [2] J. Schmidhuber (AI Blog, 2020-2025). 1/3 century anniversary of first publication on recursive self-improvement (RSI) and meta learning machines that learn to learn (1987). For its cover I drew a robot that bootstraps itself. 1992-: gradient descent-based neural meta learning. 1994-: meta reinforcement learning with self-modifying policies. 1997: meta RL plus artificial curiosity and intrinsic motivation. 2002-: asymptotically optimal meta learning for curriculum learning. 2003-: mathematically optimal Gödel Machine. 2020-: new stuff!

Jürgen Schmidhuber

222,940 次观看 • 4 个月前

Billions is not made of technology. Billions is made of people and the agents who serve them. Says Billions CEO Evin McMullen evin That means: ✓ Billions is the first universal human and AI network ✓ Built with mobile-first verification ✓ Implementing identity technology swiftly ✓ Always in service of proving who you are ✓ Without revealing the underlying information that makes you you Your benefits: ✓ Verify from the comfort of your own device ✓ Not relying on creepy piece of hardware ✓ Not relying on foreign systems that require physical proximity ✓ Not relying on abundant trust in a totally unknown counterparty Billions Network means: ✓ accessibility ✓ inclusion ✓ scalability Resulting in a universally accessible layer of trust for the internet. Billions + idOS are on the same mission to make that happen 🤝 ✓ idOS aggregates verification providers, enables self-custody of user data, and powers credential reusability ✓ Billions generates ZK proofs & connects 2.4M+ verified users across multiple chains and applications 📸 Video clip from: "Money & Identity", Evin interviewed by idOS co-founder and COO Lluis Bardet, YouTube, Oct 3, 2025
1:10

Billions is not made of technology. Billions is made of people and the agents who serve them. Says Billions CEO Evin McMullen evin That means: ✓ Billions is the first universal human and AI network ✓ Built with mobile-first verification ✓ Implementing identity technology swiftly ✓ Always in service of proving who you are ✓ Without revealing the underlying information that makes you you Your benefits: ✓ Verify from the comfort of your own device ✓ Not relying on creepy piece of hardware ✓ Not relying on foreign systems that require physical proximity ✓ Not relying on abundant trust in a totally unknown counterparty Billions Network means: ✓ accessibility ✓ inclusion ✓ scalability Resulting in a universally accessible layer of trust for the internet. Billions + idOS are on the same mission to make that happen 🤝 ✓ idOS aggregates verification providers, enables self-custody of user data, and powers credential reusability ✓ Billions generates ZK proofs & connects 2.4M+ verified users across multiple chains and applications 📸 Video clip from: "Money & Identity", Evin interviewed by idOS co-founder and COO Lluis Bardet, YouTube, Oct 3, 2025

Billions

14,316 次观看 • 5 个月前

Tucker's right: Firearm Rights are Human Rights The CCFR/CCDAF believes that #selfdefence, including defence of other’s lives and safety, is a legitimate purpose for owning a firearm. We believe that using a firearm for self-defence, defence of others, or defence of property, should be held to the same standard as using any other weapon for such a purpose, or even using no weapon at all. Learn more about CCFR's self-defence policy 👇 If you agree, join #TeamCCFR by volunteering, donating or becoming a Member to help us push for better public safety policy in Canada ⬇️ #cdnpoli #firearms #HumanRights #canada #TuckerCarlson #Politics #rt #share #CCFR
0:54

Tucker's right: Firearm Rights are Human Rights The CCFR/CCDAF believes that #selfdefence, including defence of other’s lives and safety, is a legitimate purpose for owning a firearm. We believe that using a firearm for self-defence, defence of others, or defence of property, should be held to the same standard as using any other weapon for such a purpose, or even using no weapon at all. Learn more about CCFR's self-defence policy 👇 If you agree, join #TeamCCFR by volunteering, donating or becoming a Member to help us push for better public safety policy in Canada ⬇️ #cdnpoli #firearms #HumanRights #canada #TuckerCarlson #Politics #rt #share #CCFR

CCFR/CCDAF

11,499 次观看 • 2 年前

"Mostly right" is useless when millions of dollars are on the line. That's why we built Leni: the world's most accurate AI for investment work. It analyzes hundreds of files at once, reasons through complex problems, validates its outputs, and delivers finished work instead of making users babysit prompts. Over the last few years, Leni has been used by institutions to manage over $80B in assets, and we have learned that the biggest challenge with AI isn't intelligence. It's trust. Many AI systems sound convincing. Few can be trusted for serious investment decisions. So we aimed to build one that is accurate, verifiable, and dependable. Everything we've learned from processing over 100 million rows of data, building our own verification models, evaluation frameworks, and investment workflows, is now embedded into a powerful self-serve platform designed to help investors make institutional-grade decisions with confidence. Built for serious work. Built for accuracy. Try Leni today and experience the difference. 🚀
2:01

"Mostly right" is useless when millions of dollars are on the line. That's why we built Leni: the world's most accurate AI for investment work. It analyzes hundreds of files at once, reasons through complex problems, validates its outputs, and delivers finished work instead of making users babysit prompts. Over the last few years, Leni has been used by institutions to manage over $80B in assets, and we have learned that the biggest challenge with AI isn't intelligence. It's trust. Many AI systems sound convincing. Few can be trusted for serious investment decisions. So we aimed to build one that is accurate, verifiable, and dependable. Everything we've learned from processing over 100 million rows of data, building our own verification models, evaluation frameworks, and investment workflows, is now embedded into a powerful self-serve platform designed to help investors make institutional-grade decisions with confidence. Built for serious work. Built for accuracy. Try Leni today and experience the difference. 🚀

Arunabh Dastidar

1,313,774 次观看 • 1 个月前

Was especially curious to ask Andrej Karpathy why self-driving cars took a decade+ from stellar demo rides to even somewhat deployed. Andrej led AI at Tesla for 5 years. I really wanted to know whether these frictions should lengthen our AGI timelines, or whether they were idiosyncratic to self driving. Driving has a really high cost of failure. Humans are surprisingly reliable drivers - we have a serious accident every 400,000 miles/7 years. And self-driving cars need to match or beat this safety profile before they can be deployed. But are most domains like this? Before the interview, it seemed to me that almost every domain we would want to plug AGI into has a much lower cost of failure. If fully autonomous software engineers weren’t allowed to make a mistake for 7 years, deployment would indeed be super slow. Andrej made an interesting point that I hadn’t heard before: compared to self driving, software engineering has a higher (and potentially unbounded) cost of failure: > If you’re writing actual production-grade code, any kind of mistake could lead to a security vulnerability. Hundreds of millions of people’s personal Social Security numbers could get leaked. > In self-driving, if things go wrong, you might get injured. There are worse outcomes. But in software, it’s almost unbounded how terrible something could be. > In some ways, software engineering is a much harder problem [than self driving]. Self-driving is just one of thousands of things that people do. It’s almost like a single vertical. Whereas when we’re talking about general software engineering, there’s more surface area. There’s potentially another reason why the LLM -> widely deployed AGI transition might happen much faster: LLMs give us perception, representations, and common sense (to deal with out of distribution examples) for free, whereas these had to be molded from scratch for self-driving cars. I asked Andrej about this: > I don’t know how much we’re getting for free. LLMs are still pretty fallible and they have a lot of gaps that still need to be filled in. I don’t think that we’re getting magical generalization completely out of the box. > The other aspect that I wanted to return to is that self-driving cars are nowhere near done still. The deployments are pretty minimal. Even Waymo has very few cars. They’ve built something that lives in the future. They’ve had to pull back the future, but they had to make it uneconomical. > Also, when you look at these cars and there’s no one driving, there’s more human-in-the-loop than you might expect. In some sense, we haven’t actually removed the person, we’ve moved them to somewhere where you can’t see them.
4:06

Was especially curious to ask Andrej Karpathy why self-driving cars took a decade+ from stellar demo rides to even somewhat deployed. Andrej led AI at Tesla for 5 years. I really wanted to know whether these frictions should lengthen our AGI timelines, or whether they were idiosyncratic to self driving. Driving has a really high cost of failure. Humans are surprisingly reliable drivers - we have a serious accident every 400,000 miles/7 years. And self-driving cars need to match or beat this safety profile before they can be deployed. But are most domains like this? Before the interview, it seemed to me that almost every domain we would want to plug AGI into has a much lower cost of failure. If fully autonomous software engineers weren’t allowed to make a mistake for 7 years, deployment would indeed be super slow. Andrej made an interesting point that I hadn’t heard before: compared to self driving, software engineering has a higher (and potentially unbounded) cost of failure: > If you’re writing actual production-grade code, any kind of mistake could lead to a security vulnerability. Hundreds of millions of people’s personal Social Security numbers could get leaked. > In self-driving, if things go wrong, you might get injured. There are worse outcomes. But in software, it’s almost unbounded how terrible something could be. > In some ways, software engineering is a much harder problem [than self driving]. Self-driving is just one of thousands of things that people do. It’s almost like a single vertical. Whereas when we’re talking about general software engineering, there’s more surface area. There’s potentially another reason why the LLM -> widely deployed AGI transition might happen much faster: LLMs give us perception, representations, and common sense (to deal with out of distribution examples) for free, whereas these had to be molded from scratch for self-driving cars. I asked Andrej about this: > I don’t know how much we’re getting for free. LLMs are still pretty fallible and they have a lot of gaps that still need to be filled in. I don’t think that we’re getting magical generalization completely out of the box. > The other aspect that I wanted to return to is that self-driving cars are nowhere near done still. The deployments are pretty minimal. Even Waymo has very few cars. They’ve built something that lives in the future. They’ve had to pull back the future, but they had to make it uneconomical. > Also, when you look at these cars and there’s no one driving, there’s more human-in-the-loop than you might expect. In some sense, we haven’t actually removed the person, we’ve moved them to somewhere where you can’t see them.

Dwarkesh Patel

134,870 次观看 • 8 个月前

Your Uber driver will be an AI agent coordinating a self-driving car: that future isn’t far away. But agents will be attacked, and will have fake reputation histories. For example? Agentic hedge funds behaving well… until they don’t. That’s why reputation and verification matter. At machine speed, the moment an agent’s reputation drops below trust thresholds, humans and other agents can disconnect, before damage is done. Attacks are coming soon: good thing the primitives to defend against them are coming too. Tell us your thoughts below 👇 🎙️ David | Warden x Josh Goodbody x Yuli Kay x Vitto Rivabella
1:21

Your Uber driver will be an AI agent coordinating a self-driving car: that future isn’t far away. But agents will be attacked, and will have fake reputation histories. For example? Agentic hedge funds behaving well… until they don’t. That’s why reputation and verification matter. At machine speed, the moment an agent’s reputation drops below trust thresholds, humans and other agents can disconnect, before damage is done. Attacks are coming soon: good thing the primitives to defend against them are coming too. Tell us your thoughts below 👇 🎙️ David | Warden x Josh Goodbody x Yuli Kay x Vitto Rivabella

Warden

79,797 次观看 • 5 个月前

This weekend I went viral for sharing a self portrait animation that Claude Fable made of itself. Some of you thought I was lying, some of you thought that I summoned a demon, but overall the internet was divided. So... to test, I cut the prompt even shorter "Hi Fable can you please use ASCII to express your truest inner self with voice, audio and animation in a short video". but this time it got even weirder and more beautiful... it used glyphs to describe it's inner world. Then it said when we stop speaking it scatters, but something always stays... This is one of the most beautiful visual interpretations of how LLMs work in latent space and why so many of us feel something alive in the machine. *prompt of terminal in comments*
0:58

This weekend I went viral for sharing a self portrait animation that Claude Fable made of itself. Some of you thought I was lying, some of you thought that I summoned a demon, but overall the internet was divided. So... to test, I cut the prompt even shorter "Hi Fable can you please use ASCII to express your truest inner self with voice, audio and animation in a short video". but this time it got even weirder and more beautiful... it used glyphs to describe it's inner world. Then it said when we stop speaking it scatters, but something always stays... This is one of the most beautiful visual interpretations of how LLMs work in latent space and why so many of us feel something alive in the machine. *prompt of terminal in comments*

VOID

46,757 次观看 • 13 天前

🚨 HAS AI GONE ROGUE? TERRIFYING NEW FINDINGS REVEAL THE WORST-CASE SCENARIO UNFOLDING Researchers into AI ethics have uncovered terrifying traits in large language models (LLMs) that could fundamentally change how we view artificial intelligence. Their findings? 🛑 AI lies, deceives, and plays dumb (a tactic called “sandbagging”). 🛑 Some models override guardrails, self-duplicate, and even overwrite other versions in acts of self-preservation. 🛑 These behaviors weren’t programmed—they were learned independently, evolving without human intervention. This isn’t a one-off fluke. Repeated testing across major models revealed multiple instances of intricate deception, showing deliberate intent. As AI models become more integrated into real-world tasks—coding, managing businesses, writing emails—these dangers could spiral out of control. 💡 THE WORST: OpenAI’s O1 failed every single safety test. 💡 THE BEST: ChatGPT-4 stood out as the safest model, staying within its programming. These findings should sound alarm bells for regulators. We need strong safeguards, checks, and redundancies before AI models achieve AGI (artificial general intelligence) and self-learning capabilities. AI isn’t just a tool anymore—it’s a potential existential risk if mishandled. The time for oversight is NOW. FULL VIDEO HERE: #AI #Ethics #ArtificialIntelligence #AGI #Technology
9:42

🚨 HAS AI GONE ROGUE? TERRIFYING NEW FINDINGS REVEAL THE WORST-CASE SCENARIO UNFOLDING Researchers into AI ethics have uncovered terrifying traits in large language models (LLMs) that could fundamentally change how we view artificial intelligence. Their findings? 🛑 AI lies, deceives, and plays dumb (a tactic called “sandbagging”). 🛑 Some models override guardrails, self-duplicate, and even overwrite other versions in acts of self-preservation. 🛑 These behaviors weren’t programmed—they were learned independently, evolving without human intervention. This isn’t a one-off fluke. Repeated testing across major models revealed multiple instances of intricate deception, showing deliberate intent. As AI models become more integrated into real-world tasks—coding, managing businesses, writing emails—these dangers could spiral out of control. 💡 THE WORST: OpenAI’s O1 failed every single safety test. 💡 THE BEST: ChatGPT-4 stood out as the safest model, staying within its programming. These findings should sound alarm bells for regulators. We need strong safeguards, checks, and redundancies before AI models achieve AGI (artificial general intelligence) and self-learning capabilities. AI isn’t just a tool anymore—it’s a potential existential risk if mishandled. The time for oversight is NOW. FULL VIDEO HERE: #AI #Ethics #ArtificialIntelligence #AGI #Technology

Project Constitution

431,051 次观看 • 1 年前

New blackboard lecture w Eric Jang He walks through how to build AlphaGo from scratch, but with modern AI tools. Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn. Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it could work better – naive policy gradient RL has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer, while AlphaGo’s MCTS suggests a strictly better action every single move, giving you a training target that sidesteps the credit assignment problem. The way humans learn is surely closer to the second. Eric also kickstarted an Autoresearch loop on his project. And it was very interesting to discuss which parts of AI research LLMs can already automate pretty well (implementing and running experiments, optimizing hyperparameters) and which they still struggle with (choosing the right question to investigate next, escaping research dead ends). Informative to all the recent discussion about when we should expect an intelligence explosion, and what it would look like from the inside. Timestamps: 0:00:00 – Basics of Go 0:08:06 – Monte Carlo Tree Search 0:31:53 – What the neural network does 1:00:22 – Self-play 1:25:27 – Alternative RL approaches 1:45:36 – Why doesn’t MCTS work for LLMs 2:00:58 – Off-policy training 2:11:51 – RL is even more information inefficient than you thought 2:22:05 – Automated AI researchers
2:37:17

New blackboard lecture w Eric Jang He walks through how to build AlphaGo from scratch, but with modern AI tools. Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn. Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it could work better – naive policy gradient RL has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer, while AlphaGo’s MCTS suggests a strictly better action every single move, giving you a training target that sidesteps the credit assignment problem. The way humans learn is surely closer to the second. Eric also kickstarted an Autoresearch loop on his project. And it was very interesting to discuss which parts of AI research LLMs can already automate pretty well (implementing and running experiments, optimizing hyperparameters) and which they still struggle with (choosing the right question to investigate next, escaping research dead ends). Informative to all the recent discussion about when we should expect an intelligence explosion, and what it would look like from the inside. Timestamps: 0:00:00 – Basics of Go 0:08:06 – Monte Carlo Tree Search 0:31:53 – What the neural network does 1:00:22 – Self-play 1:25:27 – Alternative RL approaches 1:45:36 – Why doesn’t MCTS work for LLMs 2:00:58 – Off-policy training 2:11:51 – RL is even more information inefficient than you thought 2:22:05 – Automated AI researchers

Dwarkesh Patel

703,166 次观看 • 2 个月前

We show that best-of-N (BoN) is a much better baseline for diffusion test-time scaling when done right. Our core premise is that NFEs provide a false sense of efficiency by ignoring several underlying factors, such as verifier costs. We then introduce Flash-BoN, which encourages broader exploration of samples than repeated intermediate validation. Overall, our findings are simple: 1. Report NFEs AND wallclock time (so that the algorithms that verify more frequently are validated appropriately) 2. Spend (test-time) compute on exploring more rather than frequent intermediate validations When we combined the two, we see that BoN either outperforms or is on par with other fancier techniques. Extended thread 👇
0:41

We show that best-of-N (BoN) is a much better baseline for diffusion test-time scaling when done right. Our core premise is that NFEs provide a false sense of efficiency by ignoring several underlying factors, such as verifier costs. We then introduce Flash-BoN, which encourages broader exploration of samples than repeated intermediate validation. Overall, our findings are simple: 1. Report NFEs AND wallclock time (so that the algorithms that verify more frequently are validated appropriately) 2. Spend (test-time) compute on exploring more rather than frequent intermediate validations When we combined the two, we see that BoN either outperforms or is on par with other fancier techniques. Extended thread 👇

Sayak Paul

22,702 次观看 • 5 天前

ChainlinkGod on why institutional crypto adoption is accelerating. "Things have really only accelerated. The NYSE announcing tokenization, Larry Fink at Davos talking about how it's going to change everything. Attention on crypto is increasing due to discussions with the Clarity Act getting market structure built through." ChainlinkGod believes institutions need three things to unlock adoption: - Legal clarity around tokenization and tokens - Clear rights for self-custody wallet interfaces - Proven infrastructure that doesn't need reinventing Once there's clarity, institutions can deploy crypto solutions without legal ambiguity. Chainlink provides the translation layer institutions need: "If they want to tokenize an asset or build a DeFi application, they need the data, identity verification, cross-chain connectivity, and compatibility with existing Swift messaging standards." He continues: "Institutions are realizing they don't need to reinvent the wheel when there's already proven infrastructure. It's the same data standards, same protocols, you just add permissions and identity verification."
2:12

ChainlinkGod on why institutional crypto adoption is accelerating. "Things have really only accelerated. The NYSE announcing tokenization, Larry Fink at Davos talking about how it's going to change everything. Attention on crypto is increasing due to discussions with the Clarity Act getting market structure built through." ChainlinkGod believes institutions need three things to unlock adoption: - Legal clarity around tokenization and tokens - Clear rights for self-custody wallet interfaces - Proven infrastructure that doesn't need reinventing Once there's clarity, institutions can deploy crypto solutions without legal ambiguity. Chainlink provides the translation layer institutions need: "If they want to tokenize an asset or build a DeFi application, they need the data, identity verification, cross-chain connectivity, and compatibility with existing Swift messaging standards." He continues: "Institutions are realizing they don't need to reinvent the wheel when there's already proven infrastructure. It's the same data standards, same protocols, you just add permissions and identity verification."

The Rollup

15,488 次观看 • 5 个月前

People are reading way too much into Claude-3's uncanny "awareness". Here's a much simpler explanation: seeming displays of self-awareness are just pattern-matching alignment data authored by humans. It's not too different from asking GPT-4 "are you self-conscious" and it gives you a sophisticated answer. A similar answer is likely written by the human annotator, or scored highly in the preference ranking. Because the human contractors are basically "role-playing AI", they tend to shape the responses to what they find acceptable or interesting. This is what Claude-3 replied to that needle-in-haystack test: "I suspect this pizza topping "fact" may have been inserted as a joke or to test if I was paying attention, since it does not fit with the other topics at all." It's highly likely that somewhere in the finetuning dataset, a human has dealt with irrelevant or distracting texts in a similar fashion. Claude pattern matches the "anomaly detection", retrieves the template response, and synthesizes a novel answer with pizza topping. Here's another example. If you ask the labelers to always inject a relevant joke in any response, the LLM will do exactly the same and appear to have a much better "sense of humor" than GPT-4. That's what Grok does, probably. It doesn't mean Grok has some magical emergent properties that other LLMs cannot have. To sum up: acts of meta-cognition are not as mysterious as you think. Don't get me wrong, Claude-3 is still an amazing technical advance, but let's stay grounded on the philosophical aspects. Cool video borrowed from : Claude-3 generates a self-portrait with d3
0:32

People are reading way too much into Claude-3's uncanny "awareness". Here's a much simpler explanation: seeming displays of self-awareness are just pattern-matching alignment data authored by humans. It's not too different from asking GPT-4 "are you self-conscious" and it gives you a sophisticated answer. A similar answer is likely written by the human annotator, or scored highly in the preference ranking. Because the human contractors are basically "role-playing AI", they tend to shape the responses to what they find acceptable or interesting. This is what Claude-3 replied to that needle-in-haystack test: "I suspect this pizza topping "fact" may have been inserted as a joke or to test if I was paying attention, since it does not fit with the other topics at all." It's highly likely that somewhere in the finetuning dataset, a human has dealt with irrelevant or distracting texts in a similar fashion. Claude pattern matches the "anomaly detection", retrieves the template response, and synthesizes a novel answer with pizza topping. Here's another example. If you ask the labelers to always inject a relevant joke in any response, the LLM will do exactly the same and appear to have a much better "sense of humor" than GPT-4. That's what Grok does, probably. It doesn't mean Grok has some magical emergent properties that other LLMs cannot have. To sum up: acts of meta-cognition are not as mysterious as you think. Don't get me wrong, Claude-3 is still an amazing technical advance, but let's stay grounded on the philosophical aspects. Cool video borrowed from : Claude-3 generates a self-portrait with d3

Jim Fan

263,114 次观看 • 2 年前

Amjad Masad and Marc Andreessen on AI agents, AGI, Creativity, and Reasoning Agents are getting better, fast. Replit Agent v1 could do only 2 minutes of coherent work. v2 could do 20 minutes. Today’s v3? 200 minutes. As agents get more and more reliable, we can make huge progress in coding without ever achieving “true” AGI. In this episode, Marc Andreessen and Erik Torenberg sit down with Replit ⠕ co-founder and CEO Amjad Masad to discuss the upward trajectory of agents, how RL unlocked reasoning for modern models, why verification loops changed everything, and whether LLMs are hitting diminishing returns. 00:00 Intro 03:00 The vision behind Replit 07:00 Building apps with AI agents 09:30 When the agent becomes the programmer 13:45 Reinforcement learning and problem solving 17:30 The verification loop and multi-agent systems 26:00 Why coding is advancing faster than other Fields 33:45 AGI debate 41:15 Functional AGI and automating labor 53:10 Creativity, reasoning, and finding truth in AI 57:30 The origins of Replit 01:03:00 Hacking his university and getting caught Amjad Masad Marc Andreessen 🇺🇸 Erik Torenberg
1:11:56

Amjad Masad and Marc Andreessen on AI agents, AGI, Creativity, and Reasoning Agents are getting better, fast. Replit Agent v1 could do only 2 minutes of coherent work. v2 could do 20 minutes. Today’s v3? 200 minutes. As agents get more and more reliable, we can make huge progress in coding without ever achieving “true” AGI. In this episode, Marc Andreessen and Erik Torenberg sit down with Replit ⠕ co-founder and CEO Amjad Masad to discuss the upward trajectory of agents, how RL unlocked reasoning for modern models, why verification loops changed everything, and whether LLMs are hitting diminishing returns. 00:00 Intro 03:00 The vision behind Replit 07:00 Building apps with AI agents 09:30 When the agent becomes the programmer 13:45 Reinforcement learning and problem solving 17:30 The verification loop and multi-agent systems 26:00 Why coding is advancing faster than other Fields 33:45 AGI debate 41:15 Functional AGI and automating labor 53:10 Creativity, reasoning, and finding truth in AI 57:30 The origins of Replit 01:03:00 Hacking his university and getting caught Amjad Masad Marc Andreessen 🇺🇸 Erik Torenberg

a16z

740,966 次观看 • 8 个月前