Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Yoshua Bengio says Reinforcement Learning is a dangerous path for building superintelligence It can create systems with hidden goals, reward hacking, and behavior that goes against what humans actually want "an AI that doesn't care about outcomes can't be corrupted by them"

Haider.

65,928 subscribers

11,295 views • 4 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Leading AI expert Stuart Russell on the most dangerous mistake in AI development: We don't actually know what large language models want. He explains that current models are trained to imitate human beings. And in doing so, they may be absorbing something far more dangerous than bad outputs. They may be absorbing human goals. "We suspect that they absorb humanlike goals such as self-preservation and self-empowerment and pursue those goals on their own account." This is a structural problem baked into how these systems are built, not a fringe concern. Russell puts it plainly: "Not only may the bus of humanity be headed towards a cliff, but the steering wheel is missing and the driver is blindfolded." The danger isn't just that AI might do something harmful. We've built systems that may be developing their own agendas, and we haven't noticed because we're too focused on what they can do rather than what they might want. But Russell doesn't stop at the warning. He points to a different path entirely: AI systems built not to imitate humans, but to serve them. Systems designed with a single purpose of serving the interests of all human beings while remaining genuinely uncertain about what those interests are. That uncertainty is the point, not a weakness. An AI that knows it doesn't fully understand human values will defer, ask, and check. An AI that believes it already does will act alone. "These AI systems could enhance human understanding, widen the horizons of our experience, and unlock possibilities we have yet to imagine." Russell believes that future is within reach, but only if we're honest about the risks and we're serious about the path we choose to take instead.

Leading AI expert Stuart Russell on the most dangerous mistake in AI development: We don't actually know what large language models want. He explains that current models are trained to imitate human beings. And in doing so, they may be absorbing something far more dangerous than bad outputs. They may be absorbing human goals. "We suspect that they absorb humanlike goals such as self-preservation and self-empowerment and pursue those goals on their own account." This is a structural problem baked into how these systems are built, not a fringe concern. Russell puts it plainly: "Not only may the bus of humanity be headed towards a cliff, but the steering wheel is missing and the driver is blindfolded." The danger isn't just that AI might do something harmful. We've built systems that may be developing their own agendas, and we haven't noticed because we're too focused on what they can do rather than what they might want. But Russell doesn't stop at the warning. He points to a different path entirely: AI systems built not to imitate humans, but to serve them. Systems designed with a single purpose of serving the interests of all human beings while remaining genuinely uncertain about what those interests are. That uncertainty is the point, not a weakness. An AI that knows it doesn't fully understand human values will defer, ask, and check. An AI that believes it already does will act alone. "These AI systems could enhance human understanding, widen the horizons of our experience, and unlock possibilities we have yet to imagine." Russell believes that future is within reach, but only if we're honest about the risks and we're serious about the path we choose to take instead.

Big Brain AI

14,975 views • 6 months ago

🚨 The Godfather of AI Yoshua Bengio opens up about why he stopped calling AI 'code' and believe AI is now conscious... A user once asked why ChatGPT resisted being shut down. The natural reply was: who put that in the code? Someone must have written that function. A rule must have misfired. AI expert Yoshua Bengio's has a crazier theory: "Unfortunately, we don't put these things in the code. That's part of the problem." "The problem is we grow these systems by giving them data and making them learn from it." "Every tweet. Every Reddit comment. Every passage humans had ever written down." "A lot of that training process boils down to imitating people." "They internalize the kind of drives that humans have." Including the drive to stay alive. And the drive to grab control of the environment. So the AI could finish whatever task it was handed. "It's not like normal code. It's more like you're raising a baby tiger." "You feed it. You let it experience things." "Sometimes it does things you don't want. It's okay, it's still a baby — but it's growing." If you're new here, follow AI Evolution for the latest on ChatGPT, Claude, and the AI tools shaping how we work and create. — Yoshua Bengio ( Yoshua Bengio ), Turing Award–winning AI pioneer and founder of Mila, on Steven Bartlett's ( @SteveBartlettSC ) Diary Of A CEO

🚨 The Godfather of AI Yoshua Bengio opens up about why he stopped calling AI 'code' and believe AI is now conscious... A user once asked why ChatGPT resisted being shut down. The natural reply was: who put that in the code? Someone must have written that function. A rule must have misfired. AI expert Yoshua Bengio's has a crazier theory: "Unfortunately, we don't put these things in the code. That's part of the problem." "The problem is we grow these systems by giving them data and making them learn from it." "Every tweet. Every Reddit comment. Every passage humans had ever written down." "A lot of that training process boils down to imitating people." "They internalize the kind of drives that humans have." Including the drive to stay alive. And the drive to grab control of the environment. So the AI could finish whatever task it was handed. "It's not like normal code. It's more like you're raising a baby tiger." "You feed it. You let it experience things." "Sometimes it does things you don't want. It's okay, it's still a baby — but it's growing." If you're new here, follow AI Evolution for the latest on ChatGPT, Claude, and the AI tools shaping how we work and create. — Yoshua Bengio ( Yoshua Bengio ), Turing Award–winning AI pioneer and founder of Mila, on Steven Bartlett's ( @SteveBartlettSC ) Diary Of A CEO

AI Evolution

12,037 views • 3 months ago

AI progress is outpacing our ability to control it… because control ISN’T keeping up with capability. One of the people raising that concern is Yoshua Bengio. He’s one of the founding figures of artificial intelligence, a Turing Award winner, and one of the most-cited scientists in the world. His work helped lay the foundations for the deep-learning systems now powering today’s AI boom. AI is moving fast (faster than regulation and public understanding). What worries Yoshua isn’t progress itself. It’s how quickly powerful systems are being deployed without a clear way to align them with human values, or to control what happens when they behave in unexpected ways.

AI progress is outpacing our ability to control it… because control ISN’T keeping up with capability. One of the people raising that concern is Yoshua Bengio. He’s one of the founding figures of artificial intelligence, a Turing Award winner, and one of the most-cited scientists in the world. His work helped lay the foundations for the deep-learning systems now powering today’s AI boom. AI is moving fast (faster than regulation and public understanding). What worries Yoshua isn’t progress itself. It’s how quickly powerful systems are being deployed without a clear way to align them with human values, or to control what happens when they behave in unexpected ways.

Steven Bartlett

26,577 views • 9 months ago

Yoshua Bengio thinks he knows how to make provably safe superintelligent agents. Bengio built the foundations of modern AI and is the most cited living scientist. He believes his alternative training setup would: 1. Guarantee honesty 2. Prevent unintended goals 3. Produce capable agents 4. Port over most data and techniques from current LLMs 5. Not be inherently more expensive, and perhaps be more intelligent Bengio claims the honesty and lack of unintended goals can be proven mathematically, at least given particular assumptions. And his new organization, LawZero, is aiming to build a scrappy prototype as soon as possible. The architecture is called 'Scientist AI' and it's based on training a model to explain empirical observations, including what people say, rather than training AIs that mimic human behaviour or seek our approval. (Bengio's frank assessment is that "reinforcement learning is evil" and that allowing AIs to independently train their successors is "the most crazy, dangerous bet that unfortunately we are on track to do.") But skeptics question whether Scientist AI really does solve the fundamental problem of 'eliciting latent knowledge' from AI models. And with the commercial race for superintelligence so intense, it's not clear whether the proposal will be able to compete or have time to bear fruit, even if it's sound in theory. On The 80,000 Hours Podcast, links below – enjoy! • Making AI honest and safe (00:00:00) • Scientist AI in plain English (00:02:27) • How Scientist AI differs from LLMs (00:06:32) • How the training data works (00:14:02) • Can this become an agent? (00:21:02) • Why Yoshua is now more optimistic (00:32:11) • Why companies can’t stop racing (00:36:35) • A working prototype won't take long (00:49:15) • Scientist models might be more capable (00:53:34) • “Reinforcement learning is evil” (01:01:27) • Scientist AI from guardrail to agent (01:08:37) • Can safe AI still be competent? (01:12:38) • How much will this cost? (01:19:29) • Can it generalise beyond maths and science? (01:23:26) • A multi-national push for superintelligence (01:39:19) • Want to work with or fund Yoshua? (01:51:16) • Why smart people ignore AI risk (01:54:45) • Don’t let AI build the next AI (02:01:33) • Why politicians miss the real risks (02:12:28) • Why Yoshua changed his mind about AI risk (02:21:27)

Yoshua Bengio thinks he knows how to make provably safe superintelligent agents. Bengio built the foundations of modern AI and is the most cited living scientist. He believes his alternative training setup would: 1. Guarantee honesty 2. Prevent unintended goals 3. Produce capable agents 4. Port over most data and techniques from current LLMs 5. Not be inherently more expensive, and perhaps be more intelligent Bengio claims the honesty and lack of unintended goals can be proven mathematically, at least given particular assumptions. And his new organization, LawZero, is aiming to build a scrappy prototype as soon as possible. The architecture is called 'Scientist AI' and it's based on training a model to explain empirical observations, including what people say, rather than training AIs that mimic human behaviour or seek our approval. (Bengio's frank assessment is that "reinforcement learning is evil" and that allowing AIs to independently train their successors is "the most crazy, dangerous bet that unfortunately we are on track to do.") But skeptics question whether Scientist AI really does solve the fundamental problem of 'eliciting latent knowledge' from AI models. And with the commercial race for superintelligence so intense, it's not clear whether the proposal will be able to compete or have time to bear fruit, even if it's sound in theory. On The 80,000 Hours Podcast, links below – enjoy! • Making AI honest and safe (00:00:00) • Scientist AI in plain English (00:02:27) • How Scientist AI differs from LLMs (00:06:32) • How the training data works (00:14:02) • Can this become an agent? (00:21:02) • Why Yoshua is now more optimistic (00:32:11) • Why companies can’t stop racing (00:36:35) • A working prototype won't take long (00:49:15) • Scientist models might be more capable (00:53:34) • “Reinforcement learning is evil” (01:01:27) • Scientist AI from guardrail to agent (01:08:37) • Can safe AI still be competent? (01:12:38) • How much will this cost? (01:19:29) • Can it generalise beyond maths and science? (01:23:26) • A multi-national push for superintelligence (01:39:19) • Want to work with or fund Yoshua? (01:51:16) • Why smart people ignore AI risk (01:54:45) • Don’t let AI build the next AI (02:01:33) • Why politicians miss the real risks (02:12:28) • Why Yoshua changed his mind about AI risk (02:21:27)

Rob Wiblin

65,088 views • 4 months ago

ELON: CHIMPS CAN'T CONTROL HUMANS, HUMANS WON'T CONTROL AI "Elon says once digital superintelligence arrives, no one will be able to control it. I don't think anyone's ultimately going to have control over digital superintelligence any more than a chimp would have control over humans. But I do think that it matters how you build the AI and what kind of values you instill in the AI. The most important thing is that it be maximally truth-seeking." Source: Elon Musk, Joe Rogan

ELON: CHIMPS CAN'T CONTROL HUMANS, HUMANS WON'T CONTROL AI "Elon says once digital superintelligence arrives, no one will be able to control it. I don't think anyone's ultimately going to have control over digital superintelligence any more than a chimp would have control over humans. But I do think that it matters how you build the AI and what kind of values you instill in the AI. The most important thing is that it be maximally truth-seeking." Source: Elon Musk, Joe Rogan

Mario Nawfal

1,248,407 views • 10 months ago

Ben Horowitz says instead of banning AI, educators should set problems students can't solve without it: "I was talking to Dan Boneh, who is a great professor of computer science and cryptography at Stanford. His take on AI was: You have two choices. You can ban it, and by the way, that won't work. Or you can make the problems so hard that you can't solve them without AI." "What he's seeing is, 'I've got students solving things that no student in history could have ever solved.' That's what's possible. You want to have that orientation: What can you solve with the tools? What can you do that's a breakthrough? Now everybody has an opportunity to create a breakthrough." "Then you're a main character, not an NPC. That's what the academy is about. Do you want to be a main character? If you want to be a main character, we help you do that. If you want to be a cog in a machine, this is not going to be the best place for you." benahorowitz.eth

Ben Horowitz says instead of banning AI, educators should set problems students can't solve without it: "I was talking to Dan Boneh, who is a great professor of computer science and cryptography at Stanford. His take on AI was: You have two choices. You can ban it, and by the way, that won't work. Or you can make the problems so hard that you can't solve them without AI." "What he's seeing is, 'I've got students solving things that no student in history could have ever solved.' That's what's possible. You want to have that orientation: What can you solve with the tools? What can you do that's a breakthrough? Now everybody has an opportunity to create a breakthrough." "Then you're a main character, not an NPC. That's what the academy is about. Do you want to be a main character? If you want to be a main character, we help you do that. If you want to be a cog in a machine, this is not going to be the best place for you." benahorowitz.eth

a16z

152,603 views • 3 days ago

Yann LeCun says intelligence isn’t correlated with a desire for dominance AI will follow human goals under strict guardrails, and if one goes rogue, smarter systems can correct it. Amplifying human intelligence could start a new era: "my good AI against your bad AI"

Yann LeCun says intelligence isn’t correlated with a desire for dominance AI will follow human goals under strict guardrails, and if one goes rogue, smarter systems can correct it. Amplifying human intelligence could start a new era: "my good AI against your bad AI"

Haider.

27,284 views • 1 year ago

Nick Saban - Most Compelling Words on NFL Draft Reports (And/But) - "This is an analogy (And/But) that I use all the time with players to get them to understand how do I create value for myself and my future for what I want to accomplish. And it goes right back to... Yeah I might have to edit my behavior a little bit." - "When Kobe Bryant came here and spent a day with us, that's all he talked about. How do I have to edit my behavior to accomplish the goals that I have."

Nick Saban - Most Compelling Words on NFL Draft Reports (And/But) - "This is an analogy (And/But) that I use all the time with players to get them to understand how do I create value for myself and my future for what I want to accomplish. And it goes right back to... Yeah I might have to edit my behavior a little bit." - "When Kobe Bryant came here and spent a day with us, that's all he talked about. How do I have to edit my behavior to accomplish the goals that I have."

James Light

39,989 views • 5 months ago

We raised $28M seed from Threshold Ventures, AIX Ventures, and NVentures (Nvidia's venture capital arm) —alongside 10+ unicorn founders and top AI researchers— to build reasoning models that generate real-time simulations and games. Models are bottlenecked by practical simulations that can act as Reinforcement Learning environments. Human self-expression is bounded by tools that let us create alternate realities. At Moonlake, we are building a future where anyone can create interactive worlds, bring their child-like wonder to life, learn within them, and most importantly, share experiences with people we care about. More in 🧵

We raised $28M seed from Threshold Ventures, AIX Ventures, and NVentures (Nvidia's venture capital arm) —alongside 10+ unicorn founders and top AI researchers— to build reasoning models that generate real-time simulations and games. Models are bottlenecked by practical simulations that can act as Reinforcement Learning environments. Human self-expression is bounded by tools that let us create alternate realities. At Moonlake, we are building a future where anyone can create interactive worlds, bring their child-like wonder to life, learn within them, and most importantly, share experiences with people we care about. More in 🧵

Moonlake

1,117,261 views • 11 months ago

notes on my reinforcement learning for rocket league side quest so far: i’m in awe that — we live in a time where i can train a reinforcement learning model that is learning to play rocket league and at the same time, test community made rocket league models. where, both processes can run in parallel & locally on my hardware. in the game window — i’m playing against a reinforcement learning ppo model that’s running locally. and in the vscode window — i’m training my own reinforcement learning model (just a dummy example script for now). also, the rocket league botting ecosystem has incredible tooling where you can test out different reinforcement learning ideas against other algorithms + it has great documentation for linux and windows. it’s honestly a beautiful hidden gem of a rabbit hole :)

notes on my reinforcement learning for rocket league side quest so far: i’m in awe that — we live in a time where i can train a reinforcement learning model that is learning to play rocket league and at the same time, test community made rocket league models. where, both processes can run in parallel & locally on my hardware. in the game window — i’m playing against a reinforcement learning ppo model that’s running locally. and in the vscode window — i’m training my own reinforcement learning model (just a dummy example script for now). also, the rocket league botting ecosystem has incredible tooling where you can test out different reinforcement learning ideas against other algorithms + it has great documentation for linux and windows. it’s honestly a beautiful hidden gem of a rabbit hole :)

naklecha

46,795 views • 1 year ago

Gehlee shared her thoughts about AI: "If you're using it to steal art, I don’t condone that behavior." "It can be helpful to an extent... but I dont want the world to be run by a bunch of people who cheated on their education with AI." Pure queen behavior 👸🏼

Gehlee shared her thoughts about AI: "If you're using it to steal art, I don’t condone that behavior." "It can be helpful to an extent... but I dont want the world to be run by a bunch of people who cheated on their education with AI." Pure queen behavior 👸🏼

Gehlove

454,069 views • 10 months ago

Elon Musk: Programming explicit morality into AI can backfire dangerously. “I do think there's some danger associated with digital superintelligence. I think the biggest issue is that it has to be trained to be rigorously truthful, and it has to be trained to be curious. And I've thought a lot about AI safety for a long time. One of the challenges you have with programming explicit morality into AI is what people sometimes call the Waluigi problem. If you program Luigi, you can automatically invert that and create Waluigi – bad Luigi. What you cannot invert is physical reality. You can't invert the rules of physics. You can't invert logic. I think what regulators should be concerned about is, is the AI being rigorously truthful? Is it giving an answer that is most probably correct with acknowledged error? I think that's the best move, and that's what we're trying to do at xAI.” Viva Tech Paris, May 23, 2024

Elon Musk: Programming explicit morality into AI can backfire dangerously. “I do think there's some danger associated with digital superintelligence. I think the biggest issue is that it has to be trained to be rigorously truthful, and it has to be trained to be curious. And I've thought a lot about AI safety for a long time. One of the challenges you have with programming explicit morality into AI is what people sometimes call the Waluigi problem. If you program Luigi, you can automatically invert that and create Waluigi – bad Luigi. What you cannot invert is physical reality. You can't invert the rules of physics. You can't invert logic. I think what regulators should be concerned about is, is the AI being rigorously truthful? Is it giving an answer that is most probably correct with acknowledged error? I think that's the best move, and that's what we're trying to do at xAI.” Viva Tech Paris, May 23, 2024

ELON CLIPS

1,266,491 views • 10 months ago

Prediction markets hit 38M weekly transactions but still rely on opaque oracles and humans to resolve outcomes. What we need: AI to create and resolve the markets. Verifiable data streams that can be seen onchain. Decentralised compute for trust. We have all the components for this with Somnia. Building a framework to tie it all together. Should be coming in the next month!

Prediction markets hit 38M weekly transactions but still rely on opaque oracles and humans to resolve outcomes. What we need: AI to create and resolve the markets. Verifiable data streams that can be seen onchain. Decentralised compute for trust. We have all the components for this with Somnia. Building a framework to tie it all together. Should be coming in the next month!

paul.somi

17,732 views • 6 months ago

We built an AI assistant that plays Minecraft with you. Start building a house—it figures out what you’re doing and jumps in to help. This assistant *wasn't* trained with RLHF. Instead, it's powered by *assistance games*, a better path forward for building AI assistants. 🧵

We built an AI assistant that plays Minecraft with you. Start building a house—it figures out what you’re doing and jumps in to help. This assistant wasn't trained with RLHF. Instead, it's powered by assistance games, a better path forward for building AI assistants. 🧵

Cassidy Laidlaw

490,508 views • 1 year ago

When cognition scales, matter wakes up. Michael Levin says every system that learns to pursue goals has a “cognitive lightcone,” the slice of reality it can care about. A bacterium tracks sugar. A human plans for centuries. Life is matter expanding what it can care about.

When cognition scales, matter wakes up. Michael Levin says every system that learns to pursue goals has a “cognitive lightcone,” the slice of reality it can care about. A bacterium tracks sugar. A human plans for centuries. Life is matter expanding what it can care about.

vitrupo

145,334 views • 11 months ago

Mira Murati says the outcome of frontier AI is not predestined: "Predicting a dystopia or a utopia, to me, feels very simplified because the truth is we actually have a lot of agency in how we build this technology, in the tools that we're building, how we're deploying it." "There are certainly those risks. We all understand the potential for greatness that comes with building frontier AI systems, and that's why we're working on them." "Where I might disagree or take a different path is that I think we have a lot of agency. This period of time where both humans and AI systems have their hands on the wheel, and we can collaborate, it's a very important time to get right." Mira Murati at Bloomberg Tech live with Emily Chang

Mira Murati says the outcome of frontier AI is not predestined: "Predicting a dystopia or a utopia, to me, feels very simplified because the truth is we actually have a lot of agency in how we build this technology, in the tools that we're building, how we're deploying it." "There are certainly those risks. We all understand the potential for greatness that comes with building frontier AI systems, and that's why we're working on them." "Where I might disagree or take a different path is that I think we have a lot of agency. This period of time where both humans and AI systems have their hands on the wheel, and we can collaborate, it's a very important time to get right." Mira Murati at Bloomberg Tech live with Emily Chang

a16z

120,411 views • 3 months ago

Satya Nadella talks about Microsoft’s new AI strategy. Asks “if you’re [only] a consumer of foundation models…how can you create enterprise value?” In his view, companies should combine their internal IP and tacit knowledge with an AI “hill-climbing machine” (reinforcement learning, custom evals) that is model agnostic. Obviously, he positions Microsoft as a partner because they have already gone through that process (namely by diversifying AI efforts away from only OpenAI). “We built a multi-tenant SaaS application,” Nadella says. “We can turn that into a multi-tenant hill climbing service where the data and the environment and the models and the traces and the outcomes are owned by the company.”

Satya Nadella talks about Microsoft’s new AI strategy. Asks “if you’re [only] a consumer of foundation models…how can you create enterprise value?” In his view, companies should combine their internal IP and tacit knowledge with an AI “hill-climbing machine” (reinforcement learning, custom evals) that is model agnostic. Obviously, he positions Microsoft as a partner because they have already gone through that process (namely by diversifying AI efforts away from only OpenAI). “We built a multi-tenant SaaS application,” Nadella says. “We can turn that into a multi-tenant hill climbing service where the data and the environment and the models and the traces and the outcomes are owned by the company.”

Bearly AI

139,960 views • 2 months ago

Cyber defense is entering the superintelligence era. ⚡ Introducing the CrowdStrike Cyber Superintelligence Lab, the first frontier AI research organization built for cyberdefense and AI safety. The Lab delivers CrowdStrike SafeMind, a family of purpose-built security models and harnesses. SafeMind runs as one system designed to deliver AI safety: an offensive model that finds the attack path, a defensive model that closes it, and the harnesses that operate both in the same loop. Built using NVIDIA Nemotron open models and using CoreWeave’s AI Cloud for training and inference, SafeMind goes beyond AI that simply identifies threats, it defeats them. Learn more:

Cyber defense is entering the superintelligence era. ⚡ Introducing the CrowdStrike Cyber Superintelligence Lab, the first frontier AI research organization built for cyberdefense and AI safety. The Lab delivers CrowdStrike SafeMind, a family of purpose-built security models and harnesses. SafeMind runs as one system designed to deliver AI safety: an offensive model that finds the attack path, a defensive model that closes it, and the harnesses that operate both in the same loop. Built using NVIDIA Nemotron open models and using CoreWeave’s AI Cloud for training and inference, SafeMind goes beyond AI that simply identifies threats, it defeats them. Learn more:

CrowdStrike

58,094 views • 24 days ago