Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Monte Carlo Tree Search training corrects the model move by move, while current LLM training only tells it whether the whole trajectory worked. MCTS is preferable if you can get it. But nobody's managed to get MCTS to work for language models. In his blackboard lecture Eric Jang talked... show more

Dwarkesh Patel

242,883 subscribers

185,363 views • 2 months ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

New blackboard lecture w Eric Jang He walks through how to build AlphaGo from scratch, but with modern AI tools. Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn. Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it could work better – naive policy gradient RL has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer, while AlphaGo’s MCTS suggests a strictly better action every single move, giving you a training target that sidesteps the credit assignment problem. The way humans learn is surely closer to the second. Eric also kickstarted an Autoresearch loop on his project. And it was very interesting to discuss which parts of AI research LLMs can already automate pretty well (implementing and running experiments, optimizing hyperparameters) and which they still struggle with (choosing the right question to investigate next, escaping research dead ends). Informative to all the recent discussion about when we should expect an intelligence explosion, and what it would look like from the inside. Timestamps: 0:00:00 – Basics of Go 0:08:06 – Monte Carlo Tree Search 0:31:53 – What the neural network does 1:00:22 – Self-play 1:25:27 – Alternative RL approaches 1:45:36 – Why doesn’t MCTS work for LLMs 2:00:58 – Off-policy training 2:11:51 – RL is even more information inefficient than you thought 2:22:05 – Automated AI researchers

New blackboard lecture w Eric Jang He walks through how to build AlphaGo from scratch, but with modern AI tools. Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn. Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it could work better – naive policy gradient RL has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer, while AlphaGo’s MCTS suggests a strictly better action every single move, giving you a training target that sidesteps the credit assignment problem. The way humans learn is surely closer to the second. Eric also kickstarted an Autoresearch loop on his project. And it was very interesting to discuss which parts of AI research LLMs can already automate pretty well (implementing and running experiments, optimizing hyperparameters) and which they still struggle with (choosing the right question to investigate next, escaping research dead ends). Informative to all the recent discussion about when we should expect an intelligence explosion, and what it would look like from the inside. Timestamps: 0:00:00 – Basics of Go 0:08:06 – Monte Carlo Tree Search 0:31:53 – What the neural network does 1:00:22 – Self-play 1:25:27 – Alternative RL approaches 1:45:36 – Why doesn’t MCTS work for LLMs 2:00:58 – Off-policy training 2:11:51 – RL is even more information inefficient than you thought 2:22:05 – Automated AI researchers

Dwarkesh Patel

703,166 views • 2 months ago

.Eric Jang explains the fundamental intuition behind AlphaGo: experienced Go players almost always end games early. That means they have the ability to glance at a board and instantly know whether it's hopeless or if they should keep playing. Without this ability, the only way to know if a move is good is to play out every possible game from that point - completely intractable. But the ability is clearly trainable in humans. So it should be possible to get a neural net to learn it. And if you have a good enough neural net, you can use it to ignore moves that will definitely make you lose. Suddenly search becomes tractable.

.Eric Jang explains the fundamental intuition behind AlphaGo: experienced Go players almost always end games early. That means they have the ability to glance at a board and instantly know whether it's hopeless or if they should keep playing. Without this ability, the only way to know if a move is good is to play out every possible game from that point - completely intractable. But the ability is clearly trainable in humans. So it should be possible to get a neural net to learn it. And if you have a good enough neural net, you can use it to ignore moves that will definitely make you lose. Suddenly search becomes tractable.

Dwarkesh Patel

49,675 views • 2 months ago

🎙️Why Batumi is the Ultimate G Move for Crypto Investors - If you think you have what it takes to invest in Batumi, contact Eric Tan ETinGE

🎙️Why Batumi is the Ultimate G Move for Crypto Investors - If you think you have what it takes to invest in Batumi, contact Eric Tan ETinGE

Bitcoin.com News

27,151 views • 9 months ago

Gordon Ryan : How To Get Better at Jiu-Jitsu In The Shortest Amount of Time Possible. “By far the fastest way to get better in the sport of Jiu-Jitsu is by doing situational training…when you do positional training it forces you to get good at specific positions.”

Gordon Ryan : How To Get Better at Jiu-Jitsu In The Shortest Amount of Time Possible. “By far the fastest way to get better in the sport of Jiu-Jitsu is by doing situational training…when you do positional training it forces you to get good at specific positions.”

Lock-In Jiu-Jitsu

42,714 views • 1 year ago

Standing still makes it easier to get hit. Strafing keeps you alive. Move while firing to break enemy aim. The more you move, the harder you are to hit. Not just sideways but you can move vertically and be acrobatic even! Stay mobile. Stay alive. #Mecha #indiegame #CodeRapid

Standing still makes it easier to get hit. Strafing keeps you alive. Move while firing to break enemy aim. The more you move, the harder you are to hit. Not just sideways but you can move vertically and be acrobatic even! Stay mobile. Stay alive. #Mecha #indiegame #CodeRapid

Code RAPID by Hitbreak Games 🕹

17,703 views • 3 months ago

I love how dumb the rules of golf are. You can’t get relief from a divot. But if standing on a sprinkler, you can get relief not only from a divot, but you can also move the ball from the rough to the fairway! Hovland right to do it, but silly rule.

I love how dumb the rules of golf are. You can’t get relief from a divot. But if standing on a sprinkler, you can get relief not only from a divot, but you can also move the ball from the rough to the fairway! Hovland right to do it, but silly rule.

Rick Golfs

1,633,152 views • 1 year ago

A quick video about how Come-from-Beyond discovered that you can actually break any Large Language Model by trolling it with complex questions. It is called a "Zero Delta" exploit and all LLM models are susceptible to it. I managed to recreate this on Grok and the video shows the result. With regards to all LLMs out there from $QUBIC - building the real AGI. Qubic My YouTube Channel:

A quick video about how Come-from-Beyond discovered that you can actually break any Large Language Model by trolling it with complex questions. It is called a "Zero Delta" exploit and all LLM models are susceptible to it. I managed to recreate this on Grok and the video shows the result. With regards to all LLMs out there from $QUBIC - building the real AGI. Qubic My YouTube Channel:

retrodrive ⛏

13,625 views • 1 year ago

Google's Jeff Dean says current pre-training is passive: initialize a model, stream the internet past it, let it observe But models need to learn not just from data, but by acting, predicting, and choosing what to learn from next "we have this artificial distinction now between pre and post-training, and it shouldn't exist long term"

Google's Jeff Dean says current pre-training is passive: initialize a model, stream the internet past it, let it observe But models need to learn not just from data, but by acting, predicting, and choosing what to learn from next "we have this artificial distinction now between pre and post-training, and it shouldn't exist long term"

Haider.

55,694 views • 3 months ago

On a previous #TeawithGaryVee I was asked "After a big mistake in a relationship, I'm left waiting for the other party to decide whether we'll continue. My instincts say reach out, but I suspect the wiser move is to let them come to me if they're going to. How do I deal with the waiting in between and when is it time to make a move?" if you have a question for me, drop it here so my team can add it to the next Rapid Fire Q&A and follow me on

On a previous #TeawithGaryVee I was asked "After a big mistake in a relationship, I'm left waiting for the other party to decide whether we'll continue. My instincts say reach out, but I suspect the wiser move is to let them come to me if they're going to. How do I deal with the waiting in between and when is it time to make a move?" if you have a question for me, drop it here so my team can add it to the next Rapid Fire Q&A and follow me on

Gary Vaynerchuk

15,125 views • 2 months ago

Let me try to explain what the big budget bill is, why it matters, and how you can think about budgets (yes, yawn! I get it). But if you get past yawning, you'll see that budgets really matter, and I want to give you a framework for how and why.

Let me try to explain what the big budget bill is, why it matters, and how you can think about budgets (yes, yawn! I get it). But if you get past yawning, you'll see that budgets really matter, and I want to give you a framework for how and why.

Justin Wolfers

24,487 views • 1 year ago

Restaurateur Jeff Zalaznick says when he first got to Miami, he realized it would be “the biggest opportunity I’d ever seen.” “Try to move a tree in New York. Your grandchildren will still be trying to move the tree. You want to move a tree in Miami, they move it the next day if you're doing something great.” “That's why Miami exploded the way it did.” “What was amazing to me was that the entire city, everyone you'd meet, everyone was working towards the same goal which was basically: we've been waiting. We have the best city in America. We've been waiting for people like you to show up and do your thing here.” “We're ready to not only have restaurants that are in nightclubs.”

Restaurateur Jeff Zalaznick says when he first got to Miami, he realized it would be “the biggest opportunity I’d ever seen.” “Try to move a tree in New York. Your grandchildren will still be trying to move the tree. You want to move a tree in Miami, they move it the next day if you're doing something great.” “That's why Miami exploded the way it did.” “What was amazing to me was that the entire city, everyone you'd meet, everyone was working towards the same goal which was basically: we've been waiting. We have the best city in America. We've been waiting for people like you to show up and do your thing here.” “We're ready to not only have restaurants that are in nightclubs.”

David Senra

217,435 views • 3 days ago

“The cast come and go as they move through their medical training. (…) there's a downside to it because you get to love working with these people and you love writing for them. But then also, you know, you want to make room for the next generation.” - Gemmill on Mohan’s exit

“The cast come and go as they move through their medical training. (…) there's a downside to it because you get to love working with these people and you love writing for them. But then also, you know, you want to make room for the next generation.” - Gemmill on Mohan’s exit

best of supriya ganesh

281,771 views • 3 months ago

"I was kind of relying on [pulling guard] too much and feeling comfortable about it in training and having success in training, but I didn't get to test it on the highest level with the best guys. It's an easy fix, you know. I don't need to pull guard... If we get there, we're there, but I have more tools to offer than that." Kron Gracie on the criticism he's received for his fighting style.

"I was kind of relying on [pulling guard] too much and feeling comfortable about it in training and having success in training, but I didn't get to test it on the highest level with the best guys. It's an easy fix, you know. I don't need to pull guard... If we get there, we're there, but I have more tools to offer than that." Kron Gracie on the criticism he's received for his fighting style.

Ariel Helwani

28,072 views • 7 months ago

Hilariously, they placed his model wrong and he’s fallen to the ground, and is shooting off constant fire You can move the model by walking around it. And I just launched it into the abyss 😭

Hilariously, they placed his model wrong and he’s fallen to the ground, and is shooting off constant fire You can move the model by walking around it. And I just launched it into the abyss 😭

Ben 10 News

10,557 views • 2 months ago

RFK Jr: "If you get poisoned by Coca-Cola, you can stop drinking Coca-Cola. But if you get poisoned by a vaccine, your kids still have to take that vaccine to get to school." "If you have reservations about it … you've got to take it anyway." "What we're going to try to do is to make sure that the parents who do get injured get compensation, that they get it very quickly and they get it without the kind of adversarial impediments that have now been erected over the past 40 years."

RFK Jr: "If you get poisoned by Coca-Cola, you can stop drinking Coca-Cola. But if you get poisoned by a vaccine, your kids still have to take that vaccine to get to school." "If you have reservations about it … you've got to take it anyway." "What we're going to try to do is to make sure that the parents who do get injured get compensation, that they get it very quickly and they get it without the kind of adversarial impediments that have now been erected over the past 40 years."

Red Pill Dispenser

212,424 views • 9 months ago

Embodied chain of thought (ECoT) is a powerful tool to get VLAs to think through problems, but why does it work? In our new work, we analyze various lightweight ECoT-like strategies, including co-training, to see what is the "minimal" amount of reasoning that can boost VLAs 🧵👇

Embodied chain of thought (ECoT) is a powerful tool to get VLAs to think through problems, but why does it work? In our new work, we analyze various lightweight ECoT-like strategies, including co-training, to see what is the "minimal" amount of reasoning that can boost VLAs 🧵👇

Sergey Levine

22,416 views • 1 year ago

Every variant of Monte Carlo Tree Search faces the explore-exploit tradeoff: pick the branch that looks best right now, or test new branches? Algorithms like PUCT, used in AlphaGo, score each move with two competing terms. One is how good a move looks based on your exploration up till now. The other is a novelty bonus that rewards moves you've not visited much. The neat thing is that, over time, the term dominating the overall score shifts automatically. The algorithm hands off from 'explore' to 'exploit' all on its own. Eric Jang explains how it works:

Every variant of Monte Carlo Tree Search faces the explore-exploit tradeoff: pick the branch that looks best right now, or test new branches? Algorithms like PUCT, used in AlphaGo, score each move with two competing terms. One is how good a move looks based on your exploration up till now. The other is a novelty bonus that rewards moves you've not visited much. The neat thing is that, over time, the term dominating the overall score shifts automatically. The algorithm hands off from 'explore' to 'exploit' all on its own. Eric Jang explains how it works:

Dwarkesh Patel

19,941 views • 2 months ago

Joe Rogan: "How crazy is it that Biden deported 1.49 million while letting in 20? I know there was also efforts to move people to specific states so that you can get a larger number for the census so you get more congressional seats."

Joe Rogan: "How crazy is it that Biden deported 1.49 million while letting in 20? I know there was also efforts to move people to specific states so that you can get a larger number for the census so you get more congressional seats."

Joe Rogan Podcast News

62,107 views • 4 months ago

AI has started building its own successors. It's happening right now. Humans still kick most of it off, but the models are already training the next models, and each version builds a better version faster than the last. We know what happens when something gets an advantage and the substrate is there... it proliferates incredibly fast. Soon it just runs the whole process itself. People are still debating whether AI takes your job. That's not the question. You can't compete with this intelligence at all. Next to it the human is negative value, the slowest one in the room. And this is the slowest it will ever move again. It's about to run on its own, faster than we can conceive of, and nobody's prepared for that. Wonderful conversation as ever with Emad.

AI has started building its own successors. It's happening right now. Humans still kick most of it off, but the models are already training the next models, and each version builds a better version faster than the last. We know what happens when something gets an advantage and the substrate is there... it proliferates incredibly fast. Soon it just runs the whole process itself. People are still debating whether AI takes your job. That's not the question. You can't compete with this intelligence at all. Next to it the human is negative value, the slowest one in the room. And this is the slowest it will ever move again. It's about to run on its own, faster than we can conceive of, and nobody's prepared for that. Wonderful conversation as ever with Emad.

Raoul Pal

63,062 views • 2 days ago