正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Task diversity is supposedly key to generalization in RL. But what does it do to continual RL, where agents face one new task distribution after another? We find that past a point, more diversity actually inhibits continual reinforcement learning 🧵

Wilka Carvalho

1,868 subscribers

15,333 次观看 • 5 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

19 条评论

Wilka Carvalho 的头像

Wilka Carvalho5 个月前

In ML, diversity is the standard route to generalization. For example, domain randomization has shown that visual diversity enables zero-shot sim-to-real and XLand showed that agents trained on diverse tasks could then solve millions of new ones zero-shot.

Wilka Carvalho 的头像

Wilka Carvalho5 个月前

Interestingly, diversity has even been key to cognitive science results. Lake & Baroni showed that the right diverse task distribution induced systematicity in neural networks! A question open since Fodor & Pylyshyn. But nobody has studied what diversity does to continual learning.

Wilka Carvalho 的头像

Wilka Carvalho5 个月前

So we built Banyan: a procedurally-generated environment where you can systematically vary the components that define a task. A Banyan task is composed of a layout & a task tree, where each tree is defined by its topology and the objects assigned to its nodes.

Wilka Carvalho 的头像

Wilka Carvalho5 个月前

Vary any single component and agents begin a new distribution at or above the same performance that they finished the previous one with! This is true for both PPO and PQN.

Wilka Carvalho 的头像

Wilka Carvalho5 个月前

We analyze backward transfer and find that varying the number of layouts or object assignments has no effect, but varying the number of topologies has a big effect! Again, this is true for both PPO and PQN.

Wilka Carvalho 的头像

Wilka Carvalho5 个月前

So what happens across a sequence of ten distributions? We co-vary the number of layouts with the number of topologies, and find...

Wilka Carvalho 的头像

Wilka Carvalho5 个月前

Task diversity inhibits continual reinforcement learning! At medium diversity (16 layouts × 16 topologies), performance climbs with each new distribution. But past that point, it monotonically gets worse.

Wilka Carvalho 的头像

Wilka Carvalho5 个月前

Strikingly, higher diversity improves backward transfer to the very first distribution! So the agent keeps getting better on old tasks even as it stops improving on new ones. This suggests it's learning the shared structure, but losing the ability to specialize.

Wilka Carvalho 的头像

Wilka Carvalho5 个月前

If you want to learn more, or see the rest of our analysis, please check out our paper! 📄 Paper: 🌐 Project page: This was a fun collaboration with Purab Seth, @neilhshah15, @gershbrain, @kjha02, and @maxhkw !

Sam Z Liu 的头像

Sam Z Liu5 个月前

Could another view on this be that RL has a saturation point on its ability to learn tasks and early task diversity saturates that more quickly? Because the improvement in earlier tasks even at the later steps suggests that diversity is somehow still beneficial

Wilka Carvalho 的头像

Wilka Carvalho4 个月前

That makes sense! Yeah, one explanation is that this could be the primacy bias in effect

Stephanie Chan 的头像

Stephanie Chan4 个月前

@samzliu Maybe also loss of plasticity?

Wilka Carvalho 的头像

Wilka Carvalho4 个月前

@samzliu Thank you for this pointer!

mint yolk 的头像

mint yolk5 个月前

Very cool! Do you have an idea of how soon the code will be released?

Wilka Carvalho 的头像

Wilka Carvalho5 个月前

aiming to have the code released in 2 months!

mint yolk 的头像

mint yolk5 个月前

😭😭 thank you anyway haha

Wilka Carvalho 的头像

Wilka Carvalho5 个月前

code isn't cleaned up so if you don't mind working with a messier version of the code and want to use it on a project, DM me! students I worked with are busy this summer so limited on time

mint yolk 的头像

mint yolk4 个月前

DM’d

Wilka Carvalho 的头像

Wilka Carvalho4 个月前

don't see the DM

相关视频

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->

Chris Paxton

34,102 次观看 • 4 个月前

1/ Introducing RL Swarm’s new backend: GenRL. A modular reinforcement learning library built for distributed, fault-tolerant training - now powering RL Swarm from the ground up. 🧵

1/ Introducing RL Swarm’s new backend: GenRL. A modular reinforcement learning library built for distributed, fault-tolerant training - now powering RL Swarm from the ground up. 🧵

gensyn

82,215 次观看 • 1 年前

capability != learning new benchtalks with Parth Asawa on continual learning, where we discuss teaching models to learn from experience, measuring learning ability, the bet on parametric models, and more 01:06 What is continual learning? 04:10 Why capability and learning are different 06:13 Why build a benchmark? 08:07 Continual Learning Bench launch and reception 09:13 Anthropic's Fable release and Continual Learning Bench 11:02 How to design tasks for continual learning 18:41 The gain metric 24:01 What good looks like on the leaderboard 29:13 Failure modes: why models can't update their beliefs 31:12 Parametric systems and future architectures 34:30 Open science and AI safety 45:42 Lightning round 49:08 How to contribute to Continual Learning Bench

capability != learning new benchtalks with Parth Asawa on continual learning, where we discuss teaching models to learn from experience, measuring learning ability, the bet on parametric models, and more 01:06 What is continual learning? 04:10 Why capability and learning are different 06:13 Why build a benchmark? 08:07 Continual Learning Bench launch and reception 09:13 Anthropic's Fable release and Continual Learning Bench 11:02 How to design tasks for continual learning 18:41 The gain metric 24:01 What good looks like on the leaderboard 29:13 Failure modes: why models can't update their beliefs 31:12 Parametric systems and future architectures 34:30 Open science and AI safety 45:42 Lightning round 49:08 How to contribute to Continual Learning Bench

vincent sunn chen

41,770 次观看 • 4 个月前

Richard Sutton argues that AI must move beyond human-generated static data into the “Era of Experience,” where agents learn through continuous interaction with the world. This will require building upon RL with better algorithms capable of continual learning and meta-learning.

Richard Sutton argues that AI must move beyond human-generated static data into the “Era of Experience,” where agents learn through continuous interaction with the world. This will require building upon RL with better algorithms capable of continual learning and meta-learning.

The Humanoid Hub

31,493 次观看 • 1 年前

This is the Claude Code moment for Reinforcement Learning. Every frontier lab knows the secret: post-training is where the magic happens. If you can eval a task, you can use RL to benchmax your model. But building and scaling the RL loop has remained a dark art reserved for the top 1% of AI Labs. Until today. Introducing Optimus — an autonomous RL Agent Harness, and the first of many we are about to ship Iacon Autonomics. We are removing the manual grind of RL and adding taste and relentlessness to the process. Just hand Optimus a codebase, an API key, and a goal. It spins up a fleet of sub-agents to run hundreds of parallel experiments. It learns from its own history, autonomously building an evolutionary tree of runs where every failure sets the next batch up to win. Train LLMs, agents, robots in sim, game npcs and much more. If you can score it, Optimus can train it. Stop doing manual RL. Let the agents cook.

This is the Claude Code moment for Reinforcement Learning. Every frontier lab knows the secret: post-training is where the magic happens. If you can eval a task, you can use RL to benchmax your model. But building and scaling the RL loop has remained a dark art reserved for the top 1% of AI Labs. Until today. Introducing Optimus — an autonomous RL Agent Harness, and the first of many we are about to ship Iacon Autonomics. We are removing the manual grind of RL and adding taste and relentlessness to the process. Just hand Optimus a codebase, an API key, and a goal. It spins up a fleet of sub-agents to run hundreds of parallel experiments. It learns from its own history, autonomously building an evolutionary tree of runs where every failure sets the next batch up to win. Train LLMs, agents, robots in sim, game npcs and much more. If you can score it, Optimus can train it. Stop doing manual RL. Let the agents cook.

Dev Shah

13,730 次观看 • 2 个月前

Today we present Morpheus, a persistent enterprise simulation platform designed to make Continual Learning a reality. Morpheus is the world’s first real world Reinforcement Learning environment. Every Reinforcement Learning environment operates in the game world. Benchmarks like Atari, OpenAI Gym, MuJoCo, and Procgen are all small, game-like worlds that reset every few minutes. But the real world never resets. A business keeps running and evolving everyday. We tested how frontier LLMs would perform in realistic and dynamic business environments 🧬on Morpheus. The main conclusion was that LLMs are not continual learners. 🧵Here’s how we did it and what we learned:

Today we present Morpheus, a persistent enterprise simulation platform designed to make Continual Learning a reality. Morpheus is the world’s first real world Reinforcement Learning environment. Every Reinforcement Learning environment operates in the game world. Benchmarks like Atari, OpenAI Gym, MuJoCo, and Procgen are all small, game-like worlds that reset every few minutes. But the real world never resets. A business keeps running and evolving everyday. We tested how frontier LLMs would perform in realistic and dynamic business environments 🧬on Morpheus. The main conclusion was that LLMs are not continual learners. 🧵Here’s how we did it and what we learned:

Skyfall AI

269,894 次观看 • 3 个月前

KL divergence is a fundamental concept used in machine learning, from optimization a neural net to RL training of LLMs. but here is what it actually means:

KL divergence is a fundamental concept used in machine learning, from optimization a neural net to RL training of LLMs. but here is what it actually means:

ℏεsam

20,381 次观看 • 9 个月前

Continual learning sometimes gets discussed as if the goal is to dissolve the context/weights distinction. Let the model just keep accumulating, fine-tuning itself on the fly. Andrej Karpathy points out, though, that this isn't how humans do it. Our working memory gets wiped regularly. What we actually have is a consolidation process (sleep) that distills stuff into the brain, in a weird and lossy way. This is very different from how people sometimes talk about continual learning. It's not obvious it's something you can get for free from doing long enough RL loops.

Continual learning sometimes gets discussed as if the goal is to dissolve the context/weights distinction. Let the model just keep accumulating, fine-tuning itself on the fly. Andrej Karpathy points out, though, that this isn't how humans do it. Our working memory gets wiped regularly. What we actually have is a consolidation process (sleep) that distills stuff into the brain, in a weird and lossy way. This is very different from how people sometimes talk about continual learning. It's not obvious it's something you can get for free from doing long enough RL loops.

Dwarkesh Patel

60,477 次观看 • 5 个月前

We asked Sholto Douglas from Anthropic about the costs of RL (Reinforcement Learning) runs. "In Dario Amodei's essay, he said that RL runs cost only $1M back in December." "RL is a more naively parallelizable and scalable than pre-training." "With pre-training, you need everything in one big data center ideally. For RL, in theory, you could scale all over the world."

We asked Sholto Douglas from Anthropic about the costs of RL (Reinforcement Learning) runs. "In Dario Amodei's essay, he said that RL runs cost only $1M back in December." "RL is a more naively parallelizable and scalable than pre-training." "With pre-training, you need everything in one big data center ideally. For RL, in theory, you could scale all over the world."

TBPN

76,696 次观看 • 1 年前

Introducing RL Environment Explorer 🧭 There are now 12M+ RL task entries across RL environments on the Hugging Face Hub. You can now explore, visualize, and run RL environments across: OpenEnv • Harbor • MiMo • Verifiers • NeMo Gym All in one place

Introducing RL Environment Explorer 🧭 There are now 12M+ RL task entries across RL environments on the Hugging Face Hub. You can now explore, visualize, and run RL environments across: OpenEnv • Harbor • MiMo • Verifiers • NeMo Gym All in one place

Adithya S K

27,719 次观看 • 23 天前

Our work, "A Primer on SO(3) Action Representations in Deep Reinforcement Learning," was accepted to #ICLR2026! We provide a systematic study of action representation choices in RL, showing that they fundamentally impact training stability and performance. #Robotics #AI #RL

Our work, "A Primer on SO(3) Action Representations in Deep Reinforcement Learning," was accepted to #ICLR2026! We provide a systematic study of action representation choices in RL, showing that they fundamentally impact training stability and performance. #Robotics #AI #RL

Learning Systems and Robotics Lab (is hiring!)

50,130 次观看 • 8 个月前

Want to scale RL with your shiny new GPU? 🚀 In our ICML24 Oral we find that RL algorithms hit a barrier when data is scaled up. Our new algorithm, SAPG, proposes a simple fix. It scales to 25k envs and solves hard tasks where PPO makes no progress. 1/n

Want to scale RL with your shiny new GPU? 🚀 In our ICML24 Oral we find that RL algorithms hit a barrier when data is scaled up. Our new algorithm, SAPG, proposes a simple fix. It scales to 25k envs and solves hard tasks where PPO makes no progress. 1/n

Ananye Agarwal

64,610 次观看 • 2 年前

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Jingyun Yang

88,312 次观看 • 3 年前

Daniel Kokotajlo says companies are making AI more agentic by scaling RL (reinforcement learning) and connecting agents to various tools. by early 2027, these agents will be fully autonomous, superhuman coders able to replace human programmers.

Daniel Kokotajlo says companies are making AI more agentic by scaling RL (reinforcement learning) and connecting agents to various tools. by early 2027, these agents will be fully autonomous, superhuman coders able to replace human programmers.

Haider.

11,572 次观看 • 1 年前

RL-100 Performant Robotic Manipulation with Real-World Reinforcement Learning

RL-100 Performant Robotic Manipulation with Real-World Reinforcement Learning

AK

15,364 次观看 • 1 年前

Introducing RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 7 real robot tasks, 900/900 successes. Up to 250 consecutive trials in one task, running 2 hours nonstop without failure. High success rate against physical disturbances, zero-shot, and few-shot adaptation Our first step toward a deployable robot learning system.

Introducing RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 7 real robot tasks, 900/900 successes. Up to 250 consecutive trials in one task, running 2 hours nonstop without failure. High success rate against physical disturbances, zero-shot, and few-shot adaptation Our first step toward a deployable robot learning system.

Kun Lei

92,774 次观看 • 1 年前

Self-supervised representation learning looks a bit like RL. What if we literally use RL as a SSL method for visual representations? Turns out that it works quite well. In new work by Dibya Ghosh, we show how this can be done:

Self-supervised representation learning looks a bit like RL. What if we literally use RL as a SSL method for visual representations? Turns out that it works quite well. In new work by Dibya Ghosh, we show how this can be done:

Sergey Levine

48,840 次观看 • 1 年前

NeurIPS 2025 Paper: LLMs are Reinforcement Learners 🤯! Surprisingly, we show that LLMs can solve RL tasks without any external component! We introduce Prompted Policy Search (ProPS), an RL method based only LLMs and in-context learning. [Paper]

NeurIPS 2025 Paper: LLMs are Reinforcement Learners 🤯! Surprisingly, we show that LLMs can solve RL tasks without any external component! We introduce Prompted Policy Search (ProPS), an RL method based only LLMs and in-context learning. [Paper]

Heni Ben Amor

51,248 次观看 • 11 个月前

Using our brain simulator, we’ve trained a reinforcement learning agent to maximize bits per second. Here is the RL policy converting brain data to cursor control in simulation:

Using our brain simulator, we’ve trained a reinforcement learning agent to maximize bits per second. Here is the RL policy converting brain data to cursor control in simulation:

Neuralink

18,410 次观看 • 1 年前

We are excited to announce CLIN 🤖: The first continually learning language agent that excels in both task adaptation and generalization to unseen tasks and environments in a pure zero-shot setup. Aristo Team at Ai2 Ai2 Website: Let's dive in 🧵 (1/n)

We are excited to announce CLIN 🤖: The first continually learning language agent that excels in both task adaptation and generalization to unseen tasks and environments in a pure zero-shot setup. Aristo Team at Ai2 Ai2 Website: Let's dive in 🧵 (1/n)

Bodhisattwa Majumder

35,062 次观看 • 3 年前