Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

Task diversity is supposedly key to generalization in RL. But what does it do to continual RL, where agents face one new task distribution after another? We find that past a point, more diversity actually inhibits continual reinforcement learning 🧵

Wilka Carvalho

1,868 subscribers

15,333 просмотров • 5 месяцев назад •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

Комментарии: 19

Фото профиля Wilka Carvalho

Wilka Carvalho5 месяцев назад

In ML, diversity is the standard route to generalization. For example, domain randomization has shown that visual diversity enables zero-shot sim-to-real and XLand showed that agents trained on diverse tasks could then solve millions of new ones zero-shot.

Фото профиля Wilka Carvalho

Wilka Carvalho5 месяцев назад

Interestingly, diversity has even been key to cognitive science results. Lake & Baroni showed that the right diverse task distribution induced systematicity in neural networks! A question open since Fodor & Pylyshyn. But nobody has studied what diversity does to continual learning.

Фото профиля Wilka Carvalho

Wilka Carvalho5 месяцев назад

So we built Banyan: a procedurally-generated environment where you can systematically vary the components that define a task. A Banyan task is composed of a layout & a task tree, where each tree is defined by its topology and the objects assigned to its nodes.

Фото профиля Wilka Carvalho

Wilka Carvalho5 месяцев назад

Vary any single component and agents begin a new distribution at or above the same performance that they finished the previous one with! This is true for both PPO and PQN.

Фото профиля Wilka Carvalho

Wilka Carvalho5 месяцев назад

We analyze backward transfer and find that varying the number of layouts or object assignments has no effect, but varying the number of topologies has a big effect! Again, this is true for both PPO and PQN.

Фото профиля Wilka Carvalho

Wilka Carvalho5 месяцев назад

So what happens across a sequence of ten distributions? We co-vary the number of layouts with the number of topologies, and find...

Фото профиля Wilka Carvalho

Wilka Carvalho5 месяцев назад

Task diversity inhibits continual reinforcement learning! At medium diversity (16 layouts × 16 topologies), performance climbs with each new distribution. But past that point, it monotonically gets worse.

Фото профиля Wilka Carvalho

Wilka Carvalho5 месяцев назад

Strikingly, higher diversity improves backward transfer to the very first distribution! So the agent keeps getting better on old tasks even as it stops improving on new ones. This suggests it's learning the shared structure, but losing the ability to specialize.

Фото профиля Wilka Carvalho

Wilka Carvalho5 месяцев назад

If you want to learn more, or see the rest of our analysis, please check out our paper! 📄 Paper: 🌐 Project page: This was a fun collaboration with Purab Seth, @neilhshah15, @gershbrain, @kjha02, and @maxhkw !

Фото профиля Sam Z Liu

Sam Z Liu4 месяцев назад

Could another view on this be that RL has a saturation point on its ability to learn tasks and early task diversity saturates that more quickly? Because the improvement in earlier tasks even at the later steps suggests that diversity is somehow still beneficial

Фото профиля Wilka Carvalho

Wilka Carvalho4 месяцев назад

That makes sense! Yeah, one explanation is that this could be the primacy bias in effect

Фото профиля Stephanie Chan

Stephanie Chan4 месяцев назад

@samzliu Maybe also loss of plasticity?

Фото профиля Wilka Carvalho

Wilka Carvalho4 месяцев назад

@samzliu Thank you for this pointer!

Фото профиля mint yolk

mint yolk5 месяцев назад

Very cool! Do you have an idea of how soon the code will be released?

Фото профиля Wilka Carvalho

Wilka Carvalho5 месяцев назад

aiming to have the code released in 2 months!

Фото профиля mint yolk

mint yolk5 месяцев назад

😭😭 thank you anyway haha

Фото профиля Wilka Carvalho

Wilka Carvalho5 месяцев назад

code isn't cleaned up so if you don't mind working with a messier version of the code and want to use it on a project, DM me! students I worked with are busy this summer so limited on time

Фото профиля mint yolk

mint yolk4 месяцев назад

DM’d

Фото профиля Wilka Carvalho

Wilka Carvalho4 месяцев назад

don't see the DM

Похожие видео

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->

Chris Paxton

34,102 просмотров • 4 месяцев назад

1/ Introducing RL Swarm’s new backend: GenRL. A modular reinforcement learning library built for distributed, fault-tolerant training - now powering RL Swarm from the ground up. 🧵

1/ Introducing RL Swarm’s new backend: GenRL. A modular reinforcement learning library built for distributed, fault-tolerant training - now powering RL Swarm from the ground up. 🧵

gensyn

82,215 просмотров • 1 год назад

capability != learning new benchtalks with Parth Asawa on continual learning, where we discuss teaching models to learn from experience, measuring learning ability, the bet on parametric models, and more 01:06 What is continual learning? 04:10 Why capability and learning are different 06:13 Why build a benchmark? 08:07 Continual Learning Bench launch and reception 09:13 Anthropic's Fable release and Continual Learning Bench 11:02 How to design tasks for continual learning 18:41 The gain metric 24:01 What good looks like on the leaderboard 29:13 Failure modes: why models can't update their beliefs 31:12 Parametric systems and future architectures 34:30 Open science and AI safety 45:42 Lightning round 49:08 How to contribute to Continual Learning Bench

capability != learning new benchtalks with Parth Asawa on continual learning, where we discuss teaching models to learn from experience, measuring learning ability, the bet on parametric models, and more 01:06 What is continual learning? 04:10 Why capability and learning are different 06:13 Why build a benchmark? 08:07 Continual Learning Bench launch and reception 09:13 Anthropic's Fable release and Continual Learning Bench 11:02 How to design tasks for continual learning 18:41 The gain metric 24:01 What good looks like on the leaderboard 29:13 Failure modes: why models can't update their beliefs 31:12 Parametric systems and future architectures 34:30 Open science and AI safety 45:42 Lightning round 49:08 How to contribute to Continual Learning Bench

vincent sunn chen

41,770 просмотров • 4 месяцев назад

Richard Sutton argues that AI must move beyond human-generated static data into the “Era of Experience,” where agents learn through continuous interaction with the world. This will require building upon RL with better algorithms capable of continual learning and meta-learning.

Richard Sutton argues that AI must move beyond human-generated static data into the “Era of Experience,” where agents learn through continuous interaction with the world. This will require building upon RL with better algorithms capable of continual learning and meta-learning.

The Humanoid Hub

31,493 просмотров • 1 год назад

This is the Claude Code moment for Reinforcement Learning. Every frontier lab knows the secret: post-training is where the magic happens. If you can eval a task, you can use RL to benchmax your model. But building and scaling the RL loop has remained a dark art reserved for the top 1% of AI Labs. Until today. Introducing Optimus — an autonomous RL Agent Harness, and the first of many we are about to ship Iacon Autonomics. We are removing the manual grind of RL and adding taste and relentlessness to the process. Just hand Optimus a codebase, an API key, and a goal. It spins up a fleet of sub-agents to run hundreds of parallel experiments. It learns from its own history, autonomously building an evolutionary tree of runs where every failure sets the next batch up to win. Train LLMs, agents, robots in sim, game npcs and much more. If you can score it, Optimus can train it. Stop doing manual RL. Let the agents cook.

This is the Claude Code moment for Reinforcement Learning. Every frontier lab knows the secret: post-training is where the magic happens. If you can eval a task, you can use RL to benchmax your model. But building and scaling the RL loop has remained a dark art reserved for the top 1% of AI Labs. Until today. Introducing Optimus — an autonomous RL Agent Harness, and the first of many we are about to ship Iacon Autonomics. We are removing the manual grind of RL and adding taste and relentlessness to the process. Just hand Optimus a codebase, an API key, and a goal. It spins up a fleet of sub-agents to run hundreds of parallel experiments. It learns from its own history, autonomously building an evolutionary tree of runs where every failure sets the next batch up to win. Train LLMs, agents, robots in sim, game npcs and much more. If you can score it, Optimus can train it. Stop doing manual RL. Let the agents cook.

Dev Shah

13,730 просмотров • 2 месяцев назад

Today we present Morpheus, a persistent enterprise simulation platform designed to make Continual Learning a reality. Morpheus is the world’s first real world Reinforcement Learning environment. Every Reinforcement Learning environment operates in the game world. Benchmarks like Atari, OpenAI Gym, MuJoCo, and Procgen are all small, game-like worlds that reset every few minutes. But the real world never resets. A business keeps running and evolving everyday. We tested how frontier LLMs would perform in realistic and dynamic business environments 🧬on Morpheus. The main conclusion was that LLMs are not continual learners. 🧵Here’s how we did it and what we learned:

Today we present Morpheus, a persistent enterprise simulation platform designed to make Continual Learning a reality. Morpheus is the world’s first real world Reinforcement Learning environment. Every Reinforcement Learning environment operates in the game world. Benchmarks like Atari, OpenAI Gym, MuJoCo, and Procgen are all small, game-like worlds that reset every few minutes. But the real world never resets. A business keeps running and evolving everyday. We tested how frontier LLMs would perform in realistic and dynamic business environments 🧬on Morpheus. The main conclusion was that LLMs are not continual learners. 🧵Here’s how we did it and what we learned:

Skyfall AI

269,894 просмотров • 3 месяцев назад

KL divergence is a fundamental concept used in machine learning, from optimization a neural net to RL training of LLMs. but here is what it actually means:

KL divergence is a fundamental concept used in machine learning, from optimization a neural net to RL training of LLMs. but here is what it actually means:

ℏεsam

20,381 просмотров • 9 месяцев назад

Continual learning sometimes gets discussed as if the goal is to dissolve the context/weights distinction. Let the model just keep accumulating, fine-tuning itself on the fly. Andrej Karpathy points out, though, that this isn't how humans do it. Our working memory gets wiped regularly. What we actually have is a consolidation process (sleep) that distills stuff into the brain, in a weird and lossy way. This is very different from how people sometimes talk about continual learning. It's not obvious it's something you can get for free from doing long enough RL loops.

Continual learning sometimes gets discussed as if the goal is to dissolve the context/weights distinction. Let the model just keep accumulating, fine-tuning itself on the fly. Andrej Karpathy points out, though, that this isn't how humans do it. Our working memory gets wiped regularly. What we actually have is a consolidation process (sleep) that distills stuff into the brain, in a weird and lossy way. This is very different from how people sometimes talk about continual learning. It's not obvious it's something you can get for free from doing long enough RL loops.

Dwarkesh Patel

60,477 просмотров • 5 месяцев назад

We asked Sholto Douglas from Anthropic about the costs of RL (Reinforcement Learning) runs. "In Dario Amodei's essay, he said that RL runs cost only $1M back in December." "RL is a more naively parallelizable and scalable than pre-training." "With pre-training, you need everything in one big data center ideally. For RL, in theory, you could scale all over the world."

We asked Sholto Douglas from Anthropic about the costs of RL (Reinforcement Learning) runs. "In Dario Amodei's essay, he said that RL runs cost only $1M back in December." "RL is a more naively parallelizable and scalable than pre-training." "With pre-training, you need everything in one big data center ideally. For RL, in theory, you could scale all over the world."

TBPN

76,696 просмотров • 1 год назад

Introducing RL Environment Explorer 🧭 There are now 12M+ RL task entries across RL environments on the Hugging Face Hub. You can now explore, visualize, and run RL environments across: OpenEnv • Harbor • MiMo • Verifiers • NeMo Gym All in one place

Introducing RL Environment Explorer 🧭 There are now 12M+ RL task entries across RL environments on the Hugging Face Hub. You can now explore, visualize, and run RL environments across: OpenEnv • Harbor • MiMo • Verifiers • NeMo Gym All in one place

Adithya S K

27,719 просмотров • 23 дней назад

Our work, "A Primer on SO(3) Action Representations in Deep Reinforcement Learning," was accepted to #ICLR2026! We provide a systematic study of action representation choices in RL, showing that they fundamentally impact training stability and performance. #Robotics #AI #RL

Our work, "A Primer on SO(3) Action Representations in Deep Reinforcement Learning," was accepted to #ICLR2026! We provide a systematic study of action representation choices in RL, showing that they fundamentally impact training stability and performance. #Robotics #AI #RL

Learning Systems and Robotics Lab (is hiring!)

50,130 просмотров • 8 месяцев назад

Want to scale RL with your shiny new GPU? 🚀 In our ICML24 Oral we find that RL algorithms hit a barrier when data is scaled up. Our new algorithm, SAPG, proposes a simple fix. It scales to 25k envs and solves hard tasks where PPO makes no progress. 1/n

Want to scale RL with your shiny new GPU? 🚀 In our ICML24 Oral we find that RL algorithms hit a barrier when data is scaled up. Our new algorithm, SAPG, proposes a simple fix. It scales to 25k envs and solves hard tasks where PPO makes no progress. 1/n

Ananye Agarwal

64,610 просмотров • 2 лет назад

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Jingyun Yang

88,312 просмотров • 3 лет назад

Daniel Kokotajlo says companies are making AI more agentic by scaling RL (reinforcement learning) and connecting agents to various tools. by early 2027, these agents will be fully autonomous, superhuman coders able to replace human programmers.

Daniel Kokotajlo says companies are making AI more agentic by scaling RL (reinforcement learning) and connecting agents to various tools. by early 2027, these agents will be fully autonomous, superhuman coders able to replace human programmers.

Haider.

11,572 просмотров • 1 год назад

RL-100 Performant Robotic Manipulation with Real-World Reinforcement Learning

RL-100 Performant Robotic Manipulation with Real-World Reinforcement Learning

AK

15,364 просмотров • 1 год назад

Introducing RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 7 real robot tasks, 900/900 successes. Up to 250 consecutive trials in one task, running 2 hours nonstop without failure. High success rate against physical disturbances, zero-shot, and few-shot adaptation Our first step toward a deployable robot learning system.

Introducing RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 7 real robot tasks, 900/900 successes. Up to 250 consecutive trials in one task, running 2 hours nonstop without failure. High success rate against physical disturbances, zero-shot, and few-shot adaptation Our first step toward a deployable robot learning system.

Kun Lei

92,774 просмотров • 1 год назад

Self-supervised representation learning looks a bit like RL. What if we literally use RL as a SSL method for visual representations? Turns out that it works quite well. In new work by Dibya Ghosh, we show how this can be done:

Self-supervised representation learning looks a bit like RL. What if we literally use RL as a SSL method for visual representations? Turns out that it works quite well. In new work by Dibya Ghosh, we show how this can be done:

Sergey Levine

48,840 просмотров • 1 год назад

NeurIPS 2025 Paper: LLMs are Reinforcement Learners 🤯! Surprisingly, we show that LLMs can solve RL tasks without any external component! We introduce Prompted Policy Search (ProPS), an RL method based only LLMs and in-context learning. [Paper]

NeurIPS 2025 Paper: LLMs are Reinforcement Learners 🤯! Surprisingly, we show that LLMs can solve RL tasks without any external component! We introduce Prompted Policy Search (ProPS), an RL method based only LLMs and in-context learning. [Paper]

Heni Ben Amor

51,248 просмотров • 11 месяцев назад

Using our brain simulator, we’ve trained a reinforcement learning agent to maximize bits per second. Here is the RL policy converting brain data to cursor control in simulation:

Using our brain simulator, we’ve trained a reinforcement learning agent to maximize bits per second. Here is the RL policy converting brain data to cursor control in simulation:

Neuralink

18,410 просмотров • 1 год назад

We are excited to announce CLIN 🤖: The first continually learning language agent that excels in both task adaptation and generalization to unseen tasks and environments in a pure zero-shot setup. Aristo Team at Ai2 Ai2 Website: Let's dive in 🧵 (1/n)

We are excited to announce CLIN 🤖: The first continually learning language agent that excels in both task adaptation and generalization to unseen tasks and environments in a pure zero-shot setup. Aristo Team at Ai2 Ai2 Website: Let's dive in 🧵 (1/n)

Bodhisattwa Majumder

35,062 просмотров • 3 лет назад