Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

Does LLM RL post-training need to be on-policy?

Kianté Brantley

1,941 subscribers

114,174 görüntüleme • 6 ay önce •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

LLM training on RTX 5090

LLM training on RTX 5090

ℏεsam

144,115 görüntüleme • 1 yıl önce

Does off-policy value-based RL scale? In LLMs, larger scale predictably improves performance. Value-based RL learns from arbitrary data and is sample-efficient, but folk wisdom says it doesn't scale 🧵⬇️We show predictability for scaling value-based RL!

Does off-policy value-based RL scale? In LLMs, larger scale predictably improves performance. Value-based RL learns from arbitrary data and is sample-efficient, but folk wisdom says it doesn't scale 🧵⬇️We show predictability for scaling value-based RL!

Oleg Rybkin

23,994 görüntüleme • 1 yıl önce

As a supporter of the open weights ecosystem, we're proud to be a post-training partner for NVIDIA Nemotron. We post-train Nemotron models for customer use cases, de-risk mainline RL runs on our AC2 platform and training stack, and contribute aggregate workload statistics for inference benchmarking. This is how open models get better, and we're excited to keep working closely with NVIDIA AI.

As a supporter of the open weights ecosystem, we're proud to be a post-training partner for NVIDIA Nemotron. We post-train Nemotron models for customer use cases, de-risk mainline RL runs on our AC2 platform and training stack, and contribute aggregate workload statistics for inference benchmarking. This is how open models get better, and we're excited to keep working closely with NVIDIA AI.

Applied Compute

25,234 görüntüleme • 1 ay önce

Rare serious post on here but it need to be said ❤️

Rare serious post on here but it need to be said ❤️

Alexandra Pembroke

10,899 görüntüleme • 4 ay önce

Tutorial Time: Run any open-source LLM locally. Now we will run an LLM on your M1/2 Mac. And its fast. All you need is moved to: @lmstudio let's get started. Good to be back. A thread

Tutorial Time: Run any open-source LLM locally. Now we will run an LLM on your M1/2 Mac. And its fast. All you need is moved to: @lmstudio let's get started. Good to be back. A thread

Linus ✦ Ekenstam

915,956 görüntüleme • 2 yıl önce

Reinforcement learning should be able to improve upon behaviors seen when training. In practice, RL agents often struggle to generalize to new long-horizon behaviors. Our new paper studies *horizon generalization*, the degree RL algorithms generalize to reaching distant goals. 1/

Reinforcement learning should be able to improve upon behaviors seen when training. In practice, RL agents often struggle to generalize to new long-horizon behaviors. Our new paper studies horizon generalization, the degree RL algorithms generalize to reaching distant goals. 1/

Vivek Myers

79,523 görüntüleme • 1 yıl önce

Need my house to look like a RL store

Need my house to look like a RL store

Prep Propaganda 👔

18,250 görüntüleme • 5 gün önce

New project! Flow Policy Gradients for Robot Control tldr; a simple online RL recipe for training and fine-tuning flow policies for robots co-led w/ Hongsuk Benjamin Choi:

New project! Flow Policy Gradients for Robot Control tldr; a simple online RL recipe for training and fine-tuning flow policies for robots co-led w/ Hongsuk Benjamin Choi:

Brent Yi

92,732 görüntüleme • 7 ay önce

How thick does ice need to be before you can walk on it? 😭

How thick does ice need to be before you can walk on it? 😭

AlphaFox

43,053 görüntüleme • 7 ay önce

Introducing autoresearch for arXiv papers: replicate and experiment on any arXiv paper using an army of Claude/Codex agents For post-training, your agents can use Tinker to launch concurrent RL runs while you watch your experiment tree grow Try it out now on OpenResearch:

Introducing autoresearch for arXiv papers: replicate and experiment on any arXiv paper using an army of Claude/Codex agents For post-training, your agents can use Tinker to launch concurrent RL runs while you watch your experiment tree grow Try it out now on OpenResearch:

alphaXiv

60,850 görüntüleme • 14 gün önce

Nature does not always need venom to be terrifying.

Nature does not always need venom to be terrifying.

NECA

36,412 görüntüleme • 1 ay önce

Does your bed board need to be replaced 😏

Does your bed board need to be replaced 😏

sytoys-us1

49,674 görüntüleme • 11 ay önce

How does high-fidelity tactile simulation help robots nail the last millimeter? We’re releasing VT-Refine, accepted to CoRL: a real-to-sim-to-real visuo-tactile policy using a GPU-parallel tactile sim for our piezoresistive skin FlexiTac. Then fine-tuning a diffusion policy with large-scale RL in simulation. Website: #CoRL2025 #RobotLearning #Sim2Real

How does high-fidelity tactile simulation help robots nail the last millimeter? We’re releasing VT-Refine, accepted to CoRL: a real-to-sim-to-real visuo-tactile policy using a GPU-parallel tactile sim for our piezoresistive skin FlexiTac. Then fine-tuning a diffusion policy with large-scale RL in simulation. Website: #CoRL2025 #RobotLearning #Sim2Real

Binghao Huang

46,982 görüntüleme • 10 ay önce

How it feels to be an LLM

How it feels to be an LLM

Beff (e/acc)

36,669 görüntüleme • 1 yıl önce

i need to post more on here 😩

i need to post more on here 😩

MS. F!NEE $HITT

124,925 görüntüleme • 1 yıl önce

🚨Current scalable RL algos train a policy w/o value func, which is limiting with learning in open-ended, non-stationary, dynamic environments. But, how to scale value-based RL with more data/compute is unclear... Not anymore: presenting scaling laws for value-based RL 🧵⬇️

🚨Current scalable RL algos train a policy w/o value func, which is limiting with learning in open-ended, non-stationary, dynamic environments. But, how to scale value-based RL with more data/compute is unclear... Not anymore: presenting scaling laws for value-based RL 🧵⬇️

Aviral Kumar

37,377 görüntüleme • 1 yıl önce

RL is painfully slow 😭 — bottlenecked by super-long CoT rollout. 🔭 Sparse attention should help, but naive sparse rollout hits a brutal efficiency–stability tradeoff: A tedious trial-and-error sparsity sweep for each dense policy is required before an actual RL run. 🐤Sparrow chirps no more pain! Introduce Sparrow: Sparse Rollout for stable and efficient long-context RL. Sparrow finds that: 💡As long as we keep the tail distribution mismatch throughout the sparse rollout above a critical threshold, the RL training will be stable. 💡Even cooler! Through comprehensive control studies of Qwen3-1.7B, 4B, 8B thinking models RL with 40K rollout max length, the critical threshold stays constant across model sizes. 💡Sparrow then finds the optimal dynamic sparse schedule to reach the threshold with minimal cost. 💡Sparrow's findings are empirically validated to generalize in Qwen3-14B, and hold on both Math and Coding RL. 🐤Sparrow empirically helps achieve 2.2× / 2.4× / 2.0× rollout speedup on Qwen3 1.7B / 4B / 8B thinking models, while keeping training stability over extended RL steps. We release the 🐤bird in the following formats. [1/n] Paper: Code: Blog:

RL is painfully slow 😭 — bottlenecked by super-long CoT rollout. 🔭 Sparse attention should help, but naive sparse rollout hits a brutal efficiency–stability tradeoff: A tedious trial-and-error sparsity sweep for each dense policy is required before an actual RL run. 🐤Sparrow chirps no more pain! Introduce Sparrow: Sparse Rollout for stable and efficient long-context RL. Sparrow finds that: 💡As long as we keep the tail distribution mismatch throughout the sparse rollout above a critical threshold, the RL training will be stable. 💡Even cooler! Through comprehensive control studies of Qwen3-1.7B, 4B, 8B thinking models RL with 40K rollout max length, the critical threshold stays constant across model sizes. 💡Sparrow then finds the optimal dynamic sparse schedule to reach the threshold with minimal cost. 💡Sparrow's findings are empirically validated to generalize in Qwen3-14B, and hold on both Math and Coding RL. 🐤Sparrow empirically helps achieve 2.2× / 2.4× / 2.0× rollout speedup on Qwen3 1.7B / 4B / 8B thinking models, while keeping training stability over extended RL steps. We release the 🐤bird in the following formats. [1/n] Paper: Code: Blog:

Infini-AI-Lab

78,984 görüntüleme • 3 ay önce

🤔 How to fine-tune an Imitation Learning policy (e.g., Diffusion Policy, ACT) with RL? As an RL practitioner, I’ve been struggling with this problem for a while. Here’s why it’s tough: 1️⃣ Special designs (usually for multimodal action distributions) in modern IL models make them non-trivial to fine-tune by RL. 2️⃣ Large policy models + RL's poor sample efficiency = a nightmare But finally, we figured out a simple solution that works for any model architecture! 🌟 Check out our #ICLR2025 paper: “Policy Decorator: Model-Agnostic Online Refinement for Large Policy Models”, led by my amazing mentee Xiu Yuan. 🔗 🧵 Read more below!

🤔 How to fine-tune an Imitation Learning policy (e.g., Diffusion Policy, ACT) with RL? As an RL practitioner, I’ve been struggling with this problem for a while. Here’s why it’s tough: 1️⃣ Special designs (usually for multimodal action distributions) in modern IL models make them non-trivial to fine-tune by RL. 2️⃣ Large policy models + RL's poor sample efficiency = a nightmare But finally, we figured out a simple solution that works for any model architecture! 🌟 Check out our #ICLR2025 paper: “Policy Decorator: Model-Agnostic Online Refinement for Large Policy Models”, led by my amazing mentee Xiu Yuan. 🔗 🧵 Read more below!

Tongzhou Mu 🤖🦾🦿

17,018 görüntüleme • 1 yıl önce