Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Does LLM RL post-training need to be on-policy?

Kianté Brantley

1,958 subscribers

114,565 Aufrufe • vor 8 Monaten •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

LLM training on RTX 5090

LLM training on RTX 5090

ℏεsam

144,115 Aufrufe • vor 1 Jahr

GPT-6 Astra test 4/n Prompt: "Implement pen spinning with a dexterous hand. Use Isaac Lab for RL training, use the Sharpa hand, and create the pen mesh yourself. Give me a trained RL policy and a visualization video. You are free to search the web and download papers or anything else you need." After running autonomously for a day and a half (including training the policy), this is what we got: Credit: my student Chengyang Li

GPT-6 Astra test 4/n Prompt: "Implement pen spinning with a dexterous hand. Use Isaac Lab for RL training, use the Sharpa hand, and create the pen mesh yourself. Give me a trained RL policy and a visualization video. You are free to search the web and download papers or anything else you need." After running autonomously for a day and a half (including training the policy), this is what we got: Credit: my student Chengyang Li

Wentao Zhu ✈️ CoRL 2026

696,018 Aufrufe • vor 1 Monat

Zombie robot RL policy

Zombie robot RL policy

Simon Kalouche

164,467 Aufrufe • vor 1 Jahr

Does off-policy value-based RL scale? In LLMs, larger scale predictably improves performance. Value-based RL learns from arbitrary data and is sample-efficient, but folk wisdom says it doesn't scale 🧵⬇️We show predictability for scaling value-based RL!

Does off-policy value-based RL scale? In LLMs, larger scale predictably improves performance. Value-based RL learns from arbitrary data and is sample-efficient, but folk wisdom says it doesn't scale 🧵⬇️We show predictability for scaling value-based RL!

Oleg Rybkin

23,994 Aufrufe • vor 1 Jahr

As a supporter of the open weights ecosystem, we're proud to be a post-training partner for NVIDIA Nemotron. We post-train Nemotron models for customer use cases, de-risk mainline RL runs on our AC2 platform and training stack, and contribute aggregate workload statistics for inference benchmarking. This is how open models get better, and we're excited to keep working closely with NVIDIA AI.

As a supporter of the open weights ecosystem, we're proud to be a post-training partner for NVIDIA Nemotron. We post-train Nemotron models for customer use cases, de-risk mainline RL runs on our AC2 platform and training stack, and contribute aggregate workload statistics for inference benchmarking. This is how open models get better, and we're excited to keep working closely with NVIDIA AI.

Applied Compute

25,683 Aufrufe • vor 3 Monaten

Rare serious post on here but it need to be said ❤️

Rare serious post on here but it need to be said ❤️

Alexandra Pembroke

10,899 Aufrufe • vor 6 Monaten

Tutorial Time: Run any open-source LLM locally. Now we will run an LLM on your M1/2 Mac. And its fast. All you need is moved to: @lmstudio let's get started. Good to be back. A thread

Tutorial Time: Run any open-source LLM locally. Now we will run an LLM on your M1/2 Mac. And its fast. All you need is moved to: @lmstudio let's get started. Good to be back. A thread

Linus ✦ Ekenstam

916,081 Aufrufe • vor 3 Jahren

training for RL 2027 🥰😋

training for RL 2027 🥰😋

jessica__

27,857 Aufrufe • vor 1 Monat

Reinforcement learning should be able to improve upon behaviors seen when training. In practice, RL agents often struggle to generalize to new long-horizon behaviors. Our new paper studies *horizon generalization*, the degree RL algorithms generalize to reaching distant goals. 1/

Reinforcement learning should be able to improve upon behaviors seen when training. In practice, RL agents often struggle to generalize to new long-horizon behaviors. Our new paper studies horizon generalization, the degree RL algorithms generalize to reaching distant goals. 1/

Vivek Myers

79,523 Aufrufe • vor 1 Jahr

干的像沙漠般的一篇，关于 RL Terms. RL 术语很多，且不好理解，例如: policy, state, action rewardreturn, value, advantage baseline, TD error, GAE clipping, KL 这篇用一个例子把 RL 的术语写给 LLM 读者（依然太干了，RL 真的很难写得生动有趣）

干的像沙漠般的一篇，关于 RL Terms. RL 术语很多，且不好理解，例如: policy, state, action rewardreturn, value, advantage baseline, TD error, GAE clipping, KL 这篇用一个例子把 RL 的术语写给 LLM 读者（依然太干了，RL 真的很难写得生动有趣）

Dongxi 东锡 NLP

15,464 Aufrufe • vor 15 Tagen

New project! Flow Policy Gradients for Robot Control tldr; a simple online RL recipe for training and fine-tuning flow policies for robots co-led w/ Hongsuk Benjamin Choi:

New project! Flow Policy Gradients for Robot Control tldr; a simple online RL recipe for training and fine-tuning flow policies for robots co-led w/ Hongsuk Benjamin Choi:

Brent Yi

94,426 Aufrufe • vor 9 Monaten

Need my house to look like a RL store

Need my house to look like a RL store

Prep Propaganda 👔

20,565 Aufrufe • vor 2 Monaten

How thick does ice need to be before you can walk on it? 😭

How thick does ice need to be before you can walk on it? 😭

AlphaFox

43,053 Aufrufe • vor 9 Monaten

Introducing autoresearch for arXiv papers: replicate and experiment on any arXiv paper using an army of Claude/Codex agents For post-training, your agents can use Tinker to launch concurrent RL runs while you watch your experiment tree grow Try it out now on OpenResearch:

Introducing autoresearch for arXiv papers: replicate and experiment on any arXiv paper using an army of Claude/Codex agents For post-training, your agents can use Tinker to launch concurrent RL runs while you watch your experiment tree grow Try it out now on OpenResearch:

alphaXiv

61,958 Aufrufe • vor 2 Monaten

New work on Training Foundation Models with a Single Algorithm: Paper: Code, Models: This work introduces Unified Training, a framework that unifies pretraining, midtraining, supervised fine-tuning and RL under a single maximum-likelihood objective. Building on MaxRL, it enables static data and RL-generated data to be trained together within the same system. Empirically, Unified Training improves accuracy and compute efficiency, generalizes better to harder reasoning problems, and reduces forgetting. Interestingly, reusing data across training stages can improve performance even when conventional staged training fails. It also produces models that adapt more effectively to new tasks, leading to stronger continual learning. See an excellent thread by Yuda Song.

New work on Training Foundation Models with a Single Algorithm: Paper: Code, Models: This work introduces Unified Training, a framework that unifies pretraining, midtraining, supervised fine-tuning and RL under a single maximum-likelihood objective. Building on MaxRL, it enables static data and RL-generated data to be trained together within the same system. Empirically, Unified Training improves accuracy and compute efficiency, generalizes better to harder reasoning problems, and reduces forgetting. Interestingly, reusing data across training stages can improve performance even when conventional staged training fails. It also produces models that adapt more effectively to new tasks, leading to stronger continual learning. See an excellent thread by Yuda Song.

Russ Salakhutdinov

42,682 Aufrufe • vor 13 Tagen

Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive. So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.

Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive. So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.

Modal

168,188 Aufrufe • vor 1 Monat

Does your bed board need to be replaced 😏

Sensitive content

Does your bed board need to be replaced 😏

sytoys-us1

49,674 Aufrufe • vor 1 Jahr

Nature does not always need venom to be terrifying.

Nature does not always need venom to be terrifying.

NECA

36,534 Aufrufe • vor 3 Monaten

How does high-fidelity tactile simulation help robots nail the last millimeter? We’re releasing VT-Refine, accepted to CoRL: a real-to-sim-to-real visuo-tactile policy using a GPU-parallel tactile sim for our piezoresistive skin FlexiTac. Then fine-tuning a diffusion policy with large-scale RL in simulation. Website: #CoRL2025 #RobotLearning #Sim2Real

How does high-fidelity tactile simulation help robots nail the last millimeter? We’re releasing VT-Refine, accepted to CoRL: a real-to-sim-to-real visuo-tactile policy using a GPU-parallel tactile sim for our piezoresistive skin FlexiTac. Then fine-tuning a diffusion policy with large-scale RL in simulation. Website: #CoRL2025 #RobotLearning #Sim2Real

Binghao Huang

46,982 Aufrufe • vor 1 Jahr