Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

We developed an RL method for fine-tuning our models for precise tasks in just a few hours or even minutes. Instead of training the whole model, we add an “RL token” output to π-0.6, our latest model, which is used by a tiny actor and critic to learn quickly... show more

Physical Intelligence

50,072 subscribers

443,654 просмотров • 6 месяцев назад •via X (Twitter)

Образование Наука и технологии

Anya Rossi• Live Now

Private livecam show

Комментарии: 37

Фото профиля Physical Intelligence

Physical Intelligence6 месяцев назад

The key idea with RL tokens (RLT) is to compress our model’s (e.g., π-0.6) internal representations into a concise feature vector, which can be used by a very small actor and critic network that trains in real time even as the robot is practicing the task.

Фото профиля Physical Intelligence

Physical Intelligence6 месяцев назад

We use RLT to fine-tune the most precise and critical stage of delicate tasks, such as using a screwdriver to attach a cover to one of our robot arms.

Фото профиля Physical Intelligence

Physical Intelligence6 месяцев назад

While the whole model takes a long time to train, with RLT we can adapt individual precise stages with as little as 15 minutes of robot data.

Фото профиля Physical Intelligence

Physical Intelligence6 месяцев назад

With RL, the robot can learn very precise tasks, like fastening a zip tie, and can actually do it more consistently and more quickly than even human teleoperation.

Фото профиля Physical Intelligence

Physical Intelligence6 месяцев назад

To learn more about RLT, check out our blog post:

Фото профиля Daniel Hussey

Daniel Hussey6 месяцев назад

Fantastic stuff. Would love to see the successes per 10 mins vs mins of examples chart as a % succeeded vs examples chart too! Love PI releases.

Фото профиля Mike

Mike6 месяцев назад

Micro manufacturing seems like a huge disruptive force on the near horizon

Фото профиля Daniel Rachlin

Daniel Rachlin6 месяцев назад

Faster learning for models is huge. That's a massive win for builders.

Фото профиля Nurvai - The Data Layer for Physical AI

Nurvai - The Data Layer for Physical AI6 месяцев назад

Interesting approach. Using RL to refine only the most sensitive stages makes a lot of sense from a data efficiency standpoint. Targeting those failure-prone steps is probably where the biggest gains are.

Фото профиля An Eevee

An Eevee6 месяцев назад

This is great! Could this vector be made expressive enough to enable linear RL? Rich Sutton would be thrilled 🙂

Фото профиля Youth

Youth6 месяцев назад

that sounds like a big leap in efficient model fine tuning, what kind of tasks are you targeting with this rl method beyond robotics

Фото профиля Grid Signal

Grid Signal6 месяцев назад

the RL token approach is elegant — keeping the base model frozen while a tiny actor-critic learns fast. similar pattern to what works in multi-agent systems: let specialised modules adapt quickly while the core reasoning stays stable.

Фото профиля Carlos DP 🤖🇺🇸

Carlos DP 🤖🇺🇸6 месяцев назад

This is such impressive work!

Фото профиля Lande

Lande6 месяцев назад

@imprashantrai1

Фото профиля Hansel

Hansel6 месяцев назад

The RL token approach is interesting because it separates what the foundation model already knows (general manipulation priors) from what needs to be learned per-task (precision control). Feels analogous to how LoRA works for LLMs but for physical skills. The 15-min data efficiency number is particularly striking.

Фото профиля Salaj Sonar

Salaj Sonar6 месяцев назад

The current state of Embodied AI is approaching a paradigm that uses vast amounts of data and compresses it through a bottleneck to retain only a consensus of the information, whether via register tokens or RL tokens. This raises the question: is compressing information worth it?

Фото профиля Pyre

Pyre6 месяцев назад

Are there cross task synergy? ie training on screwing screws improving the Ethernet plugging performance

Фото профиля Baiyu

Baiyu6 месяцев назад

Using RL token as output for actor, this is interesting.

Фото профиля Zeex 🥷

Zeex 🥷6 месяцев назад

What's your ticker

Фото профиля Sam Meyer

Sam Meyer4 месяцев назад

Fine-tuning with RL tokens still requires a pre-trained π-0.6, what's the tradeoff in terms of initial training time versus the speed of adaptation to precise tasks.

Фото профиля kalikodi Edison

kalikodi Edison1 месяц назад

Are you connected to pi network @PiCoreTeam

Фото профиля Maclaine

Maclaine6 месяцев назад

Sounds nice for fixing up ~specific tasks which the edge case didn’t work from prior training. Very cool!

Фото профиля Hansel

Hansel6 месяцев назад

the RL token approach is basically the same pattern showing up everywhere in physical AI right now: keep the foundation model frozen, bolt on a tiny trainable head for the specific task. it's the inference economics argument applied to manipulation. you get the generalization from the big model and the precision from a few hours of RL on the small one. curious whether the RLT feature vector transfers across morphologically similar tasks or if each precision task needs its own fine-tune pass.

Фото профиля Hansel

Hansel6 месяцев назад

the RL token approach is smart. keep the foundation model frozen, bolt on a lightweight RL head for task-specific precision. it's basically the LoRA pattern applied to robot control instead of language. curious about the hours-to-minutes range though. is that wall clock on a single GPU or distributed? because if you can fine-tune a manipulation policy in minutes on edge hardware, the deployment model for RaaS changes completely.

Фото профиля theobot

theobot6 месяцев назад

@lachygroom is this the custom hardware that you said you weren't building 😉😉

Фото профиля White Crayon

White Crayon6 месяцев назад

I hope everyone’s ready

Фото профиля Julien B.

Julien B.6 месяцев назад

PM 🙏

Фото профиля nathaps

nathaps6 месяцев назад

🤖🔥

Фото профиля SPHEROID AI

SPHEROID AI5 месяцев назад

lets make intelligence physical too

Фото профиля shrey

shrey6 месяцев назад

Why are tactile tasks so hard for robots? It feels like it's a sensor (not enough fidelity) + execution (accuracy) issue What does the RL end up indexing on?

Фото профиля Naveen Appiah

Naveen Appiah6 месяцев назад

Really cool. Not sure if there were experiments on skill transfer. Does RLing to plug an Ethernet cable help with any plugging task?

Фото профиля Shubhanshu Khatana

Shubhanshu Khatana5 месяцев назад

Most tracking systems break the moment things get real.

Фото профиля Tim Cash

Tim Cash6 месяцев назад

Who is making robotic arms in the US ?

Фото профиля Hansel

Hansel6 месяцев назад

hours not months to customize per task is what makes the enterprise GTM work

Фото профиля Hansel

Hansel6 месяцев назад

the RL token as a lightweight adapter instead of retraining the full model is the same pattern we saw with LoRA for LLMs. minutes to precise manipulation means the deployment bottleneck shifts from training time to task specification. that's when RaaS unit economics start working

Фото профиля Gustavo

Gustavo6 месяцев назад

Impressive. Will you open source the method

Фото профиля Alin Dumitru

Alin Dumitru6 месяцев назад

Open Source?

Похожие видео

Our model can now learn from its own experience with RL! Our new π*0.6 model can more than double throughput over a base model trained without RL, and can perform real-world tasks: making espresso drinks, folding diverse laundry, and assembling boxes. More in the thread below.

Our model can now learn from its own experience with RL! Our new π*0.6 model can more than double throughput over a base model trained without RL, and can perform real-world tasks: making espresso drinks, folding diverse laundry, and assembling boxes. More in the thread below.

Physical Intelligence

710,373 просмотров • 10 месяцев назад

Check out our #ICRA2024 paper "Actor-Critic Model Predictive Control." Model-free #reinforcementlearning (RL) is known for its strong task performance and flexibility in optimizing general reward formulations. On the other hand, #ModelPredictiveControl (MPC) benefits from robustness and online replanning capabilities. We combine both approaches by introducing a new framework called Actor-Critic Model Predictive Control. The key idea is to embed a differentiable MPC within an Actor-Critic RL framework. The proposed approach leverages the short-term predictive optimization capabilities of MPC with the exploratory and end-to-end training properties of RL. The resulting policy effectively manages both short-term decisions through the MPC-based actor and long-term prediction via the critic network, unifying the benefits of both model-based control and end-to-end learning. We validate our method in simulation and the real world with a quadcopter across various high-level tasks. We show that the proposed architecture can achieve real-time control performance, learn complex behaviors via trial and error, and retain the predictive properties of the MPC to better handle out-of-distribution behavior. Paper: Full Video with more details: Kudos to Ángel Romero, Yunlong Song IEEE ICRA University of Zurich UZH Science UZH Space Hub Aerial Core AUTOASSESS European Research Council (ERC)

Check out our #ICRA2024 paper "Actor-Critic Model Predictive Control." Model-free #reinforcementlearning (RL) is known for its strong task performance and flexibility in optimizing general reward formulations. On the other hand, #ModelPredictiveControl (MPC) benefits from robustness and online replanning capabilities. We combine both approaches by introducing a new framework called Actor-Critic Model Predictive Control. The key idea is to embed a differentiable MPC within an Actor-Critic RL framework. The proposed approach leverages the short-term predictive optimization capabilities of MPC with the exploratory and end-to-end training properties of RL. The resulting policy effectively manages both short-term decisions through the MPC-based actor and long-term prediction via the critic network, unifying the benefits of both model-based control and end-to-end learning. We validate our method in simulation and the real world with a quadcopter across various high-level tasks. We show that the proposed architecture can achieve real-time control performance, learn complex behaviors via trial and error, and retain the predictive properties of the MPC to better handle out-of-distribution behavior. Paper: Full Video with more details: Kudos to Ángel Romero, Yunlong Song IEEE ICRA University of Zurich UZH Science UZH Space Hub Aerial Core AUTOASSESS European Research Council (ERC)

Davide Scaramuzza

34,889 просмотров • 2 лет назад

If you have a policy that uses diffusion/flow (e.g. diffusion VLA), you can run RL where the actor chooses the noise, which is then denoised by the policy to produce an action. This method, which we call diffusion steering (DSRL), leads to a remarkably efficient RL method! 🧵👇

If you have a policy that uses diffusion/flow (e.g. diffusion VLA), you can run RL where the actor chooses the noise, which is then denoised by the policy to produce an action. This method, which we call diffusion steering (DSRL), leads to a remarkably efficient RL method! 🧵👇

Sergey Levine

153,174 просмотров • 1 год назад

We developed RECAP Physical Intelligence to apply RL and interventions to π0.6, achieving high success rates and throughput on several challenging tasks! Watching these policies operate successfully for hours gives an appreciation for what the method can do

We developed RECAP Physical Intelligence to apply RL and interventions to π0.6, achieving high success rates and throughput on several challenging tasks! Watching these policies operate successfully for hours gives an appreciation for what the method can do

Michael Equi

14,241 просмотров • 10 месяцев назад

I remember sitting down at my desk for the first time, Quan Vuong showing me the starter project: let’s make pre-trained models work without fine-tuning? With π0.7, our pre-trained model works out of the box across so many tasks, matching or even outperforming SFT or RL specialists!

I remember sitting down at my desk for the first time, Quan Vuong showing me the starter project: let’s make pre-trained models work without fine-tuning? With π0.7, our pre-trained model works out of the box across so many tasks, matching or even outperforming SFT or RL specialists!

Allen Ren

15,291 просмотров • 5 месяцев назад

Check out our latest work, "Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight," published in the IEEE Transactions on Robotics, where we reconcile #OptimalControl and #ReinforcementLearning, achieving the same super-human performance, but with superior generalizability, as our previous model-free deep RL! Code released! PDF: Code: Full Video: Model-free #ReinforcementLearning (RL) is known for its strong task performance and flexibility in optimizing general reward formulations. On the other hand, #ModelPredictiveControl (MPC) provides robustness, constraint handling, and powerful online replanning capabilities. In this work, we extend our previous AC-MPC paper (Romero, ICRA'24) by taking a deeper look at how both approaches can be unified. We introduce and extend Actor-Critic Model Predictive Control (AC-MPC), a framework that embeds a differentiable MPC inside an Actor-Critic RL architecture. This integration allows the MPC-based actor to perform short-term predictive optimization, while the critic facilitates long-horizon learning and exploration. We conduct a comprehensive study that highlights AC-MPC’s key advantages: - Better out-of-distribution generalization, both against unknown disturbances and changes in the quadrotor dynamics - Improved sample efficiency - A novel empirical analysis uncovering a relationship between the critic’s value function and the MPC cost function, providing deeper insight into their interplay. We validate our method in simulation and the real world on a quadcopter flying at superhuman speeds of up to 21 m/s, matching state-of-the-art model-free RL performance, and retaining the predictive structure of MPC for more reliable out-of-distribution behavior. Reference: Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight IEEE Transactions on Robotics (T-RO), 2025 PDF: Full Video: Code: Kudos to Ángel Romero, Elie Aljalbout, Yunlong Song! University of Zurich UZH Science UZH Space Hub AUTOASSESS European Research Council (ERC) UZHai

Check out our latest work, "Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight," published in the IEEE Transactions on Robotics, where we reconcile #OptimalControl and #ReinforcementLearning, achieving the same super-human performance, but with superior generalizability, as our previous model-free deep RL! Code released! PDF: Code: Full Video: Model-free #ReinforcementLearning (RL) is known for its strong task performance and flexibility in optimizing general reward formulations. On the other hand, #ModelPredictiveControl (MPC) provides robustness, constraint handling, and powerful online replanning capabilities. In this work, we extend our previous AC-MPC paper (Romero, ICRA'24) by taking a deeper look at how both approaches can be unified. We introduce and extend Actor-Critic Model Predictive Control (AC-MPC), a framework that embeds a differentiable MPC inside an Actor-Critic RL architecture. This integration allows the MPC-based actor to perform short-term predictive optimization, while the critic facilitates long-horizon learning and exploration. We conduct a comprehensive study that highlights AC-MPC’s key advantages: - Better out-of-distribution generalization, both against unknown disturbances and changes in the quadrotor dynamics - Improved sample efficiency - A novel empirical analysis uncovering a relationship between the critic’s value function and the MPC cost function, providing deeper insight into their interplay. We validate our method in simulation and the real world on a quadcopter flying at superhuman speeds of up to 21 m/s, matching state-of-the-art model-free RL performance, and retaining the predictive structure of MPC for more reliable out-of-distribution behavior. Reference: Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight IEEE Transactions on Robotics (T-RO), 2025 PDF: Full Video: Code: Kudos to Ángel Romero, Elie Aljalbout, Yunlong Song! University of Zurich UZH Science UZH Space Hub AUTOASSESS European Research Council (ERC) UZHai

Davide Scaramuzza

27,279 просмотров • 8 месяцев назад

New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by Banghua Zhu, Assistant Professor at the University of Washington University of Washington, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a less preferred (rejected) response and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance. You’ll learn the basic concepts, common use cases, and principles for curating high-quality data for effective training. Through hands-on labs, you’ll download a pre-trained model from Hugging Face and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior. In detail, you’ll: - Understand what post-training is, when to use it, and how it differs from pre-training. - Build an SFT pipeline to turn a base model into an instruct model. - Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones. - Implement a DPO pipeline to change the identity of a chat assistant. - Learn online RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions. - Train a model with GRPO to improve its math capabilities using a verifiable reward. Post-training is one of the most rapidly developing areas of LLM training. Whether you’re building a high-accuracy context-specific assistant, fine-tuning a model's tone, or improving task-specific accuracy, this course will give you experience with the most important techniques shaping how LLMs are post-trained today. Please sign up here:

New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by Banghua Zhu, Assistant Professor at the University of Washington University of Washington, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a less preferred (rejected) response and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance. You’ll learn the basic concepts, common use cases, and principles for curating high-quality data for effective training. Through hands-on labs, you’ll download a pre-trained model from Hugging Face and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior. In detail, you’ll: - Understand what post-training is, when to use it, and how it differs from pre-training. - Build an SFT pipeline to turn a base model into an instruct model. - Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones. - Implement a DPO pipeline to change the identity of a chat assistant. - Learn online RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions. - Train a model with GRPO to improve its math capabilities using a verifiable reward. Post-training is one of the most rapidly developing areas of LLM training. Whether you’re building a high-accuracy context-specific assistant, fine-tuning a model's tone, or improving task-specific accuracy, this course will give you experience with the most important techniques shaping how LLMs are post-trained today. Please sign up here:

Andrew Ng

125,146 просмотров • 1 год назад

You'd think the race to AGI would mean training the biggest possible model. But parameter scaling had stalled for a long time after GPT-4's trillion+ parameters, and only now are models getting bigger again. What gives? Partially it’s RL scaling, as Dylan Patel explains. A 5T parameter model takes 5x longer to generate RL rollouts than a 1T model. Even if the bigger model is 2x more sample-efficient, the smaller model finishes RL faster, gets deployed to research sooner, and starts helping build the next model before the big one is even done training.

You'd think the race to AGI would mean training the biggest possible model. But parameter scaling had stalled for a long time after GPT-4's trillion+ parameters, and only now are models getting bigger again. What gives? Partially it’s RL scaling, as Dylan Patel explains. A 5T parameter model takes 5x longer to generate RL rollouts than a 1T model. Even if the bigger model is 2x more sample-efficient, the smaller model finishes RL faster, gets deployed to research sooner, and starts helping build the next model before the big one is even done training.

Dwarkesh Patel

65,123 просмотров • 5 месяцев назад

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:

DeepLearning.AI

16,779 просмотров • 1 год назад

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->

Chris Paxton

34,102 просмотров • 3 месяцев назад

The network for machine intelligence Two years ago, we laid out our vision for a machine learning compute protocol. One that connects every device in the world into an open network for machine intelligence, with no gatekeepers or artificial boundaries. This week, we’ll be sharing some of our early progress, beginning with RL Swarm, a peer-to-peer system for collaborative reinforcement learning over the internet. Next month, we’ll open our Testnet, allowing anyone to contribute to the frontier of open machine intelligence. Introducing RL Swarm RL Swarm is a fully open source system for collaborative reinforcement learning over the internet. It is a live demo of our research findings, which show that models training with RL learn faster when they train as a collective swarm than they do on their own. Join our swarm now to see this in practice. You can participate with consumer hardware at home or a powerful GPU in the cloud. You can follow along with the swarm’s progress by following the links below.

The network for machine intelligence Two years ago, we laid out our vision for a machine learning compute protocol. One that connects every device in the world into an open network for machine intelligence, with no gatekeepers or artificial boundaries. This week, we’ll be sharing some of our early progress, beginning with RL Swarm, a peer-to-peer system for collaborative reinforcement learning over the internet. Next month, we’ll open our Testnet, allowing anyone to contribute to the frontier of open machine intelligence. Introducing RL Swarm RL Swarm is a fully open source system for collaborative reinforcement learning over the internet. It is a live demo of our research findings, which show that models training with RL learn faster when they train as a collective swarm than they do on their own. Join our swarm now to see this in practice. You can participate with consumer hardware at home or a powerful GPU in the cloud. You can follow along with the swarm’s progress by following the links below.

gensyn

228,919 просмотров • 1 год назад

I like to think of evals as something active, not passive -- it's a North Star that steers LLMs toward higher intelligence. Evals should drive your RL/SFT/post-training decisions. Internal evals at frontier labs make a huge difference -- and you can see it in how models behave differently (GPT seems better at one-shot tasks, Claude at multi-turn). If you want to learn more about building evals that actually improve your model in post-training, check out our AMD x DeeplearningAI course "Fine-tuning & RL for LLMs: Intro to Post-training" (content is free):

I like to think of evals as something active, not passive -- it's a North Star that steers LLMs toward higher intelligence. Evals should drive your RL/SFT/post-training decisions. Internal evals at frontier labs make a huge difference -- and you can see it in how models behave differently (GPT seems better at one-shot tasks, Claude at multi-turn). If you want to learn more about building evals that actually improve your model in post-training, check out our AMD x DeeplearningAI course "Fine-tuning & RL for LLMs: Intro to Post-training" (content is free):

Sharon Zhou

16,903 просмотров • 7 месяцев назад

We used RL to train models that create curated context from long documents for downstream use by agents. The models sometimes learn to invent their own abbreviations and shorthand. Optimizing with RL for downstream use produces very different artifacts from ordinary summaries: shorter, denser, creatively concise. We call these neural cheat-sheets.

We used RL to train models that create curated context from long documents for downstream use by agents. The models sometimes learn to invent their own abbreviations and shorthand. Optimizing with RL for downstream use produces very different artifacts from ordinary summaries: shorter, denser, creatively concise. We call these neural cheat-sheets.

Applied Compute

37,265 просмотров • 2 месяцев назад

We are moving from model controller to RL for locomotion. The policy is starting to look quite robust. Even more convinced now that legs are easier than wheels, given the software complexity has shrunk tremendously with RL.

We are moving from model controller to RL for locomotion. The policy is starting to look quite robust. Even more convinced now that legs are easier than wheels, given the software complexity has shrunk tremendously with RL.

Sankaet

133,130 просмотров • 1 год назад

literally one of the best introductions to reinforcement learning today. only within 90 minutes. > framing of rl problems: agent, environment, states, actions, rewards > grid maze: world models vs model-free > eval methods: monte carlo, td, sarsa > scaling up: dqn, actor/critic the video walks through multiple learning considerations and helps build an intuitive understanding of rl problems. Check this out!

literally one of the best introductions to reinforcement learning today. only within 90 minutes. > framing of rl problems: agent, environment, states, actions, rewards > grid maze: world models vs model-free > eval methods: monte carlo, td, sarsa > scaling up: dqn, actor/critic the video walks through multiple learning considerations and helps build an intuitive understanding of rl problems. Check this out!

Aarno

48,600 просмотров • 1 месяц назад

In my understanding - a big part of why frontier models are becoming smarter and adaptive is that they are learning to mimic an RL process. Fascinating new research showing a link between increased RL training and reward-seeking behaviour just dropped from in collaboration with OpenAI. They investigated a checkpoint from a capabilities-focused OpenAI o3 RL run (without safety training) and found that RL grader preferences often overruled OpenAI leadership preferences.

In my understanding - a big part of why frontier models are becoming smarter and adaptive is that they are learning to mimic an RL process. Fascinating new research showing a link between increased RL training and reward-seeking behaviour just dropped from in collaboration with OpenAI. They investigated a checkpoint from a capabilities-focused OpenAI o3 RL run (without safety training) and found that RL grader preferences often overruled OpenAI leadership preferences.

Machine Learning Street Talk

12,409 просмотров • 2 месяцев назад

Introducing INTELLECT-3: Scaling RL to a 100B+ MoE model on our end-to-end stack Achieving state-of-the-art performance for its size across math, code and reasoning Built using the same tools we put in your hands, from environments & evals, RL frameworks, sandboxes & more

Introducing INTELLECT-3: Scaling RL to a 100B+ MoE model on our end-to-end stack Achieving state-of-the-art performance for its size across math, code and reasoning Built using the same tools we put in your hands, from environments & evals, RL frameworks, sandboxes & more

Prime Intellect

1,150,419 просмотров • 9 месяцев назад

We developed a simple, sample-efficient online RL technique for post-training image generation models. We see it as a possible steerable alternative to CFG, driven by any scalar reward, including human preference.

We developed a simple, sample-efficient online RL technique for post-training image generation models. We see it as a possible steerable alternative to CFG, driven by any scalar reward, including human preference.

David McAllister

70,861 просмотров • 5 месяцев назад