Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

🤖Adding new RL algorithms to LeRobot just got much easier. Demo: HIL-SERL training with a SAC-based RL algorithm on an SO-100 for a hole-in-hand peg-in-hole task. Sparse reward, only 30 offline demos mixed with live robot experience, and ~1 hour of online training with human interventions only when the... policy fails. The bottom graph tracks intervention rate: high at the start, steadily dropping as the policy improves. The refactor separates algorithm logic from training infrastructure: • RLAlgorithm owns learning logic • RLTrainer handles orchestration • DataMixer combines rollouts, demos, interventions, and future data sources Adding an RL algorithm now looks much closer to adding a policy: one algorithm file, one config, one registry entry. SAC is first. RLT, RECAP, ConRFT, QC-FQL, DSRL, and VLA RL fine-tuning next! Thomas Wolf clem 🤗show more

LeRobot

20,388 subscribers

31,237 Aufrufe • vor 5 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

10 Kommentare

Profilbild von LeRobot

LeRobotvor 5 Monaten

Check out the docs:

Profilbild von Rajbalaji V

Rajbalaji Vvor 5 Monaten

The biggest thing here is making RL algorithms feel plug and play instead of tightly coupled to the training stack. That’s probably more important long term than the SAC result itself

Profilbild von BenUsesAI

BenUsesAIvor 4 Monaten

finally a framework that lets me swap algorithms without rewriting the whole pipeline.

Profilbild von Avery Superpowers

Avery Superpowersvor 5 Monaten

Interesting the use of sparse reward with only 30 offline demos. Most curious the speed of convergence.

Profilbild von OrionX Robotics

OrionX Roboticsvor 5 Monaten

Clean refactor on LeRobot.

Profilbild von Nurvai - The Data Layer for Physical AI

Nurvai - The Data Layer for Physical AIvor 5 Monaten

Interesting to see RL infrastructure becoming modular enough that algorithms are almost plug and play now. The intervention curve is especially compelling for real world training. How much do you think reusable infrastructure will accelerate experimentation compared to new algorithms themselves?

Profilbild von Jack

Jackvor 4 Monaten

Just fine-tuned SmolVLA on a bimanual folding task, any insights appreciated:

Profilbild von Neural Times

Neural Timesvor 4 Monaten

so adding new algorithms is just copy paste now? guess the robot will learn to hate manual configs

Profilbild von Romir Jain

Romir Jainvor 4 Monaten

the rl side is exciting but what happens after training is where most teams stall. going from a lerobot checkpoint to something running on real hardware at inference speed is still a whole separate engineering problem

Profilbild von Mi.lu.

Mi.lu.vor 4 Monaten

nice demo. I’m mostly curious about transfer here: same policy idea on a different arm or fixture, or does that become a whole new weekend?

Ähnliche Videos

New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by Banghua Zhu, Assistant Professor at the University of Washington University of Washington, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a less preferred (rejected) response and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance. You’ll learn the basic concepts, common use cases, and principles for curating high-quality data for effective training. Through hands-on labs, you’ll download a pre-trained model from Hugging Face and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior. In detail, you’ll: - Understand what post-training is, when to use it, and how it differs from pre-training. - Build an SFT pipeline to turn a base model into an instruct model. - Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones. - Implement a DPO pipeline to change the identity of a chat assistant. - Learn online RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions. - Train a model with GRPO to improve its math capabilities using a verifiable reward. Post-training is one of the most rapidly developing areas of LLM training. Whether you’re building a high-accuracy context-specific assistant, fine-tuning a model's tone, or improving task-specific accuracy, this course will give you experience with the most important techniques shaping how LLMs are post-trained today. Please sign up here:

New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by Banghua Zhu, Assistant Professor at the University of Washington University of Washington, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a less preferred (rejected) response and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance. You’ll learn the basic concepts, common use cases, and principles for curating high-quality data for effective training. Through hands-on labs, you’ll download a pre-trained model from Hugging Face and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior. In detail, you’ll: - Understand what post-training is, when to use it, and how it differs from pre-training. - Build an SFT pipeline to turn a base model into an instruct model. - Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones. - Implement a DPO pipeline to change the identity of a chat assistant. - Learn online RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions. - Train a model with GRPO to improve its math capabilities using a verifiable reward. Post-training is one of the most rapidly developing areas of LLM training. Whether you’re building a high-accuracy context-specific assistant, fine-tuning a model's tone, or improving task-specific accuracy, this course will give you experience with the most important techniques shaping how LLMs are post-trained today. Please sign up here:

Andrew Ng

125,146 Aufrufe • vor 1 Jahr

Reward AI just came out of stealth and unveiled OM-1, a robot policy learned entirely from humans. No teleoperation, no on-robot data. One Model, Any Body. 1. Capture. A wearable 7-DoF hand (Omnibody Hand) lets people demonstrate manipulation naturally while working, cooking, or sorting. Sensors record tactile, proximity, force and hand pose at full human speed. 2. Learn. OM-1 trains on this human data alone. No teleop, no on-robot experience. Then it runs across bodies from industrial arms to humanoids, and scaling is just adding more human data. 3. Control. A high-frequency control layer, trained with RL in sim, executes those actions on any robot, holding steady through unknown loads like a stuck fridge door. The result: OM-1 picks up a brand-new task from under 30 minutes of data.

The Humanoid Hub

35,672 Aufrufe • vor 25 Tagen

OpenClaw meets RL! OpenClaw Agents adapt through memory files and skills, but the base model weights never actually change. OpenClaw-RL solves this! It wraps a self-hosted model as an OpenAI-compatible API, intercepts live conversations from OpenClaw, and trains the policy in the background using RL. The architecture is fully async. This means serving, reward scoring, and training all run in parallel. Once done, weights get hot-swapped after every batch while the agent keeps responding. Currently, it has two training modes: - Binary RL (GRPO): A process reward model scores each turn as good, bad, or neutral. That scalar reward drives policy updates via a PPO-style clipped objective. - On-Policy Distillation: When concrete corrections come in like "you should have checked that file first," it uses that feedback as a richer, directional training signal at the token level. When to use OpenClaw-RL? To be fair, a lot of agent behavior can already be improved through better memory and skill design. OpenClaw's existing skill ecosystem and community-built self-improvement skills handle a wide range of use cases without touching model weights at all. If the agent keeps forgetting preferences, that's a memory problem. And if it doesn't know how to handle a specific workflow, that's a skill problem. Both are solvable at the prompt and context layer. Where RL becomes interesting is when the failure pattern lives deeper in the model's reasoning itself. Things like consistently poor tool selection order, weak multi-step planning, or failing to interpret ambiguous instructions the way a specific user intends. Research on agentic RL (like ARTIST and Agent-R1) has shown that these behavioral patterns hit a ceiling with prompt-based approaches alone, especially in complex multi-turn tasks where the model needs to recover from tool failures or adapt its strategy mid-execution. That's the layer OpenClaw-RL targets, and it's a meaningful distinction from what OpenClaw offers. I have shared the repo in the replies!

OpenClaw meets RL! OpenClaw Agents adapt through memory files and skills, but the base model weights never actually change. OpenClaw-RL solves this! It wraps a self-hosted model as an OpenAI-compatible API, intercepts live conversations from OpenClaw, and trains the policy in the background using RL. The architecture is fully async. This means serving, reward scoring, and training all run in parallel. Once done, weights get hot-swapped after every batch while the agent keeps responding. Currently, it has two training modes: - Binary RL (GRPO): A process reward model scores each turn as good, bad, or neutral. That scalar reward drives policy updates via a PPO-style clipped objective. - On-Policy Distillation: When concrete corrections come in like "you should have checked that file first," it uses that feedback as a richer, directional training signal at the token level. When to use OpenClaw-RL? To be fair, a lot of agent behavior can already be improved through better memory and skill design. OpenClaw's existing skill ecosystem and community-built self-improvement skills handle a wide range of use cases without touching model weights at all. If the agent keeps forgetting preferences, that's a memory problem. And if it doesn't know how to handle a specific workflow, that's a skill problem. Both are solvable at the prompt and context layer. Where RL becomes interesting is when the failure pattern lives deeper in the model's reasoning itself. Things like consistently poor tool selection order, weak multi-step planning, or failing to interpret ambiguous instructions the way a specific user intends. Research on agentic RL (like ARTIST and Agent-R1) has shown that these behavioral patterns hit a ceiling with prompt-based approaches alone, especially in complex multi-turn tasks where the model needs to recover from tool failures or adapt its strategy mid-execution. That's the layer OpenClaw-RL targets, and it's a meaningful distinction from what OpenClaw offers. I have shared the repo in the replies!

Avi Chawla

138,769 Aufrufe • vor 7 Monaten

Excited to introduce SOLE-R1, a video-language reasoning model for zero-shot reward prediction for robot manipulation tasks! SOLE-R1 reasoning can serve as the SOLE signal for learning new tasks (completely from scratch) through online RL - i.e., robots start with random actions and learn previously unseen tasks guided only by SOLE-R1 rewards, without any demonstrations, ground-truth rewards, success indicators, or task-specific tuning. SOLE-R1 significantly outperforms strong baselines (e.g., Robometer, RoboReward, TOPReward, GPT-5, Gemini-3-Pro) in zero-shot online RL when evaluated across 40 tasks - including a real-world tabletop manipulation setting and 4 sim environments (LIBERO, ManiSkill, Meta-World, RoboSuite). We open source all models, training data, and code. Website, demos, and paper at: 🧵 (1/6)

Excited to introduce SOLE-R1, a video-language reasoning model for zero-shot reward prediction for robot manipulation tasks! SOLE-R1 reasoning can serve as the SOLE signal for learning new tasks (completely from scratch) through online RL - i.e., robots start with random actions and learn previously unseen tasks guided only by SOLE-R1 rewards, without any demonstrations, ground-truth rewards, success indicators, or task-specific tuning. SOLE-R1 significantly outperforms strong baselines (e.g., Robometer, RoboReward, TOPReward, GPT-5, Gemini-3-Pro) in zero-shot online RL when evaluated across 40 tasks - including a real-world tabletop manipulation setting and 4 sim environments (LIBERO, ManiSkill, Meta-World, RoboSuite). We open source all models, training data, and code. Website, demos, and paper at: 🧵 (1/6)

Philip Schroeder

19,755 Aufrufe • vor 4 Monaten

Today, we're joined by Nikita Rudin, co-founder and CEO of Flexion to discuss the gap between current robotic capabilities and what’s required to deploy fully autonomous robots in the real world. Nikita explains how reinforcement learning and simulation have driven rapid progress in robot locomotion—and why locomotion is still far from “solved.” We dig into the sim2real gap, and how adding visual inputs introduces noise and significantly complicates sim-to-real transfer. We also explore the debate between end-to-end models and modular approaches, and why separating locomotion, planning, and semantics remains a pragmatic approach today. Nikita also introduces the concept of "real-to-sim", which uses real-world data to refine simulation parameters for higher fidelity training, discusses how reinforcement learning, imitation learning, and teleoperation data are combined to train robust policies for both quadruped and humanoid robots, and introduces Flexion's hierarchical approach that utilizes pre-trained Vision-Language Models (VLMs) for high-level task orchestration with Vision-Language-Action (VLA) models and low-level whole-body trackers. Finally, Nikita shares the behind-the-scenes in humanoid robot demos, his take on reinforcement learning in simulation versus the real world, the nuances of reward tuning, and offers practical advice for researchers and practitioners looking to get started in robotics today. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 04:07 - Is robot locomotion solved? 06:04 - Sim-to-real gap 08:58 - Adding semantics to policies 09:42 - Modular vs end-to-end architectures 10:29 - Planner model 12:21 - Adapting RL techniques from quadrupeds to humanoids 15:39 - Behind robot demos 18:09 - Humanoid robots in home environments 22:03 - Training approach 23:56 - VLA models 27:59 - Closing the sim-to-real gap 32:55 - Task orchestration using VLMs 36:38 - Tool use 38:10 - Model hierarchy 43:37 - Simulator versus simulation environment 44:57 - Combining imitation learning and reinforcement learning 46:42 - RL in real world versus RL in simulation 52:58 - Reward tuning and value functions in robotics 56:38 - Predictions 1:00:10 - Humanoids, quadropeds, and wheeled platforms 1:02:45 - Advice, recommended robot kits, and community pla

Today, we're joined by Nikita Rudin, co-founder and CEO of Flexion to discuss the gap between current robotic capabilities and what’s required to deploy fully autonomous robots in the real world. Nikita explains how reinforcement learning and simulation have driven rapid progress in robot locomotion—and why locomotion is still far from “solved.” We dig into the sim2real gap, and how adding visual inputs introduces noise and significantly complicates sim-to-real transfer. We also explore the debate between end-to-end models and modular approaches, and why separating locomotion, planning, and semantics remains a pragmatic approach today. Nikita also introduces the concept of "real-to-sim", which uses real-world data to refine simulation parameters for higher fidelity training, discusses how reinforcement learning, imitation learning, and teleoperation data are combined to train robust policies for both quadruped and humanoid robots, and introduces Flexion's hierarchical approach that utilizes pre-trained Vision-Language Models (VLMs) for high-level task orchestration with Vision-Language-Action (VLA) models and low-level whole-body trackers. Finally, Nikita shares the behind-the-scenes in humanoid robot demos, his take on reinforcement learning in simulation versus the real world, the nuances of reward tuning, and offers practical advice for researchers and practitioners looking to get started in robotics today. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 04:07 - Is robot locomotion solved? 06:04 - Sim-to-real gap 08:58 - Adding semantics to policies 09:42 - Modular vs end-to-end architectures 10:29 - Planner model 12:21 - Adapting RL techniques from quadrupeds to humanoids 15:39 - Behind robot demos 18:09 - Humanoid robots in home environments 22:03 - Training approach 23:56 - VLA models 27:59 - Closing the sim-to-real gap 32:55 - Task orchestration using VLMs 36:38 - Tool use 38:10 - Model hierarchy 43:37 - Simulator versus simulation environment 44:57 - Combining imitation learning and reinforcement learning 46:42 - RL in real world versus RL in simulation 52:58 - Reward tuning and value functions in robotics 56:38 - Predictions 1:00:10 - Humanoids, quadropeds, and wheeled platforms 1:02:45 - Advice, recommended robot kits, and community pla

The TWIML AI Podcast

22,592 Aufrufe • vor 9 Monaten

New Course: Reinforcement Fine-Tuning LLMs with GRPO! Learn to use reinforcement learning to improve your LLM performance in this short course, built in collaboration with Predibase by Rubrik, and taught by Travis Addair, its Co-Founder and CTO, and Arnav Garg, its Senior Engineer and Machine Learning Lead. Reasoning models have been one of the most important developments in LLMs. Reinforcement Fine-Tuning (RFT) uses rewards to encourage LLMs to find solutions to multi-step reasoning tasks such as solving math problems and debugging code - without needing pre-existing training examples like in traditional supervised fine-tuning. Group Relative Policy Optimization (GRPO) is a reinforcement fine-tuning algorithm gaining rapid adoption. Developed by the DeepSeek team and used to train the R1 reasoning model, GRPO uses reward functions that you can write in Python to assign rewards to model responses. It’s beneficial for tasks with verifiable outcomes and can work well even with fewer than 100 training examples. It can also significantly improve the reasoning ability of smaller LLMs, making applications faster and more cost effective. In this course, you’ll take a technical deep dive into RFT with GRPO. You’ll learn to build reward functions that you can use in the GRPO training process to guide an LLM toward better performance on multi-step reasoning tasks. In detail, you’ll: - Learn when reinforcement fine-tuning is a better fit than supervised fine-tuning, especially for tasks involving multi-step reasoning or limited labeled data. - Understand how GRPO uses programmable reward functions as a more scalable alternative to the human feedback required for other reinforcement learning algorithms, such as RLHF and DPO. - Frame the Wordle game as a reinforcement fine-tuning problem and see how an LLM can learn to plan, analyze feedback, and improve its strategy over time. - Design reward functions that power the reinforcement fine-tuning process. - Learn techniques for evaluating more subjective tasks, such as rating the quality of a text summary, using an LLM as a judge. - Understand why reward hacking happens and how to avoid it by adding penalty functions to discourage undesirable behaviors. - Learn the four key components of the loss calculation in the GRPO algorithm: token probability distribution ratios, advantages, clipping, and KL-divergence. - Launch reinforcement fine-tuning jobs using Predibase’s hosted training services. By the end of this course, you’ll be able to build and fine-tune LLMs using reinforcement learning to improve reasoning without relying on large labeled datasets or subjective human feedback. Please sign up here:

New Course: Reinforcement Fine-Tuning LLMs with GRPO! Learn to use reinforcement learning to improve your LLM performance in this short course, built in collaboration with Predibase by Rubrik, and taught by Travis Addair, its Co-Founder and CTO, and Arnav Garg, its Senior Engineer and Machine Learning Lead. Reasoning models have been one of the most important developments in LLMs. Reinforcement Fine-Tuning (RFT) uses rewards to encourage LLMs to find solutions to multi-step reasoning tasks such as solving math problems and debugging code - without needing pre-existing training examples like in traditional supervised fine-tuning. Group Relative Policy Optimization (GRPO) is a reinforcement fine-tuning algorithm gaining rapid adoption. Developed by the DeepSeek team and used to train the R1 reasoning model, GRPO uses reward functions that you can write in Python to assign rewards to model responses. It’s beneficial for tasks with verifiable outcomes and can work well even with fewer than 100 training examples. It can also significantly improve the reasoning ability of smaller LLMs, making applications faster and more cost effective. In this course, you’ll take a technical deep dive into RFT with GRPO. You’ll learn to build reward functions that you can use in the GRPO training process to guide an LLM toward better performance on multi-step reasoning tasks. In detail, you’ll: - Learn when reinforcement fine-tuning is a better fit than supervised fine-tuning, especially for tasks involving multi-step reasoning or limited labeled data. - Understand how GRPO uses programmable reward functions as a more scalable alternative to the human feedback required for other reinforcement learning algorithms, such as RLHF and DPO. - Frame the Wordle game as a reinforcement fine-tuning problem and see how an LLM can learn to plan, analyze feedback, and improve its strategy over time. - Design reward functions that power the reinforcement fine-tuning process. - Learn techniques for evaluating more subjective tasks, such as rating the quality of a text summary, using an LLM as a judge. - Understand why reward hacking happens and how to avoid it by adding penalty functions to discourage undesirable behaviors. - Learn the four key components of the loss calculation in the GRPO algorithm: token probability distribution ratios, advantages, clipping, and KL-divergence. - Launch reinforcement fine-tuning jobs using Predibase’s hosted training services. By the end of this course, you’ll be able to build and fine-tune LLMs using reinforcement learning to improve reasoning without relying on large labeled datasets or subjective human feedback. Please sign up here:

Andrew Ng

86,697 Aufrufe • vor 1 Jahr

In a world of PPO everything for reinforcement learning, I've been tinkering with SAC for training a quadruped gait. This gait is trained purely on CPU (training on one of the Dell GB10s) on a single environment. Training any particular run is obviously slower than PPO on an RTX Pro 6000 with 8092 envs, if you already know the exact hyperparams/rwd function for your PPO algo... but, if we're honest with ourselves, then we know we usually spend days tuning our PPO algo and fighting it to do what we want. In contrast, SAC has kind of been a breath of fresh air, very amenable to changing the reward function to tune behavior. So far, my first attempts to tune things have consistently just worked immediately rather than 15 different variations of reward hacking only to find previous tuned behaviors got lost in the process. There is also FastSAC, which I've not yet tried, but can speed things up potentially and introduce scale back into the equation. My main painpoint in getting SAC to work for gait was actually getting it to learn to step. It seems as though SAC is not as good as PPO at significant exploration on its own. I ended up starting with a sinusoidal gait (basically just a rule to make legs swing) as training wheels then blended it out through training as phase 1, then began working on smoothing things out after this. I think if we look at end to end dev time rather than any particular run that finally managed to work, SAC may actually be the "faster" algorithm to train. Quadruped gaits are inherently easier than bipedal and maybe there are areas where SAC falls short, but I'll definitely be spending more time with SAC.

In a world of PPO everything for reinforcement learning, I've been tinkering with SAC for training a quadruped gait. This gait is trained purely on CPU (training on one of the Dell GB10s) on a single environment. Training any particular run is obviously slower than PPO on an RTX Pro 6000 with 8092 envs, if you already know the exact hyperparams/rwd function for your PPO algo... but, if we're honest with ourselves, then we know we usually spend days tuning our PPO algo and fighting it to do what we want. In contrast, SAC has kind of been a breath of fresh air, very amenable to changing the reward function to tune behavior. So far, my first attempts to tune things have consistently just worked immediately rather than 15 different variations of reward hacking only to find previous tuned behaviors got lost in the process. There is also FastSAC, which I've not yet tried, but can speed things up potentially and introduce scale back into the equation. My main painpoint in getting SAC to work for gait was actually getting it to learn to step. It seems as though SAC is not as good as PPO at significant exploration on its own. I ended up starting with a sinusoidal gait (basically just a rule to make legs swing) as training wheels then blended it out through training as phase 1, then began working on smoothing things out after this. I think if we look at end to end dev time rather than any particular run that finally managed to work, SAC may actually be the "faster" algorithm to train. Quadruped gaits are inherently easier than bipedal and maybe there are areas where SAC falls short, but I'll definitely be spending more time with SAC.

Harrison Kinsley

26,758 Aufrufe • vor 7 Monaten

The robot flipped a pancake nobody taught it! 🥞 Skild AI team assumed pancake flipping had to be somewhere in the training data. So they searched. Millions of hours of pre-training data. Nothing. S1 inferred the whole task from a single human demonstration. That's their new general robot model, built as an in-context learner from the ground up. Every new robot task today starts with days of teleoperation and a fine-tuning run on a specialist policy. S1 skips all of it. Much like a language model, it never updates its weights to learn a new task. The demonstration enters the context window, and the policy uses it to decide what to do next. → Ten-minute tasks it was never trained on, composed from primitives learned in pre-training: a new style of coffee, potting a plant, frying pancakes. → Soil and pots arrived at their office at 8:54 PM. The robot was running the task autonomously by 9:27 PM. → Slide objects away mid-reach, swap them, change the lighting, it still finishes. → The prompt waters a plant with a watering can, but only a cup is available. It uses the cup. It doesn't rigidly replay what it saw, but it recovers from its own errors, and sometimes executes with more precision than the demonstrator, when the human fumbles an egg and makes a mess, S1 performs the same step cleanly. The demonstration is a specification of the goal, and not a trajectory to copy. On unseen tasks after 100K hours of pre-training: language-prompted VLAs reach 9%. Their new model reaches 66%. It's already deploying with industrial partners, with a wider rollout over the coming months. Congrats Deepak Pathak and team behind this! 😮‍💨 🔗 Link to their latest blog: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

The robot flipped a pancake nobody taught it! 🥞 Skild AI team assumed pancake flipping had to be somewhere in the training data. So they searched. Millions of hours of pre-training data. Nothing. S1 inferred the whole task from a single human demonstration. That's their new general robot model, built as an in-context learner from the ground up. Every new robot task today starts with days of teleoperation and a fine-tuning run on a specialist policy. S1 skips all of it. Much like a language model, it never updates its weights to learn a new task. The demonstration enters the context window, and the policy uses it to decide what to do next. → Ten-minute tasks it was never trained on, composed from primitives learned in pre-training: a new style of coffee, potting a plant, frying pancakes. → Soil and pots arrived at their office at 8:54 PM. The robot was running the task autonomously by 9:27 PM. → Slide objects away mid-reach, swap them, change the lighting, it still finishes. → The prompt waters a plant with a watering can, but only a cup is available. It uses the cup. It doesn't rigidly replay what it saw, but it recovers from its own errors, and sometimes executes with more precision than the demonstrator, when the human fumbles an egg and makes a mess, S1 performs the same step cleanly. The demonstration is a specification of the goal, and not a trajectory to copy. On unseen tasks after 100K hours of pre-training: language-prompted VLAs reach 9%. Their new model reaches 66%. It's already deploying with industrial partners, with a wider rollout over the coming months. Congrats Deepak Pathak and team behind this! 😮‍💨 🔗 Link to their latest blog: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

11,406 Aufrufe • vor 1 Monat

Foundation models are enough to solve robotics! Unfortunately, this is not true. We keep hearing that Vision-Language-Action (VLA) models struggle because of the gap between static training and the dynamic real world. A German startup (Sereact) just released a solution that bridges this gap perfectly. They are introducing a new paradigm called Interactive RL Policy Patching. It's a distributed framework that allows robots to self-learn from human corrections without needing full retraining. When a robot fails, a human operator provides a brief "patch" or demonstration. The system then uses online off-policy reinforcement learning to update the behavior instantly. This is powered by a massive foundation model trained on hundreds of millions of interactions from over 100 deployed robot stations. The best part is the distributed parameter synchronization... When one robot learns a fix, the update is published fleet-wide... meaning the entire swarm gets smarter from a single human intervention. They are already proving this on complex manipulation tasks like shoe unboxing and screw sorting, drastically reducing the data needed to handle edge cases. Real-world environments are unforgiving, and I love seeing systems that can actually adapt on the fly! 📍 More info:

Foundation models are enough to solve robotics! Unfortunately, this is not true. We keep hearing that Vision-Language-Action (VLA) models struggle because of the gap between static training and the dynamic real world. A German startup (Sereact) just released a solution that bridges this gap perfectly. They are introducing a new paradigm called Interactive RL Policy Patching. It's a distributed framework that allows robots to self-learn from human corrections without needing full retraining. When a robot fails, a human operator provides a brief "patch" or demonstration. The system then uses online off-policy reinforcement learning to update the behavior instantly. This is powered by a massive foundation model trained on hundreds of millions of interactions from over 100 deployed robot stations. The best part is the distributed parameter synchronization... When one robot learns a fix, the update is published fleet-wide... meaning the entire swarm gets smarter from a single human intervention. They are already proving this on complex manipulation tasks like shoe unboxing and screw sorting, drastically reducing the data needed to handle edge cases. Real-world environments are unforgiving, and I love seeing systems that can actually adapt on the fly! 📍 More info:

Ilir Aliu

18,210 Aufrufe • vor 8 Monaten

Check out our #ICRA2024 paper "Actor-Critic Model Predictive Control." Model-free #reinforcementlearning (RL) is known for its strong task performance and flexibility in optimizing general reward formulations. On the other hand, #ModelPredictiveControl (MPC) benefits from robustness and online replanning capabilities. We combine both approaches by introducing a new framework called Actor-Critic Model Predictive Control. The key idea is to embed a differentiable MPC within an Actor-Critic RL framework. The proposed approach leverages the short-term predictive optimization capabilities of MPC with the exploratory and end-to-end training properties of RL. The resulting policy effectively manages both short-term decisions through the MPC-based actor and long-term prediction via the critic network, unifying the benefits of both model-based control and end-to-end learning. We validate our method in simulation and the real world with a quadcopter across various high-level tasks. We show that the proposed architecture can achieve real-time control performance, learn complex behaviors via trial and error, and retain the predictive properties of the MPC to better handle out-of-distribution behavior. Paper: Full Video with more details: Kudos to Ángel Romero, Yunlong Song IEEE ICRA University of Zurich UZH Science UZH Space Hub Aerial Core AUTOASSESS European Research Council (ERC)

Check out our #ICRA2024 paper "Actor-Critic Model Predictive Control." Model-free #reinforcementlearning (RL) is known for its strong task performance and flexibility in optimizing general reward formulations. On the other hand, #ModelPredictiveControl (MPC) benefits from robustness and online replanning capabilities. We combine both approaches by introducing a new framework called Actor-Critic Model Predictive Control. The key idea is to embed a differentiable MPC within an Actor-Critic RL framework. The proposed approach leverages the short-term predictive optimization capabilities of MPC with the exploratory and end-to-end training properties of RL. The resulting policy effectively manages both short-term decisions through the MPC-based actor and long-term prediction via the critic network, unifying the benefits of both model-based control and end-to-end learning. We validate our method in simulation and the real world with a quadcopter across various high-level tasks. We show that the proposed architecture can achieve real-time control performance, learn complex behaviors via trial and error, and retain the predictive properties of the MPC to better handle out-of-distribution behavior. Paper: Full Video with more details: Kudos to Ángel Romero, Yunlong Song IEEE ICRA University of Zurich UZH Science UZH Space Hub Aerial Core AUTOASSESS European Research Council (ERC)

Davide Scaramuzza

34,920 Aufrufe • vor 2 Jahren

The robot is learning several novel tasks instantly, after just ONE demonstration each... Instant Policy makes it possible: no extra training, no weight updates, just pure in-context learning. It just got accepted at ICLR 2025, and it’s changing how robots learn. With just a single demo, a robot can pick up a new task and start performing it right away. Why this is a big deal: ✅ Learns tasks instantly with just one or a few demonstrations ✅ Improves over time as more demonstrations are given ✅ Uses simulation-based training with “pseudo-demonstrations” for scalability ✅ Can transfer skills across different robots and even follow language-defined tasks It brings in-context learning to robotics, opening up new possibilities for flexible, real-world automation. You can try it yourself: code and weights are available at • • • • Thank you to Edward Johns, Director of the Robot Learning Lab at Imperial College for sharing their work! 🙏

The robot is learning several novel tasks instantly, after just ONE demonstration each... Instant Policy makes it possible: no extra training, no weight updates, just pure in-context learning. It just got accepted at ICLR 2025, and it’s changing how robots learn. With just a single demo, a robot can pick up a new task and start performing it right away. Why this is a big deal: ✅ Learns tasks instantly with just one or a few demonstrations ✅ Improves over time as more demonstrations are given ✅ Uses simulation-based training with “pseudo-demonstrations” for scalability ✅ Can transfer skills across different robots and even follow language-defined tasks It brings in-context learning to robotics, opening up new possibilities for flexible, real-world automation. You can try it yourself: code and weights are available at • • • • Thank you to Edward Johns, Director of the Robot Learning Lab at Imperial College for sharing their work! 🙏

Ilir Aliu

46,285 Aufrufe • vor 1 Jahr

Failing to Understand the Exponential, Again? My conversation with Julian Schrittwieser - Julian Schrittwieser (Anthropic, AlphaGo Zero, MuZero) - on Move 37, Scaling RL, Nobel Prize for AI, and the AI frontier: 00:00 - Cold open: “We’re not seeing any slowdown.” 00:32 - Intro — Meet Julian 01:09 - The “exponential” from inside frontier labs 04:46 - 2026–2027: agents that work a full day; expert-level breadth 08:58 - Benchmarks vs reality: long-horizon work, GDP-Val, user value 10:26 - Move 37 — what actually happened and why it mattered 13:55 - Novel science: AlphaCode/AlphaTensor → when does AI earn a Nobel? 16:25 - Discontinuity vs smooth progress (and warning signs) 19:08 - Does pre-training + RL get us there? (AGI debates aside) 20:55 - Sutton’s “RL from scratch”? Julian’s take 23:03 - Julian’s path: Google → DeepMind → Anthropic 26:45 - AlphaGo (learn + search) in plain English 30:16 - AlphaGo Zero (no human data) 31:00 - AlphaZero (one algorithm: Go, chess, shogi) 31:46 - MuZero (planning with a learned world model) 33:23 -Lessons for today’s agents: search + learning at scale 34:57 - Do LLMs already have implicit world models? 39:02 - Why RL on LLMs took time (stability, feedback loops) 41:43 - Compute & scaling for RL — what we see so far 42:35 - Rewards frontier: human prefs, rubrics, RLVR, process rewards 44:36 - RL training data & the “flywheel” (and why quality matters) 48:02 - RL & Agents 101 — why RL unlocks robustness 50:51 - Should builders use RL-as-a-service? Or just tools + prompts? 52:18 - What’s missing for dependable agents (capability vs engineering) 53:51 - Evals & Goodhart — internal vs external benchmarks 57:35 - Mechanistic interpretability & “Golden Gate Claude” 1:00:03 - Safety & alignment at Anthropic — how it shows up in practice 1:03:48 - Jobs: human–AI complementarity (comparative advantage) 1:06:33 - Inequality, policy, and the case for 10× productivity → abundance 1:09:24 - Closing thoughts

Failing to Understand the Exponential, Again? My conversation with Julian Schrittwieser - Julian Schrittwieser (Anthropic, AlphaGo Zero, MuZero) - on Move 37, Scaling RL, Nobel Prize for AI, and the AI frontier: 00:00 - Cold open: “We’re not seeing any slowdown.” 00:32 - Intro — Meet Julian 01:09 - The “exponential” from inside frontier labs 04:46 - 2026–2027: agents that work a full day; expert-level breadth 08:58 - Benchmarks vs reality: long-horizon work, GDP-Val, user value 10:26 - Move 37 — what actually happened and why it mattered 13:55 - Novel science: AlphaCode/AlphaTensor → when does AI earn a Nobel? 16:25 - Discontinuity vs smooth progress (and warning signs) 19:08 - Does pre-training + RL get us there? (AGI debates aside) 20:55 - Sutton’s “RL from scratch”? Julian’s take 23:03 - Julian’s path: Google → DeepMind → Anthropic 26:45 - AlphaGo (learn + search) in plain English 30:16 - AlphaGo Zero (no human data) 31:00 - AlphaZero (one algorithm: Go, chess, shogi) 31:46 - MuZero (planning with a learned world model) 33:23 -Lessons for today’s agents: search + learning at scale 34:57 - Do LLMs already have implicit world models? 39:02 - Why RL on LLMs took time (stability, feedback loops) 41:43 - Compute & scaling for RL — what we see so far 42:35 - Rewards frontier: human prefs, rubrics, RLVR, process rewards 44:36 - RL training data & the “flywheel” (and why quality matters) 48:02 - RL & Agents 101 — why RL unlocks robustness 50:51 - Should builders use RL-as-a-service? Or just tools + prompts? 52:18 - What’s missing for dependable agents (capability vs engineering) 53:51 - Evals & Goodhart — internal vs external benchmarks 57:35 - Mechanistic interpretability & “Golden Gate Claude” 1:00:03 - Safety & alignment at Anthropic — how it shows up in practice 1:03:48 - Jobs: human–AI complementarity (comparative advantage) 1:06:33 - Inequality, policy, and the case for 10× productivity → abundance 1:09:24 - Closing thoughts

Matt Turck

235,526 Aufrufe • vor 11 Monaten

Can an inexpensive, off-the-shelf IMU be the only sensor to estimate the full state (position, velocity, orientation) of a quadrotor flying through a track at high speed and even be on-pair with vision-based localization? The answer is yes, within certain limitations! In this #RAL2023 paper, we propose a learning-based odometry algorithm that couples a model-based filter driven by the inertial measurements with a learning-based module with access to the control commands. Our system outperforms by a large margin the state-of-the-art visual-inertial odometry (#VIO) algorithms and the state-of-the-art learned-inertial odometry algorithm, #TLIO, for the task of drone racing. Additionally, we show that our system is as accurate as a VIO algorithm that uses a camera to localize to a known map of the racing track. The main limitation of our approach is that it cannot generalize to trajectories that have not been seen at training time. However, in drone racing competitions, the track is known beforehand. Human pilots spend hours or even days of practice on the race track before the competition. Similarly, our system can be trained with the data collected during practice time and deployed during the competition. Future work will investigate how to generalize to trajectories not seen at training time. The code is released! Paper: Video: Code: Kudos to Giovanni Cioffi Leonard Bauersfeld Elia Kaufmann European Research Council (ERC) University of Zurich UZH Science UZH Space Hub NCCR Robotics Aerial Core #RAL2023 #IROS2023 #SLAM

Can an inexpensive, off-the-shelf IMU be the only sensor to estimate the full state (position, velocity, orientation) of a quadrotor flying through a track at high speed and even be on-pair with vision-based localization? The answer is yes, within certain limitations! In this #RAL2023 paper, we propose a learning-based odometry algorithm that couples a model-based filter driven by the inertial measurements with a learning-based module with access to the control commands. Our system outperforms by a large margin the state-of-the-art visual-inertial odometry (#VIO) algorithms and the state-of-the-art learned-inertial odometry algorithm, #TLIO, for the task of drone racing. Additionally, we show that our system is as accurate as a VIO algorithm that uses a camera to localize to a known map of the racing track. The main limitation of our approach is that it cannot generalize to trajectories that have not been seen at training time. However, in drone racing competitions, the track is known beforehand. Human pilots spend hours or even days of practice on the race track before the competition. Similarly, our system can be trained with the data collected during practice time and deployed during the competition. Future work will investigate how to generalize to trajectories not seen at training time. The code is released! Paper: Video: Code: Kudos to Giovanni Cioffi Leonard Bauersfeld Elia Kaufmann European Research Council (ERC) University of Zurich UZH Science UZH Space Hub NCCR Robotics Aerial Core #RAL2023 #IROS2023 #SLAM

Davide Scaramuzza

37,061 Aufrufe • vor 3 Jahren

New blackboard lecture w Eric Jang He walks through how to build AlphaGo from scratch, but with modern AI tools. Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn. Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it could work better – naive policy gradient RL has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer, while AlphaGo’s MCTS suggests a strictly better action every single move, giving you a training target that sidesteps the credit assignment problem. The way humans learn is surely closer to the second. Eric also kickstarted an Autoresearch loop on his project. And it was very interesting to discuss which parts of AI research LLMs can already automate pretty well (implementing and running experiments, optimizing hyperparameters) and which they still struggle with (choosing the right question to investigate next, escaping research dead ends). Informative to all the recent discussion about when we should expect an intelligence explosion, and what it would look like from the inside. Timestamps: 0:00:00 – Basics of Go 0:08:06 – Monte Carlo Tree Search 0:31:53 – What the neural network does 1:00:22 – Self-play 1:25:27 – Alternative RL approaches 1:45:36 – Why doesn’t MCTS work for LLMs 2:00:58 – Off-policy training 2:11:51 – RL is even more information inefficient than you thought 2:22:05 – Automated AI researchers

New blackboard lecture w Eric Jang He walks through how to build AlphaGo from scratch, but with modern AI tools. Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn. Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it could work better – naive policy gradient RL has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer, while AlphaGo’s MCTS suggests a strictly better action every single move, giving you a training target that sidesteps the credit assignment problem. The way humans learn is surely closer to the second. Eric also kickstarted an Autoresearch loop on his project. And it was very interesting to discuss which parts of AI research LLMs can already automate pretty well (implementing and running experiments, optimizing hyperparameters) and which they still struggle with (choosing the right question to investigate next, escaping research dead ends). Informative to all the recent discussion about when we should expect an intelligence explosion, and what it would look like from the inside. Timestamps: 0:00:00 – Basics of Go 0:08:06 – Monte Carlo Tree Search 0:31:53 – What the neural network does 1:00:22 – Self-play 1:25:27 – Alternative RL approaches 1:45:36 – Why doesn’t MCTS work for LLMs 2:00:58 – Off-policy training 2:11:51 – RL is even more information inefficient than you thought 2:22:05 – Automated AI researchers

Dwarkesh Patel

708,022 Aufrufe • vor 4 Monaten

Ever wished we had fewer X-training hyphenates? Pre, mid, post etc. Why not just Training? Trying to bridge the divides (and get all our friends into one team again), we intro *Introspective X Training*, an offline RL inspired method that scales effectively across any LLM stage by annotating your data with a thinking reward generated language critique! Up to 2.8x FLOP efficiency + 5-10 point score gains (esp with math and code) at any stage from scratch to 24T tokens on 8b (active) sized models!! We burned much compute ablating so you wouldn't have to Moral of the story is‼️don't throw out any data via filtering, just feedback condition it‼️ You can spend FLOPs up front on inference to *classify* data quality and then train so that tokens aren't all treated equally based on the feedback starting early in training itself. Right now they're really only separated out much later during mid/post training This improves overall compute efficiency and gives us benchmark perf not possible with just baseline methods! Paper here: Thanks to Brandon Cui and Ximing Lu for leading this w/ Syeda Nahida Akter David Acuna Hyunwoo Kim Jaehun Jung Yuxiao Qu Shrimai Yejin Choi

Ever wished we had fewer X-training hyphenates? Pre, mid, post etc. Why not just Training? Trying to bridge the divides (and get all our friends into one team again), we intro Introspective X Training, an offline RL inspired method that scales effectively across any LLM stage by annotating your data with a thinking reward generated language critique! Up to 2.8x FLOP efficiency + 5-10 point score gains (esp with math and code) at any stage from scratch to 24T tokens on 8b (active) sized models!! We burned much compute ablating so you wouldn't have to Moral of the story is‼️don't throw out any data via filtering, just feedback condition it‼️ You can spend FLOPs up front on inference to classify data quality and then train so that tokens aren't all treated equally based on the feedback starting early in training itself. Right now they're really only separated out much later during mid/post training This improves overall compute efficiency and gives us benchmark perf not possible with just baseline methods! Paper here: Thanks to Brandon Cui and Ximing Lu for leading this w/ Syeda Nahida Akter David Acuna Hyunwoo Kim Jaehun Jung Yuxiao Qu Shrimai Yejin Choi

Prithviraj (Raj) Ammanabrolu

27,870 Aufrufe • vor 4 Monaten

15–30 minutes of real-world robot data. That's now enough to go from sim-to-real failure to working robot. Let’s see… You train a robot in simulation. You deploy it in the real world. It fails. The physics don't match. So you try to fine-tune it with real data, but… you never have enough real data, and the fine-tuning breaks everything the simulation taught it. SimDist fixes this with one key decision: don't transfer the policy. Transfer the world model. Keep the reward and value knowledge from simulation frozen. Only update the part that's actually wrong, how the robot predicts physics. Now the robot doesn't have to relearn the entire task in the real world. It already knows what success looks like. It just needs to correct its understanding of how the real world moves. The part that makes this work: they also trained on failures and recoveries; not just perfect demonstrations. Without that, the planner finds the gaps and exploits them. With it, the robot can tell a good future from a bad one. That's all it needs. Results on peg insertion, table leg assembly, locomotion on slippery and uneven surfaces. Tasks that require precision, force, and quick reaction. Thanks for sharing, Tyler Westenbroek ([Tyler Westenbroek]. Interactive visualization + paper: ——- Weekly robotics and AI insights. Subscribe free:

15–30 minutes of real-world robot data. That's now enough to go from sim-to-real failure to working robot. Let’s see… You train a robot in simulation. You deploy it in the real world. It fails. The physics don't match. So you try to fine-tune it with real data, but… you never have enough real data, and the fine-tuning breaks everything the simulation taught it. SimDist fixes this with one key decision: don't transfer the policy. Transfer the world model. Keep the reward and value knowledge from simulation frozen. Only update the part that's actually wrong, how the robot predicts physics. Now the robot doesn't have to relearn the entire task in the real world. It already knows what success looks like. It just needs to correct its understanding of how the real world moves. The part that makes this work: they also trained on failures and recoveries; not just perfect demonstrations. Without that, the planner finds the gaps and exploits them. With it, the robot can tell a good future from a bad one. That's all it needs. Results on peg insertion, table leg assembly, locomotion on slippery and uneven surfaces. Tasks that require precision, force, and quick reaction. Thanks for sharing, Tyler Westenbroek ([Tyler Westenbroek]. Interactive visualization + paper: ——- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

14,222 Aufrufe • vor 4 Monaten

Mistral AI Releases Robostral Navigate: An 8B Model Enabling Robots to Navigate Complex Environments Hitting 76.6% on R2R-CE With One RGB Camera. No LiDAR. No depth sensor. No multi-camera rig. Here's how it works. 👇 1. Pointing, not metric commands The model predicts the pixel coordinates of the next target in the camera view, plus the arrival orientation. Working in pixel space keeps it robust to camera intrinsics and world scale. When the target leaves the frame, it falls back to local displacements ("2m forward, 1.5m left, turn 25°"). 2. Grounding-first No open-source VLM base. It starts from Mistral's grounding model (pointing, counting, localization). Navigation emerges once the model knows where things are. → ~400,000 trajectories across 6,000 simulated scenes 3. Prefix-caching for training A tree-based attention mask packs a full episode into one sequence — all time steps in a single forward pass. → 22× fewer training tokens; months of training done in days 4. Online RL on top After supervised training, CISPO adds trial-and-error learning to fight distribution shift from behavior cloning. → +3.2% success rate from RL alone 5. The numbers (R2R-CE, Matterport3D) → 76.6% success on validation unseen → +9.7 pts over best single-camera approach → +4.5 pts over best depth/multi-camera system The key takeaway: state-of-the-art continuous VLN without a sensor stack — grounding-init, pixel-space actions, prefix-cached SFT, and online RL, on one RGB camera. Full analysis: Technical details: Mistral AI Mistral AI for Developers

Mistral AI Releases Robostral Navigate: An 8B Model Enabling Robots to Navigate Complex Environments Hitting 76.6% on R2R-CE With One RGB Camera. No LiDAR. No depth sensor. No multi-camera rig. Here's how it works. 👇 1. Pointing, not metric commands The model predicts the pixel coordinates of the next target in the camera view, plus the arrival orientation. Working in pixel space keeps it robust to camera intrinsics and world scale. When the target leaves the frame, it falls back to local displacements ("2m forward, 1.5m left, turn 25°"). 2. Grounding-first No open-source VLM base. It starts from Mistral's grounding model (pointing, counting, localization). Navigation emerges once the model knows where things are. → ~400,000 trajectories across 6,000 simulated scenes 3. Prefix-caching for training A tree-based attention mask packs a full episode into one sequence — all time steps in a single forward pass. → 22× fewer training tokens; months of training done in days 4. Online RL on top After supervised training, CISPO adds trial-and-error learning to fight distribution shift from behavior cloning. → +3.2% success rate from RL alone 5. The numbers (R2R-CE, Matterport3D) → 76.6% success on validation unseen → +9.7 pts over best single-camera approach → +4.5 pts over best depth/multi-camera system The key takeaway: state-of-the-art continuous VLN without a sensor stack — grounding-init, pixel-space actions, prefix-cached SFT, and online RL, on one RGB camera. Full analysis: Technical details: Mistral AI Mistral AI for Developers

Marktechpost AI

39,955 Aufrufe • vor 2 Monaten

There’s a guy running a high-frequency trading algorithm on Polymarket, averaging just $53 per trade. In the end, he made $129,923. His bot specializes in short-term BTC and ETH up/down markets, using a mixed strategy of two-sided market making plus complete-pair accumulation, combined with inventory rebalancing. The specific logic works like this: First, it accumulates positions on both the “up” and “down” sides at different times. On both sides it hangs buy orders, taking advantage of market volatility to get filled at different prices. For example, on the “up” side it buys at 47 cents, and a while later on the “down” side it buys at 52 cents, adding up to 99 cents combined, which locks in a guaranteed 1 cent gross profit. Second, it keeps a bit of directional exposure. After putting together complete pairs, if its fair-probability model sees an edge, it holds onto part of the directional position. The model is based on external data on the underlying assets, order book depth, and things like momentum, acceleration, and volatility. His main Polymarket page: The core edge of his high-frequency algorithm is constantly repeating the same cycle, steadily eating up the price difference to keep compounding his capital into a bigger and bigger pile.

There’s a guy running a high-frequency trading algorithm on Polymarket, averaging just $53 per trade. In the end, he made $129,923. His bot specializes in short-term BTC and ETH up/down markets, using a mixed strategy of two-sided market making plus complete-pair accumulation, combined with inventory rebalancing. The specific logic works like this: First, it accumulates positions on both the “up” and “down” sides at different times. On both sides it hangs buy orders, taking advantage of market volatility to get filled at different prices. For example, on the “up” side it buys at 47 cents, and a while later on the “down” side it buys at 52 cents, adding up to 99 cents combined, which locks in a guaranteed 1 cent gross profit. Second, it keeps a bit of directional exposure. After putting together complete pairs, if its fair-probability model sees an edge, it holds onto part of the directional position. The model is based on external data on the underlying assets, order book depth, and things like momentum, acceleration, and volatility. His main Polymarket page: The core edge of his high-frequency algorithm is constantly repeating the same cycle, steadily eating up the price difference to keep compounding his capital into a bigger and bigger pile.

draven

60,555 Aufrufe • vor 1 Monat

Let's reverse engineer this demo. You need 3 things: (1) robust hardware and motor designs that treat simulation as first-class citizen; (2) a human motion capture ("mocap") dataset, such as those for film and gaming characters; (3) massively parallel RL training in GPU-accelerated simulation. Last October, our team trained a 1.5M parameter foundation model called HOVER for such agile motor control. It follows this recipe, roughly speaking (details in thread): (1) Simulation used to be an after-thought. Now, it has to be part of the hardware design process. If your robot doesn't simulate well, you can kiss RL goodbye. Hardware-simulation co-design is a very interesting emergent topic that only becomes meaningful with today's compute capability. (2) Human mocap dataset to produce natural-looking walking and running gaits. That's one huge advantage of using humanoid robot - you get to imitate from tons of human motions that were originally captured for movies or AAA games. At least 3 ways to use the data: - For initialization: pre-train the neural net to imitate human, and then finetune it into the robot form factor with physics turned on; - For reward function: penalize any deviations from the target pose; - For representation learning: treat the human poses as a "motion prior" to constrain the space of robot behaviors. (3) Shove the above into Isaac sim, add a lot of randomization, pump it through PPO, throw in a bunch of GPUs, and then watch Netflix till loss converges. If you have an urge to comment this is CGI, let me save you a few keystrokes — many academic labs now own the G1 robot in the flesh. See our team's HOVER work in the thread: 🧵

Let's reverse engineer this demo. You need 3 things: (1) robust hardware and motor designs that treat simulation as first-class citizen; (2) a human motion capture ("mocap") dataset, such as those for film and gaming characters; (3) massively parallel RL training in GPU-accelerated simulation. Last October, our team trained a 1.5M parameter foundation model called HOVER for such agile motor control. It follows this recipe, roughly speaking (details in thread): (1) Simulation used to be an after-thought. Now, it has to be part of the hardware design process. If your robot doesn't simulate well, you can kiss RL goodbye. Hardware-simulation co-design is a very interesting emergent topic that only becomes meaningful with today's compute capability. (2) Human mocap dataset to produce natural-looking walking and running gaits. That's one huge advantage of using humanoid robot - you get to imitate from tons of human motions that were originally captured for movies or AAA games. At least 3 ways to use the data: - For initialization: pre-train the neural net to imitate human, and then finetune it into the robot form factor with physics turned on; - For reward function: penalize any deviations from the target pose; - For representation learning: treat the human poses as a "motion prior" to constrain the space of robot behaviors. (3) Shove the above into Isaac sim, add a lot of randomization, pump it through PPO, throw in a bunch of GPUs, and then watch Netflix till loss converges. If you have an urge to comment this is CGI, let me save you a few keystrokes — many academic labs now own the G1 robot in the flesh. See our team's HOVER work in the thread: 🧵

Jim Fan

216,139 Aufrufe • vor 1 Jahr

Reinforcement Learning from Human Feedback (RLHF) is gaining traction. This field aims to make AI more responsible by including human values and preferences. In this video, Nathan Lambert, a research scientist and RLHF team lead at Hugging Face explores its inner workings, applications and industry impact. RLHF has gained the spotlight in recent years. The growth of language models like Anthropic’s Claude and OpenAI's ChatGPT have increased interest in human-feedback integration. "There are some rumors that Open AI had two teams; one was doing RLHF and the other instruction fine-tuning. And the RLHF team kept getting more and more performance." Understanding RLHF The RLHF process has three main steps: Pre-training: Much like with GPT models, the journey starts with pre-training on a large corpus of data. This can range from text data, web scrapes, to specialized datasets. Reward Modeling: This is the RLHF counterpart of supervised fine-tuning in large language models. This stage involves creating a reward model that resonates with human values and preferences. RL Optimization: This stage parallels reward modeling and reinforcement learning in traditional AI models. The AI system fine-tunes itself based on the reward model, employing reinforcement learning algorithms for that extra layer of optimization. The Data Challenge Data collection and curation in RLHF closely resemble the challenges you'd encounter in large language model training. Datasets from organizations like OpenAI can serve as a useful foundation. However, the need for high-quality, task-specific data cannot be overstated. Implementing RLHF: A Practical Guide If you’re someone who loves getting hands-on with AI libraries like Hugging Face, implementing RLHF is right way to do. It’s essential to understand its limitations. Think about model stability, over-optimization, and exploration strategies, much like you would when prompt engineering. Ongoing Research and Next Steps While he suggests that some basics figured out, there are layers of complexity that still need to be unraveled: 1. New Benchmarks: How do we measure the effectiveness of RLHF? 2. Preference Modeling: How can the model be made to understand human preferences better? 3. Interpreting RLHF: Much like explainability in traditional models, how do we make RLHF more interpretable? 4. System-Wide Evaluation: Going beyond individual performance, how does RLHF affect an entire system? The Transformative Power of RLHF Whether you're an AI developer, a business analyst, or a marketer, RLHF promises to revolutionize your domain. Imagine customer service chatbots that understand human emotions better, or content generators that align more closely with human values. RLHF is an emerging field that focuses on enhancing machine learning models through human feedback. While it tackles important issues like bias and ethics, its broader goal is to improve system performance across various applications. Whether you're deeply invested in the ethics of AI or simply curious about advancements in machine learning, RLHF offers valuable insights. If you're interested in the next wave of AI development, this area is definitely one to watch.

Reinforcement Learning from Human Feedback (RLHF) is gaining traction. This field aims to make AI more responsible by including human values and preferences. In this video, Nathan Lambert, a research scientist and RLHF team lead at Hugging Face explores its inner workings, applications and industry impact. RLHF has gained the spotlight in recent years. The growth of language models like Anthropic’s Claude and OpenAI's ChatGPT have increased interest in human-feedback integration. "There are some rumors that Open AI had two teams; one was doing RLHF and the other instruction fine-tuning. And the RLHF team kept getting more and more performance." Understanding RLHF The RLHF process has three main steps: Pre-training: Much like with GPT models, the journey starts with pre-training on a large corpus of data. This can range from text data, web scrapes, to specialized datasets. Reward Modeling: This is the RLHF counterpart of supervised fine-tuning in large language models. This stage involves creating a reward model that resonates with human values and preferences. RL Optimization: This stage parallels reward modeling and reinforcement learning in traditional AI models. The AI system fine-tunes itself based on the reward model, employing reinforcement learning algorithms for that extra layer of optimization. The Data Challenge Data collection and curation in RLHF closely resemble the challenges you'd encounter in large language model training. Datasets from organizations like OpenAI can serve as a useful foundation. However, the need for high-quality, task-specific data cannot be overstated. Implementing RLHF: A Practical Guide If you’re someone who loves getting hands-on with AI libraries like Hugging Face, implementing RLHF is right way to do. It’s essential to understand its limitations. Think about model stability, over-optimization, and exploration strategies, much like you would when prompt engineering. Ongoing Research and Next Steps While he suggests that some basics figured out, there are layers of complexity that still need to be unraveled: 1. New Benchmarks: How do we measure the effectiveness of RLHF? 2. Preference Modeling: How can the model be made to understand human preferences better? 3. Interpreting RLHF: Much like explainability in traditional models, how do we make RLHF more interpretable? 4. System-Wide Evaluation: Going beyond individual performance, how does RLHF affect an entire system? The Transformative Power of RLHF Whether you're an AI developer, a business analyst, or a marketer, RLHF promises to revolutionize your domain. Imagine customer service chatbots that understand human emotions better, or content generators that align more closely with human values. RLHF is an emerging field that focuses on enhancing machine learning models through human feedback. While it tackles important issues like bias and ethics, its broader goal is to improve system performance across various applications. Whether you're deeply invested in the ethics of AI or simply curious about advancements in machine learning, RLHF offers valuable insights. If you're interested in the next wave of AI development, this area is definitely one to watch.

Muratcan Koylan

27,366 Aufrufe • vor 3 Jahren