Loading video...

Video Failed to Load

Go Home

New project! Flow Policy Gradients for Robot Control tldr; a simple online RL recipe for training and fine-tuning flow policies for robots co-led w/ Hongsuk Benjamin Choi:

brenthyi's profile picture

Brent Yi

1,773 subscribers

92,409 views • 6 months ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

🤔 How to fine-tune an Imitation Learning policy (e.g., Diffusion Policy, ACT) with RL? As an RL practitioner, I’ve been struggling with this problem for a while. Here’s why it’s tough: 1️⃣ Special designs (usually for multimodal action distributions) in modern IL models make them non-trivial to fine-tune by RL. 2️⃣ Large policy models + RL's poor sample efficiency = a nightmare But finally, we figured out a simple solution that works for any model architecture! 🌟 Check out our #ICLR2025 paper: “Policy Decorator: Model-Agnostic Online Refinement for Large Policy Models”, led by my amazing mentee Xiu Yuan. 🔗 🧵 Read more below!
0:16

🤔 How to fine-tune an Imitation Learning policy (e.g., Diffusion Policy, ACT) with RL? As an RL practitioner, I’ve been struggling with this problem for a while. Here’s why it’s tough: 1️⃣ Special designs (usually for multimodal action distributions) in modern IL models make them non-trivial to fine-tune by RL. 2️⃣ Large policy models + RL's poor sample efficiency = a nightmare But finally, we figured out a simple solution that works for any model architecture! 🌟 Check out our #ICLR2025 paper: “Policy Decorator: Model-Agnostic Online Refinement for Large Policy Models”, led by my amazing mentee Xiu Yuan. 🔗 🧵 Read more below!

Tongzhou Mu 🤖🦾🦿

16,997 views • 1 year ago

Reinforcement Learning (RL) has long been the dominant method for fine-tuning, powering many state-of-the-art LLMs. Methods like PPO and GRPO explore in action space. But can we instead explore directly in parameter space? YES we can. We propose a scalable framework for full-parameter fine-tuning using Evolution Strategies (ES). By skipping gradients and optimizing directly in parameter space, ES achieves more accurate, efficient, and stable fine-tuning. Paper: Code:
0:16

Reinforcement Learning (RL) has long been the dominant method for fine-tuning, powering many state-of-the-art LLMs. Methods like PPO and GRPO explore in action space. But can we instead explore directly in parameter space? YES we can. We propose a scalable framework for full-parameter fine-tuning using Evolution Strategies (ES). By skipping gradients and optimizing directly in parameter space, ES achieves more accurate, efficient, and stable fine-tuning. Paper: Code:

Yulu Gan

415,325 views • 10 months ago

How does high-fidelity tactile simulation help robots nail the last millimeter? We’re releasing VT-Refine, accepted to CoRL: a real-to-sim-to-real visuo-tactile policy using a GPU-parallel tactile sim for our piezoresistive skin FlexiTac. Then fine-tuning a diffusion policy with large-scale RL in simulation. Website: #CoRL2025 #RobotLearning #Sim2Real
0:17

How does high-fidelity tactile simulation help robots nail the last millimeter? We’re releasing VT-Refine, accepted to CoRL: a real-to-sim-to-real visuo-tactile policy using a GPU-parallel tactile sim for our piezoresistive skin FlexiTac. Then fine-tuning a diffusion policy with large-scale RL in simulation. Website: #CoRL2025 #RobotLearning #Sim2Real

Binghao Huang

46,982 views • 9 months ago

MolmoAct2 is landing in LeRobot! Ai2's open Action Reasoning Model combines a Molmo2-ER vision-language backbone with a flow-matching continuous action expert to predict robot action chunks from images, language instructions, and proprioceptive state. An open robot foundation model built for real-world control, with strong out-of-the-box performance and easy fine-tuning in LeRobot. Pick-and-place inference running on NVIDIA DGX Spark! Blog: Paper: Thanks to Ai2 Jiafei Duan Haoquan Fang
0:11

MolmoAct2 is landing in LeRobot! Ai2's open Action Reasoning Model combines a Molmo2-ER vision-language backbone with a flow-matching continuous action expert to predict robot action chunks from images, language instructions, and proprioceptive state. An open robot foundation model built for real-world control, with strong out-of-the-box performance and easy fine-tuning in LeRobot. Pick-and-place inference running on NVIDIA DGX Spark! Blog: Paper: Thanks to Ai2 Jiafei Duan Haoquan Fang

LeRobot

24,727 views • 2 months ago

I just spent weeks handwriting a 200+ page guide on the entirety of Robotics foundations from scratch. Featuring: - Kinematics (Forward, Inverse Kin, Jacobians, DoF) - Controls (PID, Force/Impedance Control, Feedforward) - SLAM (Bayes Filters, Kalman, Particle Filters, Mapping) - Motion Planning (Dijkstra, A*, RRT, RRT*) - RL (MDP, Policy Gradients, PPO, Sim2Real, Value Func) - VLA (Q/K/V, ViT, SigLIP, Diffusion/Flow-Matching) - ACT, Diffusion Policy, UMI, LeRobot tooling - World Model (Dreamer, VideoGen, Gaussian Splatting) Everything to get started, for free.
0:19

I just spent weeks handwriting a 200+ page guide on the entirety of Robotics foundations from scratch. Featuring: - Kinematics (Forward, Inverse Kin, Jacobians, DoF) - Controls (PID, Force/Impedance Control, Feedforward) - SLAM (Bayes Filters, Kalman, Particle Filters, Mapping) - Motion Planning (Dijkstra, A*, RRT, RRT*) - RL (MDP, Policy Gradients, PPO, Sim2Real, Value Func) - VLA (Q/K/V, ViT, SigLIP, Diffusion/Flow-Matching) - ACT, Diffusion Policy, UMI, LeRobot tooling - World Model (Dreamer, VideoGen, Gaussian Splatting) Everything to get started, for free.

Arjun Virk

63,701 views • 20 days ago

Success! 🍎🤖 I can now control Unitree robots on a Mac Mini M4 running macOS with a simple tweak! The unitree_sdk2_python ( thread implementation isn’t essential—getting the DDS subscriber and publisher to work is enough. I replaced Unitree’s Linux thread logic with a simple Python while loop for sensor reading and robot control. Here’s a video of me controlling a Go2 robot on macOS (M4 Mac Mini) using a Python script for low-level control and sensor reading.
0:11

Success! 🍎🤖 I can now control Unitree robots on a Mac Mini M4 running macOS with a simple tweak! The unitree_sdk2_python ( thread implementation isn’t essential—getting the DDS subscriber and publisher to work is enough. I replaced Unitree’s Linux thread logic with a simple Python while loop for sensor reading and robot control. Here’s a video of me controlling a Go2 robot on macOS (M4 Mac Mini) using a Python script for low-level control and sensor reading.

Tairan He

57,603 views • 1 year ago

Generalist robot policies need a benchmark that works across any robot and any policy. 🦾 Introducing RoboLab, a high‑fidelity simulation benchmark built on NVIDIA Isaac and Omniverse to evaluate generalist robot policies in diverse, photoreal, physics‑based environments. Coming soon to the NVIDIA Isaac Lab‑Arena roadmap for large‑scale, robotic policy evaluation. 📖 #NationalRoboticsWeek
0:15

Generalist robot policies need a benchmark that works across any robot and any policy. 🦾 Introducing RoboLab, a high‑fidelity simulation benchmark built on NVIDIA Isaac and Omniverse to evaluate generalist robot policies in diverse, photoreal, physics‑based environments. Coming soon to the NVIDIA Isaac Lab‑Arena roadmap for large‑scale, robotic policy evaluation. 📖 #NationalRoboticsWeek

NVIDIA Robotics

23,872 views • 3 months ago

Introducing CQN: Coarse-to-fine Q-Network, a value-based RL algorithm for continuous control🦾Initialized with 20~50 demonstrations, it learns to solve real-world robotic tasks within 10 mins of training, without any pre-training and shaped rewards! (1/4)
0:10

Introducing CQN: Coarse-to-fine Q-Network, a value-based RL algorithm for continuous control🦾Initialized with 20~50 demonstrations, it learns to solve real-world robotic tasks within 10 mins of training, without any pre-training and shaped rewards! (1/4)

Younggyo Seo

16,413 views • 2 years ago

New research from Databricks: LLMs Can Learn to Reason via Off-Policy RL Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL) shows you don’t need strict on-policy training to improve reasoning. It matches or beats Group Relative Policy Optimization (GRPO), stays stable with large policy lag, and uses ~3× fewer training generations. For Databricks customers, it’s a simpler, practical, and equally powerful approach to RL that Databricks is pioneering internally — and bringing directly to Databricks customers, so enterprises can improve agents using the same methods we use for our in-house agents, without complex infrastructure changes.
0:13

New research from Databricks: LLMs Can Learn to Reason via Off-Policy RL Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL) shows you don’t need strict on-policy training to improve reasoning. It matches or beats Group Relative Policy Optimization (GRPO), stays stable with large policy lag, and uses ~3× fewer training generations. For Databricks customers, it’s a simpler, practical, and equally powerful approach to RL that Databricks is pioneering internally — and bringing directly to Databricks customers, so enterprises can improve agents using the same methods we use for our in-house agents, without complex infrastructure changes.

Databricks AI Research

12,660 views • 5 months ago

D4RL is a great benchmark, but is saturated. Introducing OGBench, a new benchmark for offline goal-conditioned RL and offline RL! Tasks include HumanoidMaze, Puzzle, Drawing, and more 🙂 Project page: GitHub: 🧵↓
0:16

D4RL is a great benchmark, but is saturated. Introducing OGBench, a new benchmark for offline goal-conditioned RL and offline RL! Tasks include HumanoidMaze, Puzzle, Drawing, and more 🙂 Project page: GitHub: 🧵↓

Seohong Park

36,447 views • 1 year ago

Can robots learn without training❓ [𝗜𝘁'𝘀 𝗼𝗽𝗲𝗻 𝘀𝗼𝘂𝗿𝗰𝗲𝗱 ⬇ ] Teaching robots to do complex tasks WITHOUT spending hours training them. Sounds cool, right? That's exactly what DIAL-MPC does! The first training-free method for whole-body torque control using full-order dynamics: ✅ Instantly checks if a robot's moves are right or wrong ✅ Adapts quickly to new tasks without needing extra training ✅ Could work hand-in-hand with other robot learning methods Robots are getting smarter AND faster without the need for long training sessions. Website: Paper: Code: Saw this first Haoru Xue ✈️ CVPR 🙏
0:10

Can robots learn without training❓ [𝗜𝘁'𝘀 𝗼𝗽𝗲𝗻 𝘀𝗼𝘂𝗿𝗰𝗲𝗱 ⬇ ] Teaching robots to do complex tasks WITHOUT spending hours training them. Sounds cool, right? That's exactly what DIAL-MPC does! The first training-free method for whole-body torque control using full-order dynamics: ✅ Instantly checks if a robot's moves are right or wrong ✅ Adapts quickly to new tasks without needing extra training ✅ Could work hand-in-hand with other robot learning methods Robots are getting smarter AND faster without the need for long training sessions. Website: Paper: Code: Saw this first Haoru Xue ✈️ CVPR 🙏

Ilir Aliu

71,502 views • 1 year ago

Disney’s testing a new crowd control system for the first day of Flower & Garden today! Three lines: attractions, pins, and merch — they’ll be released at different stages to help crowd flow!
0:16

Disney’s testing a new crowd control system for the first day of Flower & Garden today! Three lines: attractions, pins, and merch — they’ll be released at different stages to help crowd flow!

Disney Food Blog

31,751 views • 5 months ago

🚀 Introducing Frank 🤖—a whole-body robot control system for day-to-day household chores. Frank has been in works for the past year—a tightly coupled hardware and a remote teleoperation interface co-led by Michelle, Nandini, and Pulkit Agrawal. Thanks to Aditya Agarwal and Shankara Narayanan for the awesome help with recording the videos! 🎥 🧵[1/6]
0:14

🚀 Introducing Frank 🤖—a whole-body robot control system for day-to-day household chores. Frank has been in works for the past year—a tightly coupled hardware and a remote teleoperation interface co-led by Michelle, Nandini, and Pulkit Agrawal. Thanks to Aditya Agarwal and Shankara Narayanan for the awesome help with recording the videos! 🎥 🧵[1/6]

Bipasha Sen

51,137 views • 1 year ago

Another feature I implemented is cutscene skipping for both games, with help from a co-worker fine tuning engine things. So, you're on your 100th play of SR2 and can't stand Moebius yapping for 6 minutes straight? SKIP HIM!
0:14

Another feature I implemented is cutscene skipping for both games, with help from a co-worker fine tuning engine things. So, you're on your 100th play of SR2 and can't stand Moebius yapping for 6 minutes straight? SKIP HIM!

Gemini REbirth

13,886 views • 1 year ago

🤖 Another zero-shot reward model is now in LeRobot: ROBOMETER. A general-purpose, zero-shot video-language reward model from University of South Carolina, UT Dallas, Massachusetts Institute of Technology (MIT), University of Washington, Ai2, and NVIDIA that predicts frame-level task progress. Trained on 1M+ trajectories from 21 robot embodiments, generalizes zero-shot to unseen tasks, scenes, and robots. 2.4–4.5x better downstream success rates across online RL, offline RL, data filtering, failure detection, and data retrieval for IL. Project: Paper:
0:14

🤖 Another zero-shot reward model is now in LeRobot: ROBOMETER. A general-purpose, zero-shot video-language reward model from University of South Carolina, UT Dallas, Massachusetts Institute of Technology (MIT), University of Washington, Ai2, and NVIDIA that predicts frame-level task progress. Trained on 1M+ trajectories from 21 robot embodiments, generalizes zero-shot to unseen tasks, scenes, and robots. 2.4–4.5x better downstream success rates across online RL, offline RL, data filtering, failure detection, and data retrieval for IL. Project: Paper:

LeRobot

32,625 views • 2 months ago

We built a bipedal robot for about $2,500. A real, mostly 3D-printed robot you can build, repair, simulate, train, and control. Today we’re releasing LeRobot Humanoid: an open robot-learning platform with hardware, runtime, identification tools, and training environments. Blog post: Repo:
0:16

We built a bipedal robot for about $2,500. A real, mostly 3D-printed robot you can build, repair, simulate, train, and control. Today we’re releasing LeRobot Humanoid: an open robot-learning platform with hardware, runtime, identification tools, and training environments. Blog post: Repo:

LeRobot

199,474 views • 2 months ago

Can we leverage VLMs for robot manipulation in the open world? Checkout our new work MOKA, a simple and effective visual prompting method!
0:16

Can we leverage VLMs for robot manipulation in the open world? Checkout our new work MOKA, a simple and effective visual prompting method!

Fangchen Liu

81,193 views • 2 years ago

It seems so simple to humans: "First watch the video with a red border, then pick up and place the same cube twice." And yet it's very difficult for policies as they are now. Learn more about benchmarking for robot memory ->
0:17

It seems so simple to humans: "First watch the video with a red border, then pick up and place the same cube twice." And yet it's very difficult for policies as they are now. Learn more about benchmarking for robot memory ->

Chris Paxton

15,439 views • 15 days ago