So you’ve trained your favorite diffusion/flow based policy, but it’s just not good enough 0-shot. Worry not, in our new work DSRL - we show how to *steer* pre-trained diffusion policies with off-policy RL, improving behavior efficiently enough for direct training in the real world! DSRL retains nice exploration... from the base policy, but allows for quick improvement beyond this base policy with RL. The method is frustratingly simple, and super easy to throw on top of your favorite pretrained policy (VLA/diffusion policy, etc). Let’s think about how it works, 🧵 (1/10)show more

What’s the key idea? Don’t touch the base policy, but instead train a lightweight policy via RL to select the initial noise in a diffusion (or flow) policy's denoising process. This initial noise modulates the behaviors, guiding it towards desired behaviors. Think of it as - absorb the base policy into the environment, and have the RL process just modulate it’s initial noise. The nice part is that since we’re just choosing initial noise, we can use whatever RL algorithm we want, including efficient off-policy RL algos like SAC! This leads to very sample-efficient improvement, and avoids challenges typically encountered with diffusion policies + RL. (2/10)

Why does this matter? - 1) retains base policy exploration, 2) enables super efficient real-world RL, efficient enough to put on a robot! Let’s look at some examples of efficient improvement of real-world diffusion policies for robotic control. We applied DSRL on several real-world robot experiments, and find that it is able to improve performance from <30% success to >90% in anywhere from 30-60 minutes of online training. (3/10)

Ok but can we steer policies that we didn’t actually pre-train ourselves? To test this, we applied DSRL to pi0, a state-of-the-art flow-based generalist policy from @physical_int. DSRL is able to improve pi0 in real-world deployment, on some tasks taking success from 25% to 90% in <90 minutes of online training. No weight access needed, just API access to sampling from the model and the ability to set initial noise. For all of you doing pre-training of VLA models, expose the choice of initial noise vector in the API and your models will be a lot more steerable! (4/10)

Can we understand what’s going on a little more visually? Watching a timelapse of DSRL on a WidowX robot tasked with picking and placing an object shows the coherence of exploration and the efficiency of learning! (5/10)

Why does this actually help? Since any "action" played by the noise-space RL policy is just initial noise for the denoising process, even early in training the denoised actions look like actions from a BC-trained policy, rather than an unconverged RL policy. This makes exploration far more coherent than if we had started training from scratch with an algorithm like RLPD. We can visualize this exploration behavior through a video of the robots actual behavior, and 3D plots of end-effector position (6/10)

What’s the catch? We’ve changed the action space of the RL policy from the original actions a to initial noise z. The challenge is that offline data or interventions are collected in the (s, a, s’, r) space, not in the (s, z, s’, r) space. To allow this data to still be useful, we use a simple trick, which we refer to as noise aliasing - trains two Q-functions, one on the noise space via DSRL (Q(s, z)) and another on the original action space (Q(s, a)). Then force these two Q functions to be consistent for action a = pi(s, z). This allows for learning from offline data or interventions, and improves online sample efficiency by as much as 2x. This algorithm takes advantage of a diffusion policy's tendency to map different noise to similar actions, and allows for off-policy training. (7/10)

What are the numbers? In simulation, we find that DSRL substantially outperforms all existing approaches to finetuning diffusion policies online on benchmarks such as Robomimic and Mujoco tasks. (8/10)

Thanks to noise-aliasing - DSRL is also a competitive offline RL procedure: first training a diffusion/flow policy on an entire offline dataset, then applying DSRL to steer it to high-reward behavior using the offline data performs on par with state-of-the-art offline RL methods on OGBench. (9/10)

Ok what are the takeaways? 1. Wrap a base diffusion policy into the environment and just control initial noise to steer it. Yields a frustratingly simple, scalable and efficient algorithm that retains good exploration while enabling sample efficient improvement. 2. Train two Q functions via noise-aliasing to be able to use offline data easily and improve sample efficiency. 3. Use this to improve any base policy through API access only, when initial noise for diffusion can be chosen! Kudos to @ajwagenmaker for leading this work along with @mitsuhiko_nm, @yunchuzh, along with collaborators @seohong_park, Waleed Yagoub, Anusha Nagabandi. And as always, a pleasure to work with @svlevine! :) Please see the paper and website for additional results! Website: Paper: (10/10)

Stellar work! We’ve been exploring similar off-policy “steering” to fine-tune plant-care agents in live grow rooms—bridging sim-trained diffusion models to real-world climate swings. Love seeing DSRL push sim2real efficiency even further. Excited to dive into the paper!