Diffusion policies have demonstrated impressive performance in robot control, yet are difficult to improve online when 0-shot performance isn’t enough. To address this challenge, we introduce DSRL: Diffusion Steering via Reinforcement Learning. (1/n)

DSRL trains a lightweight policy via RL to select input noise to the diffusion policy's denoising process, steering it to desired behaviors. We find this leads to very sample-efficient improvement, and avoids challenges typically encountered with diffusion policies + RL. (2/n)

DSRL enables efficient improvement of real-world diffusion policies for robotic control. We apply it to several different robot embodiments, and find that it is able to improve performance from <30% success to >90% in anywhere from 30-60 minutes of online training. (3/n)

We also apply DSRL to state-of-the-art flow-based generalist policies, in particular pi0 from Physical Intelligence. DSRL is able to improve pi0 in real-world deployment, on some tasks taking success from 25% to 90% in <90 minutes of online training. (4/n)

Uncut training timelapse of DSRL on WidowX pick-and-place task. (5/n)

Real-world DSRL training is highly stable. As any "action" played by the noise-space RL policy is just initial noise for the denoising process, even early in training the denoised actions look like actions from a BC-trained policy, rather than an unconverged RL policy. (6/n)

DSRL can, in principle, be instantiated with any RL algorithm. We propose, however, a SAC-based variant, Noise-Aliased DSRL, that takes advantage of a diffusion policy's tendency to map different noise to similar actions, and allows for fully off-policy training. (7/n)

Noise-Aliased DSRL trains two Q-functions, one on the original action space via TD learning, and one on the noise action space by distilling the first Q function. This allows learning from offline data and improves online sample efficiency by as much as 2x. (8/n)

In simulation, we find that DSRL substantially outperforms all existing approaches to improving diffusion policies online on benchmarks such as Robomimic. (9/n)

DSRL is also a competitive offline RL procedure: we find that first training a diffusion/flow policy on an offline dataset, then applying DSRL to steer it to high-reward behavior using the offline data performs on par with state-of-the-art offline RL methods on OGBench. (10/n)

Fun collaboration with @mitsuhiko_nm, @yunchuzh, @seohong_park, Waleed Yagoub, Anusha Nagabandi, @abhishekunique7, @svlevine! (11/n) Please see the paper and website for additional results! Website: Paper:

looks awesome and the presentation video is extremely clear! I was looking at the real world training and felt that there isn’t much randomization of the start state eg pick/place the cube into the bowl, cube is always fixed. Have you tried larger reset regions? What’s the sample efficiency in those scenarios?

We've played around with this a bit, and it still works, it just takes somewhat longer to train (the exact amount probably depends how diverse initial state is, but didn't seem to be significantly more from what we saw).

Congratulations on your work Andrew! This is actually highly related to our work (just presented at ICML) Outsourced Diffusion Sampling. In our work we learn to sample from high value tilted distributions from generative model priors by learning a policy over the latent space. We actually parameterize that policy itself as a diffusion model, trained with off-policy RL. It is suitable for offline RL too, and naturally enforces conservatism because it samples from the Bayesian posterior p_{bc}(s | a)exp(beta*Q(s, a))

Looks like the link to the code is broken

We will post code soon!

@AdilZtn ✨👀

Really cool work! In case you're interested, we found that latent noises for diffusion corresponding to high sample reward tend to live in these disjoint "balls" in noise space, and that this same noise initialization trick works really well for steering images too!

Thanks for the pointer! Will definitely take a look at this

Interesting to see that rewards converge early on most RL tasks. Does the selection of the initial noise follow the same pattern in the latent space or is there a delayed effect?

Very cool work! Are you planning on publishing your code? I had a question about how you did real-world policy rollouts, but I saw the code button on the website links to the website itself.

Yes, we do plan to publish code, hopefully in the near future!

Your post is fascinating, especially for someone like me who’s always looking to optimise. It’s just clicked for me that diffusion is right at the heart of why robots move in that ‘robotic’ way and it’s directly tied to the whole step-by-step refinement process.

Congratulations, cool work! Do you find that the space of diffusion noise is well-organized? Do different types of noise map to semantic or interpretable behavior modes?

We ran some very preliminary experiments exploring this, and our results suggested that the noise space does exhibit some semantic structure. This really requires further investigation though, and would be a cool direction for future work!

Thanks for the insights, congrats again!