Diffusion policies have demonstrated impressive performance in robot control, yet are difficult to…
@ajwagenmaker1,4 B takipçi
65.303 görüntüleme
25 Yorum
Yorumlar X (Twitter) üzerindeki orijinal gönderiden alınmıştır

@ajwagenmaker
DSRL trains a lightweight policy via RL to select input noise to the diffusion policy's denoising process, steering it to desired behaviors. We find this leads to very sample-efficient improvement, and avoids challenges typically encountered with diffusion policies + RL. (2/n)
26 beğeni

@ajwagenmaker
DSRL enables efficient improvement of real-world diffusion policies for robotic control. We apply it to several different robot embodiments, and find that it is able to improve performance from <30% success to >90% in anywhere from 30-60 minutes of online training. (3/n)
14 beğeni

@ajwagenmaker
We also apply DSRL to state-of-the-art flow-based generalist policies, in particular pi0 from Physical Intelligence. DSRL is able to improve pi0 in real-world deployment, on some tasks taking success from 25% to 90% in <90 minutes of online training. (4/n)
13 beğeni

@ajwagenmaker
Uncut training timelapse of DSRL on WidowX pick-and-place task. (5/n)
13 beğeni

@ajwagenmaker
Real-world DSRL training is highly stable. As any "action" played by the noise-space RL policy is just initial noise for the denoising process, even early in training the denoised actions look like actions from a BC-trained policy, rather than an unconverged RL policy. (6/n)
16 beğeni

@ajwagenmaker
DSRL can, in principle, be instantiated with any RL algorithm. We propose, however, a SAC-based variant, Noise-Aliased DSRL, that takes advantage of a diffusion policy's tendency to map different noise to similar actions, and allows for fully off-policy training. (7/n)
10 beğeni

@ajwagenmaker
Noise-Aliased DSRL trains two Q-functions, one on the original action space via TD learning, and one on the noise action space by distilling the first Q function. This allows learning from offline data and improves online sample efficiency by as much as 2x. (8/n)
7 beğeni

@ajwagenmaker
In simulation, we find that DSRL substantially outperforms all existing approaches to improving diffusion policies online on benchmarks such as Robomimic. (9/n)
10 beğeni

@ajwagenmaker
DSRL is also a competitive offline RL procedure: we find that first training a diffusion/flow policy on an offline dataset, then applying DSRL to steer it to high-reward behavior using the offline data performs on par with state-of-the-art offline RL methods on OGBench. (10/n)
11 beğeni

@ajwagenmaker
Fun collaboration with @mitsuhiko_nm, @yunchuzh, @seohong_park, Waleed Yagoub, Anusha Nagabandi, @abhishekunique7, @svlevine! (11/n) Please see the paper and website for additional results! Website: Paper:
12 beğeni

@Stone_Tao
looks awesome and the presentation video is extremely clear! I was looking at the real world training and felt that there isn’t much randomization of the start state eg pick/place the cube into the bowl, cube is always fixed. Have you tried larger reset regions? What’s the sample efficiency in those scenarios?
2 beğeni

@ajwagenmaker
We've played around with this a bit, and it still works, it just takes somewhat longer to train (the exact amount probably depends how diverse initial state is, but didn't seem to be significantly more from what we saw).
1 beğeni

@siddarthv66
Congratulations on your work Andrew! This is actually highly related to our work (just presented at ICML) Outsourced Diffusion Sampling. In our work we learn to sample from high value tilted distributions from generative model priors by learning a policy over the latent space. We actually parameterize that policy itself as a diffusion model, trained with off-policy RL. It is suitable for offline RL too, and naturally enforces conservatism because it samples from the Bayesian posterior p_{bc}(s | a)exp(beta*Q(s, a))
2 beğeni

@real_vaishak
Looks like the link to the code is broken
2 beğeni

@ajwagenmaker
We will post code soon!
1 beğeni

@_fracapuano
@AdilZtn ✨👀
2 beğeni

@aakaran31
Really cool work! In case you're interested, we found that latent noises for diffusion corresponding to high sample reward tend to live in these disjoint "balls" in noise space, and that this same noise initialization trick works really well for steering images too!

@ajwagenmaker
Thanks for the pointer! Will definitely take a look at this
1 beğeni

@HeMuyu0327
Interesting to see that rewards converge early on most RL tasks. Does the selection of the initial noise follow the same pattern in the latent space or is there a delayed effect?

@sirwart
Very cool work! Are you planning on publishing your code? I had a question about how you did real-world policy rollouts, but I saw the code button on the website links to the website itself.

