Gezinmeyi atla
Diffusion policies have demonstrated impressive performance in robot control, yet are difficult to…
0:00 / 2:53

Diffusion policies have demonstrated impressive performance in robot control, yet are difficult to…

Andrew Wagenmaker @ CoRL

@ajwagenmaker1,4 B takipçi

X profili
307 beğeniİndirOrijinal gönderi

65.303 görüntüleme

Diffusion policies have demonstrated impressive performance in robot control, yet are difficult to improve online when 0-shot performance isn’t enough. To address this challenge, we introduce DSRL: Diffusion Steering via Reinforcement Learning. (1/n)

25 Yorum

Yorumlar X (Twitter) üzerindeki orijinal gönderiden alınmıştır

  • Andrew Wagenmaker profil fotoğrafı

    @ajwagenmaker

    DSRL trains a lightweight policy via RL to select input noise to the diffusion policy's denoising process, steering it to desired behaviors. We find this leads to very sample-efficient improvement, and avoids challenges typically encountered with diffusion policies + RL. (2/n)

    26 beğeni

  • Andrew Wagenmaker profil fotoğrafı

    @ajwagenmaker

    DSRL enables efficient improvement of real-world diffusion policies for robotic control. We apply it to several different robot embodiments, and find that it is able to improve performance from <30% success to >90% in anywhere from 30-60 minutes of online training. (3/n)

    14 beğeni

  • Andrew Wagenmaker profil fotoğrafı

    @ajwagenmaker

    We also apply DSRL to state-of-the-art flow-based generalist policies, in particular pi0 from Physical Intelligence. DSRL is able to improve pi0 in real-world deployment, on some tasks taking success from 25% to 90% in <90 minutes of online training. (4/n)

    13 beğeni

  • Andrew Wagenmaker profil fotoğrafı

    @ajwagenmaker

    Uncut training timelapse of DSRL on WidowX pick-and-place task. (5/n)

    13 beğeni

  • Andrew Wagenmaker profil fotoğrafı

    @ajwagenmaker

    Real-world DSRL training is highly stable. As any "action" played by the noise-space RL policy is just initial noise for the denoising process, even early in training the denoised actions look like actions from a BC-trained policy, rather than an unconverged RL policy. (6/n)

    16 beğeni

  • Andrew Wagenmaker profil fotoğrafı

    @ajwagenmaker

    DSRL can, in principle, be instantiated with any RL algorithm. We propose, however, a SAC-based variant, Noise-Aliased DSRL, that takes advantage of a diffusion policy's tendency to map different noise to similar actions, and allows for fully off-policy training. (7/n)

    10 beğeni

  • Andrew Wagenmaker profil fotoğrafı

    @ajwagenmaker

    Noise-Aliased DSRL trains two Q-functions, one on the original action space via TD learning, and one on the noise action space by distilling the first Q function. This allows learning from offline data and improves online sample efficiency by as much as 2x. (8/n)

    7 beğeni

  • Andrew Wagenmaker profil fotoğrafı

    @ajwagenmaker

    In simulation, we find that DSRL substantially outperforms all existing approaches to improving diffusion policies online on benchmarks such as Robomimic. (9/n)

    10 beğeni

  • Andrew Wagenmaker profil fotoğrafı

    @ajwagenmaker

    DSRL is also a competitive offline RL procedure: we find that first training a diffusion/flow policy on an offline dataset, then applying DSRL to steer it to high-reward behavior using the offline data performs on par with state-of-the-art offline RL methods on OGBench. (10/n)

    11 beğeni

  • Andrew Wagenmaker profil fotoğrafı

    @ajwagenmaker

    Fun collaboration with @mitsuhiko_nm, @yunchuzh, @seohong_park, Waleed Yagoub, Anusha Nagabandi, @abhishekunique7, @svlevine! (11/n) Please see the paper and website for additional results! Website: Paper:

    12 beğeni

  • Stone Tao profil fotoğrafı

    @Stone_Tao

    looks awesome and the presentation video is extremely clear! I was looking at the real world training and felt that there isn’t much randomization of the start state eg pick/place the cube into the bowl, cube is always fixed. Have you tried larger reset regions? What’s the sample efficiency in those scenarios?

    2 beğeni

  • Andrew Wagenmaker profil fotoğrafı

    @ajwagenmaker

    We've played around with this a bit, and it still works, it just takes somewhat longer to train (the exact amount probably depends how diverse initial state is, but didn't seem to be significantly more from what we saw).

    1 beğeni

  • Siddarth profil fotoğrafı

    @siddarthv66

    Congratulations on your work Andrew! This is actually highly related to our work (just presented at ICML) Outsourced Diffusion Sampling. In our work we learn to sample from high value tilted distributions from generative model priors by learning a policy over the latent space. We actually parameterize that policy itself as a diffusion model, trained with off-policy RL. It is suitable for offline RL too, and naturally enforces conservatism because it samples from the Bayesian posterior p_{bc}(s | a)exp(beta*Q(s, a))

    2 beğeni

  • Vaishak Kumar profil fotoğrafı

    @real_vaishak

    Looks like the link to the code is broken

    2 beğeni

  • Andrew Wagenmaker profil fotoğrafı

    @ajwagenmaker

    We will post code soon!

    1 beğeni

  • Francesco Capuano profil fotoğrafı

    @_fracapuano

    @AdilZtn ✨👀

    2 beğeni

  • Aayush Karan profil fotoğrafı

    @aakaran31

    Really cool work! In case you're interested, we found that latent noises for diffusion corresponding to high sample reward tend to live in these disjoint "balls" in noise space, and that this same noise initialization trick works really well for steering images too!

  • Andrew Wagenmaker profil fotoğrafı

    @ajwagenmaker

    Thanks for the pointer! Will definitely take a look at this

    1 beğeni

  • Muyu He profil fotoğrafı

    @HeMuyu0327

    Interesting to see that rewards converge early on most RL tasks. Does the selection of the initial noise follow the same pattern in the latent space or is there a delayed effect?

  • Brian Smith profil fotoğrafı

    @sirwart

    Very cool work! Are you planning on publishing your code? I had a question about how you did real-world policy rollouts, but I saw the code button on the website links to the website itself.