跳过导航
If you have a policy that uses diffusion/flow (e.g.
0:00 / 0:39

If you have a policy that uses diffusion/flow (e.g.

Sergey Levine

@svlevine14万 位关注者

在 X 上查看
1,164 个赞下载原帖

153,323 次观看

If you have a policy that uses diffusion/flow (e.g. diffusion VLA), you can run RL where the actor chooses the noise, which is then denoised by the policy to produce an action. This method, which we call diffusion steering (DSRL), leads to a remarkably efficient RL method! 🧵👇

9 条评论

评论来自 X (Twitter) 上的原帖

  • Sergey Levine 的头像

    @svlevine

    DSRL trains an actor and Q-function, treating the diffusion noise as the action space. Because samples from the noise prior map to reasonable actions for the policy, DSRL essentially explores "inside" the set of reasonable pre-trained behaviors, making it extremely efficient.

    18 个赞

  • Sergey Levine 的头像

    @svlevine

    DSRL learns essentially in real time, with good results in as little as 50 trials (it's so efficient that a person can literally sit in front of the robot and push a button to assign sparse rewards).

    27 个赞

  • Sergey Levine 的头像

    @svlevine

    This was a really fun collaboration led by @ajwagenmaker Project website with paper: To find out more, check out his thread here:

    21 个赞

  • ahad 的头像

    @ahadj0

    would a supervised learning version of this work? where the noise distribution is a parmeter that is also optimized along with policy weights

  • ahad 的头像

    @ahadj0

    how long would it take to get that first sparse reward with this method?

  • Himanshu Kumar 的头像

    @codewithimanshu

    Controlling the noise instead of the action itself is a surprisingly effective approach.

  • Andres Franco 的头像

    @theinsiderzclub

    This is pretty amazing, and the visualization made everything so easy to understand😅

  • Ran Cheng 的头像

    @RanCheng10

    Will making the initial noise distribution a learnable parameter reduce randomness and thus make the model more prone to overfitting?

  • Joanne Mercado 的头像

    @square26circle

    😅🥹