Everyone knows action chunking is great for imitation learning. It turns out that we can extend its success to RL to better leverage prior data for improved exploration and online sample efficiency! The recipe to achieve this is incredibly simple. 🧵 1/N

1. Run Q-learning on temporally extended (‘chunked’) action space 2. Impose a behavior constraint to keep the policy close to the offline data. This allows temporally coherent actions for better exploration and unbiased n-step return backups for faster value learning. How? 2/N

n-step return backups are nice because the value signals are propagated n times faster than the standard 1-step backup, but it is biased when used naïvely. Q-chunking uses n-step return backups but avoids the bias issue by modeling the Q-value of the whole n-step sequence! 3/N

The behavior constraint brings the policy close to the offline data, allowing us to leverage temporally coherent behaviors in the prior data for better exploration. Plus, we can get all the benefits of action chunking we see in IL (e.g., handling non-Markovian behaviors)! 4/N

In principle, our recipe synergizes well with any offline/offline-to-online RL methods as they are often already equipped with some forms of behavior constraints. But in reality, more considerations must be taken into account when the offline data is complex. 5/N

Handling complex offline behavior distribution requires a more expressive policy class like diffusion/flow policies, and it is not immediately obvious how we can practically train them to simultaneously enforce a behavior constraint while maximizing Q-values. 6/N

We found a very simple and elegant solution that works. We train a behavior flow policy that clones the offline data and then use best-of-N sampling (sample N actions and take the best) to maximize Q-value. The algorithm can be summarized in just a few lines! 7/N

We also apply Q-chunking to one of our recent work, flow-Q learning (FQL) which uses one-step distillation of the behavior flow policy to tackle the value maximization challenge. This Q-chunking variant (QC-FQL) also works quite well across the board. 8/N

In practice, we do see that Q-chunking results in better temporal coherency in actions, thanks to action chunking policy and the behavior constraint we impose. This leads to better state coverage, more effective exploration, and consequently better online sample-efficiency! 9/N

I will be presenting at the ICML EXAIT workshop 07/19 (Sat) 11:45-14:15 in Rm 205-207. Come and learn more if you are around and interested! Code: Paper: Big thanks to my amazing collaborators @zhiyuan_zhou_ and @svlevine 10/N=10

Great work - I liked the intuition for the improved state coverage/exploration from Nachum et al., which might apply here: random walks with time-correlated noise -> higher variance/coverage in the end distribution vs. independent noise. Cool to see how much it helps in O2O!

Thanks John! I really like this intuition as well! One nuance is that not all time-correlated noises are good (imagine going always left-right-left-right-… in a maze). It is often nice to use some prior (eg skill). BC with an action chunking policy is a simple way of doing this