🚨 New (Offline) RL Method 🦾 🚨 Introducing Q-Transformer - new RL approach that works at scale with large models and many tasks. This is the best method we found so far that works with demos and autonomous (also negative) data at large scale. 🧵

Scaling Q-learning to large, diverse datasets and large models has been proven difficult. Most commonly used RL algorithms at that scale tend to be on-policy algorithms (e.g. PPO in RLHF) In Q-Transformer, we demonstrate a few important design decisions that made it work for us.

• we represent each discretized action dimension as a separate token in a Transformer - you can think of it as every action dim being a separate MDP step • we add a regularization term for unseen action bins • we set Bellman update to max(MC-return, next Q) to speed it up

To compare it to other methods, we reduce the original RT-1 dataset to <100 episodes per task and add negative data (otherwise, there is no room for improvement). Q-Transformer is ~2x better than the next baseline! See lots of ablations + more sim comparisons in the paper.

Since this is a "proper" RL method, we can also use its value function as an affordance metric (for example for which works better than other alternatives. Q-learning FTW!

This was a long-term effort led by @YevgenChebotar and @QuanVng with an amazing team of collaborators. Tagging a few on twitter: @AlexIrpan @xf1280 @aviral_kumar2 @TianheYu @julianibarz @AlexHerzog001 @keerthanpg @chelseabfinn @svlevine

Google is using large transformers to solve general purpose robotics, this is awesome

Releasing code soon?