Introducing Masked Trajectory Modeling (MTM), a new general-purpose framework for sequential decision making. A single transformer trained with MTM can exhibit multiple capabilities by simply choosing different masking patterns at inference time. Accepted at ICML 2023. 🧵👇

MTM trains by reconstructing a masked input sequence using random autoregressive masking (think BERT with a prediction prior). We prompt the model with different masking patterns to achieve specific capabilities such as behavior cloning and inverse dynamics.

We also evaluate MTM in offline RL settings, such as D4RL. At test time we prompt the model with a desired return. We find that MTM is comparable to prior work while also also having the capability to be used for other tasks.

MTM is able to train on datasets with missing data modes. Ex: we might have a large dataset of state trajectories, but only a small amount with action labels. MTM is able to train on both without any change to the algorithm, simply by treating missing data modes as a forced mask.

We test if MTM is able to leverage the trajectories without actions. Here we compare various methods that only train on the labeled subset and our Heteromodal MTM which also trains on trajectories without action labels. Heteromodal MTM is able to improve task performance.

MTM can also be used to learn representations. After pretraining on generic trajectories, we use TD3 on top of the learned MTM representations. We find that these representations help with faster learning and in some cases, improve on asymptotic performance.

If you are interested in this line of work, also checkout MaskDP ( from my lab mate @fangchenliu. Our work extends on some of the ideas presented here and explores a different set of capabilities and features of this paradigm.

If you want to get a quick start with MTM, checkout this notebook which illustrates how to train and use MTM on a simple sinusoidal dataset.

Thanks for the help and assistance of amazing collaborators! @arjunmajum, @kevinleestone, @yixin_lin_, @IMordatch, @pabbeel, @aravindr93 If you are interested in more details checkout the links below! Website: Code:

Paper now out on arxiv!

Nice. It finally worked. Congrats. 🎉

😂 thanks kourosh!!