Are you training your VLAs or WAMs correctly? Your action expert may be learning vision–action shortcuts that undermine generalization beyond the training distribution. Introducing Latent Interface Training (LIT 🔥): Learn to act first, then learn how to use vision. 🧵👇

1/🧵 In training data, backgrounds, camera angles, or lighting can correlate with demonstrated actions. An action expert may learn to rely on these cues, then fail when those correlations change in a new scene. LIT constrains visual conditioning while preserving goal-relevant spatial information.

2/🧵 Standard VLAs/WAMs train their action experts directly on visual representations, alongside language and robot state. This can encourage reliance on visual shortcuts. LIT restructures training into two stages: 1️⃣ Learn to act first. Train the action expert without images, conditioned on language, robot state, and each demonstrated action chunk’s terminal end-effector pose. 2️⃣ Learn how to use vision. Initialize the action expert from Stage 1, then jointly train the policy with visual conditioning routed exclusively through a latent interface. Supervise this interface to reconstruct the same terminal pose, encouraging it to retain goal-relevant spatial information.

3/🧵 Why does LIT make intuitive sense? First, give the action expert a spatial goal and teach it how to move there without images. It can learn goal-directed motion without picking up visual shortcuts. Then, introduce vision through a latent interface trained to recover that same goal. This encourages visual information to support the spatial understanding the action expert has already learned. The two stages work together: establish goal-directed action generation, then guide visual conditioning toward what matters for the action.

4/🧵 Across four architectures, LIT improves average LIBERO-Plus success: π0.5: 68.97 → 79.67%📈 MolmoAct2: 63.62 → 71.92%📈 FAST-WAM: 51.44 → 60.63%📈 ImageWAM: 83.02 → 86.89%📈 Average in-distribution LIBERO success also improves for all four.

5/🧵 On a real-world experiment, MolmoAct2 learnt three tasks from 300 demonstrations. Success across tasks, MolmoAct2 → MolmoAct2+LIT: New lighting: 53.3 → 70.0% Top camera only: 30.0 → 46.7% Distractors: 50.0 → 63.3% In-distribution: 74.7 → 88.0%.

6/🧵 Does the policy respond to the right changes? 🔍 Adding a distractor or blurring the image leaves the task unchanged. The baseline’s predicted trajectories shift substantially, while LIT’s remain close to their original paths. Change the instructed goal instead, and LIT redirects toward the new target, while the baseline continues toward the original one. The insight: a robust policy should resist irrelevant visual changes while responding to changes in intent. These examples support LIT’s reduced reliance on vision–action shortcuts.

7/🧵 Our takeaway: how vision reaches the action expert matters for generalization. LIT links action learning and visual conditioning through a shared spatial goal and improves robustness across both VLAs and WAMs. Project: Paper: Code:

Great work done by students @jianmanlinnn9l , @Shailes_h_ and Zhongyi Luo! A first for me too with @NUSComputing !

Impressive

It’s so weird when people frame their research as some gossip magazine. Yes anyone who has trained and deployed a VLA knows about taking shortcuts from the training set. It’s well known.

The shortcut problem is such a good trap to surface early. I treat awkward or unresolved states as product notes, then walk them explicitly instead of letting the demo pass hide them.

On our hardest held-out VLM packs, frontier models still land under 20%, and 0% on exact object counts. Happy to share the breakdown if that distinction is useful.

Did you guys try just augmenting the background out?

Not yet, but could try. Many engineering perturbation could be done, but we want the model to break away from these habits