What makes a good representation for diffusion models? Surprisingly, more information isn't always better. In our new work, Purify Before You Align, we show that removing locally predictable information can substantially improve diffusion alignment. 🎥 A quick overview 👇 🧵 1/n

Instead of aligning diffusion models to everything a visual encoder knows, we remove locally predictable content and align to what remains. Simple change, substantial gains. 🧵 2/n

With the same DINOv2 encoder: - No alignment: 53.92 FID - Predictable component: 56.71 - Residual: 41.47 Aligning to the predictable part is worse than no alignment. The residual retains the gain. 🧵 3/n

Our solution is Local Appearance Purification (LAP). We predict the locally recoverable part of a teacher's features and subtract it before alignment. Only the alignment target changes. No changes to the diffusion architecture or inference. 🧵 4/n

LAP turns MAE into a competitive alignment teacher. On ImageNet-256 (SiT-B/2): MAE + REPA: 53.05 FID MAE + LAP-N: 42.30 FID DINOv2 + REPA: 41.75 FID Purification also improves AIMv2 and CLIP. 🧵 5/n

Then, we ask a deeper question: can we predict which encoders need purification before training diffusion? We derive a constant-readout score that measures alignment achievable without the denoiser's hidden state. Across 13 encoders at 100K steps, it predicts LAP-L gains with Spearman ρ = 0.94. 🧵 6/n

Takeaway: in diffusion alignment, what a representation emphasizes matters, NOT just how much it contains. LAP works across teachers and REPA recipes. 💼 Project: 📜 Paper: 🧑‍💻 Code: 🧵 7/n

This is joint work with amazing collaborators and advisors Yaoqiang Li, Yaqin Wu, @thoma_gu, @chrismdesa, @volokuleshov. 🧵 8/8

@chrismdesa @volokuleshov @_akhaliq @huggingface @HuggingPapers

The shortcut reward angle here is spot on. Locally predictable features act like high-frequency noise that the reward model overfits to, so pruning them forces the alignment step to actually operate on global semantics rather than surface textures. Curious if you noticed any shift in inference-time diversity once that local info was stripped?

Great question! We haven't explicitly measured diversity yet. One nuance is that LAP removes locally predictable components from the alignment targets, not local details from generated images. It'd be interesting to see how this affects diversity and coverage!

Fair enough. As long as decoupling the objective preserves generative entropy, that trade-off seems totally worth it.