Introducing Modality Forcing, a recipe for post-training T2I models for SOTA RGB-Depth generation! Text-to-image (T2I) models learn rich representations of the spatial world. How do we build on this prior for high-quality depth generation? 🧵 [1/6]

Modality Forcing trains one DiT backbone with a separate noise level per modality. A modality’s noise level sets whether it’s generated or conditioned on. A single model does image→ depth, depth→ image and joint RGB-D generation. [2/6] [2/6]

Does the T2I prior actually help with depth? We train T2I models from scratch across a range of scales and find a clean trend: bigger T2I models trained on more data give better depth. The spatial prior scales. [3/6]

Applied to FLUX.2-klein-9B, Modality Forcing matches the best dedicated depth models (MoGe-2, PPD) — creating crisp, robust depth maps with few floating artifacts. It even generalizes well beyond the training distribution, e.g., animals and humans. [4/6]

Furthermore, traversing any denoising trajectory enables conditioning at arbitrary strength. Introducing Corgi-Anything: a demo to show many animals can be reshaped to a Corgi! Try it out yourself on the website 🐶 [5/6]

Big thanks to all collaborators @theworldlabs @CarnegieMellon - @RamananDeva @jeff_ichnowski @jcjohnss @KeunhongP Project Page: arXiv: Code: HF Demo 🤗: [6/6]

@BDuisterhof nice!

@BDuisterhof This is awesome! Please cite the datasets used in this work (i.e., include them in the bibliography) in the arXiv paper when you have a chance 😅

Thank you! And apologies, totally slipped and will update asap. Hypersim is awesome!

@BDuisterhof This is very cool! Do you know whether the backbone's t2i performance improves after Modality Forcing (post-training with depth constraints)?

Thank you! The dept-conditioned generation becomes more aligned, but I don’t believe generic image generation improves with only post-training FLUX. If you had access to training data and co-trained it might be different.

@BDuisterhof Using the spatial world model already in T2I and forcing it toward depth output is elegant. How does interpretability of the depth features compare to the original representation space?