(1/10) 🔥Thrilled to introduce OneDiffusion—our latest work in unified diffusion modeling! 🚀 This model bridges the gap between image synthesis and understanding, excelling in a wide range of tasks: T2I, conditional generation, image understanding, identity preservation, multiview generation, and even camera pose estimation. Learn more at: Project: arXiv: Code... (on the way):show more

(2/10) In OneDiffusion, we frame image generation with multimodal conditions as a sequential modeling problem. By treating all tasks as frame sequences with different noise scales during training, our approach is both simple and highly effective. This design enables any frame to serve as a conditioning image during inference.

(3/10) To train our model we create the One-Gen dataset, which integrates high-quality data across a variety of sources. This includes standard T2I data along with synthetic outputs from state-of-the-art models to support a range of tasks such as depth estimation, segmentation, pose estimation, etc. Our dataset also incorporates data for ID customization and multiview generation, providing diverse conditioning setups. With One-Gen, our model handles text-to-image generation across different art styles, from artistic to photorealistic.

(4/10) For ID customization, unlike prior methods that rely on face embeddings and often fail to generalize, our model demonstrates superior generalization. It effectively adjusts facial expressions and gaze directions (first row), changes viewpoints (second row), and even customizes non-human IDs (third row).

(5/10) For multi-view generation, our model supports a variable number of conditional inputs and, due to its flexible denoising framework, can incorporate additional conditional views even when their camera poses are unknown. Starting from a single front-view image, our model generates consistent views across different azimuths and elevations.

(6/10) We conducted image-to-image translation experiments across various source domains, such as HED, depth maps, human poses, semantic maps, and bounding boxes. The results show that OneDiffusion effectively aligns with the conditioning image across diverse input conditions, leveraging pure attention mechanisms and additional context from captions during generation.

(7/10) We also tested our model on monocular depth estimation, where it performs comparable with (task specific) diffusion-based depth estimators like Marigold. It shows exceptional robustness in handling open-world images, including paintings, hazy weather, and unconventional textures.

(8/10) In Camera Pose Estimation, we use six rendered images per synthetic object and estimate camera poses by denoising the corresponding ray embeddings. A qualitative comparison on the GSO dataset shows that OneDiffusion outperforms RayDiffusion, delivering more accurate predictions.

(9/10) Our mode's capability can be further grown. We further finetuned the model on the Subject-200K dataset (along with all other tasks) for additional 40k steps. The model is now capable of subject-driven generation 😀

(10/10) This incredible work was led by @lehduonggg, in collaboration with @tunkhongmatday, Sangho Lee, Christopher Clark, @anikembhavi, @StephanMandt, and @RanjayKrishna

Nice work Jiasen!

Thanks, Credit goes to leading authors (Duong, and Tuan).