Stable Diffusion generates beautiful images, but can it be used for open-world recognition? Try Demo! Our #CVPR2023 paper shows that the pre-trained diffusion model indeed is a good image parser, allows for open-vocabulary segmentation and detection.

This comes from an observation of using pre-trained diffusion features for clustering. If we use the UNet from Stable Diffusion to extract the image feature, and apply KMeans, we will obtain the following clusters. Already looks like a good segmentation? 2/n

With this observation, our method ODISE: Open-vocabulary DIffusion-based panoptic SEgmentation, can provide high-resolution segmentation and gives each segment a detailed description. We are not just segmenting truck, but also trailer truck and pickup truck. 3/n

Our approach leverages both the diffusion model and large-scale image-text data for training. 4/n

Besides standard image datasets, we also apply our model to egocentric videos (Ego4D), without any fine-tuning. To me, this is a very exciting result since it enables a lot of robotics applications as well. (some flickering because there is no tracking on instances) 5/n

The work is done by Jiarui @Jerry_XU_Jiarui when he is interning with Shalini @shalinidemello in @nvidia , with collaborators: @SifeiL @ArashVahdat @wonmin_byeon Arxiv: Website: 6/n

Jiarui presenting GroupVIT last year.

Some time ago on a group hike

The good old days.

Some impressive work again @Jerry_XU_Jiarui ! Congrats. Maybe we should make it available in Diffusers? 😉👌

@memdotai mem it