正在加载视频...
视频加载失败
Ever wonder why well-trained Vision Transformers still exhibit noises? We introduce Denoising Vision Transformers (DVT), led by amazing Jiawei Yang Katie Luo Jeff Li, and with long-term collaborators Yonglong Tian Kilian Weinberger. Website: Code: Paper:
8 条评论

The problem we study is that Vision Transformers exhibit grid-like and position-related noises, which hamper their performance on downstream tasks. This problem was first observed by Jiawei back to last summer when we worked on EmerNeRF ( and concurrently studied in Register (Darcet et al.). We propose a novel and universally applicable solution that capitalizes on feature invariances and neural fields to remove these noises. We use an instant-NGP to represent 2D features and enforce their invariance to geometric augmentation. Therefore, position-related noises (not invariant to geometric data augmentation) are removed and not cached in the neural fields. Then, to enable real-time tasks, we use the denoised features as pseudo labels to learn a generalizable denoiser (a one-layer Transformer block). Our method doesn't require a full re-training of existing ViTs (which can be very expensive due to data/model scale) Finally, we show that we can obtain better feature visualizations (PCA & Kmeans) and improve dense prediction tasks such as semantic segmentation and depth prediction. See more results in our paper and website.

Great work! I saw those artifacts when working on my MAGIC-VFM paper and ended up using DinoV1 rather than DinoV2. DinoV1 doesn't experience the same artifacts, but not sure why that's the case. I'll definitely try your method in my future work when using DinoV2.

@JiaweiYang118 @katielulula @jiefengli_jeff @YonglongT @KilianQW Excellent! Let us know how it works for your case. We're also happy to help :)

@JiaweiYang118 @katielulula @jiefengli_jeff @YonglongT @KilianQW Cool! Congrats @yuewang314

@JiaweiYang118 @katielulula @jiefengli_jeff @YonglongT @KilianQW Thanks Sungjin!

@JiaweiYang118 @katielulula @jiefengli_jeff @YonglongT @KilianQW This advancement could be pivotal in improving diagnostic precision. Impressive work! Eager to see its impact in medical imaging modalities, such as MRI and CT scan for clinical validation.

@JiaweiYang118 @katielulula @jiefengli_jeff @YonglongT @KilianQW Do you think this noise occurs with hierarchical attention models like swin transformers?

@JiaweiYang118 @katielulula @jiefengli_jeff @YonglongT @KilianQW What is this can you tell me please ?
