Загрузка видео...

Не удалось загрузить видео

На главную

Christian Rupprecht explains their interpretability research in 3D computer vision, testing if (and where in the model) multi-view transformers like VGGT, DepthAnything 3, and DUSt3R use point/patch correspondences to make sense of 3D scene geometry.

74,801 просмотров • 7 месяцев назад •via X (Twitter)

Комментарии: 9

Фото профиля Chris Offner
Chris Offner7 месяцев назад

Full talk:

Фото профиля Chris Offner
Chris Offner7 месяцев назад

Paper:

Фото профиля Chris Offner
Chris Offner7 месяцев назад

Q: Are these truly geometric correspondences or just the semantic correspondences we get from the DINO encoder, or even from diffusion models like Stable Diffusion? A: Early layers are good at semantic correspondences but bad at geometric ones, and vice versa for later layers.

Фото профиля Chris Offner
Chris Offner7 месяцев назад

Related:

Фото профиля Mazeyar
Mazeyar7 месяцев назад

Can we extract these correspondences from VGGT then run classic bundle adjustment?

Фото профиля Chris Offner
Chris Offner7 месяцев назад

Possibly, but why would you want to do that? I'd expect them to be much less precise than, e.g., RoMA 2, not least because they're only patch-level (not pixel-level) correspondences.

Фото профиля Mazeyar
Mazeyar7 месяцев назад

Im thinking they're less precise too, but maybe more robust

Фото профиля Kemajou Charlesquin
Kemajou Charlesquin7 месяцев назад

Interesting!

Фото профиля Prateeth Rao
Prateeth Rao7 месяцев назад

That's a very good analysis to verify if models are truly able to learn epipolar geometry for monocular VSLAM. Every matching/depth estimation model that I have tested, produce few epipolar correspondences until explicitly trained to follow geometry.

Похожие видео

DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior paper page: present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation.

AK

161,530 просмотров • 3 лет назад