(1/N) Will this be the BERT/GPT moment for 3D vision？ Finally, unsupervised pre-training for 3D works. Led by Qitao Zhao , we present E-RayZer — a fully self-supervised 3D reconstruction model that: 🔥Matches or surpasses supervised methods like VGGT 👀Learns transferable 3D representations, outperforming CroCo, VideoMAE, and DINO 📈Scales... with more unlabeled data A new recipe for scalable 3D foundation models.show more

(2/N) So why do we insist self-supervised representation learning with 3D priors? Two reasons: 🧭3D-domain perspective (practical): Self-supervised learning directly addresses the scarcity of labeled 3D data. 🌍Vision-as-a-whole perspective (philosophical): Vision is inherently 3D as the world is 3D. David Marr: “vision is knowing what is where by looking.” Yet today’s rep-learners only learn what (semantics/recognition). They don’t learn where (3D spatial awareness). If we want vision-first intelligence, we must learn both. E-RayZer is an exploration towards spatial-aware visual pre-training. (Image is Jitendra’s talk slide)

(3/N) Why can RayZer / E-RayZer become the framework for spatial representation learning? Because it combines the strengths of two worlds: 🔁Contrastive learning: Contrastive methods align representations of the same instance under different augmentations. In (E-)RayZer, multi-view images are a natural augmentation. They are partial observations of the same scene. The multi-view encoder aligns these representations across views. 🧩Masked image modeling: MIM predicts masked pixel patches. In (E-)RayZer, the model predicts masked views. It has a flavor of contrastive + masked modeling, but in 3D.

(4/N) Then what makes E-RayZer a better representation learner than our prior work RayZer? We bring minimal inductive bias of Explicit 3D back. We found RayZer is with too less 3D inductive bias and can learn undesirable shortcut solutions, e.g., frame interpolation, to produce high-quality renderings. Grounding the latent representation with explicit geometry also helps correct representation learning. From this perspective, E-RayZer is the model with minimal 3D inductive bias; LVSM still has strong 3D inductive bias of using 3D supervision, or its statement is true only from a model architecture perspective.

(5/N) But models with explicit 3D are harder to train and scale. We find the key is controlling the training data difficulty in a fine-grained manner, by approximating visual co-visibility between input frames. For example, we can start from samples with nearly no camera motion and make camera predictor gives near identity poses. We then gradually increase the data difficulty. In my perspective, this is EM-like updating. This curriculum makes the gradient consistent within a batch, and make E-RayZer scale to heterogeneous data with diverse camera motion patterns (better than a brutal curriculum of frame interval).

(6/N) We compare self-supervised E-RayZer with supervised SoTA method VGGT (trained under the same setting for fair comparisons). E-RayZer performs on par and sometimes better, although is purely self-supervised. This implies the performance upper bound is decided by the data quality, rather than whether we use supervision. Besides, E-RayZer also improves supervised model as pre-training, and exhibit similar scaling pattern with supervised VGGT (illustrated by the grid color)

(7/N) We compare E-RayZer with other representation learners for feature transferring to our downstream tasks. E-RayZer outperforms on tasks that need 3D understanding!

(N/N) All credit to amazing @qitao_zhao ! This is my first time serving as a mentor, co-lead with Shubham Tulsiani @shubhtuls In collaboration with Hao Tan @HaoTan5 , Qianqian Wang @QianqianWang5 , Sai Bi, Kai Zhang @KaiZhang9546 , and Kalyan Sunkavalli Project: Paper: Code:

@qitao_zhao E-RayZer sets a new bar for self-supervised 3D reconstruction: no labeled data, yet matching or surpassing supervised methods, and learning transferable 3D representations. Excited to see public benchmarks, code, and real-world applications next.

@qitao_zhao Nice angle-explicit 3D bias helps avoid shortcuts. How do you quantify co-visibility and keep training scalable in practice?

@qitao_zhao We tried two strategies, 1. using actual covisibility model trained with 3D labels (ideally our oracle), 2. using DINO similarity as its approximation, because DINO is 2D spatial awareness, and our curriculum is just based on an image level score. We find them comparable.

@qitao_zhao Solid setup. Oracle covisibility gives a strong signal; DINO as a scalable proxy makes it practical.

@qitao_zhao Seems like everyone’s hyping up 3D, but idk if it’s the game changer 🤔