Introducing VGGT-Ω: scaling feed-forward reconstruction across static and dynamic scenes, and studying whether the learned geometric representations transfer beyond reconstruction.

[2/9] Details are available here: 🌐 Project: 📄 arXiv: 💻 Code: 🤗 Demo:

[3/9] We find that the quality of reconstruction models scales predictably with model and data size. By increasing model capacity from 0.2B to 10B parameters and expanding training data from a few thousand to 2M sequences, we observe consistent power-law-like improvements in reconstruction accuracy.

[4/9] First, we noticed that geometric tasks still lack a general-purpose scene representation. This led us to revisit registers: prior ViT work used them as auxiliary tokens and often discarded them at inference time. In VGGT-Ω, we find that registers are natural carriers of global scene information. Register attention replaces part of full global attention with attention restricted to registers, making them the bottleneck for exchanging information across frames. With register attention, they aggregate and redistribute multi-frame information, yielding features that (a) benefit VLA models, (b) align well with text embeddings, and (c) improve efficiency.

[5/9] To make scaling feasible, VGGT-Ω introduces architectural and training improvements. Besides register attention, we replace memory-heavy high-resolution convs in dense predictors with a single MLP followed by pixel shuffle. We also keep multi-task losses, but retain only one dense head for depth and one sparse head for camera prediction. Together, these changes save 70% GPU memory during training and modestly improve inference speed, with no performance degradation.

[6/9] Scaling reconstruction also requires scaling data: amount, diversity, and quality are all critical. In particular, handling dynamic content unlocks orders of magnitude more Internet-style videos for training. We build a high-quality annotation pipeline combining VLM-based pre-filtering, VGGT, COLMAP, modern image matching, and supervised geometric post-filtering. From about 40M Internet-style videos, it retains 0.8M accurate sequences, roughly one-third dynamic. Combined with existing real and synthetic datasets, this yields 4M diverse scenes/sequences, more than 15x VGGT.

[7/9] Self-supervised training is another promising path for scaling. Inspired by DINO-style momentum teacher-student methods, we initialize both networks from a supervised VGGT-Ω checkpoint. They process the same video under different augmentations and frame permutations; after aligning the frame order, the student matches the teacher's predictions and feature distribution, while the teacher is updated by EMA. This lets us train on 18M unlabeled videos.

[8/9] VGGT-Ω achieves state-of-the-art results on three static and three dynamic benchmarks by a wide margin. It substantially outperforms post-optimization methods such as MegaSaM and recent feed-forward methods such as Depth Anything 3. For example, on Sintel, VGGT-Ω improves camera AUC@3° from 22.5 to 40.0 and depth δ1.25 from 74.1 to 93.5.

[9/9] We also include empirical observations from developing VGGT-Ω: where information is stored via model souping, whether motion awareness emerges, synthetic/real data mixtures, practical fine-tuning recommendations, and so on. In the Discussion, we further explain why we prioritize simplicity, the benefits of feed-forward reconstruction, and the role of 3D/4D in the era of large models. Hope these could be useful for the community! Enjoy 😄

Building a project like this takes patience, trust, resilience, and a shared belief that the work is worth doing. Thanks my great co-authors: @MinghaoChen23 Shangzhan Zhang, @n_karaev, Johannes Schönberger, @monsieurlabatut, @p_bojanowski, @davnov134, Andrea Vedaldi, Christian Rupprecht!

I love feed-forward 3D models!

embracing deep learning!

Amazing!! Congratulations 🎉

🔥

Nice job @jianyuan_wang ! You think there is any promise in bootstrapping the EMA SSL objective?

Thanks David! I think ssl is an important topic, but I have not found a promising implementation / training protocol. That's why I shared the discussion of ssl in the paper, that we hope it could insipire some people who may have interest. The underlying concern here, which is also the reason I am not entirely satisfied with the SSL protocol presented in this paper, is that it cannot be trained from scratch. Perhaps it is just an unrealistic pursuit of elegance on my part, but I have always hoped that such an SSL approach could be trained entirely from the ground up.

@davnords Any reason why SSL should only be done from scratch? 👀 Did u try the Rayzer objective with your pretrained model? 😉

I was hoping the ultimate model will not be limited by the 'limited' supervised data. Personally I feel it is more about learning a representation rather than training a reconstruction model, and that's why I care about 'from scratch'. But honestly if aiming for optimizing the reconstruction accuracy, I feel 'from scratch' is merely a superfluous pursuit of aesthetics.

I agree that scaling for vision is really about representation learning 💯 I think representation learning is strongly guided by the objective - no need to train for one task be it supervised or pseudo or SSL? 😛 I guess I'm betting that using supervised data aka 3D, 2D, Flow, Segmentation, ... for bootstrapping something more scalable might be a scalable recipe for representation learning on any length videos & unconstrained image collections 👀

yeah I agree. Personally I feel, it should be closer to the paradigm of: large-scale unsupervised training plus small-scale supervised data for 'grounding'/'guiding'/'aligning', kinda like SFT in LLM. I was not that happy with using pretrained model because it is the reverse, small-scale supervised data + large-scale unsupervised data. But it is likely that both of them will work well.

@davnords Yup I agree - should push from both directions 💯💪

@jianyuan_wang I also like the rayzer/xfactor rationale of having the purest possible vision objective. Though, when I was listening to your talk @Nik__V__ I was quite drawn to your Memor framework. I liked the feeling of putting the complex scene representation in center and having lw decoders

You cooked brother 🤝

Looks incredible!

Great work!

Thanks a lot Aleks! And grateful for our last discussion that inspired a portion of our 'insights' section!

@dimadamen 🚀

how about adding a bundle adjustment at the end?

yeah its always possible

Cool work, looks like a step function in perf! Any reason for completely ditching the 2D matching head and testing if BA improves camera pose? I know this goes against the philosophy of feed forward 3D recon, but why not have both?

I guess it should be possible. My idea was, once a strong enough backbone is trained, all such ideas can be explored in a cheap and efficient manner. So I hoped to first build a simple paradigm for scaling up.

makes sense, thanks!

Wow