Introducing St4RTrack!🖖 Simultaneous 4D Reconstruction and Tracking in the world coordinate feed-forwardly, just by changing the meaning of two pointmaps!

From static to dynamic, MonST3R reconstructs pointmaps at their own times. St4RTrack instead estimates both at the same moment—predicting how points in frame1 move to frame2 and reconstructing geometry of frame2. Same architecture, now for simultaneous tracking + reconstruction.

Such representation can be learned supervisedly from small-scale, synthetic 4D datasets. But to better generalize to real scenes, St4RTrack can also adapt to new videos *without any 4D labels*, using only 2D reprojection cues like trajectories & monocular depth.

To evaluate the method, we propose a benchmark, *WroldTrack*, for world coordinate tracking and dynamic 3d reconstruction

More interestingly is our joint reconstruction and tracking results, even the fully feed-forward mode gives promising results ⬇️

We present a "more interactive" results on our webpage. Come and check it out!

Great collaboration with @HavenFeng (co-lead), @QianqianWang5, @yufei_ye, @pengcheng_147, @Michael_J_Black, @trevordarrell, @akanazawa 🥰

With CoinStats, you can track all your assets in one place and get advanced analytics. Connect your portfolio to start tracking your Profit & Loss today:

cool work!

Congrats Junyi!

🔥