跳过导航
4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos
0:00 / 5:19

4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos

MrNeRF

@janusch_patas1.6万 位关注者

在 X 上查看
414 个赞下载原帖

34,782 次观看

4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos Abstract: We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We introduced a novel density control strategy in training, which allows our 4DGT to handle longer space-time input while maintaining efficient rendering at runtime. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can significantly outperform prior Gaussian-based networks in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos.

11 条评论

评论来自 X (Twitter) 上的原帖

  • MrNeRF 的头像

    @janusch_patas

    Paper: not yet Project: "4DGT takes a series of monocular frames with poses as input. During training, we subsample the temporal frames at different granularity and use all images for supervision. In stage one, we train 4DGT to predict pixel-aligned Gaussians at coarse resolution. In stage two, we prune a majority of non-activated Gaussians based on the histograms of per-patch activation channels and densify the Gaussian prediction by increasing the input token samples in both space and time. At inference time, we run the 4DGT network trained after stage two, which supports dense video frames input at high resolution."

    24 个赞

  • MrNeRF 的头像

    @janusch_patas

    Paper:

    6 个赞

  • Pablo Vela 的头像

    @pablovelagomez1

    Wow this looks really sick

    4 个赞

  • MrNeRF 的头像

    @janusch_patas

    Yeah, and the clip is super long.

    1 个赞

  • Micky Abir 的头像

    @MickyAbir

    people don’t realize how huge this is

    1 个赞

  • MrNeRF 的头像

    @janusch_patas

    long long videos, yeah!

  • TessyVFXR 的头像

    @TessyXR

    The fact that I can't get my head off this for the past few days... For me, it is that much needed tool that unlocks a lot.

    1 个赞

  • James | 🤖 的头像

    @_troglobyte

    Awesome. Looking forward to trying this out!

    1 个赞

  • MrNeRF 的头像

    @janusch_patas

    I'm crafting an email newsletter that turns my daily updates into a captivating weekly digest, complete with exclusive content. Although it's not live yet, you can sign up now! If you're curious, visit my website and join the subscriber list today!

    1 个赞

  • Mars (parody) 的头像

    @marknadal

    the future is beaming into reality gaaah this is so exciting

    1 个赞

  • MrNeRF 的头像

    @janusch_patas

    Pretty good for monocular footage. The videos are also very long!

    2 个赞