Learning Temporally Consistent Video Depth from Video Diffusion Priors
81,666 次观看
9 条评论
评论来自 X (Twitter) 上的原帖

@_akhaliq
developing a depth estimator from scratch, we reformulate the prediction task into a conditional generation problem. This allows us to leverage the prior knowledge embedded in existing video generation models, thereby reducing learn- ing difficulty and enhancing
2 个赞

@_akhaliq
generalizability. Concretely, we study how to tame the public Stable Video Diffusion (SVD) to predict reliable depth from input videos using a mixture of image depth and video depth datasets. We empirically confirm that a procedural training strategy - first optimizing the
2 个赞

@_akhaliq
spatial layers of SVD and then optimizing the temporal layers while keeping the spatial layers frozen - yields the best results in terms of both spatial accuracy and temporal consistency. We further examine the sliding window strategy for inference on arbitrarily long
2 个赞

@_akhaliq
videos. Our observations indicate a trade-off between efficiency and performance, with a one-frame overlap already producing favorable results. Extensive experimental results demonstrate the superiority of our approach, termed ChronoDepth, over existing
3 个赞

@_akhaliq
alternatives, particularly in terms of the temporal consistency of the estimated depth. Additionally, we highlight the benefits of more consistent video depth in two practical applications: depth-conditioned video generation and novel view synthesis.
2 个赞

@_akhaliq
paper page:
16 个赞

@_akhaliq
daily papers:
4 个赞

@NFkTys
@NandoDF Amazing progress 🚀🔥

@AbhinavGirdhar
Great insights on the new features and security measures! It's good to see continuous improvements in user experience and safety. Looking forward to more updates.