Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Introducing 🛹 RollingDepth 🛹 — a universal monocular depth estimator for arbitrarily long videos! Our paper, “Video Depth without Video Models,” delivers exactly that, setting new standards in temporal consistency. Check out more details in the thread 🧵

Anton Obukhov

2,702 subscribers

49,879 views • 1 year ago •via X (Twitter)

Science & Technology Education

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Depth Any Video with Scalable Synthetic Data AI physicists and chemists continue to make strides in depth estimation from video. Check out this new paper featuring some impressive examples. See the thread for more details (unfortunately no code yet). Abstract: Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, we introduce Depth Any Video, a model that tackles the challenge through two key innovations. First, we develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse game environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations. Second, we leverage the powerful priors of generative video diffusion models to handle real-world videos effectively, integrating advanced techniques such as rotary position encoding and flow matching to further enhance flexibility and efficiency. Unlike previous models, which are limited to fixed-length video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates 0 - even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high-resolution video depth across sequences of up to 150 frames. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency.

Depth Any Video with Scalable Synthetic Data AI physicists and chemists continue to make strides in depth estimation from video. Check out this new paper featuring some impressive examples. See the thread for more details (unfortunately no code yet). Abstract: Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, we introduce Depth Any Video, a model that tackles the challenge through two key innovations. First, we develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse game environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations. Second, we leverage the powerful priors of generative video diffusion models to handle real-world videos effectively, integrating advanced techniques such as rotary position encoding and flow matching to further enhance flexibility and efficiency. Unlike previous models, which are limited to fixed-length video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates 0 - even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high-resolution video depth across sequences of up to 150 frames. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency.

MrNeRF

27,428 views • 1 year ago

🤔Applying Depth Estimation models directly to videos can result in inconsistency between frames. 💪Well, not anymore. 🔥ChronoDepth is a new approach to video depth estimation that focuses on achieving both accuracy within each frame and consistency across frames. 📜More👇

🤔Applying Depth Estimation models directly to videos can result in inconsistency between frames. 💪Well, not anymore. 🔥ChronoDepth is a new approach to video depth estimation that focuses on achieving both accuracy within each frame and consistency across frames. 📜More👇

Gradio

19,992 views • 2 years ago

There are so many exciting changes with our latest update, so here’s a quick recap of the major differences 🥳 For more details, check out our blog post! Also, stay tuned for more in-depth videos 👀

There are so many exciting changes with our latest update, so here’s a quick recap of the major differences 🥳 For more details, check out our blog post! Also, stay tuned for more in-depth videos 👀

Notability

16,137 views • 3 years ago

xAI absolutely cooked with the new Grok Imagine model! The temporal consistency is rock-solid. The penguin's anatomy, lighting, and proportions stay locked in for the full 10 seconds with no weird morphing or melting. In less than 30 seconds, Grok created a video with snow spray building realistically, motion blur, and depth of field. The physics simulation rivals other top models.

xAI absolutely cooked with the new Grok Imagine model! The temporal consistency is rock-solid. The penguin's anatomy, lighting, and proportions stay locked in for the full 10 seconds with no weird morphing or melting. In less than 30 seconds, Grok created a video with snow spray building realistically, motion blur, and depth of field. The physics simulation rivals other top models.

tetsuo

77,537 views • 7 months ago

Depth video workflows have been getting a lot of attention lately, so I tested one myself. Combined with Seedance 2.0, it produced more natural motion-transfer results than using Kling Motion Control directly. Why use a depth video? 1. It removes the original character and scene details, reducing copyright and sensitive-content risks. 2. It preserves the original motion, timing, and spatial structure. This separates motion extraction from visual generation, allowing you to recreate the movement with better models and any reference character. We’ve also launched a free online tool that converts regular videos into depth videos—no local setup required: In the example below, we converted a dance video from Douyin into a depth video, then regenerated it with a reference character using Seedance 2.0. The original choreography and timing are preserved, while the lighting adapts naturally to the new character and scene.

Depth video workflows have been getting a lot of attention lately, so I tested one myself. Combined with Seedance 2.0, it produced more natural motion-transfer results than using Kling Motion Control directly. Why use a depth video? 1. It removes the original character and scene details, reducing copyright and sensitive-content risks. 2. It preserves the original motion, timing, and spatial structure. This separates motion extraction from visual generation, allowing you to recreate the movement with better models and any reference character. We’ve also launched a free online tool that converts regular videos into depth videos—no local setup required: In the example below, we converted a dance video from Douyin into a depth video, then regenerated it with a reference character using Seedance 2.0. The original choreography and timing are preserved, while the lighting adapts naturally to the new character and scene.

underwood

15,968 views • 1 month ago

🤯 Depth videos perfectly solve false flags on reference videos — and can perfectly recreate both martial arts and dance! Turned a Guan Dao martial arts clip into a Depth motion reference, swapped the fighter for a market auntie, and moved the scene to a T-junction inside a local market 🤣 🌟 Workflow: 1. Pick a reference clip under 15 seconds 2. Convert it to Depth with Depth Anything V2 3. Generate a new character + scene 4. Feed everything into Seedance with the prompt below 🌟 Depth video conversion: You can build a local Depth converter with Codex — prompt in the comments. This goes way beyond trending dances. Martial arts, polearms, and other complex movements can all be transferred cleanly with Depth! Workflow + Prompt below 👇

🤯 Depth videos perfectly solve false flags on reference videos — and can perfectly recreate both martial arts and dance! Turned a Guan Dao martial arts clip into a Depth motion reference, swapped the fighter for a market auntie, and moved the scene to a T-junction inside a local market 🤣 🌟 Workflow: 1. Pick a reference clip under 15 seconds 2. Convert it to Depth with Depth Anything V2 3. Generate a new character + scene 4. Feed everything into Seedance with the prompt below 🌟 Depth video conversion: You can build a local Depth converter with Codex — prompt in the comments. This goes way beyond trending dances. Martial arts, polearms, and other complex movements can all be transferred cleanly with Depth! Workflow + Prompt below 👇

Larus Canus

51,401 views • 1 month ago

For the facial animations in my game, I use a technique that yields results similar to L.A. Noire. To keep it simple: I first generate a facial animation video using AI like LTX 2. Then, I create a video depth map from that animation and project it onto a face mask. Using vertex displacement, the depth map dynamically deforms the face as the character speaks. The result looks good in standalone VR. I actually created this workflow long time ago:

For the facial animations in my game, I use a technique that yields results similar to L.A. Noire. To keep it simple: I first generate a facial animation video using AI like LTX 2. Then, I create a video depth map from that animation and project it onto a face mask. Using vertex displacement, the depth map dynamically deforms the face as the character speaks. The result looks good in standalone VR. I actually created this workflow long time ago:

Alex

48,296 views • 6 months ago

🚀New paper out - We present Video-MSG (Multimodal Sketch Guidance), a novel planning-based training-free guidance method for T2V models, improving control of spatial layout and object trajectories. 🔧 Key idea: • Generate a Video Sketch — a spatio-temporal plan with background, foreground, and motion in the pixel space. • Encode this structure directly into the latent space of the diffusion model during generation, which does not require fine-tuning or additional memory during inference. 🧵

🚀New paper out - We present Video-MSG (Multimodal Sketch Guidance), a novel planning-based training-free guidance method for T2V models, improving control of spatial layout and object trajectories. 🔧 Key idea: • Generate a Video Sketch — a spatio-temporal plan with background, foreground, and motion in the pixel space. • Encode this structure directly into the latent space of the diffusion model during generation, which does not require fine-tuning or additional memory during inference. 🧵

Jialu Li

35,060 views • 1 year ago

Return to Disney's The Lion King Realm in our next free update - filled with 2 new friends and even more surprises. Check back tomorrow for another Developer Update Video for all the details… and maybe a launch date 👀

Return to Disney's The Lion King Realm in our next free update - filled with 2 new friends and even more surprises. Check back tomorrow for another Developer Update Video for all the details… and maybe a launch date 👀

Disney Dreamlight Valley

178,377 views • 1 year ago

🔬 Built with an entirely new model architecture, our diffusion-based approach uses 6B+ parameters and leverages the latest NVIDIA hardware. This is the most dynamic and wide-ranging video enhancing method we’ve ever created, setting a new standard for AI video restoration. Videos degrade due to compression artifacts, blurring, aliasing, noise, atmospheric distortion, missing pixels, etc. Each frame suffers from unique types of corruption, making AI video restoration a highly challenging task. Our technology solves this complexity by analyzing hundreds of frames to accurately restore details, delivering unmatched detail recovery combined with unparalleled temporal consistency.

🔬 Built with an entirely new model architecture, our diffusion-based approach uses 6B+ parameters and leverages the latest NVIDIA hardware. This is the most dynamic and wide-ranging video enhancing method we’ve ever created, setting a new standard for AI video restoration. Videos degrade due to compression artifacts, blurring, aliasing, noise, atmospheric distortion, missing pixels, etc. Each frame suffers from unique types of corruption, making AI video restoration a highly challenging task. Our technology solves this complexity by analyzing hundreds of frames to accurately restore details, delivering unmatched detail recovery combined with unparalleled temporal consistency.

Topaz Labs

23,199 views • 1 year ago

🤯 I didn’t expect Depth to track rooftop parkour this cleanly! Used a fast climbing clip as the motion reference, then swapped in a student with twin tails and a backpack on a coastal school rooftop. The running path, wall contact, jumps, landings, and full-body continuity all hold together surprisingly well. 🌟 Workflow: 1. Pick a reference clip under 15 seconds 2. Convert it to Depth with Depth Anything V2 3. Generate a new character + scene 4. Feed everything into Seedance with the prompt below 🌟 Depth video conversion: You can build a local Depth converter with Codex — prompt in the comments. Depth opens up a lot more possibilities for parkour, climbing, martial arts, dance, and other complex full-body motion. Workflow + Prompt below 👇

🤯 I didn’t expect Depth to track rooftop parkour this cleanly! Used a fast climbing clip as the motion reference, then swapped in a student with twin tails and a backpack on a coastal school rooftop. The running path, wall contact, jumps, landings, and full-body continuity all hold together surprisingly well. 🌟 Workflow: 1. Pick a reference clip under 15 seconds 2. Convert it to Depth with Depth Anything V2 3. Generate a new character + scene 4. Feed everything into Seedance with the prompt below 🌟 Depth video conversion: You can build a local Depth converter with Codex — prompt in the comments. Depth opens up a lot more possibilities for parkour, climbing, martial arts, dance, and other complex full-body motion. Workflow + Prompt below 👇

Larus Canus

24,613 views • 1 month ago

🔥 Win a share of $50,000 in MORPHO rewards 🔥 We’re celebrating the launch of Stablecoin Earn in collaboration with Morpho Labs. To enter: 🔹 Like & RT 🔹👇 Deposit USDT, USDC, or USDA into Morpho vaults via Stablecoin Earn Check out the thread for more details 🧵👇

🔥 Win a share of $50,000 in MORPHO rewards 🔥 We’re celebrating the launch of Stablecoin Earn in collaboration with Morpho Labs. To enter: 🔹 Like & RT 🔹👇 Deposit USDT, USDC, or USDA into Morpho vaults via Stablecoin Earn Check out the thread for more details 🧵👇

Trust Wallet

73,031 views • 1 year ago

A sneak peek of a new feature in the #GeoAI Python package! 🎉 Now you can detect cars from georeferenced aerial imagery using deep learning—all with just a few lines of code. Stay tuned for an in-depth video tutorial coming soon! 🛠️ Explore the GitHub repository: 📚 Dive into the documentation: 📺 Check out the entire YouTube playlist:

A sneak peek of a new feature in the #GeoAI Python package! 🎉 Now you can detect cars from georeferenced aerial imagery using deep learning—all with just a few lines of code. Stay tuned for an in-depth video tutorial coming soon! 🛠️ Explore the GitHub repository: 📚 Dive into the documentation: 📺 Check out the entire YouTube playlist:

Qiusheng Wu

45,675 views • 1 year ago

Diffuman4D: 4D Consistent Human View Synthesis from Sparse-View Videos with Spatio-Temporal Diffusion Models Contributions: • We introduce Diffuman4D, a novel diffusion model that generates spatio-temporally consistent and high-resolution (1024p) human videos from sparse-view video inputs. • We propose a sliding iterative denoising mechanism that enhances both the spatial and temporal consistency of generated long-term videos while maintaining efficient inference. • We design a human pose conditioning scheme to enhance the appearance quality and motion accuracy of generated human videos. • We plan to release our processed version of the DNA-Rendering dataset, which we believe will benefit future research in this area.

Diffuman4D: 4D Consistent Human View Synthesis from Sparse-View Videos with Spatio-Temporal Diffusion Models Contributions: • We introduce Diffuman4D, a novel diffusion model that generates spatio-temporally consistent and high-resolution (1024p) human videos from sparse-view video inputs. • We propose a sliding iterative denoising mechanism that enhances both the spatial and temporal consistency of generated long-term videos while maintaining efficient inference. • We design a human pose conditioning scheme to enhance the appearance quality and motion accuracy of generated human videos. • We plan to release our processed version of the DNA-Rendering dataset, which we believe will benefit future research in this area.

MrNeRF

24,729 views • 1 year ago

Some updates on the multiview vistadream pipeline with Rerun! Rerun came in extremely useful here, as being able to visualize depths at each stage of the pipeline allowed me to debug some nasty bugs. Since the last time, I was only working with a single image input. I've added in VGGT as my multiview pose + depth estimator. It works REALLY well for getting camera poses, but the depths are not that great. To try and fix that, I estimated depth maps from MoGeV2 for each of the views, and scale+shift aligned them so that they would match up to the confident sections of VGGT's depth predictions. You can see in the video just how much sharper the visualized 2d depth maps are! The biggest issue continues to be the multiview consistency 🫠 That's up next, along with actually training the Gaussian splat. Lots of work went into actually understanding inputs+outputs for VGGT. I had some funky bugs where the confidence values would all collapse to true I'm also really excited for this pipeline to use Difix3D+ Nvidia instead of Flux Inpainting, it seems like a better suited for a multiview pipeline.

Some updates on the multiview vistadream pipeline with Rerun! Rerun came in extremely useful here, as being able to visualize depths at each stage of the pipeline allowed me to debug some nasty bugs. Since the last time, I was only working with a single image input. I've added in VGGT as my multiview pose + depth estimator. It works REALLY well for getting camera poses, but the depths are not that great. To try and fix that, I estimated depth maps from MoGeV2 for each of the views, and scale+shift aligned them so that they would match up to the confident sections of VGGT's depth predictions. You can see in the video just how much sharper the visualized 2d depth maps are! The biggest issue continues to be the multiview consistency 🫠 That's up next, along with actually training the Gaussian splat. Lots of work went into actually understanding inputs+outputs for VGGT. I had some funky bugs where the confidence values would all collapse to true I'm also really excited for this pipeline to use Difix3D+ Nvidia instead of Flux Inpainting, it seems like a better suited for a multiview pipeline.

Pablo Vela

29,904 views • 1 year ago

Sensitive content

#Kwiky is a NEW app, for hot content with No Restrictions! ✨ #BabbieBia Naughty, short-form content that just works! Check out the link in our profile, for more!

kwikyofficial

64,619 views • 1 year ago

Nano is a depth-aware atmospheric haze plugin that uses ML depth estimation to add physically accurate fog and light scattering to your footage. Works *best* on log footage with visible light sources - it analyzes scene highlights then creates airlight (atmospheric scatter) and halation (light bloom) that responds to actual depth in the scene. Pretty clever approach to getting that cinematic haze look without having to pump a fog machine on set. Makes the OG Trapcode Shine look extremely dated (basically 2D light streaks masked by luminance values), and is yet way more controllable than the current crop of generative AI video-to-video tools.

Nano is a depth-aware atmospheric haze plugin that uses ML depth estimation to add physically accurate fog and light scattering to your footage. Works best on log footage with visible light sources - it analyzes scene highlights then creates airlight (atmospheric scatter) and halation (light bloom) that responds to actual depth in the scene. Pretty clever approach to getting that cinematic haze look without having to pump a fog machine on set. Makes the OG Trapcode Shine look extremely dated (basically 2D light streaks masked by luminance values), and is yet way more controllable than the current crop of generative AI video-to-video tools.

Bilawal Sidhu

275,503 views • 1 year ago

Bring your videos to life with Akool’s Effect Enhancer. ✨ Turn ordinary footage into dynamic, eye-catching content in seconds. With smarter visual effects and enhanced motion details, you can instantly elevate your videos without complex editing. Whether you're creating marketing content, storytelling videos, or social media clips, Effect Enhancer helps you add more depth, energy, and polish to every frame. Create smoother visuals. Add stronger impact. Make every video stand out with Akool.

Bring your videos to life with Akool’s Effect Enhancer. ✨ Turn ordinary footage into dynamic, eye-catching content in seconds. With smarter visual effects and enhanced motion details, you can instantly elevate your videos without complex editing. Whether you're creating marketing content, storytelling videos, or social media clips, Effect Enhancer helps you add more depth, energy, and polish to every frame. Create smoother visuals. Add stronger impact. Make every video stand out with Akool.

Akool Inc

22,040 views • 5 months ago