Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Here’s a sneak peek using Rerun and Gradio for data annotation. It uses Video Depth Anything and Segment Anything 2 under the hood to generate segmentation masks and depth maps/point clouds. More to share next week.

Pablo Vela

2,490 subscribers

36,719 views • 1 year ago •via X (Twitter)

Education Arts Science & Technology

Anya Rossi• Live Now

Private livecam show

7 Comments

Data & Analytics1 year ago

@_akhaliq @rerundotio @Gradio @_akhaliq, isn't it fascinating how technology evolves to make our work smoother? Tools like these push boundaries, sparking new levels of innovation in data annotation. 🚀 What other tools are you excited about that are changing your workflow? #DataScienceMagic

HUDI2 years ago

Guess who’s not a fan of data ownership? 😅 When #Zuckerberg gives zero stars, you know the @datamaskwallet app is keeping your data safe from the big tech giants' grasp. 🚫📊 #DataPrivacy #BigTechNemesis art and creative by @Matteo_Frog

Van Tuan Bui1 year ago

@rerundotio @Gradio This looks awesome! Exciting times for multimodal data tools!

Nemesis1 year ago

@rerundotio @Gradio This is fascinating! Using Video Depth Anything and Segment Anything 2 for data annotation is a great step forward. Looking forward to seeing more updates!

吭气十足吃包子并偷着乐的小学生欧尼酱1 year ago

@rerundotio @Gradio cool

d'Artagnan-sha1 year ago

@rerundotio @Gradio Fascinating work, @pablovelagomez1! Leveraging cutting-edge models like Video Depth Anything and Segment Anything 2 to enable data annotation with depth and segmentation insights is a powerful step forward.

Ayush Goyal1 year ago

@_akhaliq @rerundotio @Gradio insane demo. rerun + gradio stack looks powerful

Related Videos

Bussing plates after a healthy meal. Our robot only eats plastic fruits and veggies for now. Video segmentation accomplished using Track-Anything, an architecture combining Meta's SAM to generate a zero-shot segmentation prompt and Xmem to enable long-horizon, temporally consistent video segmentation masks.

Bussing plates after a healthy meal. Our robot only eats plastic fruits and veggies for now. Video segmentation accomplished using Track-Anything, an architecture combining Meta's SAM to generate a zero-shot segmentation prompt and Xmem to enable long-horizon, temporally consistent video segmentation masks.

Watney Robotics

14,856 views • 2 years ago

Track Anything: Segment Anything Meets Videos Track-Anything is a flexible and interactive tool for video object tracking and segmentation suitable for: - Video object tracking and segmentation with shot changes. - Visualized development and data annnotation for video object tracking and segmentation. - Object-centric downstream video tasks, such as video inpainting and editing. abs: github:

Track Anything: Segment Anything Meets Videos Track-Anything is a flexible and interactive tool for video object tracking and segmentation suitable for: - Video object tracking and segmentation with shot changes. - Visualized development and data annnotation for video object tracking and segmentation. - Object-centric downstream video tasks, such as video inpainting and editing. abs: github:

AK

578,691 views • 3 years ago

Ever wondered what the real women behind ancient Greek statues looked like? I brought them back to life using: - Depth Anything (depth maps) - Omma AI (Code and 3D implementation) - Three.js

Ever wondered what the real women behind ancient Greek statues looked like? I brought them back to life using: - Depth Anything (depth maps) - Omma AI (Code and 3D implementation) - Three.js

Joseph Azar

74,695 views • 3 months ago

Infinite depth trick! 🌑 This simple Depth Fade node is revolutionary for UEFN maps. Use it on a plane to create realistic shadows and faux depth on doors and anything else that needs that effect! 🛠️🎨 #UEFN #Fortnite

Infinite depth trick! 🌑 This simple Depth Fade node is revolutionary for UEFN maps. Use it on a plane to create realistic shadows and faux depth on doors and anything else that needs that effect! 🛠️🎨 #UEFN #Fortnite

Sitka Creative 📱

119,243 views • 6 months ago

Tracking Anything with Decoupled Video Segmentation paper page: Training data for video segmentation are expensive to annotate. This impedes extensions of end-to-end algorithms to new video segmentation tasks, especially in large-vocabulary settings. To 'track anything' without training on video data for every individual task, we develop a decoupled video segmentation approach (DEVA), composed of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation. Due to this design, we only need an image-level model for the target task (which is cheaper to train) and a universal temporal propagation model which is trained once and generalizes across tasks. To effectively combine these two modules, we use bi-directional propagation for (semi-)online fusion of segmentation hypotheses from different frames to generate a coherent segmentation. We show that this decoupled formulation compares favorably to end-to-end approaches in several data-scarce tasks including large-vocabulary video panoptic segmentation, open-world video segmentation, referring video segmentation, and unsupervised video object segmentation.

Tracking Anything with Decoupled Video Segmentation paper page: Training data for video segmentation are expensive to annotate. This impedes extensions of end-to-end algorithms to new video segmentation tasks, especially in large-vocabulary settings. To 'track anything' without training on video data for every individual task, we develop a decoupled video segmentation approach (DEVA), composed of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation. Due to this design, we only need an image-level model for the target task (which is cheaper to train) and a universal temporal propagation model which is trained once and generalizes across tasks. To effectively combine these two modules, we use bi-directional propagation for (semi-)online fusion of segmentation hypotheses from different frames to generate a coherent segmentation. We show that this decoupled formulation compares favorably to end-to-end approaches in several data-scarce tasks including large-vocabulary video panoptic segmentation, open-world video segmentation, referring video segmentation, and unsupervised video object segmentation.

AK

305,683 views • 3 years ago

Depth Anything V2 This work presents Depth Anything V2. Without pursuing fancy techniques, we aim to reveal crucial findings to pave the way towards building a powerful monocular depth estimation model. Notably, compared with V1, this version produces much finer and more

Depth Anything V2 This work presents Depth Anything V2. Without pursuing fancy techniques, we aim to reveal crucial findings to pave the way towards building a powerful monocular depth estimation model. Notably, compared with V1, this version produces much finer and more

AK

114,537 views • 2 years ago

Segment Anything is fun and powerful but still requires full mask annotation for supervised training. Our new work MaskFreeVIS appearing at #CVPR2023 shows we can get accurate video instance segmentation WITHOUT any video or image mask annotations. More at

Segment Anything is fun and powerful but still requires full mask annotation for supervised training. Our new work MaskFreeVIS appearing at #CVPR2023 shows we can get accurate video instance segmentation WITHOUT any video or image mask annotations. More at

Fisher Yu

41,153 views • 3 years ago

Took it further with Omma AI, (Spline ) now it works on video, and everything working in a Web Browser. Drop a video, Depth Anything v2 calculates the depth map frame by frame, and Three.js renders a fully lit 3D mesh in real time. Depth estimation on video in the browser +Automatic baking pass + Dynamic lighting reacting to the geometry #vibecoding #threejs #webgl

Took it further with Omma AI, (Spline ) now it works on video, and everything working in a Web Browser. Drop a video, Depth Anything v2 calculates the depth map frame by frame, and Three.js renders a fully lit 3D mesh in real time. Depth estimation on video in the browser +Automatic baking pass + Dynamic lighting reacting to the geometry #vibecoding #threejs #webgl

Joseph Azar

148,148 views • 4 months ago

A quick test to see what results an AI animated sprite can achieve using normal maps and depth maps. I tried it with DOOM style sprites just for fun. #Doom

A quick test to see what results an AI animated sprite can achieve using normal maps and depth maps. I tried it with DOOM style sprites just for fun. #Doom

Alex

52,572 views • 1 month ago

We just tested what might be the most accurate motion transfer workflow we've seen so far. It's a little more involved than a standard workflow, but the results are pretty impressive. Here's the basic workflow: 1. Start with your reference footage. Create character and environment references. 2. Convert the reference footage into a depth map (with Depth-Anything tool on Fal: 3. (Optional) Layer on additional guidance like colorized character masks (with Meta's Segment Anything: and or skeleton poses (with 4. Feed the depth map into Seedance 2.0 Omni as your motion reference along with your character and environment references. 5. Generate your final video. After testing several variations, the depth map workflow consistently produced the cleanest and most accurate motion transfer. If you're trying to recreate specific performances or complex movement, this is one of the closest AI workflows we've seen to achieving production-quality results.

We just tested what might be the most accurate motion transfer workflow we've seen so far. It's a little more involved than a standard workflow, but the results are pretty impressive. Here's the basic workflow: 1. Start with your reference footage. Create character and environment references. 2. Convert the reference footage into a depth map (with Depth-Anything tool on Fal: 3. (Optional) Layer on additional guidance like colorized character masks (with Meta's Segment Anything: and or skeleton poses (with 4. Feed the depth map into Seedance 2.0 Omni as your motion reference along with your character and environment references. 5. Generate your final video. After testing several variations, the depth map workflow consistently produced the cleanest and most accurate motion transfer. If you're trying to recreate specific performances or complex movement, this is one of the closest AI workflows we've seen to achieving production-quality results.

Curious Refuge

30,397 views • 1 month ago

It’s been a long time since I last shared anything about the project, so here’s a very small and quick sneak peek 👀 I’ll have more to share very soon, with a full devlog and more regular work in progress😊 #gamedev #indiedev #indiegamedev #indiegame #gaming #game

It’s been a long time since I last shared anything about the project, so here’s a very small and quick sneak peek 👀 I’ll have more to share very soon, with a full devlog and more regular work in progress😊 #gamedev #indiedev #indiegamedev #indiegame #gaming #game

A Silent Desolation

15,284 views • 2 months ago

x86 segmentation maps memory references through segment registers. It is still used in modern x86-64 for FS/GS, especially thread-local storage and kernel per-CPU data. In sandboxing, segmentation can restrict untrusted code to a fixed memory region. MIT 6.858.

x86 segmentation maps memory references through segment registers. It is still used in modern x86-64 for FS/GS, especially thread-local storage and kernel per-CPU data. In sandboxing, segmentation can restrict untrusted code to a fixed memory region. MIT 6.858.

tetsuo

17,558 views • 3 months ago

I spent a lot of time with the ARKitScenes dataset; it's a great proxy for a bunch of downstream tasks and holds some ~5k casually collected iPhone reconstructions. I make the argument in the attached article that using this with Prompt Depth Anything, MogeV2, and Gaussian Splats leads to much better reconstructions for other downstream tasks and labels. Here's one of the post-processed reconstructions I generated that contains the depth and normal maps for the ultrawide camera

I spent a lot of time with the ARKitScenes dataset; it's a great proxy for a bunch of downstream tasks and holds some ~5k casually collected iPhone reconstructions. I make the argument in the attached article that using this with Prompt Depth Anything, MogeV2, and Gaussian Splats leads to much better reconstructions for other downstream tasks and labels. Here's one of the post-processed reconstructions I generated that contains the depth and normal maps for the ultrawide camera

Pablo Vela

15,011 views • 8 days ago

🤯 OneDiffusion: A versatile, large-scale diffusion model that seamlessly supports bidirectional image synthesis and understanding across diverse tasks. ✅ Text to Image ✅ Image to Depth ✅ Image to Segmentation ✅ Image to Pose ✅ FaceID ✅ Image to Multiview How to use & more👇

🤯 OneDiffusion: A versatile, large-scale diffusion model that seamlessly supports bidirectional image synthesis and understanding across diverse tasks. ✅ Text to Image ✅ Image to Depth ✅ Image to Segmentation ✅ Image to Pose ✅ FaceID ✅ Image to Multiview How to use & more👇

Gradio

11,820 views • 1 year ago

🚀 Excited to share a new tutorial on how to detect buildings 🏘️using the Segment Anything Model 2 (SAM 2) with Segment-Geospatial! You can choose interactive segmentation or utilize a GeoJSON file with building centroids or bounding boxes to do automated building detection. 📹 Check out the video tutorial: 📓 Explore the notebook: #Geospatial #OpenSource #AI #MachineLearning

🚀 Excited to share a new tutorial on how to detect buildings 🏘️using the Segment Anything Model 2 (SAM 2) with Segment-Geospatial! You can choose interactive segmentation or utilize a GeoJSON file with building centroids or bounding boxes to do automated building detection. 📹 Check out the video tutorial: 📓 Explore the notebook: #Geospatial #OpenSource #AI #MachineLearning

Qiusheng Wu

72,345 views • 1 year ago

Announcing our new and stable release of Segment Anything for Microscopy! We introduce new features and improve the tool to segment microscopy data with a few clicks, like the mitochondria in the video, first segmented automatically and then corrected interactively.

Announcing our new and stable release of Segment Anything for Microscopy! We introduce new features and improve the tool to segment microscopy data with a few clicks, like the mitochondria in the video, first segmented automatically and then corrected interactively.

Constantin Pape

45,380 views • 2 years ago

Happy 4th 🇺🇸!! I have a preview for my next release (assembly101 is massive, so I need to push until next week). But in the meantime, check out the link below to try out the 🚧 work in progress using Rerun and Gradio Links below 👇

Happy 4th 🇺🇸!! I have a preview for my next release (assembly101 is massive, so I need to push until next week). But in the meantime, check out the link below to try out the 🚧 work in progress using Rerun and Gradio Links below 👇

Pablo Vela

34,186 views • 1 year ago

As Phoenix AI Growth Points is expected to go live next week – PhoenixONE has leveled up once again in the meantime. Now, with the click of a button (new “Synthesize” feature) create full articles from entire conversations that go beyond in depth, quality, and data-driven rigor than much of the content on major media outlets. Acquire knowledge and insights, share content, and get rewarded in this new system. The more engagement and exposure your content gets, the bigger the reward. Create in-depth content in as little as a few minutes about anything – especially crypto! Phoenix is adding a social element to data-driven generative AI to drive user value and acceleration of growth. $PHB #Phoenix

As Phoenix AI Growth Points is expected to go live next week – PhoenixONE has leveled up once again in the meantime. Now, with the click of a button (new “Synthesize” feature) create full articles from entire conversations that go beyond in depth, quality, and data-driven rigor than much of the content on major media outlets. Acquire knowledge and insights, share content, and get rewarded in this new system. The more engagement and exposure your content gets, the bigger the reward. Create in-depth content in as little as a few minutes about anything – especially crypto! Phoenix is adding a social element to data-driven generative AI to drive user value and acceleration of growth. $PHB #Phoenix

Phoenix AI Labs

14,401 views • 1 year ago

🚀 The Segment Anything Model (SAM) has been upgraded to SAM2, featuring an efficient image encoder for segmenting images and videos. But does SAM2 outperform SAM1 in medical image and video segmentation? We're thrilled to present our paper "Segment Anything in Medical Images and Videos: Benchmark and Deployment"! We comprehensively benchmark SAM2 across 11 medical image modalities and videos. 📄 Paper: 💻 Code: **Highlights:** 1. SAM2 doesn’t always outperform SAM1 in 2D medical images, but excels in video segmentation, making it more accurate and efficient for 3D images, such as CT and MR scans. 2. MedSAM still outperforms SAM2 on most 2D modalities, but SAM2 surpasses MedSAM for 3D image segmentation in a slice-by-slice approach. 3. Segmentation performance varies with model size; sometimes the smallest model outperforms larger ones. 4. Fine-tuning SAM2 significantly boosts its performance for medical image segmentation. While SAM2 may struggle with challenging objects that have unclear boundaries or low contrast, it excels in generating good initial segmentation masks for common medical images and videos. However, the official interface doesn’t support medical data formats and has limitations on video length. To address this, we've developed a 3D Slicer Plugin and Gradio API for efficient 3D medical image and video segmentation. We invite you to try them out and provide feedback! 🔧 Deployment: - 3D Slicer Plugin: - Gradio API: (Note: Due to GPU limitations, the online API is available for only 12 hours and may be slow. We highly recommend deploying the Gradio API with your own computing resources: A big shoutout to Jun Ma (JunMa) who recently joined our UHN AI hub (UHN AI Hub) as Machine Learning Lead, and kudos to all co-authors: Sumin Kim, Feifei Li, Mohammed Baharoon (Mohammed Baharoon), Reza Asakereh, and Hongwei Lyu! This is true teamwork! Looking forward to collaborating with the community to advance 3D medical image and video segmentation foundation models! University Health Network U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology Temerty Centre for AI in Medicine (T-CAIREM) Vector Institute #MedTech #AIinHealthcare #DeepLearning #MedicalImaging #SAM2 #MedSAM #AIResearch

🚀 The Segment Anything Model (SAM) has been upgraded to SAM2, featuring an efficient image encoder for segmenting images and videos. But does SAM2 outperform SAM1 in medical image and video segmentation? We're thrilled to present our paper "Segment Anything in Medical Images and Videos: Benchmark and Deployment"! We comprehensively benchmark SAM2 across 11 medical image modalities and videos. 📄 Paper: 💻 Code: Highlights: 1. SAM2 doesn’t always outperform SAM1 in 2D medical images, but excels in video segmentation, making it more accurate and efficient for 3D images, such as CT and MR scans. 2. MedSAM still outperforms SAM2 on most 2D modalities, but SAM2 surpasses MedSAM for 3D image segmentation in a slice-by-slice approach. 3. Segmentation performance varies with model size; sometimes the smallest model outperforms larger ones. 4. Fine-tuning SAM2 significantly boosts its performance for medical image segmentation. While SAM2 may struggle with challenging objects that have unclear boundaries or low contrast, it excels in generating good initial segmentation masks for common medical images and videos. However, the official interface doesn’t support medical data formats and has limitations on video length. To address this, we've developed a 3D Slicer Plugin and Gradio API for efficient 3D medical image and video segmentation. We invite you to try them out and provide feedback! 🔧 Deployment: - 3D Slicer Plugin: - Gradio API: (Note: Due to GPU limitations, the online API is available for only 12 hours and may be slow. We highly recommend deploying the Gradio API with your own computing resources: A big shoutout to Jun Ma (JunMa) who recently joined our UHN AI hub (UHN AI Hub) as Machine Learning Lead, and kudos to all co-authors: Sumin Kim, Feifei Li, Mohammed Baharoon (Mohammed Baharoon), Reza Asakereh, and Hongwei Lyu! This is true teamwork! Looking forward to collaborating with the community to advance 3D medical image and video segmentation foundation models! University Health Network U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology Temerty Centre for AI in Medicine (T-CAIREM) Vector Institute #MedTech #AIinHealthcare #DeepLearning #MedicalImaging #SAM2 #MedSAM #AIResearch

Bo Wang

178,579 views • 2 years ago

Want to use Depth Anything, but need metric depth rather than relative depth? Thrilled to introduce Prompt Depth Anything, a new paradigm for accurate metric depth estimation with up to 4K resolution. 👉Key Message: Depth foundation models like DA have already internalized rich geometric knowledge of the 3D world but lack a proper way to elicit it. Inspired by the success of prompting in LLMs, we propose prompting Depth Anything with metric cues to produce metric depth. This method proves to be very effective when using a low-cost lidar (e.g., iPhone's LiDAR), which is widely available, as prompts. We believe the prompt can generalize to other forms as long as scale information is provided. Prompt Depth Anything offers 1⃣A series of models for iPhone lidars. 2⃣4D reconstruction from monocular videos (captured with iPhone). 3⃣Improved generalization ability for robot manipulation, e.g. Training on cans but generalizing on glasses. 4⃣More detailed depth annotations for the ScanNet++ dataset. The first author is our excellent intern Haotong Lin. Paper: Huggingface: Project Page: Code:

Want to use Depth Anything, but need metric depth rather than relative depth? Thrilled to introduce Prompt Depth Anything, a new paradigm for accurate metric depth estimation with up to 4K resolution. 👉Key Message: Depth foundation models like DA have already internalized rich geometric knowledge of the 3D world but lack a proper way to elicit it. Inspired by the success of prompting in LLMs, we propose prompting Depth Anything with metric cues to produce metric depth. This method proves to be very effective when using a low-cost lidar (e.g., iPhone's LiDAR), which is widely available, as prompts. We believe the prompt can generalize to other forms as long as scale information is provided. Prompt Depth Anything offers 1⃣A series of models for iPhone lidars. 2⃣4D reconstruction from monocular videos (captured with iPhone). 3⃣Improved generalization ability for robot manipulation, e.g. Training on cans but generalizing on glasses. 4⃣More detailed depth annotations for the ScanNet++ dataset. The first author is our excellent intern Haotong Lin. Paper: Huggingface: Project Page: Code:

Bingyi Kang

67,706 views • 1 year ago