Loading video...

Video Failed to Load

Go Home

Segment Any 3D Gaussians paper page: Interactive 3D segmentation in radiance fields is an appealing task since its importance in 3D scene understanding and manipulation. However, existing methods face challenges in either achieving fine-grained, multi-granularity segmentation or contending with substantial computational overhead, inhibiting real-time interaction. In this paper, we...

_akhaliq's profile picture

AK

406,190 subscribers

69,542 views • 2 years ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

From 2D to 3D segmentation: theory and applications of u-Segment3D by Felix Zhou et al.
0:14

From 2D to 3D segmentation: theory and applications of u-Segment3D by Felix Zhou et al.

Reto Fiolka

17,593 views • 2 years ago

We are presenting **SAD** (Segment Any RGBD): SAD is able to perform 3D segmentation (segment out any 3D object) with RGBD inputs (or rendered depth images only). - Code: - Demo Hugging Face:
0:11

We are presenting **SAD** (Segment Any RGBD): SAD is able to perform 3D segmentation (segment out any 3D object) with RGBD inputs (or rendered depth images only). - Code: - Demo Hugging Face:

Ziwei Liu

195,642 views • 3 years ago

Today we're releasing the Segment Anything Model (SAM) — a step toward the first foundation model for image segmentation. SAM is capable of one-click segmentation of any object from any photo or video + zero-shot transfer to other segmentation tasks ➡️
0:15

Today we're releasing the Segment Anything Model (SAM) — a step toward the first foundation model for image segmentation. SAM is capable of one-click segmentation of any object from any photo or video + zero-shot transfer to other segmentation tasks ➡️

AI at Meta

3,570,334 views • 3 years ago

Most 3DGS segmentation tools either pre‑train per scene or lock errors into a feature field you can’t undo. ArtisanGS instead turns a few 2D masks into editable 3D object selections via Cutie tracking + black‑box splat aggregation, then lets you iteratively correct mistakes with consistent 2D/3D selection modes. 📗 #NVIDIAResearch paper:
0:15

Most 3DGS segmentation tools either pre‑train per scene or lock errors into a feature field you can’t undo. ArtisanGS instead turns a few 2D masks into editable 3D object selections via Cutie tracking + black‑box splat aggregation, then lets you iteratively correct mistakes with consistent 2D/3D selection modes. 📗 #NVIDIAResearch paper:

NVIDIA AI Developer

42,277 views • 4 months ago

DroneSplat: 3D Gaussian Splatting for Robust 3D Reconstruction from In-the-Wild Drone Imagery Abstract: Drones have become essential tools for reconstructing wild scenes due to their outstanding maneuverability. Recent advances in radiance field methods have achieved remarkable rendering quality, providing a new avenue for 3D reconstruction from drone imagery. However, dynamic distractors in wild environments challenge the static scene assumption in radiance fields, while limited view constraints hinder the accurate capture of underlying scene geometry. To address these challenges, we introduce DroneSplat, a novel framework designed for robust 3D reconstruction from in-the-wild drone imagery. Our method adaptively adjusts masking thresholds by integrating local-global segmentation heuristics with statistical approaches, enabling precise identification and elimination of dynamic distractors in static scenes. We enhance 3D Gaussian Splatting with multi-view stereo predictions and a voxel-guided optimization strategy, supporting high-quality rendering under limited view constraints. For comprehensive evaluation, we provide a drone-captured 3D reconstruction dataset encompassing both dynamic and static scenes. Extensive experiments demonstrate that DroneSplat outperforms both 3DGS and NeRF baselines in handling in-the-wild drone imagery.
0:19

DroneSplat: 3D Gaussian Splatting for Robust 3D Reconstruction from In-the-Wild Drone Imagery Abstract: Drones have become essential tools for reconstructing wild scenes due to their outstanding maneuverability. Recent advances in radiance field methods have achieved remarkable rendering quality, providing a new avenue for 3D reconstruction from drone imagery. However, dynamic distractors in wild environments challenge the static scene assumption in radiance fields, while limited view constraints hinder the accurate capture of underlying scene geometry. To address these challenges, we introduce DroneSplat, a novel framework designed for robust 3D reconstruction from in-the-wild drone imagery. Our method adaptively adjusts masking thresholds by integrating local-global segmentation heuristics with statistical approaches, enabling precise identification and elimination of dynamic distractors in static scenes. We enhance 3D Gaussian Splatting with multi-view stereo predictions and a voxel-guided optimization strategy, supporting high-quality rendering under limited view constraints. For comprehensive evaluation, we provide a drone-captured 3D reconstruction dataset encompassing both dynamic and static scenes. Extensive experiments demonstrate that DroneSplat outperforms both 3DGS and NeRF baselines in handling in-the-wild drone imagery.

MrNeRF

21,346 views • 1 year ago

🚀 The Segment Anything Model (SAM) has been upgraded to SAM2, featuring an efficient image encoder for segmenting images and videos. But does SAM2 outperform SAM1 in medical image and video segmentation? We're thrilled to present our paper "Segment Anything in Medical Images and Videos: Benchmark and Deployment"! We comprehensively benchmark SAM2 across 11 medical image modalities and videos. 📄 Paper: 💻 Code: **Highlights:** 1. SAM2 doesn’t always outperform SAM1 in 2D medical images, but excels in video segmentation, making it more accurate and efficient for 3D images, such as CT and MR scans. 2. MedSAM still outperforms SAM2 on most 2D modalities, but SAM2 surpasses MedSAM for 3D image segmentation in a slice-by-slice approach. 3. Segmentation performance varies with model size; sometimes the smallest model outperforms larger ones. 4. Fine-tuning SAM2 significantly boosts its performance for medical image segmentation. While SAM2 may struggle with challenging objects that have unclear boundaries or low contrast, it excels in generating good initial segmentation masks for common medical images and videos. However, the official interface doesn’t support medical data formats and has limitations on video length. To address this, we've developed a 3D Slicer Plugin and Gradio API for efficient 3D medical image and video segmentation. We invite you to try them out and provide feedback! 🔧 Deployment: - 3D Slicer Plugin: - Gradio API: (Note: Due to GPU limitations, the online API is available for only 12 hours and may be slow. We highly recommend deploying the Gradio API with your own computing resources: A big shoutout to Jun Ma (JunMa) who recently joined our UHN AI hub (UHN AI Hub) as Machine Learning Lead, and kudos to all co-authors: Sumin Kim, Feifei Li, Mohammed Baharoon (Mohammed Baharoon), Reza Asakereh, and Hongwei Lyu! This is true teamwork! Looking forward to collaborating with the community to advance 3D medical image and video segmentation foundation models! University Health Network U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology Temerty Centre for AI in Medicine (T-CAIREM) Vector Institute #MedTech #AIinHealthcare #DeepLearning #MedicalImaging #SAM2 #MedSAM #AIResearch
0:22

🚀 The Segment Anything Model (SAM) has been upgraded to SAM2, featuring an efficient image encoder for segmenting images and videos. But does SAM2 outperform SAM1 in medical image and video segmentation? We're thrilled to present our paper "Segment Anything in Medical Images and Videos: Benchmark and Deployment"! We comprehensively benchmark SAM2 across 11 medical image modalities and videos. 📄 Paper: 💻 Code: **Highlights:** 1. SAM2 doesn’t always outperform SAM1 in 2D medical images, but excels in video segmentation, making it more accurate and efficient for 3D images, such as CT and MR scans. 2. MedSAM still outperforms SAM2 on most 2D modalities, but SAM2 surpasses MedSAM for 3D image segmentation in a slice-by-slice approach. 3. Segmentation performance varies with model size; sometimes the smallest model outperforms larger ones. 4. Fine-tuning SAM2 significantly boosts its performance for medical image segmentation. While SAM2 may struggle with challenging objects that have unclear boundaries or low contrast, it excels in generating good initial segmentation masks for common medical images and videos. However, the official interface doesn’t support medical data formats and has limitations on video length. To address this, we've developed a 3D Slicer Plugin and Gradio API for efficient 3D medical image and video segmentation. We invite you to try them out and provide feedback! 🔧 Deployment: - 3D Slicer Plugin: - Gradio API: (Note: Due to GPU limitations, the online API is available for only 12 hours and may be slow. We highly recommend deploying the Gradio API with your own computing resources: A big shoutout to Jun Ma (JunMa) who recently joined our UHN AI hub (UHN AI Hub) as Machine Learning Lead, and kudos to all co-authors: Sumin Kim, Feifei Li, Mohammed Baharoon (Mohammed Baharoon), Reza Asakereh, and Hongwei Lyu! This is true teamwork! Looking forward to collaborating with the community to advance 3D medical image and video segmentation foundation models! University Health Network U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology Temerty Centre for AI in Medicine (T-CAIREM) Vector Institute #MedTech #AIinHealthcare #DeepLearning #MedicalImaging #SAM2 #MedSAM #AIResearch

Bo Wang

178,455 views • 1 year ago

This is RICULOUSLY good, TRELLIS 3D Generation model by Microsoft! 🔥 Generate high-quality 3D assets from text or image prompts. Supports various formats like Radiance Fields, 3D Gaussians, and meshes Available for FREE on Hugging Face!
0:14

This is RICULOUSLY good, TRELLIS 3D Generation model by Microsoft! 🔥 Generate high-quality 3D assets from text or image prompts. Supports various formats like Radiance Fields, 3D Gaussians, and meshes Available for FREE on Hugging Face!

Vaibhav (VB) Srivastav

62,437 views • 1 year ago

We’re introducing Segmentation. SVG masks from prompt, points, or box. SOTA on benchmarks.
0:13

We’re introducing Segmentation. SVG masks from prompt, points, or box. SOTA on benchmarks.

moondream

153,393 views • 7 months ago

Moondream segmenting delivers pixel-accurate vector masks for arms, packages, and conveyor bins. A free, open-source model with state-of-the-art segmentation benchmarks, built for real-world automation.
0:11

Moondream segmenting delivers pixel-accurate vector masks for arms, packages, and conveyor bins. A free, open-source model with state-of-the-art segmentation benchmarks, built for real-world automation.

moondream

15,141 views • 6 months ago

(1/2) Check out 𝐆𝐚𝐮𝐬𝐬𝐢𝐚𝐧𝐀𝐯𝐚𝐭𝐚𝐫𝐬: Photorealistic Head Avatars with Rigged 3D Gaussians! We create photorealistic head avatars by animating 3D Gaussians on a parametric face model - edited and rendered in *real-time*!
0:14

(1/2) Check out 𝐆𝐚𝐮𝐬𝐬𝐢𝐚𝐧𝐀𝐯𝐚𝐭𝐚𝐫𝐬: Photorealistic Head Avatars with Rigged 3D Gaussians! We create photorealistic head avatars by animating 3D Gaussians on a parametric face model - edited and rendered in *real-time*!

Matthias Niessner

126,971 views • 2 years ago

Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians paper page: Creating high-fidelity 3D head avatars has always been a research hotspot, but there remains a great challenge under lightweight sparse view setups. In this paper, we propose Gaussian Head Avatar represented by controllable 3D Gaussians for high-fidelity head avatar modeling. We optimize the neutral 3D Gaussians and a fully learned MLP-based deformation field to capture complex expressions. The two parts benefit each other, thereby our method can model fine-grained dynamic details while ensuring expression accuracy. Furthermore, we devise a well-designed geometry-guided initialization strategy based on implicit SDF and Deep Marching Tetrahedra for the stability and convergence of the training procedure. Experiments show our approach outperforms other state-of-the-art sparse-view methods, achieving ultra high-fidelity rendering quality at 2K resolution even under exaggerated expressions.
0:12

Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians paper page: Creating high-fidelity 3D head avatars has always been a research hotspot, but there remains a great challenge under lightweight sparse view setups. In this paper, we propose Gaussian Head Avatar represented by controllable 3D Gaussians for high-fidelity head avatar modeling. We optimize the neutral 3D Gaussians and a fully learned MLP-based deformation field to capture complex expressions. The two parts benefit each other, thereby our method can model fine-grained dynamic details while ensuring expression accuracy. Furthermore, we devise a well-designed geometry-guided initialization strategy based on implicit SDF and Deep Marching Tetrahedra for the stability and convergence of the training procedure. Experiments show our approach outperforms other state-of-the-art sparse-view methods, achieving ultra high-fidelity rendering quality at 2K resolution even under exaggerated expressions.

AK

65,834 views • 2 years ago

Sora 2 can make a depth map, heat map, edge map and segmentation view, simultaneously in a grid. > start with a close up view of a woman in a studio, then split into a 2x2 grid showing in each cell the depth map, heat map, edge detection and segmentation of the scene
0:10

Sora 2 can make a depth map, heat map, edge map and segmentation view, simultaneously in a grid. > start with a close up view of a woman in a studio, then split into a 2x2 grid showing in each cell the depth map, heat map, edge detection and segmentation of the scene

fofr

39,754 views • 8 months ago

A teaser of the latest 3D DUSt3R based models we’re presenting at @cvprconference Discover MUSt3R & Pow3R, universal encoder DUNE + research in navigation, vizloc, segmentation & human motion understanding! All our #CVPR2025 papers are here ➡️
0:12

A teaser of the latest 3D DUSt3R based models we’re presenting at @cvprconference Discover MUSt3R & Pow3R, universal encoder DUNE + research in navigation, vizloc, segmentation & human motion understanding! All our #CVPR2025 papers are here ➡️

NAVER LABS Europe

14,222 views • 1 year ago

Meet 3D Gaussian Splatting - a 3D AI breakthrough that’s changing the game for real-time 3D rendering. It generates 3D world with state-of-the-art visuals at fast speeds: 100+ fps at 1080p. Here’s the magic behind it:
0:13

Meet 3D Gaussian Splatting - a 3D AI breakthrough that’s changing the game for real-time 3D rendering. It generates 3D world with state-of-the-art visuals at fast speeds: 100+ fps at 1080p. Here’s the magic behind it:

el.cine

60,115 views • 1 year ago

NeuralGS Bridging Neural Fields and 3D Gaussian Splatting for Compact 3D Representations
0:10

NeuralGS Bridging Neural Fields and 3D Gaussian Splatting for Compact 3D Representations

AK

68,496 views • 1 year ago

We will be in ACM SIGGRAPH 2023 with "3D Gaussian Splatting for Real-Time Radiance Field Rendering", have you ever seen radiance fields with 100+ FPS and MipNeRF360 quality? Check out our website here:
0:14

We will be in ACM SIGGRAPH 2023 with "3D Gaussian Splatting for Real-Time Radiance Field Rendering", have you ever seen radiance fields with 100+ FPS and MipNeRF360 quality? Check out our website here:

George Kopanas

135,782 views • 3 years ago

Human Hair Reconstruction with Strand-Aligned 3D Gaussians Contributions (cited): – We propose a new 3D line lifting scheme that uses a modified 3DGS reconstruction technique to lift 2D orientation maps into a 3D field while also providing refinement of the camera parameters; – We introduce a dual representation of hair strand polylines and 3D Gaussians to achieve differentiable rasterization of hair strands and leverage photometric constraints for strand-based hair reconstruction; – Based on these components, we propose a coarse-to-fine optimization method for prior-guided hair reconstruction that leverages both latent and explicit representations of the hairstyle.
0:13

Human Hair Reconstruction with Strand-Aligned 3D Gaussians Contributions (cited): – We propose a new 3D line lifting scheme that uses a modified 3DGS reconstruction technique to lift 2D orientation maps into a 3D field while also providing refinement of the camera parameters; – We introduce a dual representation of hair strand polylines and 3D Gaussians to achieve differentiable rasterization of hair strands and leverage photometric constraints for strand-based hair reconstruction; – Based on these components, we propose a coarse-to-fine optimization method for prior-guided hair reconstruction that leverages both latent and explicit representations of the hairstyle.

MrNeRF

106,497 views • 1 year ago

Lightplane Highly-Scalable Components for Neural 3D Fields Contemporary 3D research, particularly in reconstruction and generation, heavily relies on 2D images for inputs or supervision. However, current designs for these 2D-3D mapping are memory-intensive, posing a
0:12

Lightplane Highly-Scalable Components for Neural 3D Fields Contemporary 3D research, particularly in reconstruction and generation, heavily relies on 2D images for inputs or supervision. However, current designs for these 2D-3D mapping are memory-intensive, posing a

AK

18,896 views • 2 years ago

I've figured out how to constantly switch between 2D and 3D in order my gaslight the chat. I can start out as a 2D model then zoom out to reveal we were in 3D the whole time.
0:15

I've figured out how to constantly switch between 2D and 3D in order my gaslight the chat. I can start out as a 2D model then zoom out to reveal we were in 3D the whole time.

shindigs

54,090 views • 1 year ago