Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Nuvo: Neural UV Mapping! It's super difficult to UV map/texture atlas geometry produced by 3D reconstruction and generation pipelines. Nuvo works on all kinds of "unruly" 3D representations (NeRF, DreamFusion, etc.) and enables easy appearance editing! 1/3

_pratul_'s profile picture

Pratul Srinivasan

1,573 subscribers

20,842 Aufrufe • vor 2 Jahren •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

As announced in partnership with NVIDIA at CES, we’re excited to introduce Stable Point Aware 3D (SPAR3D), setting a new standard in 3D generation. Ideal for running on NVIDIA RTX AI PCs, SPAR3D enables real-time editing and complete structure generation of 3D objects from a single image in under a second. You can download the weights on Hugging Face and code on GitHub, or access the model through the Stability AI API. Learn more here: (1/3)
0:13

As announced in partnership with NVIDIA at CES, we’re excited to introduce Stable Point Aware 3D (SPAR3D), setting a new standard in 3D generation. Ideal for running on NVIDIA RTX AI PCs, SPAR3D enables real-time editing and complete structure generation of 3D objects from a single image in under a second. You can download the weights on Hugging Face and code on GitHub, or access the model through the Stability AI API. Learn more here: (1/3)

Stability AI

181,441 Aufrufe • vor 1 Jahr

📢GeomHair: Reconstruction of Hair Strands from Colorless 3D Scans📢 We present a novel method to reconstruct hair strands from colorless 3D scans by extracting orientation cues directly from the mesh surface geometry by finding local characteristic lines and from shaded renderings using a neural 2D line detector. We enhance the reconstruction with a diffusion prior trained on synthetic hair data and adapted to each scan using a tailored text prompt, allowing us to recover both simple and complex hairstyles without relying on color input. To support further research, we also introduce Strands400, the largest publicly available dataset of 3D hair strand reconstructions from real-world scans of 400 different people, featuring complicated hairstyles, such as ponytails and buns. 🌍 📷 Great work by Rachmadio Noval L. Artem Sevastopolsky Egor Zakharov @ness_pris
0:15

📢GeomHair: Reconstruction of Hair Strands from Colorless 3D Scans📢 We present a novel method to reconstruct hair strands from colorless 3D scans by extracting orientation cues directly from the mesh surface geometry by finding local characteristic lines and from shaded renderings using a neural 2D line detector. We enhance the reconstruction with a diffusion prior trained on synthetic hair data and adapted to each scan using a tailored text prompt, allowing us to recover both simple and complex hairstyles without relying on color input. To support further research, we also introduce Strands400, the largest publicly available dataset of 3D hair strand reconstructions from real-world scans of 400 different people, featuring complicated hairstyles, such as ponytails and buns. 🌍 📷 Great work by Rachmadio Noval L. Artem Sevastopolsky Egor Zakharov @ness_pris

Matthias Niessner

12,466 Aufrufe • vor 1 Jahr

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.
0:12

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

MrNeRF

52,801 Aufrufe • vor 1 Jahr

📢Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image📢 We directly regress neural parametric head models (NPHMs) from a single image — fast, stable, and significantly more expressive than classical 3DMMs such as FLAME. Face tracking & 3D reconstruction are often limited by the representational capacity of PCA-based face models. By lifting NPHMs to a first-class reconstruction primitive, we enable more accurate geometry, richer expressions, and finer animation control. Pix2NPHM obtains fast and reliable NPHM reconstructions on real-world data. Inference-time optimization against surface normals and canonical point maps can further increase fidelity. Key to successful and generalized training of our ViT-based network are: (1) large-scale registration of existing 3D head datasets, and (2) self-supervised training on vast in-the-wild 2D video datasets using pseudo ground-truth surface normals. Finally, we show that geometry-aware pretraining on pixel-aligned reconstruction tasks significantly outperforms generic visual pretraining (e.g., DINO-style features) in terms of generalization. 🌍 🎥 Great work by Simon Giebenhain, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Zhe Chen
0:17

📢Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image📢 We directly regress neural parametric head models (NPHMs) from a single image — fast, stable, and significantly more expressive than classical 3DMMs such as FLAME. Face tracking & 3D reconstruction are often limited by the representational capacity of PCA-based face models. By lifting NPHMs to a first-class reconstruction primitive, we enable more accurate geometry, richer expressions, and finer animation control. Pix2NPHM obtains fast and reliable NPHM reconstructions on real-world data. Inference-time optimization against surface normals and canonical point maps can further increase fidelity. Key to successful and generalized training of our ViT-based network are: (1) large-scale registration of existing 3D head datasets, and (2) self-supervised training on vast in-the-wild 2D video datasets using pseudo ground-truth surface normals. Finally, we show that geometry-aware pretraining on pixel-aligned reconstruction tasks significantly outperforms generic visual pretraining (e.g., DINO-style features) in terms of generalization. 🌍 🎥 Great work by Simon Giebenhain, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Zhe Chen

Matthias Niessner

37,850 Aufrufe • vor 6 Monaten

#Ghaati First Single #Sailore promo out now and looking forward to share our song with all of you tomorrow at 3:33 PM. ▶️ 😃😍🧿🤗 A @NagavelliV musical 🎼 #Ghaati GRAND RELEASE WORLDWIDE ON JULY 11th. #GhaatiFromJuly11th ⭐ing Vikram Prabhu 🎥 Directed by the phenomenal Krish Jagarlamudi 🏢 Proudly produced by UV Creations & First Frame Entertainments 🎼 Music on Aditya Music
0:15

#Ghaati First Single #Sailore promo out now and looking forward to share our song with all of you tomorrow at 3:33 PM. ▶️ 😃😍🧿🤗 A @NagavelliV musical 🎼 #Ghaati GRAND RELEASE WORLDWIDE ON JULY 11th. #GhaatiFromJuly11th ⭐ing Vikram Prabhu 🎥 Directed by the phenomenal Krish Jagarlamudi 🏢 Proudly produced by UV Creations & First Frame Entertainments 🎼 Music on Aditya Music

Anushka Shetty

141,190 Aufrufe • vor 1 Jahr

ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxies Contributions: 1) We propose ImmerseGen, a novel agent-guided 3D environment generation framework. It uses simplified geometric proxies with alpha-textured meshes to produce compact, photorealistic worlds ready for real-time mobile VR rendering. 2) We propose a novel RGBA texturing paradigm. It first synthesizes 8K terrain textures using a geometry-conditioned panorama generator via user-centric mapping, and then directly generates alpha-textured proxy assets, avoiding fidelity loss typically resulting from mesh decimation. 3) To automate scene creation from user prompts, we introduce VLM-based modeling agents equipped with a novel grid-based semantic analysis. This enables 3D spatial reasoning from 2D observations and ensures accurate asset placement. ImmerseGen further enhances immersion with dynamic effects and ambient audio for a multisensory experience. 4) Experiments on multiple scene-generation scenarios and live mobile VR applications show that ImmerseGen outperforms previous methods in visual quality, realism, spatial coherence, and rendering efficiency for immersive real-time VR experiences.
0:17

ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxies Contributions: 1) We propose ImmerseGen, a novel agent-guided 3D environment generation framework. It uses simplified geometric proxies with alpha-textured meshes to produce compact, photorealistic worlds ready for real-time mobile VR rendering. 2) We propose a novel RGBA texturing paradigm. It first synthesizes 8K terrain textures using a geometry-conditioned panorama generator via user-centric mapping, and then directly generates alpha-textured proxy assets, avoiding fidelity loss typically resulting from mesh decimation. 3) To automate scene creation from user prompts, we introduce VLM-based modeling agents equipped with a novel grid-based semantic analysis. This enables 3D spatial reasoning from 2D observations and ensures accurate asset placement. ImmerseGen further enhances immersion with dynamic effects and ambient audio for a multisensory experience. 4) Experiments on multiple scene-generation scenarios and live mobile VR applications show that ImmerseGen outperforms previous methods in visual quality, realism, spatial coherence, and rendering efficiency for immersive real-time VR experiences.

MrNeRF

14,225 Aufrufe • vor 1 Jahr

📢 Our lab has been exploring 3D world models for years — and we’re thrilled to share **PhysTwin**: a milestone that reconstructs object appearance, geometry, and dynamics from just a few seconds of interaction! Led by the amazing Hanxiao Jiang 👉 PhysTwin combines **Gaussian splatting** with **inverse dynamics optimization** based on simple **spring-mass** systems. ⚙️ The result? Real-time, action-conditioned 3D video prediction under novel interactions (i.e., 3D world models). 🔑 A few key takeaways: 1. Having the right structure (e.g., particles/masses) helps navigate the trade-off between sample efficiency, generalization, and broad applicability. 2. Visual foundation models (VFMs) have matured to the point where they can provide rich supervision for world modeling (e.g., tracking, shape completion). 3. Beyond VFMs, many crucial components have come together in recent years: Gaussian splats for rendering, NVIDIA Warp for high-performance simulation, and scene/asset generation from a wide range of labs and companies. The future of 3D world models is looking bright! ✨ 4. The resulting digital twin supports a wide range of downstream applications—especially in data generation and policy evaluation, thanks to its realistic rendering and simulation capabilities. 🎥 All code and data to reproduce the results, along with interactive demos, are available on the website. Check the following visualizations of: (1) observations, (2) reconstructed state/actions, (3) interactive digital twins, and (4) the overlays between real-world robot teleoperation and our model’s open-loop predictions.
0:05

📢 Our lab has been exploring 3D world models for years — and we’re thrilled to share **PhysTwin**: a milestone that reconstructs object appearance, geometry, and dynamics from just a few seconds of interaction! Led by the amazing Hanxiao Jiang 👉 PhysTwin combines **Gaussian splatting** with **inverse dynamics optimization** based on simple **spring-mass** systems. ⚙️ The result? Real-time, action-conditioned 3D video prediction under novel interactions (i.e., 3D world models). 🔑 A few key takeaways: 1. Having the right structure (e.g., particles/masses) helps navigate the trade-off between sample efficiency, generalization, and broad applicability. 2. Visual foundation models (VFMs) have matured to the point where they can provide rich supervision for world modeling (e.g., tracking, shape completion). 3. Beyond VFMs, many crucial components have come together in recent years: Gaussian splats for rendering, NVIDIA Warp for high-performance simulation, and scene/asset generation from a wide range of labs and companies. The future of 3D world models is looking bright! ✨ 4. The resulting digital twin supports a wide range of downstream applications—especially in data generation and policy evaluation, thanks to its realistic rendering and simulation capabilities. 🎥 All code and data to reproduce the results, along with interactive demos, are available on the website. Check the following visualizations of: (1) observations, (2) reconstructed state/actions, (3) interactive digital twins, and (4) the overlays between real-world robot teleoperation and our model’s open-loop predictions.

Yunzhu Li

25,279 Aufrufe • vor 1 Jahr

More coming soon ! I manage to integrate for the first time volumetric 3D video made with Kartel.ai inside the AI world generated in gaussian splatting by World Labs !! I coded all in three.js! And it's possible to integrate elements on the fly ;) And so you can imagine soon what we will produce with that en AI : consistency of character and environment, relighting etc...
0:14

More coming soon ! I manage to integrate for the first time volumetric 3D video made with Kartel.ai inside the AI world generated in gaussian splatting by World Labs !! I coded all in three.js! And it's possible to integrate elements on the fly ;) And so you can imagine soon what we will produce with that en AI : consistency of character and environment, relighting etc...

Lovis Odin

33,984 Aufrufe • vor 1 Jahr

NeuRBF: A Neural Fields Representation with Adaptive Radial Basis Functions paper page: present a novel type of neural fields that uses general radial bases for signal representation. State-of-the-art neural fields typically rely on grid-based representations for storing local neural features and N-dimensional linear kernels for interpolating features at continuous query points. The spatial positions of their neural features are fixed on grid nodes and cannot well adapt to target signals. Our method instead builds upon general radial bases with flexible kernel position and shape, which have higher spatial adaptivity and can more closely fit target signals. To further improve the channel-wise capacity of radial basis functions, we propose to compose them with multi-frequency sinusoid functions. This technique extends a radial basis to multiple Fourier radial bases of different frequency bands without requiring extra parameters, facilitating the representation of details. Moreover, by marrying adaptive radial bases with grid-based ones, our hybrid combination inherits both adaptivity and interpolation smoothness. We carefully designed weighting schemes to let radial bases adapt to different types of signals effectively. Our experiments on 2D image and 3D signed distance field representation demonstrate the higher accuracy and compactness of our method than prior arts. When applied to neural radiance field reconstruction, our method achieves state-of-the-art rendering quality, with small model size and comparable training speed.
0:15

NeuRBF: A Neural Fields Representation with Adaptive Radial Basis Functions paper page: present a novel type of neural fields that uses general radial bases for signal representation. State-of-the-art neural fields typically rely on grid-based representations for storing local neural features and N-dimensional linear kernels for interpolating features at continuous query points. The spatial positions of their neural features are fixed on grid nodes and cannot well adapt to target signals. Our method instead builds upon general radial bases with flexible kernel position and shape, which have higher spatial adaptivity and can more closely fit target signals. To further improve the channel-wise capacity of radial basis functions, we propose to compose them with multi-frequency sinusoid functions. This technique extends a radial basis to multiple Fourier radial bases of different frequency bands without requiring extra parameters, facilitating the representation of details. Moreover, by marrying adaptive radial bases with grid-based ones, our hybrid combination inherits both adaptivity and interpolation smoothness. We carefully designed weighting schemes to let radial bases adapt to different types of signals effectively. Our experiments on 2D image and 3D signed distance field representation demonstrate the higher accuracy and compactness of our method than prior arts. When applied to neural radiance field reconstruction, our method achieves state-of-the-art rendering quality, with small model size and comparable training speed.

AK

194,469 Aufrufe • vor 2 Jahren

WeatherEdit: Controllable Weather Editing with 4D Gaussian Field Contributions: 1. Based on our analysis of weather editing characteristics, we introduce WeatherEdit, a comprehensive and efficient framework for realistic and controllable weather generation. Compared with existing methods that focus on either background editing or static weather effects, a progressive 2D-to-4D transformation process in WeatherEdit enhances adaptability across a wider range of scenarios. 2. We introduce an all-in-one adapter to enable a diffusion model for multi-weather (snowy, rainy, and fog) synthesis, along with a Temporal-View attention to ensure consistent editing across multi-frame and multi-view. 3. We design a 4D Gaussian field for weather particle modeling, enabling plausible simulation of raindrops, snowflakes, and fog with controllable severity. 4. We demonstrate WeatherEdit’s effectiveness in generating realistic, consistent, and controllable weather effects in 3D driving scenes, showcasing its applicability to real-world scenarios.
0:13

WeatherEdit: Controllable Weather Editing with 4D Gaussian Field Contributions: 1. Based on our analysis of weather editing characteristics, we introduce WeatherEdit, a comprehensive and efficient framework for realistic and controllable weather generation. Compared with existing methods that focus on either background editing or static weather effects, a progressive 2D-to-4D transformation process in WeatherEdit enhances adaptability across a wider range of scenarios. 2. We introduce an all-in-one adapter to enable a diffusion model for multi-weather (snowy, rainy, and fog) synthesis, along with a Temporal-View attention to ensure consistent editing across multi-frame and multi-view. 3. We design a 4D Gaussian field for weather particle modeling, enabling plausible simulation of raindrops, snowflakes, and fog with controllable severity. 4. We demonstrate WeatherEdit’s effectiveness in generating realistic, consistent, and controllable weather effects in 3D driving scenes, showcasing its applicability to real-world scenarios.

MrNeRF

10,607 Aufrufe • vor 1 Jahr

Introducing Kaleido💮 from AI at Meta — a universal generative neural rendering engine for photorealistic, unified object and scene view synthesis. Kaleido is built on a simple but powerful design philosophy: 3D perception is a form of visual common sense. Following this idea, we formulate rendering purely as a sequence-to-sequence generation problem, successfully unifying neural rendering with the architecture principles behind modern language and video models. Unlike traditional neural rendering methods, Kaleido learns 3D purely in a data-driven way, without explicit 3D representations or structures. It acquires spatial understanding directly through large-scale video pretraining, then multi-view 3D data finetuning, inspired by how LLMs acquire textual common sense from large corpora before specialising in domains like coding. Through extensive ablations, we progressively modernised the architecture design and training strategies and tackled key scaling challenges in sequence-to-sequence generative rendering, arriving at a design that’s simple, versatile, and scalable. Kaleido significantly outperforms prior generative models in few-view settings, and remarkably is the first zero-shot generative method matches InstantNGP-level rendering quality in multi-view settings. We view Kaleido also as an alternative step towards world modeling that flexibly spans a spectrum of “realities": with many views, it faithfully reconstructs grounded reality; with fewer views, it imagines plausible unseen details. 🔗 Explore more results and paper:
0:12

Introducing Kaleido💮 from AI at Meta — a universal generative neural rendering engine for photorealistic, unified object and scene view synthesis. Kaleido is built on a simple but powerful design philosophy: 3D perception is a form of visual common sense. Following this idea, we formulate rendering purely as a sequence-to-sequence generation problem, successfully unifying neural rendering with the architecture principles behind modern language and video models. Unlike traditional neural rendering methods, Kaleido learns 3D purely in a data-driven way, without explicit 3D representations or structures. It acquires spatial understanding directly through large-scale video pretraining, then multi-view 3D data finetuning, inspired by how LLMs acquire textual common sense from large corpora before specialising in domains like coding. Through extensive ablations, we progressively modernised the architecture design and training strategies and tackled key scaling challenges in sequence-to-sequence generative rendering, arriving at a design that’s simple, versatile, and scalable. Kaleido significantly outperforms prior generative models in few-view settings, and remarkably is the first zero-shot generative method matches InstantNGP-level rendering quality in multi-view settings. We view Kaleido also as an alternative step towards world modeling that flexibly spans a spectrum of “realities": with many views, it faithfully reconstructs grounded reality; with fewer views, it imagines plausible unseen details. 🔗 Explore more results and paper:

Shikun Liu

22,315 Aufrufe • vor 9 Monaten

World Model is trending— let's revisit our HunyuanWorld journey. We’ve been pioneering open-source 3D world generation in the past two months, and this ride’s only getting started. 🌍 📅 July: HunyuanWorld 1.0 📌 First open-source 3D world model compatible with CG pipelines (Unity/Unreal/Blender) 📌 Hit 2K+ GitHub stars in just two months ⭐—thank you for the love! 📅 August: 1.0-Lite 📌Same top-tier quality, running on consumer GPUs! 📅 September: 1.0-Voyager 📌 Direct 3D output + world memory—taking exploration further! Seamlessly integrated into CG pipelines with layered 3D modeling (assets, terrain, skybox) and fully open-sourced.. we’re fully committed to building open-source spatial intelligence for all! 🚀 💡 Why it matters? ✅ Seamless CG Pipeline Integration: Export generated 3D scenes as standard mesh formats, effortlessly integrating into industry-standard tools like Blender, Unity, and Unreal Engine for direct editing, animation, and physical simulation. ✅ Hierarchical Scene Editing: Deconstruct scenes into semantic layers (sky, background, foreground objects) via instance recognition and layer decomposition, allowing for atomic-level control—independently modify, relocate, or replace objects without rebuilding the entire world. Project page: Github: Amazing creations by Stijn Spanhove camenduru GENEL | AIを用いた動画制作 apolinario 🌐 とりにく Directive Creator 🪥 👇 #AI #3DGeneration #OpenSource #WorldModels #Hunyuan3D #HunyuanWorld
0:28

World Model is trending— let's revisit our HunyuanWorld journey. We’ve been pioneering open-source 3D world generation in the past two months, and this ride’s only getting started. 🌍 📅 July: HunyuanWorld 1.0 📌 First open-source 3D world model compatible with CG pipelines (Unity/Unreal/Blender) 📌 Hit 2K+ GitHub stars in just two months ⭐—thank you for the love! 📅 August: 1.0-Lite 📌Same top-tier quality, running on consumer GPUs! 📅 September: 1.0-Voyager 📌 Direct 3D output + world memory—taking exploration further! Seamlessly integrated into CG pipelines with layered 3D modeling (assets, terrain, skybox) and fully open-sourced.. we’re fully committed to building open-source spatial intelligence for all! 🚀 💡 Why it matters? ✅ Seamless CG Pipeline Integration: Export generated 3D scenes as standard mesh formats, effortlessly integrating into industry-standard tools like Blender, Unity, and Unreal Engine for direct editing, animation, and physical simulation. ✅ Hierarchical Scene Editing: Deconstruct scenes into semantic layers (sky, background, foreground objects) via instance recognition and layer decomposition, allowing for atomic-level control—independently modify, relocate, or replace objects without rebuilding the entire world. Project page: Github: Amazing creations by Stijn Spanhove camenduru GENEL | AIを用いた動画制作 apolinario 🌐 とりにく Directive Creator 🪥 👇 #AI #3DGeneration #OpenSource #WorldModels #Hunyuan3D #HunyuanWorld

Tencent HY

20,178 Aufrufe • vor 10 Monaten

Two weeks ago I fixed one of my teeth with algorithms I wrote a couple of years ago! I got hooked by 3D scanning when I started to work for a software shop in Zurich that was programming 3D computational geometry algorithms for denture scanning to produce crowns (and more). Back then, a typical reconstruction pipeline was like: scan the patient’s teeth using an intraoral scanner, reconstruct the surface mesh, design the restoration digitally, and finally mill the crown out of ceramic. We were working mostly with point clouds and meshes, but it wasn’t just math, it was craftsmanship translated into a digital process. Every micron mattered. You could literally see how a good algorithm meant a better fit in someone’s mouth. Gaussian Splatting isn’t about surface reconstruction, it’s about appearance reconstruction. It doesn’t care about explicit topology, it captures how light interacts with the scene. In a sense, it’s the opposite philosophy of the dental world: instead of modeling what the object is, it models how the object looks. 3D Gaussian Splatting enables applications like training self driving cars, teaching robots to understand their environment, creating virtual worlds, or monitoring real sites. It represents scenes as millions of small Gaussians rendered in real time without the need for meshes or textures. Coming from a world where precision geometry was everything, this shift felt natural. It’s still about reconstruction, but with a different goal: not manufacturing a perfect object, but reproducing how the world actually looks. Two weeks ago I got my first dental crown, made with the same software, reconstruction algorithms, and Swiss precision I once helped develop. I haven’t worked there in two years, but sitting in that chair and seeing the process from the other side was a proud moment. It reminded me why I love this field.
0:18

Two weeks ago I fixed one of my teeth with algorithms I wrote a couple of years ago! I got hooked by 3D scanning when I started to work for a software shop in Zurich that was programming 3D computational geometry algorithms for denture scanning to produce crowns (and more). Back then, a typical reconstruction pipeline was like: scan the patient’s teeth using an intraoral scanner, reconstruct the surface mesh, design the restoration digitally, and finally mill the crown out of ceramic. We were working mostly with point clouds and meshes, but it wasn’t just math, it was craftsmanship translated into a digital process. Every micron mattered. You could literally see how a good algorithm meant a better fit in someone’s mouth. Gaussian Splatting isn’t about surface reconstruction, it’s about appearance reconstruction. It doesn’t care about explicit topology, it captures how light interacts with the scene. In a sense, it’s the opposite philosophy of the dental world: instead of modeling what the object is, it models how the object looks. 3D Gaussian Splatting enables applications like training self driving cars, teaching robots to understand their environment, creating virtual worlds, or monitoring real sites. It represents scenes as millions of small Gaussians rendered in real time without the need for meshes or textures. Coming from a world where precision geometry was everything, this shift felt natural. It’s still about reconstruction, but with a different goal: not manufacturing a perfect object, but reproducing how the world actually looks. Two weeks ago I got my first dental crown, made with the same software, reconstruction algorithms, and Swiss precision I once helped develop. I haven’t worked there in two years, but sitting in that chair and seeing the process from the other side was a proud moment. It reminded me why I love this field.

MrNeRF

290,140 Aufrufe • vor 8 Monaten

✨ From Mesh to 3D Gaussians - Shaded in Deferred Style ✨ By Stefano Scolari (with link to repo in the comment) When converting a mesh to 3DGS model, the Gaussians are embedded with PBR properties and normal info during conversion. This prepares them for shading, similar to traditional geometry. In this scenario, Stefano used a deferred approach for shading to demonstrate how seamlessly it translates from triangle-based pipelines. 🔥 Fun fact: The mesh-to-splat conversion only needs to occur once, but it's so fast you could actually run it every frame. Here’s the full pipeline: - Mesh-to-splat conversion pass - Depth pre-pass - Sort - Gaussian pre-pass (projective computations, etc.) - GBuffer rendering pass - Shadow pass - Deferred lighting pass This is the same classic deferred pipeline, but with Gaussians instead of triangles. Real-time relighting of converted assets? Totally possible.
0:15

✨ From Mesh to 3D Gaussians - Shaded in Deferred Style ✨ By Stefano Scolari (with link to repo in the comment) When converting a mesh to 3DGS model, the Gaussians are embedded with PBR properties and normal info during conversion. This prepares them for shading, similar to traditional geometry. In this scenario, Stefano used a deferred approach for shading to demonstrate how seamlessly it translates from triangle-based pipelines. 🔥 Fun fact: The mesh-to-splat conversion only needs to occur once, but it's so fast you could actually run it every frame. Here’s the full pipeline: - Mesh-to-splat conversion pass - Depth pre-pass - Sort - Gaussian pre-pass (projective computations, etc.) - GBuffer rendering pass - Shadow pass - Deferred lighting pass This is the same classic deferred pipeline, but with Gaussians instead of triangles. Real-time relighting of converted assets? Totally possible.

MrNeRF

19,768 Aufrufe • vor 1 Jahr

As a graphics engine coder I think when you look at a flickering bug like this one in the video below it’s not immediately obvious what is going on. The key here is observation - to study this flickering/bugged render carefully - what do we see? Firstly for me it was very obvious that nearly all of the scene shadows were flashing on and off - but (but!) there was a secondary issue where some buildings and parts of the sky were also flashing purple. Hmmmm. Interesting. I initially thought then this might be two separate bugs - but because the sky purple element could only based on full screen post fx and not 3D rendering I looked at this first with a few GPU captures to step through all our post processing to find the rendering stage which made these pixels turn purple: When I did this I found the colour 3D texture LUT grading that makes our different biomes have unique colour palettes was going very wrong - colours near 0 or 1 were wrapping and making the purple elements that we see in the said sky and base parts. The only way this could happen was if the texture was corrupt (which it was not) or if the 3D texture sampling was wrapping and not clamped as intended. That was the Eureka moment - because if the post fx had the wrong texture sampler then the disappearing shadows which also require an exact texture sampler for comparing depth might be also wrong because of the same kind of texture sampling issue! So with this idea that the engine was using the wrong texture samplers, but only in very high draw call scenes like the big base here I the looked at some engine limits and found the bug very quickly - a circular dx12 descriptor buffer for samplers running out over multiple frames, reusing the wrong data for new scenes inflight. Hence the flickering, as the GPU randomly got wrong samplers for some post textures or shadow depth. Easy to fix with triple limits for future expansion and also adding an assert/debug spam in case this limit is ever reached again - QA testers would see this message and report if they ever saw a flicker with this style of bug. My bug and my bad from 2017 porting NMS to DX12 without foreseeing how massively complex bases and our game would grow.
0:14

As a graphics engine coder I think when you look at a flickering bug like this one in the video below it’s not immediately obvious what is going on. The key here is observation - to study this flickering/bugged render carefully - what do we see? Firstly for me it was very obvious that nearly all of the scene shadows were flashing on and off - but (but!) there was a secondary issue where some buildings and parts of the sky were also flashing purple. Hmmmm. Interesting. I initially thought then this might be two separate bugs - but because the sky purple element could only based on full screen post fx and not 3D rendering I looked at this first with a few GPU captures to step through all our post processing to find the rendering stage which made these pixels turn purple: When I did this I found the colour 3D texture LUT grading that makes our different biomes have unique colour palettes was going very wrong - colours near 0 or 1 were wrapping and making the purple elements that we see in the said sky and base parts. The only way this could happen was if the texture was corrupt (which it was not) or if the 3D texture sampling was wrapping and not clamped as intended. That was the Eureka moment - because if the post fx had the wrong texture sampler then the disappearing shadows which also require an exact texture sampler for comparing depth might be also wrong because of the same kind of texture sampling issue! So with this idea that the engine was using the wrong texture samplers, but only in very high draw call scenes like the big base here I the looked at some engine limits and found the bug very quickly - a circular dx12 descriptor buffer for samplers running out over multiple frames, reusing the wrong data for new scenes inflight. Hence the flickering, as the GPU randomly got wrong samplers for some post textures or shadow depth. Easy to fix with triple limits for future expansion and also adding an assert/debug spam in case this limit is ever reached again - QA testers would see this message and report if they ever saw a flicker with this style of bug. My bug and my bad from 2017 porting NMS to DX12 without foreseeing how massively complex bases and our game would grow.

Martin Griffiths

72,701 Aufrufe • vor 1 Jahr

$KNDX 🤖 Theres 3 big narratives that are sending coins left right and centre rn. 🚀 #AI, #Gamefi, & #NFTs 🔹Theres 50% mindshare for #AI. 🤖 🔹#GameFi mcap is hitting ATH's with #OfftheGrid, $XBG and $SUPER making spectacular moves. 🎮 🔹NFTs and the #Metaverse are making a strong comeback with $APE up 100% over the weekend. 🐵 What if there's a project that touches all these trending narratives with groundbreaking technology to disrupt all 3 of them? 🔥 💡- That's where $KNDX comes in. -💡 Kondux is a cutting-edge Web3 SaaS platform, combining NVIDIA’s Omniverse, AI, Blockchain, and dynamic NFTs to revolutionize secure asset management across industries. 👏 Their flagship product, kNFTs, are 3D digital assets usable across Metaverse and Gaming platforms, AR/VR/XR environments, and manufacturing applications. Kondux’s scalable model opens new revenue streams by enabling effective digital asset monetization. 💰 Kondux is the first Web3 project to integrate VFX pipelines with NVIDIA’s Omniverse and bringing it onto the Blockchain. ⛓️ It is also the only Web3 project with a *Select Status Partnership* with NVIDIA, operating under NVIDIA NDAs and working with them directly for more than 2 years. About their NVIDIA Integrations: 🤖 🔹There are three areas of the Kondux tech stack that coincide with three divisions of NVIDIA: 📡GDN (Graphics Delivery Network, the backbone of GeForce Now) 💡Omniverse for 3D aspects such as, geospatial data, real world physics, lighting, and raytracing 🤖NVIDIA AI Foundation, which covers many aspects of #AI, including inference and deployment scaling. The convergence of all these components lie within .USD file format . 🔹 They are the first blockchain project to integrate NVIDIA’s Omniverse Cloud and Graphics Delivery Network (GDN) to provide high-quality 3D content accessible on any device without requiring high-end hardware. 🔹 This setup streamlines content management, democratises access to resource-intensive 3D content, and enables real-time interaction with 3D NFTs. Now, I haven’t seen any crypto project so deeply connected with NVIDIA and NVIDIA technology. GDN is a HUGE competitive advantage. With it, the need for #GPU’s basically goes out the window. 🤯 Now lets take a look at some of the other main features... 👀 OpenUSD (Universal Scene Description): 📽️ 🔹 Kondux is leveraging USD technology, developed by Pixar and used by Meta, Apple, Microsoft and other industry leaders to enhance 3D graphics and interoperability within its creative ecosystem. 🔹 Originally created for high-end film production, USD now supports a variety of applications, including gaming and virtual reality, making it a key asset for Kondux. kNFT's: 🎨 🔹 Kondux is pioneering a new category of NFTs known as kNFTs, which aim to redefine NFT utility through innovative features. 🔹 A standout feature is the upgradeable aspect provided by Kondux DNA, allowing kNFTs to transform and combine with other NFTs, creating limitless possibilities in art, gaming, and music. 🔹Through the Kondux AI portal it will be possible to communicate with kNFTs. They can learn and adapt. This AI technology is revolutionary because it makes human to kNFT interaction possible, turning it into a unique, personalized experience. Check out the clip of kNFTs in Unreal Engine 5 gameplay below. 👇 Kondux is a very obvious utility play with huge upside because it’s multi narrative. 📈 It's seriously groundbreaking stuff that they’re about to launch. 🚀 After speaking with the team there’s no doubt in my mind this will do crazy big numbers in the next months. 🤑
0:17

$KNDX 🤖 Theres 3 big narratives that are sending coins left right and centre rn. 🚀 #AI, #Gamefi, & #NFTs 🔹Theres 50% mindshare for #AI. 🤖 🔹#GameFi mcap is hitting ATH's with #OfftheGrid, $XBG and $SUPER making spectacular moves. 🎮 🔹NFTs and the #Metaverse are making a strong comeback with $APE up 100% over the weekend. 🐵 What if there's a project that touches all these trending narratives with groundbreaking technology to disrupt all 3 of them? 🔥 💡- That's where $KNDX comes in. -💡 Kondux is a cutting-edge Web3 SaaS platform, combining NVIDIA’s Omniverse, AI, Blockchain, and dynamic NFTs to revolutionize secure asset management across industries. 👏 Their flagship product, kNFTs, are 3D digital assets usable across Metaverse and Gaming platforms, AR/VR/XR environments, and manufacturing applications. Kondux’s scalable model opens new revenue streams by enabling effective digital asset monetization. 💰 Kondux is the first Web3 project to integrate VFX pipelines with NVIDIA’s Omniverse and bringing it onto the Blockchain. ⛓️ It is also the only Web3 project with a *Select Status Partnership* with NVIDIA, operating under NVIDIA NDAs and working with them directly for more than 2 years. About their NVIDIA Integrations: 🤖 🔹There are three areas of the Kondux tech stack that coincide with three divisions of NVIDIA: 📡GDN (Graphics Delivery Network, the backbone of GeForce Now) 💡Omniverse for 3D aspects such as, geospatial data, real world physics, lighting, and raytracing 🤖NVIDIA AI Foundation, which covers many aspects of #AI, including inference and deployment scaling. The convergence of all these components lie within .USD file format . 🔹 They are the first blockchain project to integrate NVIDIA’s Omniverse Cloud and Graphics Delivery Network (GDN) to provide high-quality 3D content accessible on any device without requiring high-end hardware. 🔹 This setup streamlines content management, democratises access to resource-intensive 3D content, and enables real-time interaction with 3D NFTs. Now, I haven’t seen any crypto project so deeply connected with NVIDIA and NVIDIA technology. GDN is a HUGE competitive advantage. With it, the need for #GPU’s basically goes out the window. 🤯 Now lets take a look at some of the other main features... 👀 OpenUSD (Universal Scene Description): 📽️ 🔹 Kondux is leveraging USD technology, developed by Pixar and used by Meta, Apple, Microsoft and other industry leaders to enhance 3D graphics and interoperability within its creative ecosystem. 🔹 Originally created for high-end film production, USD now supports a variety of applications, including gaming and virtual reality, making it a key asset for Kondux. kNFT's: 🎨 🔹 Kondux is pioneering a new category of NFTs known as kNFTs, which aim to redefine NFT utility through innovative features. 🔹 A standout feature is the upgradeable aspect provided by Kondux DNA, allowing kNFTs to transform and combine with other NFTs, creating limitless possibilities in art, gaming, and music. 🔹Through the Kondux AI portal it will be possible to communicate with kNFTs. They can learn and adapt. This AI technology is revolutionary because it makes human to kNFT interaction possible, turning it into a unique, personalized experience. Check out the clip of kNFTs in Unreal Engine 5 gameplay below. 👇 Kondux is a very obvious utility play with huge upside because it’s multi narrative. 📈 It's seriously groundbreaking stuff that they’re about to launch. 🚀 After speaking with the team there’s no doubt in my mind this will do crazy big numbers in the next months. 🤑

Altcoin Miyagi🇯🇵

17,303 Aufrufe • vor 1 Jahr

i built an open world minigame that's controlled with hand movements only here's a step-by-step tutorial: > started with my mediapipe + threejs template (see QT) > added a 3D model made by quaternius > sent a few prompts to gemini 3... --- prompt #1: repurpose the attached script, but now I want to import a gltf model (assets/model.gltf) and use that in the scene instead of the cube when the user moves their hand, a waypoint indicator should move around the scene, and the 3D model should smoothly move there when the user makes a fist, the model should jump the 3D model contains bundled animations in it. use the "idle", "run", and "jump" animations --- prompt #2: make this a procedural open world adventure. generate very simple procedural voxel terrain when the model gets near the edges of the screen, the camera should move, allowing the model to keep going in that direction --- prompt #3: add some floating glowing gems around the map that the user can collect --- i made some manual tweaks for styling and game feel, but that's the gist of it thanks for reading if you got all the way here full code is available at my link in bio
0:15

i built an open world minigame that's controlled with hand movements only here's a step-by-step tutorial: > started with my mediapipe + threejs template (see QT) > added a 3D model made by quaternius > sent a few prompts to gemini 3... --- prompt #1: repurpose the attached script, but now I want to import a gltf model (assets/model.gltf) and use that in the scene instead of the cube when the user moves their hand, a waypoint indicator should move around the scene, and the 3D model should smoothly move there when the user makes a fist, the model should jump the 3D model contains bundled animations in it. use the "idle", "run", and "jump" animations --- prompt #2: make this a procedural open world adventure. generate very simple procedural voxel terrain when the model gets near the edges of the screen, the camera should move, allowing the model to keep going in that direction --- prompt #3: add some floating glowing gems around the map that the user can collect --- i made some manual tweaks for styling and game feel, but that's the gist of it thanks for reading if you got all the way here full code is available at my link in bio

AA

131,799 Aufrufe • vor 7 Monaten

The architecture of this new world model is one of the most interesting things I've seen lately: Let me first explain how most world models work: They predict and render one frame at a time. If you are navigating in one of these worlds, and you look left, the model draws whatever looks right in the moment. Every time you change your viewpoint, the model has to imagine what should be there again, so it's very common for these models to "forget" what's in the world. For example, if you put a toy on the table, look away, then look back, the toy might not be there anymore. Tripo AI is releasing its Project Eden model, which works very differently: The model builds the world first, and then renders it based on that map. That map holds the real state of the world: the geometry, every object, where things are, what's already happened. The picture you see on screen gets generated from the map. This architecture flips the whole thing. Now, you get the following: 1. The world stops forgetting. Leave, come back, and the toy is still on the table because it lives in the map, not in the last frame you saw. 2. You can edit the world, and those changes persist for anyone who enters later. 3. Multiple people and AI agents can coexist in the world and see it from different perspectives. This is early research, but it's looking really promising. They just raised nearly $200M across two rounds to build it out. Tripo will be at SIGGRAPH 2026 (July 19–23, Los Angeles Convention Center). If you work in 3D, embodied AI, simulation, or anything spatial, go connect with them there.
0:10

The architecture of this new world model is one of the most interesting things I've seen lately: Let me first explain how most world models work: They predict and render one frame at a time. If you are navigating in one of these worlds, and you look left, the model draws whatever looks right in the moment. Every time you change your viewpoint, the model has to imagine what should be there again, so it's very common for these models to "forget" what's in the world. For example, if you put a toy on the table, look away, then look back, the toy might not be there anymore. Tripo AI is releasing its Project Eden model, which works very differently: The model builds the world first, and then renders it based on that map. That map holds the real state of the world: the geometry, every object, where things are, what's already happened. The picture you see on screen gets generated from the map. This architecture flips the whole thing. Now, you get the following: 1. The world stops forgetting. Leave, come back, and the toy is still on the table because it lives in the map, not in the last frame you saw. 2. You can edit the world, and those changes persist for anyone who enters later. 3. Multiple people and AI agents can coexist in the world and see it from different perspectives. This is early research, but it's looking really promising. They just raised nearly $200M across two rounds to build it out. Tripo will be at SIGGRAPH 2026 (July 19–23, Los Angeles Convention Center). If you work in 3D, embodied AI, simulation, or anything spatial, go connect with them there.

Santiago

30,189 Aufrufe • vor 24 Tagen

Colmap 4.0 was very recently released, so it inspired me to do some work to better understand it and its new capabilities with Rerun. I want to really understand how Colmap, and in particular, pycolmap, works outside of just calling it via the CLI. So my goal is to use the low-level pycolmap API to log every part of the pipeline. The explicit goal is to have an alternative to the SQLite database that I can utilize. Instead of SQLite, I want to try logging everything directly to rerun and use RRD. This means I can have deep inspectability and still save the features/matches/2D view geometry, but be able to view it directly in rerun. I think this is one of the superpowers that rerun provides; data and visualizations are deeply integrated. As I'm often working with sequential data (videos), I'm going to specifically focus on four things: 1. Monocular Video Simple: Calls high-level APIs such as pycolmap.extract_features, pycolmap.match_sequential, pycolmap.incremental_mapping. These are basically identical to the CLI options and provide a good baseline. 2. Monocular Video Streamed: Take the above high-level APIs and break them down to their iterator version, logging each component in a streamed manner. This way, I can stream the intermediate features to rerun while the extraction/matching/mapping is happening. 3. Rig with unknown calibration: <- WHAT THE VIDEO SHOWS This is probably the most interesting version and the first one I've been working on. It allows one to set a rig between known sensors, such as in VR/AR devices, leading to much better reconstructions with multiple cameras. This is the case where we don't know the calibration a priori, so we have to run a reconstruction twice: once as a normal Colmap reconstruction with no rig constraints, use this to generate the constraints, and then do it again with the newly found rig. 4. Rig with known calibration: This is the RoboCap example, where we have a pre-calibrated set of sensors, so we don't need to run the two reconstructions and also gain better matching between cameras, both spatially and temporally. Again, this leads to a much better reconstruction! Along with all this, GLOMAP has become a first-class global mapper, making it super easy to use directly within pycolmap! I'm excited to do more with this and compare it to things like pycuvslam, vipe, and other alternatives.
0:20

Colmap 4.0 was very recently released, so it inspired me to do some work to better understand it and its new capabilities with Rerun. I want to really understand how Colmap, and in particular, pycolmap, works outside of just calling it via the CLI. So my goal is to use the low-level pycolmap API to log every part of the pipeline. The explicit goal is to have an alternative to the SQLite database that I can utilize. Instead of SQLite, I want to try logging everything directly to rerun and use RRD. This means I can have deep inspectability and still save the features/matches/2D view geometry, but be able to view it directly in rerun. I think this is one of the superpowers that rerun provides; data and visualizations are deeply integrated. As I'm often working with sequential data (videos), I'm going to specifically focus on four things: 1. Monocular Video Simple: Calls high-level APIs such as pycolmap.extract_features, pycolmap.match_sequential, pycolmap.incremental_mapping. These are basically identical to the CLI options and provide a good baseline. 2. Monocular Video Streamed: Take the above high-level APIs and break them down to their iterator version, logging each component in a streamed manner. This way, I can stream the intermediate features to rerun while the extraction/matching/mapping is happening. 3. Rig with unknown calibration: <- WHAT THE VIDEO SHOWS This is probably the most interesting version and the first one I've been working on. It allows one to set a rig between known sensors, such as in VR/AR devices, leading to much better reconstructions with multiple cameras. This is the case where we don't know the calibration a priori, so we have to run a reconstruction twice: once as a normal Colmap reconstruction with no rig constraints, use this to generate the constraints, and then do it again with the newly found rig. 4. Rig with known calibration: This is the RoboCap example, where we have a pre-calibrated set of sensors, so we don't need to run the two reconstructions and also gain better matching between cameras, both spatially and temporally. Again, this leads to a much better reconstruction! Along with all this, GLOMAP has become a first-class global mapper, making it super easy to use directly within pycolmap! I'm excited to do more with this and compare it to things like pycuvslam, vipe, and other alternatives.

Pablo Vela

30,070 Aufrufe • vor 3 Monaten

🚀 The Segment Anything Model (SAM) has been upgraded to SAM2, featuring an efficient image encoder for segmenting images and videos. But does SAM2 outperform SAM1 in medical image and video segmentation? We're thrilled to present our paper "Segment Anything in Medical Images and Videos: Benchmark and Deployment"! We comprehensively benchmark SAM2 across 11 medical image modalities and videos. 📄 Paper: 💻 Code: **Highlights:** 1. SAM2 doesn’t always outperform SAM1 in 2D medical images, but excels in video segmentation, making it more accurate and efficient for 3D images, such as CT and MR scans. 2. MedSAM still outperforms SAM2 on most 2D modalities, but SAM2 surpasses MedSAM for 3D image segmentation in a slice-by-slice approach. 3. Segmentation performance varies with model size; sometimes the smallest model outperforms larger ones. 4. Fine-tuning SAM2 significantly boosts its performance for medical image segmentation. While SAM2 may struggle with challenging objects that have unclear boundaries or low contrast, it excels in generating good initial segmentation masks for common medical images and videos. However, the official interface doesn’t support medical data formats and has limitations on video length. To address this, we've developed a 3D Slicer Plugin and Gradio API for efficient 3D medical image and video segmentation. We invite you to try them out and provide feedback! 🔧 Deployment: - 3D Slicer Plugin: - Gradio API: (Note: Due to GPU limitations, the online API is available for only 12 hours and may be slow. We highly recommend deploying the Gradio API with your own computing resources: A big shoutout to Jun Ma (JunMa) who recently joined our UHN AI hub (UHN AI Hub) as Machine Learning Lead, and kudos to all co-authors: Sumin Kim, Feifei Li, Mohammed Baharoon (Mohammed Baharoon), Reza Asakereh, and Hongwei Lyu! This is true teamwork! Looking forward to collaborating with the community to advance 3D medical image and video segmentation foundation models! University Health Network U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology Temerty Centre for AI in Medicine (T-CAIREM) Vector Institute #MedTech #AIinHealthcare #DeepLearning #MedicalImaging #SAM2 #MedSAM #AIResearch
0:22

🚀 The Segment Anything Model (SAM) has been upgraded to SAM2, featuring an efficient image encoder for segmenting images and videos. But does SAM2 outperform SAM1 in medical image and video segmentation? We're thrilled to present our paper "Segment Anything in Medical Images and Videos: Benchmark and Deployment"! We comprehensively benchmark SAM2 across 11 medical image modalities and videos. 📄 Paper: 💻 Code: **Highlights:** 1. SAM2 doesn’t always outperform SAM1 in 2D medical images, but excels in video segmentation, making it more accurate and efficient for 3D images, such as CT and MR scans. 2. MedSAM still outperforms SAM2 on most 2D modalities, but SAM2 surpasses MedSAM for 3D image segmentation in a slice-by-slice approach. 3. Segmentation performance varies with model size; sometimes the smallest model outperforms larger ones. 4. Fine-tuning SAM2 significantly boosts its performance for medical image segmentation. While SAM2 may struggle with challenging objects that have unclear boundaries or low contrast, it excels in generating good initial segmentation masks for common medical images and videos. However, the official interface doesn’t support medical data formats and has limitations on video length. To address this, we've developed a 3D Slicer Plugin and Gradio API for efficient 3D medical image and video segmentation. We invite you to try them out and provide feedback! 🔧 Deployment: - 3D Slicer Plugin: - Gradio API: (Note: Due to GPU limitations, the online API is available for only 12 hours and may be slow. We highly recommend deploying the Gradio API with your own computing resources: A big shoutout to Jun Ma (JunMa) who recently joined our UHN AI hub (UHN AI Hub) as Machine Learning Lead, and kudos to all co-authors: Sumin Kim, Feifei Li, Mohammed Baharoon (Mohammed Baharoon), Reza Asakereh, and Hongwei Lyu! This is true teamwork! Looking forward to collaborating with the community to advance 3D medical image and video segmentation foundation models! University Health Network U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology Temerty Centre for AI in Medicine (T-CAIREM) Vector Institute #MedTech #AIinHealthcare #DeepLearning #MedicalImaging #SAM2 #MedSAM #AIResearch

Bo Wang

178,481 Aufrufe • vor 1 Jahr