Загрузка видео...

Не удалось загрузить видео

На главную

Semantically annotating 3D gaussian splats on the fly using gemini 3.1 + sparkjs 1. Load any 3D scene and hit scan 2. Get 2D detections from VLM 3. Cluster outputs & project into 3D world space 4. Save as a persistent 3D semantic layer Inspired by Alexander Chen's experiments...

bilawalsidhu's profile picture

Bilawal Sidhu

106,566 subscribers

56,125 просмотров • 27 дней назад •via X (Twitter)

ИскусствоОбразованиеНаука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

LLaVA-3D A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D-awareness for 3D scene understanding has been hindered by the lack of large-scale 3D vision-language datasets and powerful 3D encoders. In this paper, we introduce a simple yet effective framework called LLaVA-3D. Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D efficiently adapts LLaVA for 3D scene understanding without compromising 2D understanding capabilities. To achieve this, we employ a simple yet effective representation, 3D Patch, which connects 2D CLIP patch features with their corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs and employing joint 2D and 3D vision-language instruction tuning, we establish a unified architecture for both 2D image understanding and 3D scene understanding. Experimental results show that LLaVA-3D converges 3.5x faster than existing 3D LMMs when trained on 3D vision-language datasets. Moreover, LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks but also maintains comparable 2D image understanding and vision-language conversation capabilities with LLaVA.
0:31

LLaVA-3D A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D-awareness for 3D scene understanding has been hindered by the lack of large-scale 3D vision-language datasets and powerful 3D encoders. In this paper, we introduce a simple yet effective framework called LLaVA-3D. Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D efficiently adapts LLaVA for 3D scene understanding without compromising 2D understanding capabilities. To achieve this, we employ a simple yet effective representation, 3D Patch, which connects 2D CLIP patch features with their corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs and employing joint 2D and 3D vision-language instruction tuning, we establish a unified architecture for both 2D image understanding and 3D scene understanding. Experimental results show that LLaVA-3D converges 3.5x faster than existing 3D LMMs when trained on 3D vision-language datasets. Moreover, LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks but also maintains comparable 2D image understanding and vision-language conversation capabilities with LLaVA.

AK

41,713 просмотров • 1 год назад

📢WorldAgents: 3D worlds only from 2D image models - without any training! We propose an agentic approach with a Director (VLM) to plan the scene, a Generator (Flux or NanoBanana) for new views, and a Verifier (VLM) for selection / 3D consistency. -> High-fidelity 3D worlds from a single text prompt. What's remarkable: our agents find consistent views from 2D image models to obtain 3D-consistent worlds; this shows that image models contain world priors - agents just need to find them! Great work by Ziya Erkoç Angela Dai
0:25

📢WorldAgents: 3D worlds only from 2D image models - without any training! We propose an agentic approach with a Director (VLM) to plan the scene, a Generator (Flux or NanoBanana) for new views, and a Verifier (VLM) for selection / 3D consistency. -> High-fidelity 3D worlds from a single text prompt. What's remarkable: our agents find consistent views from 2D image models to obtain 3D-consistent worlds; this shows that image models contain world priors - agents just need to find them! Great work by Ziya Erkoç Angela Dai

Matthias Niessner

18,854 просмотров • 2 месяцев назад

Turning movies shots into 3D scenes using 3D Gaussian Splats 🤯 Next experiment will be about taking footages from movies, turning them in 3D and then take them into Unreal Engine to add/edit. Remember this scene ?
0:20

Turning movies shots into 3D scenes using 3D Gaussian Splats 🤯 Next experiment will be about taking footages from movies, turning them in 3D and then take them into Unreal Engine to add/edit. Remember this scene ?

Bad Decisions Studio

742,879 просмотров • 2 лет назад

Another AI Experiment. Using SHARP, Apple's ML framework - magical stuff. 2D Images to 3D Gaussian splats. Plotted and flown in Three.js
0:57

Another AI Experiment. Using SHARP, Apple's ML framework - magical stuff. 2D Images to 3D Gaussian splats. Plotted and flown in Three.js

M. Ali Kapadia

26,845 просмотров • 3 месяцев назад

The challenge of creating this scene was from a 3D environment to a 2D painted environment. We used the character to bait-and-switch out the backgrounds. #2d #3d #animation
0:34

The challenge of creating this scene was from a 3D environment to a 2D painted environment. We used the character to bait-and-switch out the backgrounds. #2d #3d #animation

THE LINE

696,289 просмотров • 3 лет назад

shader effects for 3d gaussian splats are a fun way to add reality bending effects to your 3d scans. made using sparkjs which is an open source 3dgs renderer for threejs.
0:40

shader effects for 3d gaussian splats are a fun way to add reality bending effects to your 3d scans. made using sparkjs which is an open source 3dgs renderer for threejs.

Bilawal Sidhu

45,202 просмотров • 3 месяцев назад

What could be better than a portal?! A portal that lets you switch between point clouds and 3D Gaussian Splats. Built a web-based 3D Gaussian Splat viewer with SparkJS, letting me freely fly through the three locations I just captured using the #PortalCam. #GaussianSplatting
1:33

What could be better than a portal?! A portal that lets you switch between point clouds and 3D Gaussian Splats. Built a web-based 3D Gaussian Splat viewer with SparkJS, letting me freely fly through the three locations I just captured using the #PortalCam. #GaussianSplatting

Hugues Bruyère

28,092 просмотров • 4 месяцев назад

You can build your 3D models directly from a prompt into Claude code. All thanks to this beautiful python project, build123 a parametric modeling framework for 2D and 3D CAD. It is built on the Open Cascade geometric kernel and it is suitable for 3D printing.
0:36

You can build your 3D models directly from a prompt into Claude code. All thanks to this beautiful python project, build123 a parametric modeling framework for 2D and 3D CAD. It is built on the Open Cascade geometric kernel and it is suitable for 3D printing.

Marco Franzon

20,743 просмотров • 2 месяцев назад

3D Gaussian Splats as an art form? 🎨 Beautiful scan by Eric Paré. Check it out on SuperSplat:
0:33

3D Gaussian Splats as an art form? 🎨 Beautiful scan by Eric Paré. Check it out on SuperSplat:

Will Eastcott

14,905 просмотров • 1 год назад

SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections abs: project page:
0:58

SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections abs: project page:

AK

317,619 просмотров • 3 лет назад

Robots that navigate grocery stores with semantic 3D maps created by language-augmented gaussian splats
0:26

Robots that navigate grocery stores with semantic 3D maps created by language-augmented gaussian splats

Towaki Takikawa / 瀧川永遠希

48,814 просмотров • 9 месяцев назад

Virtual production democratized. 3D scan any space or place, reconstruct it using 3D Gaussian Splatting (eg Polycam) and use it as an environment you can film in — all using the iPhone in your pocket 🤯
0:39

Virtual production democratized. 3D scan any space or place, reconstruct it using 3D Gaussian Splatting (eg Polycam) and use it as an environment you can film in — all using the iPhone in your pocket 🤯

Bilawal Sidhu

32,189 просмотров • 1 год назад

🤯 Gemini 3.1 Pro Google DeepMind generates parametric 3D models straight from images! 🚀Paired with Rodin Gen-2 Edit's Refine, #Gemini effortlessly delivers million-face detailed #3D models. 🔥3D Generation workflows are evolving! See it in action 👇 #CG #Hyper3D
0:39

🤯 Gemini 3.1 Pro Google DeepMind generates parametric 3D models straight from images! 🚀Paired with Rodin Gen-2 Edit's Refine, #Gemini effortlessly delivers million-face detailed #3D models. 🔥3D Generation workflows are evolving! See it in action 👇 #CG #Hyper3D

Hyper3D by Deemos

77,496 просмотров • 3 месяцев назад

3D Animator Nicolas Roth and Digital Artist Boris have combined 2D and 3D techniques to achieve a truly unique art style in their latest mixed media project. Breakdown of the project: #3danimation #2danimation #2d #3d #3dcg #vfx #b3d #blender #mixedmedia
0:24

3D Animator Nicolas Roth and Digital Artist Boris have combined 2D and 3D techniques to achieve a truly unique art style in their latest mixed media project. Breakdown of the project: #3danimation #2danimation #2d #3d #3dcg #vfx #b3d #blender #mixedmedia

80 LEVEL

57,988 просмотров • 1 год назад

Combining 2D and 3D in After Effects. In Foldspace, not every surface needs to be bent. Keep one flat for 2D artwork, and let the others exist in 3D space. This allows you to combine 2D and 3D objects in a single animation with no masks or tricks needed.
0:34

Combining 2D and 3D in After Effects. In Foldspace, not every surface needs to be bent. Keep one flat for 2D artwork, and let the others exist in 3D space. This allows you to combine 2D and 3D objects in a single animation with no masks or tricks needed.

Motion Princess

21,588 просмотров • 2 месяцев назад

HoloDreamer can generate enclosed 3D scenes from text descriptions! It does so by first creating a high-quality equirectangular panorama and then rapidly reconstructing the 3D scene using 3D Gaussian Splatting. Links ⬇️
0:36

HoloDreamer can generate enclosed 3D scenes from text descriptions! It does so by first creating a high-quality equirectangular panorama and then rapidly reconstructing the 3D scene using 3D Gaussian Splatting. Links ⬇️

Dreaming Tulpa 🥓👑

59,361 просмотров • 1 год назад

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.
0:50

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

249,494 просмотров • 2 лет назад

Amodal3R: Amodal 3D Reconstruction from Occluded 2D Images TL;DR: Given partially visible objects within images, Amodal3R reconstructs semantically meaningful 3D assets with reasonable geometry and plausible appearance.
1:16

Amodal3R: Amodal 3D Reconstruction from Occluded 2D Images TL;DR: Given partially visible objects within images, Amodal3R reconstructs semantically meaningful 3D assets with reasonable geometry and plausible appearance.

Alexandre Morgand

23,300 просмотров • 1 год назад

Instant Video-to-3D with DUST3R Dust3r generates a whole 3D scene from just a couple of images. What if it could: 1. Accept a VIDEO 2. Extract the video frames 3. Turn them into 3D? So added a Gradio Video Component. Here's me generating 3D from a video cc: NAVER LABS Europe
1:13

Instant Video-to-3D with DUST3R Dust3r generates a whole 3D scene from just a couple of images. What if it could: 1. Accept a VIDEO 2. Extract the video frames 3. Turn them into 3D? So added a Gradio Video Component. Here's me generating 3D from a video cc: NAVER LABS Europe

cocktail peanut

41,029 просмотров • 2 лет назад

The transition from 2D animation to 3D was really amazing #Advent3D
0:24

The transition from 2D animation to 3D was really amazing #Advent3D

やよい軒🎲💎

1,091,478 просмотров • 1 год назад