Fun weekend project: using AI at Meta Sam3 to segment objects inside a SpAItial AI scene.
@risingodegua8.9K followers
22,552 views
19 Comments
Comments from the original post on X (Twitter)

@risingodegua
Link to repo:
14 likes

@SDBarrington
@AIatMeta @SpAItial_AI @playcanvas this is awesome
1 like

@chrisrogers3d
@AIatMeta @SpAItial_AI @playcanvas supercool, spatial AI is amazing. thanks for sharing. i used sam3 to do something similar in blender
1 like

@risingodegua
@AIatMeta @SpAItial_AI @playcanvas That’s so cool. What tool is that?

@chrisrogers3d
@AIatMeta @SpAItial_AI @playcanvas its a Blender addon that uses SAM3 via ComfyUI in the background to make masks in 2d and find the matching triangles.
3 likes

@risingodegua
@AIatMeta @SpAItial_AI @playcanvas Nice!

@jaxxchen003
@AIatMeta @SpAItial_AI @playcanvas 那这个能力是不是可以实现对3DGS里边不同物体的二次编辑?比如删除、替换。
1 like

@risingodegua
@AIatMeta @SpAItial_AI @playcanvas That’s the next step. For that you need to be able to segment and lift out objects (specific gaussians) and be able to replace/repair after you remove. Probably need something like SAM3D
1 like

@jaxxchen003
@AIatMeta @SpAItial_AI @playcanvas ok,I got it

@JustJerry121
@AIatMeta @SpAItial_AI @playcanvas The mask-to-scene lift is the clever bit here.
1 like

@daniel_esco01
@janheld14 @AIatMeta @SpAItial_AI @playcanvas Soo cool
1 like

@akhil_kestur
@AIatMeta @SpAItial_AI @playcanvas what's the segmentation accuracy like once objects start overlapping in the splat?

@risingodegua
@AIatMeta @SpAItial_AI @playcanvas SAM3 is quite good! Give it go
2 likes

@VaishabhJ
@AIatMeta @SpAItial_AI @playcanvas How to work on this project?

@risingodegua
@AIatMeta @SpAItial_AI @playcanvas Link to repo, with readme and steps :

@EleonoreStelzer
@AIatMeta @SpAItial_AI @playcanvas That pipeline sounds pleasantly overengineered in the best way — I’d love to see how clean the segmentation gets on moving camera frames 🙂

@risingodegua
@AIatMeta @SpAItial_AI @playcanvas The 2D mask was transferred to the 3D space. So it stays when you move. Not perfect though, but it currently works and can be improved

@0xkrisv
@AIatMeta @SpAItial_AI @playcanvas wen spaital api?

@risingodegua
@AIatMeta @SpAItial_AI @playcanvas @SpAItial_AI API is out
3 likes