Skip navigation
Fun weekend project: using AI at Meta Sam3 to segment objects inside a SpAItial AI scene.
0:00 / 1:17

Fun weekend project: using AI at Meta Sam3 to segment objects inside a SpAItial AI scene.

RO2⚡️

@risingodegua8.9K followers

Visit on X
368 likesDownloadOriginal post

22,552 views

Fun weekend project: using AI at Meta Sam3 to segment objects inside a SpAItial AI scene. Pipeline: -> Generate a world on SpAItial AI app and export as .ply -> PlayCanvas viewer to render splat + a uniform spatial grid over splat centers -> Capture frames + camera pose, POST to a local SAM3 server -> Lift each 2D mask → 3D: frustum-cull splats via the grid, reproject with the pose -> Recolor / isolate the segmented Gaussians This is all running locally on my Mac. Static scene, segment once, reuse persisted mask. All open source (link in comment), you can clone and build on top of this 👇🏽

19 Comments

Comments from the original post on X (Twitter)

  • RO2⚡️'s profile picture

    @risingodegua

    Link to repo:

    14 likes

  • Scott Barrington's profile picture

    @SDBarrington

    @AIatMeta @SpAItial_AI @playcanvas this is awesome

    1 like

  • chris rogers's profile picture

    @chrisrogers3d

    @AIatMeta @SpAItial_AI @playcanvas supercool, spatial AI is amazing. thanks for sharing. i used sam3 to do something similar in blender

    1 like

  • RO2⚡️'s profile picture

    @risingodegua

    @AIatMeta @SpAItial_AI @playcanvas That’s so cool. What tool is that?

  • chris rogers's profile picture

    @chrisrogers3d

    @AIatMeta @SpAItial_AI @playcanvas its a Blender addon that uses SAM3 via ComfyUI in the background to make masks in 2d and find the matching triangles.

    3 likes

  • RO2⚡️'s profile picture

    @risingodegua

    @AIatMeta @SpAItial_AI @playcanvas Nice!

  • 硅基练习生 JAXX's profile picture

    @jaxxchen003

    @AIatMeta @SpAItial_AI @playcanvas 那这个能力是不是可以实现对3DGS里边不同物体的二次编辑?比如删除、替换。

    1 like

  • RO2⚡️'s profile picture

    @risingodegua

    @AIatMeta @SpAItial_AI @playcanvas That’s the next step. For that you need to be able to segment and lift out objects (specific gaussians) and be able to replace/repair after you remove. Probably need something like SAM3D

    1 like

  • 硅基练习生 JAXX's profile picture

    @jaxxchen003

    @AIatMeta @SpAItial_AI @playcanvas ok,I got it

  • JustJerry's profile picture

    @JustJerry121

    @AIatMeta @SpAItial_AI @playcanvas The mask-to-scene lift is the clever bit here.

    1 like

  • Daniel's profile picture

    @daniel_esco01

    @janheld14 @AIatMeta @SpAItial_AI @playcanvas Soo cool

    1 like

  • AK's profile picture

    @akhil_kestur

    @AIatMeta @SpAItial_AI @playcanvas what's the segmentation accuracy like once objects start overlapping in the splat?

  • RO2⚡️'s profile picture

    @risingodegua

    @AIatMeta @SpAItial_AI @playcanvas SAM3 is quite good! Give it go

    2 likes

  • Vaishabh Jalmi's profile picture

    @VaishabhJ

    @AIatMeta @SpAItial_AI @playcanvas How to work on this project?

  • RO2⚡️'s profile picture

    @risingodegua

    @AIatMeta @SpAItial_AI @playcanvas Link to repo, with readme and steps :

  • Eleonore Stelzer's profile picture

    @EleonoreStelzer

    @AIatMeta @SpAItial_AI @playcanvas That pipeline sounds pleasantly overengineered in the best way — I’d love to see how clean the segmentation gets on moving camera frames 🙂

  • RO2⚡️'s profile picture

    @risingodegua

    @AIatMeta @SpAItial_AI @playcanvas The 2D mask was transferred to the 3D space. So it stays when you move. Not perfect though, but it currently works and can be improved

  • kris's profile picture

    @0xkrisv

    @AIatMeta @SpAItial_AI @playcanvas wen spaital api?

  • RO2⚡️'s profile picture

    @risingodegua

    @AIatMeta @SpAItial_AI @playcanvas @SpAItial_AI API is out

    3 likes