Used Apple Foundation Models Framework to ease the UX in my Shared Space 3DGS viewer app You can ask it to recolor certain areas of your gaussian splats via computer vision models Or rotate, flip, and even ask to put your model in a "showcase mode" where it will... slowly spinshow more

That's really impressive. How is it doing the segmentation on a gaussian splat? It's not like there's a texture map for it.

So for each splat, when you first do a recolor request: 1) 7 different "camera" views are "screenshotted" around the model. Which is cached for the rest of the session in case might be doing further recolors (don't want it having to rasterize for each request) 2) Apple foundation models figures out the language request, and then sends that to a small/locally running SAM 3 on headset, which tries to pinpoint each relevant location in the each view and then match them together via a mask 3) Each mask is turned into splat indices through the view's buffer.. which doesn't exactly work perfectly all the time. But is the best situation I have so far

That's a really cool workflow. Well done!

I ‘m a big fan of Rem and have many models for her ，thank you for the work！

Is there anywhere where I could try this?

Not yet. I hope to have the app release just in time for visionOS 27 release

That’s useful. Nice recolor animation and clean design.

Format of splats? .splat, .ksplat, .sog etc?

The app supports .splat, .ply, and .usdz (with splats in them)

Holy shit this is good

Thank you. I have one more "moonshot" functionality I still want to add to this that I think would be pretty fun

What other commands is there? 😂😂😂

What splatting engine does it use and how many splats is it able to render before performance crashes?

RealityKit’s own. Up to 400K max (not a lot, but meant clearly for objects)

Thats more than the 200k number I’ve seen floating around. Does Reality kit also handle the segmentation? The model reloads upon changes so it’s a single buffer. I assume you semantically chunk the splats or the model already comes that way. Or how does that magic work?

It started as 200K in earlier 27 betas. But Apple (silently) bumped it up in later ones

That’s nice. I’m planning on diving into something similar next week. Any pointers?

Are you on the M2 or the M5? I’ve been wanting to dive into Apple Foundation for on device deployment but didn’t know how well the M2 would hold up.

I am using M5 right now. But I think this should work on M2 too. Haven’t tested yet

I need play more with 3dgs

👌