more magic/AI mesh selection, this one w/ SAM2 which takes an initial box/points instead of a text prompt. SAM2 is much faster, in 'stream' mode I can send it frames like it's processing a video (this capture is 1x, SAM2 running locally on 8GB 3070ti).

very surprising how good it is at contextual selection from just a bounding box. For the shoe laces (which are just barely there, and smoothly disappear into the shoe in lots of places) I had to set a few positive/negative markers.

I like this a lot. I was thinking about how to build a material assignment studio of sorts given arbitrary CAD meshes and such, and this might be a good approach

Didn't even know it was available for local use - impressive!

Very cool. This almost certainly could be used for reverse engineering workflows...

This is awesome! Check my understanding here, SAM2 is selecting pixels from a video stream, there is still a mapping process needed to get to triangles?

Yes, I’m doing pretty basic stuff there, checking sample points on each tri and occlusion testing

Whoa. SAM2 for mesh selection on a 3070ti? That's wild. I need to play around with this.

You running this local?

yes

looks cool but not sure what's going on here maybe you should explain more for uncontexted viewers 🙂

AI model is magically selecting triangles

how does he know to stop at the shoe and not crawl up the pant ?

incidentally how can it be so smart to select only the shoe with that much precision and dumb enough to select part of the other shoe miles away

it knows because of its training :)

This is using a very reduced/quiantized model (for speed) which causes some artifacts

ohh ts the secret sauce i see, and you have to rotate the camera because sam is image to something not sure what lol so it would be interesting to use sam to train another AI that does 3D object to triangles instead right ?

the new ai will select shoe such that on every camera angle it aligns with sam's prediction