Okay this is insane... WebGPU-accelerated semantic video tracking, powered by DINOv3 and Transformers.js! 🤯 This will revolutionize AI-powered video editors... which can now run 100% locally in your browser, no server inference required (costs $0)! 😍 Who's building this?

How does it work? 🤔 1️⃣ Generate and cache image features for each frame 2️⃣ Create a list of embeddings for selected patch(es) 3️⃣ Compute cosine similarity between each patch and the selected patch(es) 4️⃣ Highlight those whose score is above some threshold ... et voilà! 🥳

You can also make selections across frames to improve temporal consistency! This is super useful if the object changes its appearance slightly throughout the video.

As always, I've published the demo and source code on Hugging Face 👇 Try it out! Everything in a single HTML file! 🤯

I wonder how this compares to predator algorithm back in the day.

this looks interesting for @opencutapp 👀

This is crazy

yoo check this you if you need python api, multiple classes or more control:

rotoscoping in browser thou, ok. it's a new day.

This is amazing😍🤩

crazy

Demo doesn't work unfortunately, frame width 0 in console for my videos and demos

What's the difference between this and sam2? Any drawbacks?

Crazy work 🔥

does not look usable to me.

What if there are multiple horses? It will detect all of them, not good

Not working... Error: Could not load models. no available backend found. ERR: [webgpu] Error: Failed to get GPU adapter. You may need to enable flag "--enable-unsafe-webgpu" if you are using Chrome. (even when the flag is enabled.. )