Visualizing Gemini intelligence with things in my life 🔊🧵
@alexanderchen14K followers
13,528 views
12 Comments
Comments from the original post on X (Twitter)

@alexanderchen
I made it with Gemini spatial intelligence + opencv.js in @GoogleAIStudio. Link here: It's been fun testing on old videos from my photo library as Gemini is able to uncover new details.
24 likes

@alexanderchen
How it works: I prompt Gemini with the first frame of video to give me (x, y) coordinates with descriptions. Then I use optical flow to track those positions over time. Of course optical flow has limitations, but it was a fast lightweight way to prototype the idea quickly.
12 likes

@alexanderchen
The labels are interactive and pick colors dynamically for visual glanceability:
9 likes

@alexanderchen
V2 updated version of @GoogleAIStudio app supports videos of any aspect ratio, and you can also pause/scrub the video to run analysis on that frame:
4 likes

@tmztmobile
Can this be done with PaliGemma on the device?
1 like

@alexanderchen
I haven't tried PaliGemma. But the first step is just generating points of interest with (x, y) coordinates and description. The tracking after that point is basic optical flow with opencv.js. So in theory yes you could swap other models / techniques for that first step. But note that in my prompt, I'm asking Gemini to decide for itself what five points of interest are, I do not explicitly ask for particular aspects. So that may be where the reasoning capabilities of a general purpose model like Gemini differ from using other specialized vision models.

@alexanderchen
That said, I'd be very excited to see someone implement something like this with PaliGemma or other on-device models! Lots of benefits with latency, cost, user benefit of keeping videos on-device, etc. From what I've seen, you should definitely be able to connect on-device VLMs to do something like this.

@alexanderchen
Perhaps @osanseviero might know a bit more about PaliGemma or on device models for this

@Merzmensch
Brilliant!
1 like

@felixturner
Amazing
1 like

@alexanderchen
Thanks!

@SimEdw
I like this quite a bit. This would have been really hard to build just a few years ago. If you layered in a segmentation model, you could highlight the whole object. Could make for an interesting interface.
1 like