Skip navigation
Visualizing Gemini intelligence with things in my life 🔊🧵
0:00 / 1:02

Visualizing Gemini intelligence with things in my life 🔊🧵

Alexander Chen

@alexanderchen14K followers

Visit on X
101 likesDownloadOriginal post

13,528 views

Visualizing Gemini intelligence with things in my life 🔊🧵

12 Comments

Comments from the original post on X (Twitter)

  • Alexander Chen's profile picture

    @alexanderchen

    I made it with Gemini spatial intelligence + opencv.js in @GoogleAIStudio. Link here: It's been fun testing on old videos from my photo library as Gemini is able to uncover new details.

    24 likes

  • Alexander Chen's profile picture

    @alexanderchen

    How it works: I prompt Gemini with the first frame of video to give me (x, y) coordinates with descriptions. Then I use optical flow to track those positions over time. Of course optical flow has limitations, but it was a fast lightweight way to prototype the idea quickly.

    12 likes

  • Alexander Chen's profile picture

    @alexanderchen

    The labels are interactive and pick colors dynamically for visual glanceability:

    9 likes

  • Alexander Chen's profile picture

    @alexanderchen

    V2 updated version of @GoogleAIStudio app supports videos of any aspect ratio, and you can also pause/scrub the video to run analysis on that frame:

    4 likes

  • Timothy Meade's profile picture

    @tmztmobile

    Can this be done with PaliGemma on the device?

    1 like

  • Alexander Chen's profile picture

    @alexanderchen

    I haven't tried PaliGemma. But the first step is just generating points of interest with (x, y) coordinates and description. The tracking after that point is basic optical flow with opencv.js. So in theory yes you could swap other models / techniques for that first step. But note that in my prompt, I'm asking Gemini to decide for itself what five points of interest are, I do not explicitly ask for particular aspects. So that may be where the reasoning capabilities of a general purpose model like Gemini differ from using other specialized vision models.

  • Alexander Chen's profile picture

    @alexanderchen

    That said, I'd be very excited to see someone implement something like this with PaliGemma or other on-device models! Lots of benefits with latency, cost, user benefit of keeping videos on-device, etc. From what I've seen, you should definitely be able to connect on-device VLMs to do something like this.

  • Alexander Chen's profile picture

    @alexanderchen

    Perhaps @osanseviero might know a bit more about PaliGemma or on device models for this

  • Merzmensch Kosmopol🧑‍🎨🤖's profile picture

    @Merzmensch

    Brilliant!

    1 like

  • Felix Turner's profile picture

    @felixturner

    Amazing

    1 like

  • Alexander Chen's profile picture

    @alexanderchen

    Thanks!

  • Simon Edwardsson's profile picture

    @SimEdw

    I like this quite a bit. This would have been really hard to build just a few years ago. If you layered in a segmentation model, you could highlight the whole object. Could make for an interesting interface.

    1 like