Introducing Gemini Cursor ✨ – a second multimodal AI cursor for your desktop that's open-source and free! Link below 👇 This experiment 🧪 reimagines how we interact with our computers because visual cues 👀 help us make sense of what we see on a screen. In this demo, I... had my friend test it out by trying to add a payment method 💳 to Amazon. The cursor walks through the entire process 💬 while talking and pointing 🖱️ to the right parts of the website. Powered by Gemini 2.0 Flash (Experimental)⚡ from Google and their live multimodal API. Shoutout to Alexander Chen for sharing the starter code that powers most of this app 🙌🔥show more

GitHub Code

This is awesome and a great use of Gemini 2.0 Flash. Feature request: One thing that really bugged me with the existing screen sharing experience is when I used it as a pair programmer, it would read out lines of code to reference it which was so annoying. If you could get it to point to lines and also just display certain things instead of say them (and know when to show vs tell) that would be awesome!

Ohhh that would be cool. Do you have some examples in mind for what it would display?

Let’s say you are writing an essay and you start talking to it about trying to improve a sentence/paragraph. If it suggests a new version, it should say “what about this” and show it. If you like it, you can copy it to clipboard. If it has to read out the sentence each time it would be too slow

Hmm that is a valid need but I think we need to analyse other use cases and the best or decent UX for them because I don’t want to overfit for a few. An argument could be made for generative UI/UX depending on the use case which is fine but there must be some constraints and consistency in that too or else it will end up being too confusing for most users. For example, expanding on your code editor requirement, an integration with the IDE would be the best so that it can apply diffs. However sometimes you might want some commands for your shell and the ability to copy paste commands and run might appear to be a good parallel with the essay requirement but I would argue that a google doc comment style UX would be better for the essay and a window with the list of terminal commands with a button to directly run them would be better(maybe not if you need to enter into a venv for eg). I need to think more about this but my initial thoughts are that this app would be good for taking in info from your screen across apps, intermediate guidance and final output plus next step suggestions but the actions to perform would be best done via an integration with the apps directly because it opens up possibilities for better UX. If the cursor could actually interact with the screen like a second cursor then the most primitive interaction would be directly editing with the apps(gdocs, code editor, etc) on your behalf but that takes away the possibility for a diff preview in various formats(gdoc comment, code diff, etc) which is why I don’t want to jump ahead. Obviously these interactions would only be possible if the apps “want” integrations which they might not for obvious reasons. Now that I think about it what if these diff preview interfaces could be overlays? Oooo now this is definitely worth thinking about. Thank you!! This was a cool exercise ✨

That starter code is by @hapticdata. Glad it's useful!

@hapticdata I wouldn’t have been able to build it so fast without the starter code. Thank you @hapticdata 🙌

This is next-level interaction! A multimodal AI cursor that guides users visually and verbally is a game-changer. Excited to see where this goes!

This is very cool

Very cool, well done! There are real issues with Gemini multimedia api. Limited to 2 minute sessions, even shorter, depending on how many tokens you have used prior to the 2 minutes limit. That makes it pretty useless. And plain Gemini bounding boxes is just too slow >10 secs!

Thank you! I think we can restart sessions and summarise last sessions based on an avg time from observations to get around that. Haven’t implemented it tho. I’m asking for coordinates not bounding boxes and I haven’t faced such significant lags as you can see in the demo so not sure why.

Yes I have noticed you are asking for coordinates and for each point it still takes 2-3 seconds at least in the tests I have done (using your repo). That is still too slow. I wonder whether Google would manage to make that better in future. Otherwise for real time apps a no no.

Nice! Are you using bounding box positions for determining cursor target positions?

No I’m using direct coordinates with pretty much the same prompt used in the Spatial Understanding starter app. I tried bounding boxes but had better success by asking for points. I would prefer bounding boxes so that I can make the cursor point to the center of the bottom edge so that it doesn’t block the area it wants to point to. Could be an issue with my prompt because I’m asking for a list of descriptions of areas to point to in the function call of the realtime api and then making another api call to get the coordinates because it gives much better accuracy. I still need to work on the timings cuz it’s guessing based on some examples.

Oh nice. Glad pointing is working. Really neat you're combining function calling in Multimodal Live API + Gemini API as a separate tool call. Lots of potential with that approach.

Yup because the spatial understanding capability works best with text input as you mentioned and my tests revealed the same.

cc @rileybrown_ai @RayFernando1337 - vibe coding gona change forever

Amazing! That's a qualitative "user" interview!

Well done to both of you, excellent work!

This is so cool, impressive first implementation of the realtime api. I started it and it immediately helped me fix a touchdesigner node setup

Yeah Gemini has been underrated a lot for quite sometime , I am seeing many demos like this

Neat! How has function calling reliability been for you? For me, it got unreliable after a few calls and sometimes Gemini would read out the function call JSON response that it got back.

As the docs say it is unreliable when there’s audio input and output. So here I don’t ask it for the coordinates directly instead I ask for descriptions of where to point and in the tool call I make another Gemini API call to get the actual coordinates because it’s text in and out which lets the model work well. Of course this means there will be a delay but for now it’s a trade off we’ll have to deal with until it gets better. What are some function calls you’ve found to be unreliable? Just curious

Ah, that's clever! I'll have to think about whether I can apply this to my use-case.

real time AI tutor is here. :) nice

Game changer!

very interesting it would really be cool if it could act like a research assistant, proofread, evaluate arugments, help find papers and translate documents etc.

Gemini Cursor to ciekawy krok w stronę bardziej intuicyjnej interakcji z komputerem. AI jako asystent wskazujący i tłumaczący elementy interfejsu może zrewolucjonizować sposób, w jaki korzystamy z technologii. Na pewno warto obserwować rozwój tego projektu! 🚀 #AI #GeminiCursor

Insane! Thanks, it works very well!

Superb! Pretty useful

damn bro it's actually a great idea for a service or support purpose for any company.

Will it also click and interact with objects, or only point?

Gemini is dogshit in coding even their thinking model struggles with medium complexity stuff on claude and deepseek

vv cool, what do you feel about adding a voice input for a use case like data entry where : user dictates output -> formats using LLM does such a case required something like a multimodal live with json as output (coz its audio native) or just whisper like model will do?

Paid ?

Free and open source

Wow

Isn't streaming time session is 10 miniute on AI studio. BTW, does the grounding by default On while sharing screening while using API?

So the spatial understanding capability does not work well with audio in and out as the docs mention. The workaround I’m using is tool call to generate descriptions of places on the screen the point to and use a text in and out API call in the tool call to get the coordinates using the last frame. This ensures we get the models best performance.

Getting very odd things on a big screen --- wide screen 5K2K monitor and it's getting confused as to the position of elements; thinks the top left IS in the right place but the bottom left is a foot from the left to the just up from the bottom.

Hmm. Just to confirm is the main control window minimised after you enable screen sharing?

What is the current rate limit on flash for something like this?

Here are the rate limits and session duration supported according to the docs.

how about an agent that does it for you?

A second cursor that talks you through Amazon checkout is the honest demo. Visual pointing is the part chat never got right. Open-source Gemini Flash on the desktop is where you find out if it sticks. 😉