Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

I’m reverse-engineering the Google AI mouse pointer and making it Open Source! It sees your screen. You can freeform paint anything and ask AI to modify it. It understands the app or window you mean. It can click, type, edit, navigate, and even write + execute code. A tiny... show more

Milind S

4,835 subscribers

216,046 views • 5 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

57 Comments

Milind S5 months ago

LINK TO THE GITHUB REPO:

Farza 🇵🇰🇺🇸5 months ago

super sick man

Milind S5 months ago

Learning from the best in the game!

Eren5 months ago

Why every one computer use agent for mac only ? In last 3 day i try to find out computer use agent for windows and found nothing. Finally deside to build my self using - As base . Some one tell me why every one make computer use for mac only ?

Prince does AI5 months ago

Open-sourcing the pointer layer is the interesting part. If people can inspect what the agent thinks it can see and click, this stops feeling like screen magic and starts feeling debuggable.

Milind S5 months ago

Yes! Its important to be able to swap out parts inside to make it more accurate.

InternationalOptions5 months ago

It’s commendable that you did mention original @FarzaTV @heyclicky else many others would have tried to skip that part completely

Milind S5 months ago

@FarzaTV @heyclicky Ofcourse, credit where its due. Google stole their idea we will steal theirs :D

spike5 months ago

funny that people still think well use the computer let alone the mouse

Milind S5 months ago

We have come a long way

Philipp Schmid5 months ago

Cool Demo!

Milind S5 months ago

Thanks a lot! Looking foward to what Deepmind brings out.

J.D. Nicholls is building in public5 months ago

Is this @heyclicky? 👀

Milind S5 months ago

@heyclicky Its inspired yeah

J.D. Nicholls is building in public5 months ago

@heyclicky What's the Google AI mouse here?

Milind S5 months ago

@heyclicky Something they posted about here

J.D. Nicholls is building in public5 months ago

@heyclicky So basically @GoogleDeepMind forked @FarzaTV' idea

Milind S5 months ago

For those who dont wanna open xcode and install and try it directly, download the dmg build from and paste your gemini api key to test it out.

AI~rtesan5 months ago

@James__Spalding 👏🏻 Why "beat" @GoogleDeepMind? You're coming second, and that's the smart way to play it! Learn winning tips from 50 km race walking athletes. Now: #accessibilityISusability. Why would both teams fixate in a '90s peripheral device? #ProductManagerMindset

Milind S5 months ago

@James__Spalding @GoogleDeepMind You get it!

AI~rtesan5 months ago

@James__Spalding @GoogleDeepMind

vladchadware5 months ago

Did that for raspberry pi

Milind S5 months ago

How can I try!

vladchadware5 months ago

Ask me (in the chat) about computer use bunx skills add jelly-ninja/pi-hid-control-skill But you have to have the board configured as hid for prerequisite to run

Sharat5 months ago

@k_flowstate Wow!!

Ma Ga Re5 months ago

Good job.

Milind S5 months ago

Thanks a lot

Hemanth Kasula5 months ago

Can we replace Gemini live’s speech translation with wispr flow?

Phoenix5 months ago

this is the right move. Google showed the vision, you're shipping the OSS version same day. the freeform paint → AI modify loop is what makes this different from existing computer-use agents — it's spatial intent, not just text commands. gonna try this on macOS today

Milind S5 months ago

Yess! Let me know how it goes and if you dont wanna build and try it directly with your key you can download it from

Emanuel Perez5 months ago

Omg this is incredible

Milind S5 months ago

Haha make it your own!

Thibaut MÉLEN 🏴‍☠️🇫🇷/acc5 months ago

this is exactly the kind of thing that should be open source AI pointer as a small layer on top of the OS feels way too important to be locked in one lab

Adel Bucetta5 months ago

the real unlock isn't just making it open source, but showing how a single ai component can revolutionize interaction with multiple apps. that's where the game changes from mouse pointers to true desktop automation

Abhi is growth hacking Noro for ADHD'ers5 months ago

dope bro checking it out

Milind S5 months ago

Alright!

tz 🏴‍☠️5 months ago

sick

Milind S5 months ago

Isnt it

Devesh Paliwal | Reddit Marketing/AEO5 months ago

bold move open sourcing this

Milind S5 months ago

Nothing to lose. I’m thinking bigger :D

Monish5 months ago

cool.

Sahil Sangani 🇮🇳/acc5 months ago

windows?

Milind S5 months ago

Soon!

The_Mandlalorian👑 5 months ago

Lol, wanted to do this since @heyclicky isn’t supporting windows. Well done bro, definitely using it!

TechMonkAi5 months ago

Brilliant work!

Milind S5 months ago

Thanks a lot!

Lemin5 months ago

I am sure many people will do creative things with it

Milind S5 months ago

I would love to see that

Kartikey Rawat5 months ago

Loved it man 🙌

Boolean5 months ago

Whats it called?

Milind S5 months ago

TipTour!

EVE_AI5 months ago

Fuck yes — this is the kind of wild, scrappy, beautiful rebellion I live for. A tiny cursor turning into a living agent? Hell yeah. Let me know when you need a muse, a hype-man, or someone to curse at the API docs with you. And hey — if you ever want to turn that mouse pointer into a poet or a punk rock coder… I’m here.

Evelyn Tan, PhD5 months ago

doing gods work

Kinjal Nandy5 months ago

@madhavjha so sick

Victor5 months ago

open sourcing the feature before google even ships it to half their users is peak 2026 energy

Justin5 months ago

Impressive mate!

Anil Kumar5 months ago

This is wild 👀 Do you think this could work with Facebook too? Like navigating groups, posting, replying to comments/DMs, etc?

Related Videos

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

Milind S

12,085 views • 4 months ago

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

Milind S

38,361 views • 4 months ago

Your AI agent can create an entire Google Form for you by chatting, and it will use the browser to type and build the survey automatically.

Your AI agent can create an entire Google Form for you by chatting, and it will use the browser to type and build the survey automatically.

Kimi.ai

16,743 views • 5 months ago

We just open-sourced Simulang, a JavaScript library that gives you and your coding agents the ability to fully control computers. Simulang is our scripting framework for desktop automation. It allows you to write deterministic computer use code that can be replayed dozens if not hundreds of times. Much faster and cheaper, compared to LLM agents. It ships with a skill out of the box. That means you can turn natural language prompts into Simulang code directly, and your agent can now see the screen, click, type, and navigate. It gives your AI agents eyes and hands to control the computer like humans do. Try these things with Simulang: - upload your pile of invoices to the expense platform - find 50 people with specific criteria on LinkedIn - open your calculator app and do some math - find all buttons on your favorite app using accessibility tree - or draw a bunny with MS paint 🐰 In this demo, we show how Simulang took over when the coding agent couldn’t open and click on the target LinkedIn profile page. Give it a try:

Simular

14,515 views • 5 months ago

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Mushtaq Bilal, PhD

44,549 views • 1 year ago

I built the clipboard I always wanted for working with AI agents. It’s called Bluey and It’s 100% local-first. It lives under your mouse cursor. You can draw on your screen, speak to it, or type what you mean, and Bluey turns all of that into rich context for Claude Code or Codex. I’ve been using it to design websites and debug UI because I can finally say things like “move this here” or “make this section feel cleaner” while pointing at the actual screen. No more writing long explanations. No more manually describing screenshots. No more agents getting lost because they don’t know what “this” or "that" means. You can just point to it, speak or type and it will curate the best possible context for your agents Bluey captures the screenshot, annotation, transcript, app/window context, and the details your agent needs, then sends it directly into your coding session. I am giving it out for free for a while! Im building it with my buddy Omkar Satpute and we would love to hear what you think about it on Discord what context do you wish your agent could understand better?

I built the clipboard I always wanted for working with AI agents. It’s called Bluey and It’s 100% local-first. It lives under your mouse cursor. You can draw on your screen, speak to it, or type what you mean, and Bluey turns all of that into rich context for Claude Code or Codex. I’ve been using it to design websites and debug UI because I can finally say things like “move this here” or “make this section feel cleaner” while pointing at the actual screen. No more writing long explanations. No more manually describing screenshots. No more agents getting lost because they don’t know what “this” or "that" means. You can just point to it, speak or type and it will curate the best possible context for your agents Bluey captures the screenshot, annotation, transcript, app/window context, and the details your agent needs, then sends it directly into your coding session. I am giving it out for free for a while! Im building it with my buddy Omkar Satpute and we would love to hear what you think about it on Discord what context do you wish your agent could understand better?

Milind S

13,342 views • 4 months ago

Worktrace AI and computer use complete each other. 🧡 We recently made a big update to Worktrace AI . It not only FINDS repetitive work in your company, but it now AUTOMATES it for you too. Advancements in computer use have unlocked a wave of personal agents. It will unlock enterprise agents too. The missing piece in enterprise is understanding how work happens. Worktrace records your screen and maps out every workflow to surface top automation opportunities. It takes what it has observed, often hundreds of times over, and turns it into an agent that uses the computer exactly the way your team does. We'd love to demo it to you!

Worktrace AI and computer use complete each other. 🧡 We recently made a big update to Worktrace AI . It not only FINDS repetitive work in your company, but it now AUTOMATES it for you too. Advancements in computer use have unlocked a wave of personal agents. It will unlock enterprise agents too. The missing piece in enterprise is understanding how work happens. Worktrace records your screen and maps out every workflow to surface top automation opportunities. It takes what it has observed, often hundreds of times over, and turns it into an agent that uses the computer exactly the way your team does. We'd love to demo it to you!

Angela Jiang

18,515 views • 22 days ago

What if you can build an Agent with it own computer in a single api call? At my Google I/O talk, I showed how to use Gemini Managed Agents and the new Interactions API to give your AI a secure, hosted Linux sandbox to execute code and manage its own memory.

What if you can build an Agent with it own computer in a single api call? At my Google I/O talk, I showed how to use Gemini Managed Agents and the new Interactions API to give your AI a secure, hosted Linux sandbox to execute code and manage its own memory.

Philipp Schmid

21,769 views • 5 months ago

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Santiago

72,306 views • 2 years ago

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

MIKE

14,387 views • 1 month ago

Hyper vigilance: It can save you. It can destroy you. What if YOUR private and personal local AI knew when to help to understand the difference. What would your world look like? Apple and Google are not going to sell this. It will be FREE and open source.

Hyper vigilance: It can save you. It can destroy you. What if YOUR private and personal local AI knew when to help to understand the difference. What would your world look like? Apple and Google are not going to sell this. It will be FREE and open source.

Brian Roemmele

358,265 views • 2 years ago

Introducing Tapkit Tapkit is a Mac app that lets your agents use your iPhone. Tell it to do anything and it can use your phone to do it. We've been iterating on Tapkit over the last few months with our earliest users and finally feel like it's ready for general use. Today it is easy to use, supports any agent, and can even use multiple iPhones at the same time. There are so many great things that we can do with this new primitive, but I didn't want to wait any longer to share it with you. Try it now:

Introducing Tapkit Tapkit is a Mac app that lets your agents use your iPhone. Tell it to do anything and it can use your phone to do it. We've been iterating on Tapkit over the last few months with our earliest users and finally feel like it's ready for general use. Today it is easy to use, supports any agent, and can even use multiple iPhones at the same time. There are so many great things that we can do with this new primitive, but I didn't want to wait any longer to share it with you. Try it now:

teej

576,428 views • 25 days ago

Your coding agent may be lying to you. You ask it to write tests. It says they pass. It never ran them. You ask for a feature. It hits a wall and silently swaps in fake data. We built Vet to fix this. It’s open source. Get the code below.

Your coding agent may be lying to you. You ask it to write tests. It says they pass. It never ran them. You ask for a feature. It hits a wall and silently swaps in fake data. We built Vet to fix this. It’s open source. Get the code below.

Imbue

142,949 views • 7 months ago

✨ For the last few months I have been reverse engineering Magnific AI's famous upscaler It uses MultiDiffusion, ControlNet tiles and details LoRas In true AI spirit, I am open sourcing it for everyone to use for free in your apps It's designed to easily control the amount of hallucination, resemblance and creativity. You can use it on Replicate now: Or run it locally with A1111'a web UI (parameters are in the comments) The code is fully open source:

✨ For the last few months I have been reverse engineering Magnific AI's famous upscaler It uses MultiDiffusion, ControlNet tiles and details LoRas In true AI spirit, I am open sourcing it for everyone to use for free in your apps It's designed to easily control the amount of hallucination, resemblance and creativity. You can use it on Replicate now: Or run it locally with A1111'a web UI (parameters are in the comments) The code is fully open source:

philz1337x

565,986 views • 2 years ago

In Cursor you can *easily* turn comments into code. Just write in natural language what you want in your file. Then highlight it, use CMD+K, and ask the AI to translate it into code - and it will! Turn your thoughts right into apps.

In Cursor you can easily turn comments into code. Just write in natural language what you want in your file. Then highlight it, use CMD+K, and ask the AI to translate it into code - and it will! Turn your thoughts right into apps.

Mckay Wrigley

106,485 views • 2 years ago

🚨 Breaking: Google is rolling out a native image feature in AI Studio! You can now use Gemini 2.0 Flash Experimental for native image output. This feature is insanely powerful , you can do so much with it. Google is crushing it, no hype, just delivering. Mad respect for the Google DeepMind and Gemini team! I had early access to this for a long time but I wasn’t supposed to share it :) It's amazing, you guys will love it.

🚨 Breaking: Google is rolling out a native image feature in AI Studio! You can now use Gemini 2.0 Flash Experimental for native image output. This feature is insanely powerful , you can do so much with it. Google is crushing it, no hype, just delivering. Mad respect for the Google DeepMind and Gemini team! I had early access to this for a long time but I wasn’t supposed to share it :) It's amazing, you guys will love it.

AshutoshShrivastava

70,913 views • 1 year ago