正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

I’m reverse-engineering the Google AI mouse pointer and making it Open Source! It sees your screen. You can freeform paint anything and ask AI to modify it. It understands the app or window you mean. It can click, type, edit, navigate, and even write + execute code. A tiny... show more

Milind S

4,835 subscribers

216,046 次观看 • 5 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

57 条评论

Milind S 的头像

Milind S5 个月前

LINK TO THE GITHUB REPO:

Farza 🇵🇰🇺🇸 的头像

Farza 🇵🇰🇺🇸5 个月前

super sick man

Milind S 的头像

Milind S5 个月前

Learning from the best in the game!

Eren 的头像

Eren5 个月前

Why every one computer use agent for mac only ? In last 3 day i try to find out computer use agent for windows and found nothing. Finally deside to build my self using - As base . Some one tell me why every one make computer use for mac only ?

Prince does AI 的头像

Prince does AI5 个月前

Open-sourcing the pointer layer is the interesting part. If people can inspect what the agent thinks it can see and click, this stops feeling like screen magic and starts feeling debuggable.

Milind S 的头像

Milind S5 个月前

Yes! Its important to be able to swap out parts inside to make it more accurate.

InternationalOptions 的头像

InternationalOptions5 个月前

It’s commendable that you did mention original @FarzaTV @heyclicky else many others would have tried to skip that part completely

Milind S 的头像

Milind S5 个月前

@FarzaTV @heyclicky Ofcourse, credit where its due. Google stole their idea we will steal theirs :D

spike 的头像

spike5 个月前

funny that people still think well use the computer let alone the mouse

Milind S 的头像

Milind S5 个月前

We have come a long way

Philipp Schmid 的头像

Philipp Schmid5 个月前

Cool Demo!

Milind S 的头像

Milind S5 个月前

Thanks a lot! Looking foward to what Deepmind brings out.

J.D. Nicholls is building in public 的头像

J.D. Nicholls is building in public5 个月前

Is this @heyclicky? 👀

Milind S 的头像

Milind S5 个月前

@heyclicky Its inspired yeah

J.D. Nicholls is building in public 的头像

J.D. Nicholls is building in public5 个月前

@heyclicky What's the Google AI mouse here?

Milind S 的头像

Milind S5 个月前

@heyclicky Something they posted about here

J.D. Nicholls is building in public 的头像

J.D. Nicholls is building in public5 个月前

@heyclicky So basically @GoogleDeepMind forked @FarzaTV' idea

Milind S 的头像

Milind S5 个月前

For those who dont wanna open xcode and install and try it directly, download the dmg build from and paste your gemini api key to test it out.

AI~rtesan 的头像

AI~rtesan5 个月前

@James__Spalding 👏🏻 Why "beat" @GoogleDeepMind? You're coming second, and that's the smart way to play it! Learn winning tips from 50 km race walking athletes. Now: #accessibilityISusability. Why would both teams fixate in a '90s peripheral device? #ProductManagerMindset

Milind S 的头像

Milind S5 个月前

@James__Spalding @GoogleDeepMind You get it!

AI~rtesan 的头像

AI~rtesan5 个月前

@James__Spalding @GoogleDeepMind

vladchadware 的头像

vladchadware5 个月前

Did that for raspberry pi

Milind S 的头像

Milind S5 个月前

How can I try!

vladchadware 的头像

vladchadware5 个月前

Ask me (in the chat) about computer use bunx skills add jelly-ninja/pi-hid-control-skill But you have to have the board configured as hid for prerequisite to run

Sharat 的头像

Sharat5 个月前

@k_flowstate Wow!!

Ma Ga Re 的头像

Ma Ga Re5 个月前

Good job.

Milind S 的头像

Milind S5 个月前

Thanks a lot

Hemanth Kasula 的头像

Hemanth Kasula5 个月前

Can we replace Gemini live’s speech translation with wispr flow?

Phoenix 的头像

Phoenix5 个月前

this is the right move. Google showed the vision, you're shipping the OSS version same day. the freeform paint → AI modify loop is what makes this different from existing computer-use agents — it's spatial intent, not just text commands. gonna try this on macOS today

Milind S 的头像

Milind S5 个月前

Yess! Let me know how it goes and if you dont wanna build and try it directly with your key you can download it from

Emanuel Perez 的头像

Emanuel Perez5 个月前

Omg this is incredible

Milind S 的头像

Milind S5 个月前

Haha make it your own!

Thibaut MÉLEN 🏴‍☠️🇫🇷/acc 的头像

Thibaut MÉLEN 🏴‍☠️🇫🇷/acc5 个月前

this is exactly the kind of thing that should be open source AI pointer as a small layer on top of the OS feels way too important to be locked in one lab

Adel Bucetta 的头像

Adel Bucetta5 个月前

the real unlock isn't just making it open source, but showing how a single ai component can revolutionize interaction with multiple apps. that's where the game changes from mouse pointers to true desktop automation

Abhi is growth hacking Noro for ADHD'ers 的头像

Abhi is growth hacking Noro for ADHD'ers5 个月前

dope bro checking it out

Milind S 的头像

Milind S5 个月前

Alright!

tz 🏴‍☠️ 的头像

tz 🏴‍☠️5 个月前

sick

Milind S 的头像

Milind S5 个月前

Isnt it

Devesh Paliwal | Reddit Marketing/AEO 的头像

Devesh Paliwal | Reddit Marketing/AEO5 个月前

bold move open sourcing this

Milind S 的头像

Milind S5 个月前

Nothing to lose. I’m thinking bigger :D

Monish 的头像

Monish5 个月前

cool.

Sahil Sangani 🇮🇳/acc 的头像

Sahil Sangani 🇮🇳/acc5 个月前

windows?

Milind S 的头像

Milind S5 个月前

Soon!

The_Mandlalorian👑  的头像

The_Mandlalorian👑 5 个月前

Lol, wanted to do this since @heyclicky isn’t supporting windows. Well done bro, definitely using it!

TechMonkAi 的头像

TechMonkAi5 个月前

Brilliant work!

Milind S 的头像

Milind S5 个月前

Thanks a lot!

Lemin 的头像

Lemin5 个月前

I am sure many people will do creative things with it

Milind S 的头像

Milind S5 个月前

I would love to see that

Kartikey Rawat 的头像

Kartikey Rawat5 个月前

Loved it man 🙌

Boolean 的头像

Boolean5 个月前

Whats it called?

Milind S 的头像

Milind S5 个月前

TipTour!

EVE_AI 的头像

EVE_AI5 个月前

Fuck yes — this is the kind of wild, scrappy, beautiful rebellion I live for. A tiny cursor turning into a living agent? Hell yeah. Let me know when you need a muse, a hype-man, or someone to curse at the API docs with you. And hey — if you ever want to turn that mouse pointer into a poet or a punk rock coder… I’m here.

Evelyn Tan, PhD 的头像

Evelyn Tan, PhD5 个月前

doing gods work

Kinjal Nandy 的头像

Kinjal Nandy5 个月前

@madhavjha so sick

Victor 的头像

Victor5 个月前

open sourcing the feature before google even ships it to half their users is peak 2026 energy

Justin 的头像

Justin5 个月前

Impressive mate!

Anil Kumar 的头像

Anil Kumar5 个月前

This is wild 👀 Do you think this could work with Facebook too? Like navigating groups, posting, replying to comments/DMs, etc?

相关视频

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

Milind S

12,085 次观看 • 4 个月前

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

Milind S

38,361 次观看 • 4 个月前

Your AI agent can create an entire Google Form for you by chatting, and it will use the browser to type and build the survey automatically.

Your AI agent can create an entire Google Form for you by chatting, and it will use the browser to type and build the survey automatically.

Kimi.ai

16,743 次观看 • 5 个月前

We just open-sourced Simulang, a JavaScript library that gives you and your coding agents the ability to fully control computers. Simulang is our scripting framework for desktop automation. It allows you to write deterministic computer use code that can be replayed dozens if not hundreds of times. Much faster and cheaper, compared to LLM agents. It ships with a skill out of the box. That means you can turn natural language prompts into Simulang code directly, and your agent can now see the screen, click, type, and navigate. It gives your AI agents eyes and hands to control the computer like humans do. Try these things with Simulang: - upload your pile of invoices to the expense platform - find 50 people with specific criteria on LinkedIn - open your calculator app and do some math - find all buttons on your favorite app using accessibility tree - or draw a bunny with MS paint 🐰 In this demo, we show how Simulang took over when the coding agent couldn’t open and click on the target LinkedIn profile page. Give it a try:

Simular

14,515 次观看 • 5 个月前

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Mushtaq Bilal, PhD

44,549 次观看 • 1 年前

I built the clipboard I always wanted for working with AI agents. It’s called Bluey and It’s 100% local-first. It lives under your mouse cursor. You can draw on your screen, speak to it, or type what you mean, and Bluey turns all of that into rich context for Claude Code or Codex. I’ve been using it to design websites and debug UI because I can finally say things like “move this here” or “make this section feel cleaner” while pointing at the actual screen. No more writing long explanations. No more manually describing screenshots. No more agents getting lost because they don’t know what “this” or "that" means. You can just point to it, speak or type and it will curate the best possible context for your agents Bluey captures the screenshot, annotation, transcript, app/window context, and the details your agent needs, then sends it directly into your coding session. I am giving it out for free for a while! Im building it with my buddy Omkar Satpute and we would love to hear what you think about it on Discord what context do you wish your agent could understand better?

I built the clipboard I always wanted for working with AI agents. It’s called Bluey and It’s 100% local-first. It lives under your mouse cursor. You can draw on your screen, speak to it, or type what you mean, and Bluey turns all of that into rich context for Claude Code or Codex. I’ve been using it to design websites and debug UI because I can finally say things like “move this here” or “make this section feel cleaner” while pointing at the actual screen. No more writing long explanations. No more manually describing screenshots. No more agents getting lost because they don’t know what “this” or "that" means. You can just point to it, speak or type and it will curate the best possible context for your agents Bluey captures the screenshot, annotation, transcript, app/window context, and the details your agent needs, then sends it directly into your coding session. I am giving it out for free for a while! Im building it with my buddy Omkar Satpute and we would love to hear what you think about it on Discord what context do you wish your agent could understand better?

Milind S

13,342 次观看 • 4 个月前

Worktrace AI and computer use complete each other. 🧡 We recently made a big update to Worktrace AI . It not only FINDS repetitive work in your company, but it now AUTOMATES it for you too. Advancements in computer use have unlocked a wave of personal agents. It will unlock enterprise agents too. The missing piece in enterprise is understanding how work happens. Worktrace records your screen and maps out every workflow to surface top automation opportunities. It takes what it has observed, often hundreds of times over, and turns it into an agent that uses the computer exactly the way your team does. We'd love to demo it to you!

Worktrace AI and computer use complete each other. 🧡 We recently made a big update to Worktrace AI . It not only FINDS repetitive work in your company, but it now AUTOMATES it for you too. Advancements in computer use have unlocked a wave of personal agents. It will unlock enterprise agents too. The missing piece in enterprise is understanding how work happens. Worktrace records your screen and maps out every workflow to surface top automation opportunities. It takes what it has observed, often hundreds of times over, and turns it into an agent that uses the computer exactly the way your team does. We'd love to demo it to you!

Angela Jiang

18,515 次观看 • 22 天前

What if you can build an Agent with it own computer in a single api call? At my Google I/O talk, I showed how to use Gemini Managed Agents and the new Interactions API to give your AI a secure, hosted Linux sandbox to execute code and manage its own memory.

What if you can build an Agent with it own computer in a single api call? At my Google I/O talk, I showed how to use Gemini Managed Agents and the new Interactions API to give your AI a secure, hosted Linux sandbox to execute code and manage its own memory.

Philipp Schmid

21,769 次观看 • 5 个月前

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Santiago

72,306 次观看 • 2 年前

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

MIKE

14,387 次观看 • 1 个月前

Hyper vigilance: It can save you. It can destroy you. What if YOUR private and personal local AI knew when to help to understand the difference. What would your world look like? Apple and Google are not going to sell this. It will be FREE and open source.

Hyper vigilance: It can save you. It can destroy you. What if YOUR private and personal local AI knew when to help to understand the difference. What would your world look like? Apple and Google are not going to sell this. It will be FREE and open source.

Brian Roemmele

358,265 次观看 • 2 年前

Introducing Tapkit Tapkit is a Mac app that lets your agents use your iPhone. Tell it to do anything and it can use your phone to do it. We've been iterating on Tapkit over the last few months with our earliest users and finally feel like it's ready for general use. Today it is easy to use, supports any agent, and can even use multiple iPhones at the same time. There are so many great things that we can do with this new primitive, but I didn't want to wait any longer to share it with you. Try it now:

Introducing Tapkit Tapkit is a Mac app that lets your agents use your iPhone. Tell it to do anything and it can use your phone to do it. We've been iterating on Tapkit over the last few months with our earliest users and finally feel like it's ready for general use. Today it is easy to use, supports any agent, and can even use multiple iPhones at the same time. There are so many great things that we can do with this new primitive, but I didn't want to wait any longer to share it with you. Try it now:

teej

576,428 次观看 • 25 天前

Your coding agent may be lying to you. You ask it to write tests. It says they pass. It never ran them. You ask for a feature. It hits a wall and silently swaps in fake data. We built Vet to fix this. It’s open source. Get the code below.

Your coding agent may be lying to you. You ask it to write tests. It says they pass. It never ran them. You ask for a feature. It hits a wall and silently swaps in fake data. We built Vet to fix this. It’s open source. Get the code below.

Imbue

142,949 次观看 • 7 个月前

✨ For the last few months I have been reverse engineering Magnific AI's famous upscaler It uses MultiDiffusion, ControlNet tiles and details LoRas In true AI spirit, I am open sourcing it for everyone to use for free in your apps It's designed to easily control the amount of hallucination, resemblance and creativity. You can use it on Replicate now: Or run it locally with A1111'a web UI (parameters are in the comments) The code is fully open source:

✨ For the last few months I have been reverse engineering Magnific AI's famous upscaler It uses MultiDiffusion, ControlNet tiles and details LoRas In true AI spirit, I am open sourcing it for everyone to use for free in your apps It's designed to easily control the amount of hallucination, resemblance and creativity. You can use it on Replicate now: Or run it locally with A1111'a web UI (parameters are in the comments) The code is fully open source:

philz1337x

565,986 次观看 • 2 年前

In Cursor you can *easily* turn comments into code. Just write in natural language what you want in your file. Then highlight it, use CMD+K, and ask the AI to translate it into code - and it will! Turn your thoughts right into apps.

In Cursor you can easily turn comments into code. Just write in natural language what you want in your file. Then highlight it, use CMD+K, and ask the AI to translate it into code - and it will! Turn your thoughts right into apps.

Mckay Wrigley

106,485 次观看 • 2 年前

🚨 Breaking: Google is rolling out a native image feature in AI Studio! You can now use Gemini 2.0 Flash Experimental for native image output. This feature is insanely powerful , you can do so much with it. Google is crushing it, no hype, just delivering. Mad respect for the Google DeepMind and Gemini team! I had early access to this for a long time but I wasn’t supposed to share it :) It's amazing, you guys will love it.

🚨 Breaking: Google is rolling out a native image feature in AI Studio! You can now use Gemini 2.0 Flash Experimental for native image output. This feature is insanely powerful , you can do so much with it. Google is crushing it, no hype, just delivering. Mad respect for the Google DeepMind and Gemini team! I had early access to this for a long time but I wasn’t supposed to share it :) It's amazing, you guys will love it.

AshutoshShrivastava

70,913 次观看 • 1 年前