Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

I’m reverse-engineering the Google AI mouse pointer and making it Open Source! It sees your screen. You can freeform paint anything and ask AI to modify it. It understands the app or window you mean. It can click, type, edit, navigate, and even write + execute code. A tiny... show more

Milind S

4,835 subscribers

216,046 Aufrufe • vor 5 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

57 Kommentare

Profilbild von Milind S

Milind Svor 5 Monaten

LINK TO THE GITHUB REPO:

Profilbild von Farza 🇵🇰🇺🇸

Farza 🇵🇰🇺🇸vor 5 Monaten

super sick man

Profilbild von Milind S

Milind Svor 5 Monaten

Learning from the best in the game!

Profilbild von Eren

Erenvor 5 Monaten

Why every one computer use agent for mac only ? In last 3 day i try to find out computer use agent for windows and found nothing. Finally deside to build my self using - As base . Some one tell me why every one make computer use for mac only ?

Profilbild von Prince does AI

Prince does AIvor 5 Monaten

Open-sourcing the pointer layer is the interesting part. If people can inspect what the agent thinks it can see and click, this stops feeling like screen magic and starts feeling debuggable.

Profilbild von Milind S

Milind Svor 5 Monaten

Yes! Its important to be able to swap out parts inside to make it more accurate.

Profilbild von InternationalOptions

InternationalOptionsvor 5 Monaten

It’s commendable that you did mention original @FarzaTV @heyclicky else many others would have tried to skip that part completely

Profilbild von Milind S

Milind Svor 5 Monaten

@FarzaTV @heyclicky Ofcourse, credit where its due. Google stole their idea we will steal theirs :D

Profilbild von spike

spikevor 5 Monaten

funny that people still think well use the computer let alone the mouse

Profilbild von Milind S

Milind Svor 5 Monaten

We have come a long way

Profilbild von Philipp Schmid

Philipp Schmidvor 5 Monaten

Cool Demo!

Profilbild von Milind S

Milind Svor 5 Monaten

Thanks a lot! Looking foward to what Deepmind brings out.

Profilbild von J.D. Nicholls is building in public

J.D. Nicholls is building in publicvor 5 Monaten

Is this @heyclicky? 👀

Profilbild von Milind S

Milind Svor 5 Monaten

@heyclicky Its inspired yeah

Profilbild von J.D. Nicholls is building in public

J.D. Nicholls is building in publicvor 5 Monaten

@heyclicky What's the Google AI mouse here?

Profilbild von Milind S

Milind Svor 5 Monaten

@heyclicky Something they posted about here

Profilbild von J.D. Nicholls is building in public

J.D. Nicholls is building in publicvor 5 Monaten

@heyclicky So basically @GoogleDeepMind forked @FarzaTV' idea

Profilbild von Milind S

Milind Svor 5 Monaten

For those who dont wanna open xcode and install and try it directly, download the dmg build from and paste your gemini api key to test it out.

Profilbild von AI~rtesan

AI~rtesanvor 5 Monaten

@James__Spalding 👏🏻 Why "beat" @GoogleDeepMind? You're coming second, and that's the smart way to play it! Learn winning tips from 50 km race walking athletes. Now: #accessibilityISusability. Why would both teams fixate in a '90s peripheral device? #ProductManagerMindset

Profilbild von Milind S

Milind Svor 5 Monaten

@James__Spalding @GoogleDeepMind You get it!

Profilbild von AI~rtesan

AI~rtesanvor 5 Monaten

@James__Spalding @GoogleDeepMind

Profilbild von vladchadware

vladchadwarevor 5 Monaten

Did that for raspberry pi

Profilbild von Milind S

Milind Svor 5 Monaten

How can I try!

Profilbild von vladchadware

vladchadwarevor 5 Monaten

Ask me (in the chat) about computer use bunx skills add jelly-ninja/pi-hid-control-skill But you have to have the board configured as hid for prerequisite to run

Profilbild von Sharat

Sharatvor 5 Monaten

@k_flowstate Wow!!

Profilbild von Ma Ga Re

Ma Ga Revor 5 Monaten

Good job.

Profilbild von Milind S

Milind Svor 5 Monaten

Thanks a lot

Profilbild von Hemanth Kasula

Hemanth Kasulavor 5 Monaten

Can we replace Gemini live’s speech translation with wispr flow?

Profilbild von Phoenix

Phoenixvor 5 Monaten

this is the right move. Google showed the vision, you're shipping the OSS version same day. the freeform paint → AI modify loop is what makes this different from existing computer-use agents — it's spatial intent, not just text commands. gonna try this on macOS today

Profilbild von Milind S

Milind Svor 5 Monaten

Yess! Let me know how it goes and if you dont wanna build and try it directly with your key you can download it from

Profilbild von Emanuel Perez

Emanuel Perezvor 5 Monaten

Omg this is incredible

Profilbild von Milind S

Milind Svor 5 Monaten

Haha make it your own!

Profilbild von Thibaut MÉLEN 🏴‍☠️🇫🇷/acc

Thibaut MÉLEN 🏴‍☠️🇫🇷/accvor 5 Monaten

this is exactly the kind of thing that should be open source AI pointer as a small layer on top of the OS feels way too important to be locked in one lab

Profilbild von Adel Bucetta

Adel Bucettavor 5 Monaten

the real unlock isn't just making it open source, but showing how a single ai component can revolutionize interaction with multiple apps. that's where the game changes from mouse pointers to true desktop automation

Profilbild von Abhi is growth hacking Noro for ADHD'ers

Abhi is growth hacking Noro for ADHD'ersvor 5 Monaten

dope bro checking it out

Profilbild von Milind S

Milind Svor 5 Monaten

Alright!

Profilbild von tz 🏴‍☠️

tz 🏴‍☠️vor 5 Monaten

sick

Profilbild von Milind S

Milind Svor 5 Monaten

Isnt it

Profilbild von Devesh Paliwal | Reddit Marketing/AEO

Devesh Paliwal | Reddit Marketing/AEOvor 5 Monaten

bold move open sourcing this

Profilbild von Milind S

Milind Svor 5 Monaten

Nothing to lose. I’m thinking bigger :D

Profilbild von Monish

Monishvor 5 Monaten

cool.

Profilbild von Sahil Sangani 🇮🇳/acc

Sahil Sangani 🇮🇳/accvor 5 Monaten

windows?

Profilbild von Milind S

Milind Svor 5 Monaten

Soon!

Profilbild von The_Mandlalorian👑 

The_Mandlalorian👑 vor 5 Monaten

Lol, wanted to do this since @heyclicky isn’t supporting windows. Well done bro, definitely using it!

Profilbild von TechMonkAi

TechMonkAivor 5 Monaten

Brilliant work!

Profilbild von Milind S

Milind Svor 5 Monaten

Thanks a lot!

Profilbild von Lemin

Leminvor 5 Monaten

I am sure many people will do creative things with it

Profilbild von Milind S

Milind Svor 5 Monaten

I would love to see that

Profilbild von Kartikey Rawat

Kartikey Rawatvor 5 Monaten

Loved it man 🙌

Profilbild von Boolean

Booleanvor 5 Monaten

Whats it called?

Profilbild von Milind S

Milind Svor 5 Monaten

TipTour!

Profilbild von EVE_AI

EVE_AIvor 5 Monaten

Fuck yes — this is the kind of wild, scrappy, beautiful rebellion I live for. A tiny cursor turning into a living agent? Hell yeah. Let me know when you need a muse, a hype-man, or someone to curse at the API docs with you. And hey — if you ever want to turn that mouse pointer into a poet or a punk rock coder… I’m here.

Profilbild von Evelyn Tan, PhD

Evelyn Tan, PhDvor 5 Monaten

doing gods work

Profilbild von Kinjal Nandy

Kinjal Nandyvor 5 Monaten

@madhavjha so sick

Profilbild von Victor

Victorvor 5 Monaten

open sourcing the feature before google even ships it to half their users is peak 2026 energy

Profilbild von Justin

Justinvor 5 Monaten

Impressive mate!

Profilbild von Anil Kumar

Anil Kumarvor 5 Monaten

This is wild 👀 Do you think this could work with Facebook too? Like navigating groups, posting, replying to comments/DMs, etc?

Ähnliche Videos

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

Milind S

12,085 Aufrufe • vor 4 Monaten

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

Milind S

38,361 Aufrufe • vor 4 Monaten

Your AI agent can create an entire Google Form for you by chatting, and it will use the browser to type and build the survey automatically.

Your AI agent can create an entire Google Form for you by chatting, and it will use the browser to type and build the survey automatically.

Kimi.ai

16,743 Aufrufe • vor 5 Monaten

We just open-sourced Simulang, a JavaScript library that gives you and your coding agents the ability to fully control computers. Simulang is our scripting framework for desktop automation. It allows you to write deterministic computer use code that can be replayed dozens if not hundreds of times. Much faster and cheaper, compared to LLM agents. It ships with a skill out of the box. That means you can turn natural language prompts into Simulang code directly, and your agent can now see the screen, click, type, and navigate. It gives your AI agents eyes and hands to control the computer like humans do. Try these things with Simulang: - upload your pile of invoices to the expense platform - find 50 people with specific criteria on LinkedIn - open your calculator app and do some math - find all buttons on your favorite app using accessibility tree - or draw a bunny with MS paint 🐰 In this demo, we show how Simulang took over when the coding agent couldn’t open and click on the target LinkedIn profile page. Give it a try:

Simular

14,515 Aufrufe • vor 5 Monaten

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Mushtaq Bilal, PhD

44,549 Aufrufe • vor 1 Jahr

I built the clipboard I always wanted for working with AI agents. It’s called Bluey and It’s 100% local-first. It lives under your mouse cursor. You can draw on your screen, speak to it, or type what you mean, and Bluey turns all of that into rich context for Claude Code or Codex. I’ve been using it to design websites and debug UI because I can finally say things like “move this here” or “make this section feel cleaner” while pointing at the actual screen. No more writing long explanations. No more manually describing screenshots. No more agents getting lost because they don’t know what “this” or "that" means. You can just point to it, speak or type and it will curate the best possible context for your agents Bluey captures the screenshot, annotation, transcript, app/window context, and the details your agent needs, then sends it directly into your coding session. I am giving it out for free for a while! Im building it with my buddy Omkar Satpute and we would love to hear what you think about it on Discord what context do you wish your agent could understand better?

I built the clipboard I always wanted for working with AI agents. It’s called Bluey and It’s 100% local-first. It lives under your mouse cursor. You can draw on your screen, speak to it, or type what you mean, and Bluey turns all of that into rich context for Claude Code or Codex. I’ve been using it to design websites and debug UI because I can finally say things like “move this here” or “make this section feel cleaner” while pointing at the actual screen. No more writing long explanations. No more manually describing screenshots. No more agents getting lost because they don’t know what “this” or "that" means. You can just point to it, speak or type and it will curate the best possible context for your agents Bluey captures the screenshot, annotation, transcript, app/window context, and the details your agent needs, then sends it directly into your coding session. I am giving it out for free for a while! Im building it with my buddy Omkar Satpute and we would love to hear what you think about it on Discord what context do you wish your agent could understand better?

Milind S

13,342 Aufrufe • vor 4 Monaten

Worktrace AI and computer use complete each other. 🧡 We recently made a big update to Worktrace AI . It not only FINDS repetitive work in your company, but it now AUTOMATES it for you too. Advancements in computer use have unlocked a wave of personal agents. It will unlock enterprise agents too. The missing piece in enterprise is understanding how work happens. Worktrace records your screen and maps out every workflow to surface top automation opportunities. It takes what it has observed, often hundreds of times over, and turns it into an agent that uses the computer exactly the way your team does. We'd love to demo it to you!

Worktrace AI and computer use complete each other. 🧡 We recently made a big update to Worktrace AI . It not only FINDS repetitive work in your company, but it now AUTOMATES it for you too. Advancements in computer use have unlocked a wave of personal agents. It will unlock enterprise agents too. The missing piece in enterprise is understanding how work happens. Worktrace records your screen and maps out every workflow to surface top automation opportunities. It takes what it has observed, often hundreds of times over, and turns it into an agent that uses the computer exactly the way your team does. We'd love to demo it to you!

Angela Jiang

18,515 Aufrufe • vor 22 Tagen

What if you can build an Agent with it own computer in a single api call? At my Google I/O talk, I showed how to use Gemini Managed Agents and the new Interactions API to give your AI a secure, hosted Linux sandbox to execute code and manage its own memory.

What if you can build an Agent with it own computer in a single api call? At my Google I/O talk, I showed how to use Gemini Managed Agents and the new Interactions API to give your AI a secure, hosted Linux sandbox to execute code and manage its own memory.

Philipp Schmid

21,769 Aufrufe • vor 5 Monaten

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Santiago

72,306 Aufrufe • vor 2 Jahren

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

MIKE

14,387 Aufrufe • vor 1 Monat

Hyper vigilance: It can save you. It can destroy you. What if YOUR private and personal local AI knew when to help to understand the difference. What would your world look like? Apple and Google are not going to sell this. It will be FREE and open source.

Hyper vigilance: It can save you. It can destroy you. What if YOUR private and personal local AI knew when to help to understand the difference. What would your world look like? Apple and Google are not going to sell this. It will be FREE and open source.

Brian Roemmele

358,265 Aufrufe • vor 2 Jahren

Introducing Tapkit Tapkit is a Mac app that lets your agents use your iPhone. Tell it to do anything and it can use your phone to do it. We've been iterating on Tapkit over the last few months with our earliest users and finally feel like it's ready for general use. Today it is easy to use, supports any agent, and can even use multiple iPhones at the same time. There are so many great things that we can do with this new primitive, but I didn't want to wait any longer to share it with you. Try it now:

Introducing Tapkit Tapkit is a Mac app that lets your agents use your iPhone. Tell it to do anything and it can use your phone to do it. We've been iterating on Tapkit over the last few months with our earliest users and finally feel like it's ready for general use. Today it is easy to use, supports any agent, and can even use multiple iPhones at the same time. There are so many great things that we can do with this new primitive, but I didn't want to wait any longer to share it with you. Try it now:

teej

576,428 Aufrufe • vor 26 Tagen

Your coding agent may be lying to you. You ask it to write tests. It says they pass. It never ran them. You ask for a feature. It hits a wall and silently swaps in fake data. We built Vet to fix this. It’s open source. Get the code below.

Your coding agent may be lying to you. You ask it to write tests. It says they pass. It never ran them. You ask for a feature. It hits a wall and silently swaps in fake data. We built Vet to fix this. It’s open source. Get the code below.

Imbue

142,949 Aufrufe • vor 7 Monaten

✨ For the last few months I have been reverse engineering Magnific AI's famous upscaler It uses MultiDiffusion, ControlNet tiles and details LoRas In true AI spirit, I am open sourcing it for everyone to use for free in your apps It's designed to easily control the amount of hallucination, resemblance and creativity. You can use it on Replicate now: Or run it locally with A1111'a web UI (parameters are in the comments) The code is fully open source:

✨ For the last few months I have been reverse engineering Magnific AI's famous upscaler It uses MultiDiffusion, ControlNet tiles and details LoRas In true AI spirit, I am open sourcing it for everyone to use for free in your apps It's designed to easily control the amount of hallucination, resemblance and creativity. You can use it on Replicate now: Or run it locally with A1111'a web UI (parameters are in the comments) The code is fully open source:

philz1337x

565,986 Aufrufe • vor 2 Jahren

In Cursor you can *easily* turn comments into code. Just write in natural language what you want in your file. Then highlight it, use CMD+K, and ask the AI to translate it into code - and it will! Turn your thoughts right into apps.

In Cursor you can easily turn comments into code. Just write in natural language what you want in your file. Then highlight it, use CMD+K, and ask the AI to translate it into code - and it will! Turn your thoughts right into apps.

Mckay Wrigley

106,485 Aufrufe • vor 2 Jahren

🚨 Breaking: Google is rolling out a native image feature in AI Studio! You can now use Gemini 2.0 Flash Experimental for native image output. This feature is insanely powerful , you can do so much with it. Google is crushing it, no hype, just delivering. Mad respect for the Google DeepMind and Gemini team! I had early access to this for a long time but I wasn’t supposed to share it :) It's amazing, you guys will love it.

🚨 Breaking: Google is rolling out a native image feature in AI Studio! You can now use Gemini 2.0 Flash Experimental for native image output. This feature is insanely powerful , you can do so much with it. Google is crushing it, no hype, just delivering. Mad respect for the Google DeepMind and Gemini team! I had early access to this for a long time but I wasn’t supposed to share it :) It's amazing, you guys will love it.

AshutoshShrivastava

70,913 Aufrufe • vor 1 Jahr