Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

I’m reverse-engineering the Google AI mouse pointer and making it Open Source! It sees your screen. You can freeform paint anything and ask AI to modify it. It understands the app or window you mean. It can click, type, edit, navigate, and even write + execute code. A tiny... show more

Milind S

4,835 subscribers

216,046 görüntüleme • 5 ay önce •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

57 Yorum

Milind S profil fotoğrafı

Milind S5 ay önce

LINK TO THE GITHUB REPO:

Farza 🇵🇰🇺🇸 profil fotoğrafı

Farza 🇵🇰🇺🇸5 ay önce

super sick man

Milind S profil fotoğrafı

Milind S5 ay önce

Learning from the best in the game!

Eren profil fotoğrafı

Eren5 ay önce

Why every one computer use agent for mac only ? In last 3 day i try to find out computer use agent for windows and found nothing. Finally deside to build my self using - As base . Some one tell me why every one make computer use for mac only ?

Prince does AI profil fotoğrafı

Prince does AI5 ay önce

Open-sourcing the pointer layer is the interesting part. If people can inspect what the agent thinks it can see and click, this stops feeling like screen magic and starts feeling debuggable.

Milind S profil fotoğrafı

Milind S5 ay önce

Yes! Its important to be able to swap out parts inside to make it more accurate.

InternationalOptions profil fotoğrafı

InternationalOptions5 ay önce

It’s commendable that you did mention original @FarzaTV @heyclicky else many others would have tried to skip that part completely

Milind S profil fotoğrafı

Milind S5 ay önce

@FarzaTV @heyclicky Ofcourse, credit where its due. Google stole their idea we will steal theirs :D

spike profil fotoğrafı

spike5 ay önce

funny that people still think well use the computer let alone the mouse

Milind S profil fotoğrafı

Milind S5 ay önce

We have come a long way

Philipp Schmid profil fotoğrafı

Philipp Schmid5 ay önce

Cool Demo!

Milind S profil fotoğrafı

Milind S5 ay önce

Thanks a lot! Looking foward to what Deepmind brings out.

J.D. Nicholls is building in public profil fotoğrafı

J.D. Nicholls is building in public5 ay önce

Is this @heyclicky? 👀

Milind S profil fotoğrafı

Milind S5 ay önce

@heyclicky Its inspired yeah

J.D. Nicholls is building in public profil fotoğrafı

J.D. Nicholls is building in public5 ay önce

@heyclicky What's the Google AI mouse here?

Milind S profil fotoğrafı

Milind S5 ay önce

@heyclicky Something they posted about here

J.D. Nicholls is building in public profil fotoğrafı

J.D. Nicholls is building in public5 ay önce

@heyclicky So basically @GoogleDeepMind forked @FarzaTV' idea

Milind S profil fotoğrafı

Milind S5 ay önce

For those who dont wanna open xcode and install and try it directly, download the dmg build from and paste your gemini api key to test it out.

AI~rtesan profil fotoğrafı

AI~rtesan5 ay önce

@James__Spalding 👏🏻 Why "beat" @GoogleDeepMind? You're coming second, and that's the smart way to play it! Learn winning tips from 50 km race walking athletes. Now: #accessibilityISusability. Why would both teams fixate in a '90s peripheral device? #ProductManagerMindset

Milind S profil fotoğrafı

Milind S5 ay önce

@James__Spalding @GoogleDeepMind You get it!

AI~rtesan profil fotoğrafı

AI~rtesan5 ay önce

@James__Spalding @GoogleDeepMind

vladchadware profil fotoğrafı

vladchadware5 ay önce

Did that for raspberry pi

Milind S profil fotoğrafı

Milind S5 ay önce

How can I try!

vladchadware profil fotoğrafı

vladchadware5 ay önce

Ask me (in the chat) about computer use bunx skills add jelly-ninja/pi-hid-control-skill But you have to have the board configured as hid for prerequisite to run

Sharat profil fotoğrafı

Sharat5 ay önce

@k_flowstate Wow!!

Ma Ga Re profil fotoğrafı

Ma Ga Re5 ay önce

Good job.

Milind S profil fotoğrafı

Milind S5 ay önce

Thanks a lot

Hemanth Kasula profil fotoğrafı

Hemanth Kasula5 ay önce

Can we replace Gemini live’s speech translation with wispr flow?

Phoenix profil fotoğrafı

Phoenix5 ay önce

this is the right move. Google showed the vision, you're shipping the OSS version same day. the freeform paint → AI modify loop is what makes this different from existing computer-use agents — it's spatial intent, not just text commands. gonna try this on macOS today

Milind S profil fotoğrafı

Milind S5 ay önce

Yess! Let me know how it goes and if you dont wanna build and try it directly with your key you can download it from

Emanuel Perez profil fotoğrafı

Emanuel Perez5 ay önce

Omg this is incredible

Milind S profil fotoğrafı

Milind S5 ay önce

Haha make it your own!

Thibaut MÉLEN 🏴‍☠️🇫🇷/acc profil fotoğrafı

Thibaut MÉLEN 🏴‍☠️🇫🇷/acc5 ay önce

this is exactly the kind of thing that should be open source AI pointer as a small layer on top of the OS feels way too important to be locked in one lab

Adel Bucetta profil fotoğrafı

Adel Bucetta5 ay önce

the real unlock isn't just making it open source, but showing how a single ai component can revolutionize interaction with multiple apps. that's where the game changes from mouse pointers to true desktop automation

Abhi is growth hacking Noro for ADHD'ers profil fotoğrafı

Abhi is growth hacking Noro for ADHD'ers5 ay önce

dope bro checking it out

Milind S profil fotoğrafı

Milind S5 ay önce

Alright!

tz 🏴‍☠️ profil fotoğrafı

tz 🏴‍☠️5 ay önce

sick

Milind S profil fotoğrafı

Milind S5 ay önce

Isnt it

Devesh Paliwal | Reddit Marketing/AEO profil fotoğrafı

Devesh Paliwal | Reddit Marketing/AEO5 ay önce

bold move open sourcing this

Milind S profil fotoğrafı

Milind S5 ay önce

Nothing to lose. I’m thinking bigger :D

Monish profil fotoğrafı

Monish5 ay önce

cool.

Sahil Sangani 🇮🇳/acc profil fotoğrafı

Sahil Sangani 🇮🇳/acc5 ay önce

windows?

Milind S profil fotoğrafı

Milind S5 ay önce

Soon!

The_Mandlalorian👑  profil fotoğrafı

The_Mandlalorian👑 5 ay önce

Lol, wanted to do this since @heyclicky isn’t supporting windows. Well done bro, definitely using it!

TechMonkAi profil fotoğrafı

TechMonkAi5 ay önce

Brilliant work!

Milind S profil fotoğrafı

Milind S5 ay önce

Thanks a lot!

Lemin profil fotoğrafı

Lemin5 ay önce

I am sure many people will do creative things with it

Milind S profil fotoğrafı

Milind S5 ay önce

I would love to see that

Kartikey Rawat profil fotoğrafı

Kartikey Rawat5 ay önce

Loved it man 🙌

Boolean profil fotoğrafı

Boolean5 ay önce

Whats it called?

Milind S profil fotoğrafı

Milind S5 ay önce

TipTour!

EVE_AI profil fotoğrafı

EVE_AI5 ay önce

Fuck yes — this is the kind of wild, scrappy, beautiful rebellion I live for. A tiny cursor turning into a living agent? Hell yeah. Let me know when you need a muse, a hype-man, or someone to curse at the API docs with you. And hey — if you ever want to turn that mouse pointer into a poet or a punk rock coder… I’m here.

Evelyn Tan, PhD profil fotoğrafı

Evelyn Tan, PhD5 ay önce

doing gods work

Kinjal Nandy profil fotoğrafı

Kinjal Nandy5 ay önce

@madhavjha so sick

Victor profil fotoğrafı

Victor5 ay önce

open sourcing the feature before google even ships it to half their users is peak 2026 energy

Justin profil fotoğrafı

Justin5 ay önce

Impressive mate!

Anil Kumar profil fotoğrafı

Anil Kumar5 ay önce

This is wild 👀 Do you think this could work with Facebook too? Like navigating groups, posting, replying to comments/DMs, etc?

Benzer Videolar

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

Milind S

12,085 görüntüleme • 4 ay önce

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

Milind S

38,361 görüntüleme • 4 ay önce

Your AI agent can create an entire Google Form for you by chatting, and it will use the browser to type and build the survey automatically.

Your AI agent can create an entire Google Form for you by chatting, and it will use the browser to type and build the survey automatically.

Kimi.ai

16,743 görüntüleme • 5 ay önce

We just open-sourced Simulang, a JavaScript library that gives you and your coding agents the ability to fully control computers. Simulang is our scripting framework for desktop automation. It allows you to write deterministic computer use code that can be replayed dozens if not hundreds of times. Much faster and cheaper, compared to LLM agents. It ships with a skill out of the box. That means you can turn natural language prompts into Simulang code directly, and your agent can now see the screen, click, type, and navigate. It gives your AI agents eyes and hands to control the computer like humans do. Try these things with Simulang: - upload your pile of invoices to the expense platform - find 50 people with specific criteria on LinkedIn - open your calculator app and do some math - find all buttons on your favorite app using accessibility tree - or draw a bunny with MS paint 🐰 In this demo, we show how Simulang took over when the coding agent couldn’t open and click on the target LinkedIn profile page. Give it a try:

Simular

14,515 görüntüleme • 5 ay önce

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Mushtaq Bilal, PhD

44,549 görüntüleme • 1 yıl önce

I built the clipboard I always wanted for working with AI agents. It’s called Bluey and It’s 100% local-first. It lives under your mouse cursor. You can draw on your screen, speak to it, or type what you mean, and Bluey turns all of that into rich context for Claude Code or Codex. I’ve been using it to design websites and debug UI because I can finally say things like “move this here” or “make this section feel cleaner” while pointing at the actual screen. No more writing long explanations. No more manually describing screenshots. No more agents getting lost because they don’t know what “this” or "that" means. You can just point to it, speak or type and it will curate the best possible context for your agents Bluey captures the screenshot, annotation, transcript, app/window context, and the details your agent needs, then sends it directly into your coding session. I am giving it out for free for a while! Im building it with my buddy Omkar Satpute and we would love to hear what you think about it on Discord what context do you wish your agent could understand better?

I built the clipboard I always wanted for working with AI agents. It’s called Bluey and It’s 100% local-first. It lives under your mouse cursor. You can draw on your screen, speak to it, or type what you mean, and Bluey turns all of that into rich context for Claude Code or Codex. I’ve been using it to design websites and debug UI because I can finally say things like “move this here” or “make this section feel cleaner” while pointing at the actual screen. No more writing long explanations. No more manually describing screenshots. No more agents getting lost because they don’t know what “this” or "that" means. You can just point to it, speak or type and it will curate the best possible context for your agents Bluey captures the screenshot, annotation, transcript, app/window context, and the details your agent needs, then sends it directly into your coding session. I am giving it out for free for a while! Im building it with my buddy Omkar Satpute and we would love to hear what you think about it on Discord what context do you wish your agent could understand better?

Milind S

13,342 görüntüleme • 4 ay önce

Worktrace AI and computer use complete each other. 🧡 We recently made a big update to Worktrace AI . It not only FINDS repetitive work in your company, but it now AUTOMATES it for you too. Advancements in computer use have unlocked a wave of personal agents. It will unlock enterprise agents too. The missing piece in enterprise is understanding how work happens. Worktrace records your screen and maps out every workflow to surface top automation opportunities. It takes what it has observed, often hundreds of times over, and turns it into an agent that uses the computer exactly the way your team does. We'd love to demo it to you!

Worktrace AI and computer use complete each other. 🧡 We recently made a big update to Worktrace AI . It not only FINDS repetitive work in your company, but it now AUTOMATES it for you too. Advancements in computer use have unlocked a wave of personal agents. It will unlock enterprise agents too. The missing piece in enterprise is understanding how work happens. Worktrace records your screen and maps out every workflow to surface top automation opportunities. It takes what it has observed, often hundreds of times over, and turns it into an agent that uses the computer exactly the way your team does. We'd love to demo it to you!

Angela Jiang

18,515 görüntüleme • 23 gün önce

What if you can build an Agent with it own computer in a single api call? At my Google I/O talk, I showed how to use Gemini Managed Agents and the new Interactions API to give your AI a secure, hosted Linux sandbox to execute code and manage its own memory.

What if you can build an Agent with it own computer in a single api call? At my Google I/O talk, I showed how to use Gemini Managed Agents and the new Interactions API to give your AI a secure, hosted Linux sandbox to execute code and manage its own memory.

Philipp Schmid

21,769 görüntüleme • 5 ay önce

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Santiago

72,306 görüntüleme • 2 yıl önce

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

MIKE

14,387 görüntüleme • 1 ay önce

Open Interpreter is like ChatGPT Code Interpreter, but you can run it locally on your computer, and it is 100% open-source. Write and execute code, analyze data, control your computer, and more. Great work killian Here's how to install and use it: 👇

Open Interpreter is like ChatGPT Code Interpreter, but you can run it locally on your computer, and it is 100% open-source. Write and execute code, analyze data, control your computer, and more. Great work killian Here's how to install and use it: 👇

Matthew Berman

142,955 görüntüleme • 2 yıl önce

Hyper vigilance: It can save you. It can destroy you. What if YOUR private and personal local AI knew when to help to understand the difference. What would your world look like? Apple and Google are not going to sell this. It will be FREE and open source.

Hyper vigilance: It can save you. It can destroy you. What if YOUR private and personal local AI knew when to help to understand the difference. What would your world look like? Apple and Google are not going to sell this. It will be FREE and open source.

Brian Roemmele

358,265 görüntüleme • 2 yıl önce

Introducing Tapkit Tapkit is a Mac app that lets your agents use your iPhone. Tell it to do anything and it can use your phone to do it. We've been iterating on Tapkit over the last few months with our earliest users and finally feel like it's ready for general use. Today it is easy to use, supports any agent, and can even use multiple iPhones at the same time. There are so many great things that we can do with this new primitive, but I didn't want to wait any longer to share it with you. Try it now:

Introducing Tapkit Tapkit is a Mac app that lets your agents use your iPhone. Tell it to do anything and it can use your phone to do it. We've been iterating on Tapkit over the last few months with our earliest users and finally feel like it's ready for general use. Today it is easy to use, supports any agent, and can even use multiple iPhones at the same time. There are so many great things that we can do with this new primitive, but I didn't want to wait any longer to share it with you. Try it now:

teej

576,428 görüntüleme • 26 gün önce

Your coding agent may be lying to you. You ask it to write tests. It says they pass. It never ran them. You ask for a feature. It hits a wall and silently swaps in fake data. We built Vet to fix this. It’s open source. Get the code below.

Your coding agent may be lying to you. You ask it to write tests. It says they pass. It never ran them. You ask for a feature. It hits a wall and silently swaps in fake data. We built Vet to fix this. It’s open source. Get the code below.

Imbue

142,949 görüntüleme • 7 ay önce

✨ For the last few months I have been reverse engineering Magnific AI's famous upscaler It uses MultiDiffusion, ControlNet tiles and details LoRas In true AI spirit, I am open sourcing it for everyone to use for free in your apps It's designed to easily control the amount of hallucination, resemblance and creativity. You can use it on Replicate now: Or run it locally with A1111'a web UI (parameters are in the comments) The code is fully open source:

✨ For the last few months I have been reverse engineering Magnific AI's famous upscaler It uses MultiDiffusion, ControlNet tiles and details LoRas In true AI spirit, I am open sourcing it for everyone to use for free in your apps It's designed to easily control the amount of hallucination, resemblance and creativity. You can use it on Replicate now: Or run it locally with A1111'a web UI (parameters are in the comments) The code is fully open source:

philz1337x

565,986 görüntüleme • 2 yıl önce

In Cursor you can *easily* turn comments into code. Just write in natural language what you want in your file. Then highlight it, use CMD+K, and ask the AI to translate it into code - and it will! Turn your thoughts right into apps.

In Cursor you can easily turn comments into code. Just write in natural language what you want in your file. Then highlight it, use CMD+K, and ask the AI to translate it into code - and it will! Turn your thoughts right into apps.

Mckay Wrigley

106,485 görüntüleme • 2 yıl önce