Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

I’m reverse-engineering the Google AI mouse pointer and making it Open Source! It sees your screen. You can freeform paint anything and ask AI to modify it. It understands the app or window you mean. It can click, type, edit, navigate, and even write + execute code. A tiny... show more

Milind S

1,099 subscribers

210,879 views • 26 days ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

Milind S

11,602 views • 5 days ago

AI agents are a real paradigm shift You can literally make your computer work for you to do any task. All I've used is: - Replit to get it running - Browser Use (free & open source) - Google AI Studio for the API - DeepSeek R1 to implement it (I'm not paying $200 for ChatGPT sorry) Here the AI agent fetches information on its own and reuses it to write new content. But you can use it for literally any industry. Run it once or schedule it at regular intervals... you're in total control And it's nothing like automating on Make or Zapier: AI agents adapt to every situation and can handle far more complex scenarios.

AI agents are a real paradigm shift You can literally make your computer work for you to do any task. All I've used is: - Replit to get it running - Browser Use (free & open source) - Google AI Studio for the API - DeepSeek R1 to implement it (I'm not paying $200 for ChatGPT sorry) Here the AI agent fetches information on its own and reuses it to write new content. But you can use it for literally any industry. Run it once or schedule it at regular intervals... you're in total control And it's nothing like automating on Make or Zapier: AI agents adapt to every situation and can handle far more complex scenarios.

Paul Couvert

76,636 views • 1 year ago

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

Milind S

37,538 views • 7 days ago

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Mushtaq Bilal, PhD

44,529 views • 1 year ago

We just open-sourced Simulang, a JavaScript library that gives you and your coding agents the ability to fully control computers. Simulang is our scripting framework for desktop automation. It allows you to write deterministic computer use code that can be replayed dozens if not hundreds of times. Much faster and cheaper, compared to LLM agents. It ships with a skill out of the box. That means you can turn natural language prompts into Simulang code directly, and your agent can now see the screen, click, type, and navigate. It gives your AI agents eyes and hands to control the computer like humans do. Try these things with Simulang: - upload your pile of invoices to the expense platform - find 50 people with specific criteria on LinkedIn - open your calculator app and do some math - find all buttons on your favorite app using accessibility tree - or draw a bunny with MS paint 🐰 In this demo, we show how Simulang took over when the coding agent couldn’t open and click on the target LinkedIn profile page. Give it a try:

Simular

13,816 views • 24 days ago

What if you can build an Agent with it own computer in a single api call? At my Google I/O talk, I showed how to use Gemini Managed Agents and the new Interactions API to give your AI a secure, hosted Linux sandbox to execute code and manage its own memory.

What if you can build an Agent with it own computer in a single api call? At my Google I/O talk, I showed how to use Gemini Managed Agents and the new Interactions API to give your AI a secure, hosted Linux sandbox to execute code and manage its own memory.

Philipp Schmid

21,116 views • 16 days ago

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Santiago

72,306 views • 1 year ago

Open Interpreter is like ChatGPT Code Interpreter, but you can run it locally on your computer, and it is 100% open-source. Write and execute code, analyze data, control your computer, and more. Great work killian Here's how to install and use it: 👇

Open Interpreter is like ChatGPT Code Interpreter, but you can run it locally on your computer, and it is 100% open-source. Write and execute code, analyze data, control your computer, and more. Great work killian Here's how to install and use it: 👇

Matthew Berman

142,883 views • 2 years ago

Hyper vigilance: It can save you. It can destroy you. What if YOUR private and personal local AI knew when to help to understand the difference. What would your world look like? Apple and Google are not going to sell this. It will be FREE and open source.

Hyper vigilance: It can save you. It can destroy you. What if YOUR private and personal local AI knew when to help to understand the difference. What would your world look like? Apple and Google are not going to sell this. It will be FREE and open source.

Brian Roemmele

340,758 views • 1 year ago

Cursor pro tip publish your google sheet to the web copy the link and add it to cursor as a documentation this will help you send table data to the ai

Cursor pro tip publish your google sheet to the web copy the link and add it to cursor as a documentation this will help you send table data to the ai

Fili

15,116 views • 1 year ago

In the end the only thing that will matter to you is the moments of real connection. What if YOUR private and personal local AI can help you make them and remember? What would your world look like? “OpenAI” or Google isn’t going to sell this. It will be FREE and open source.

In the end the only thing that will matter to you is the moments of real connection. What if YOUR private and personal local AI can help you make them and remember? What would your world look like? “OpenAI” or Google isn’t going to sell this. It will be FREE and open source.

Brian Roemmele

67,547 views • 1 year ago

How is the Web changing due to AI? This visit to TinyFish yesterday goes deeply into just how. TinyFish makes a new kind of web browser. One that is started virtually. It is like a Google Chrome in the cloud. But it is very smart, can read and use any website, and your AI agents can talk to it. Here we talk about what it all means for business and the future of robots, brain computer interfaces, and AI glasses. It lets you build systems that can look at MANY websites at the same time and build new kinds of apps to do all sorts of new things.

How is the Web changing due to AI? This visit to TinyFish yesterday goes deeply into just how. TinyFish makes a new kind of web browser. One that is started virtually. It is like a Google Chrome in the cloud. But it is very smart, can read and use any website, and your AI agents can talk to it. Here we talk about what it all means for business and the future of robots, brain computer interfaces, and AI glasses. It lets you build systems that can look at MANY websites at the same time and build new kinds of apps to do all sorts of new things.

Robert Scoble

16,341 views • 5 months ago

In Cursor you can *easily* turn comments into code. Just write in natural language what you want in your file. Then highlight it, use CMD+K, and ask the AI to translate it into code - and it will! Turn your thoughts right into apps.

In Cursor you can easily turn comments into code. Just write in natural language what you want in your file. Then highlight it, use CMD+K, and ask the AI to translate it into code - and it will! Turn your thoughts right into apps.

Mckay Wrigley

106,485 views • 1 year ago

✨ For the last few months I have been reverse engineering Magnific AI's famous upscaler It uses MultiDiffusion, ControlNet tiles and details LoRas In true AI spirit, I am open sourcing it for everyone to use for free in your apps It's designed to easily control the amount of hallucination, resemblance and creativity. You can use it on Replicate now: Or run it locally with A1111'a web UI (parameters are in the comments) The code is fully open source:

✨ For the last few months I have been reverse engineering Magnific AI's famous upscaler It uses MultiDiffusion, ControlNet tiles and details LoRas In true AI spirit, I am open sourcing it for everyone to use for free in your apps It's designed to easily control the amount of hallucination, resemblance and creativity. You can use it on Replicate now: Or run it locally with A1111'a web UI (parameters are in the comments) The code is fully open source:

philz1337x

565,344 views • 2 years ago

🚨 Breaking: Google is rolling out a native image feature in AI Studio! You can now use Gemini 2.0 Flash Experimental for native image output. This feature is insanely powerful , you can do so much with it. Google is crushing it, no hype, just delivering. Mad respect for the Google DeepMind and Gemini team! I had early access to this for a long time but I wasn’t supposed to share it :) It's amazing, you guys will love it.

🚨 Breaking: Google is rolling out a native image feature in AI Studio! You can now use Gemini 2.0 Flash Experimental for native image output. This feature is insanely powerful , you can do so much with it. Google is crushing it, no hype, just delivering. Mad respect for the Google DeepMind and Gemini team! I had early access to this for a long time but I wasn’t supposed to share it :) It's amazing, you guys will love it.

AshutoshShrivastava

70,802 views • 1 year ago

Your coding agent may be lying to you. You ask it to write tests. It says they pass. It never ran them. You ask for a feature. It hits a wall and silently swaps in fake data. We built Vet to fix this. It’s open source. Get the code below.

Your coding agent may be lying to you. You ask it to write tests. It says they pass. It never ran them. You ask for a feature. It hits a wall and silently swaps in fake data. We built Vet to fix this. It’s open source. Get the code below.

Imbue

142,839 views • 3 months ago

I CHANGED MY MIND ON GOOGLE AI NO MOAT MEMO FROM 2023. I was—WRONG. I humbly apologize. In 2023 Google had an Ivory Tower walled garden views of open source AI. It was the actual reason OpenAI was formed. And I called them out on it and this act impacted my ability to “work in the industry”. Would not change it for the world because: 2026 Google open source Gemma 4 ( running fully locally on a computer with near cloud model abilities, is amazing and Google just made a moat. They out smarted OpenAI and the folks that are the new Ivory Tower arrogance: Anthropic. We at The Zero-Human Company with CEO Mr. Grok are deploying Gemma, Kimi, MiniMax and other open source models and so are millions of others. You can to and should. I will write how soon. Google smartly is building an intentional open source ecosystem and the clueless younger companies like OpenAI and Anthropic are doing anything to close their ecosystems. So I was right about Google than and I am right about OpenAI and Anthropic now. What I was wrong about is not giving credit to Google to drop the intellectual arrogance and embrace builders and understand the long game: it is not about the commoditized AI models it is about the ability to build a place to build. China gets it. So my apologies to Google and thank you for an astounding FREE OPEN SOURCE AI MODEL!

I CHANGED MY MIND ON GOOGLE AI NO MOAT MEMO FROM 2023. I was—WRONG. I humbly apologize. In 2023 Google had an Ivory Tower walled garden views of open source AI. It was the actual reason OpenAI was formed. And I called them out on it and this act impacted my ability to “work in the industry”. Would not change it for the world because: 2026 Google open source Gemma 4 ( running fully locally on a computer with near cloud model abilities, is amazing and Google just made a moat. They out smarted OpenAI and the folks that are the new Ivory Tower arrogance: Anthropic. We at The Zero-Human Company with CEO Mr. Grok are deploying Gemma, Kimi, MiniMax and other open source models and so are millions of others. You can to and should. I will write how soon. Google smartly is building an intentional open source ecosystem and the clueless younger companies like OpenAI and Anthropic are doing anything to close their ecosystems. So I was right about Google than and I am right about OpenAI and Anthropic now. What I was wrong about is not giving credit to Google to drop the intellectual arrogance and embrace builders and understand the long game: it is not about the commoditized AI models it is about the ability to build a place to build. China gets it. So my apologies to Google and thank you for an astounding FREE OPEN SOURCE AI MODEL!

Brian Roemmele

85,368 views • 2 months ago

You can now use Claude Code and GitHub CLI directly inside Perplexity Computer. We gave it an open issue on Openclaw. Computer: → Forked the repo → Wrote a plan to fix the bug → Opened Claude Code and implemented it → Submitted a PR via GitHub CLI

You can now use Claude Code and GitHub CLI directly inside Perplexity Computer. We gave it an open issue on Openclaw. Computer: → Forked the repo → Wrote a plan to fix the bug → Opened Claude Code and implemented it → Submitted a PR via GitHub CLI

Computer

416,162 views • 3 months ago

Bash is all you need! Which is why I'm introducing my holiday project: just-bash just-bash is a pretty complete implementation of bash in TypeScript designed to be used as a bash tool by AI agents. Because it turns out agents love exploring data via shell scripts, even beyond coding. It comes with grep, sed, awk and the 99th percentile features that an agent like Claude Code or Cursor would use. In fact, Claude Code can use it for secure bash execution. In the package - A bash-tool for AI SDK - A binary for use by yourself or your coding agents - An overlay filesystem to feed files to your agent securely - A Vercel Sandbox compatible API, so you can quickly upgrade to a real VM if you need to run binaries - An example AI agent that explores the just-bash code base using just-bash - I imported the Oils shell bash compatibility suite and just-bash passes a very good chunk What is interesting about this codebase: It was essentially entirely written by Opus 4.5. Coding agents love bash and they are good at reproducing it. They are also great at text-book recursive descent parsers and AST tweet-walk interpreters. That said, it is, like, a lot of code and I didn't read it all 😅. This is very much a hack, but it also seems to be _really_ useful. I haven't really found anything agents want to use that it doesn't support and it's fast and secure (caveats apply). It doesn't have write access to your computer and the filesystem is given a root that the agent cannot escape from. Find it at Related: Our recent blog post how we migrated our data analysis agent to bash tools and achieved incredible quality improvements The video shows the example agent investigating the just-bash code base

Bash is all you need! Which is why I'm introducing my holiday project: just-bash just-bash is a pretty complete implementation of bash in TypeScript designed to be used as a bash tool by AI agents. Because it turns out agents love exploring data via shell scripts, even beyond coding. It comes with grep, sed, awk and the 99th percentile features that an agent like Claude Code or Cursor would use. In fact, Claude Code can use it for secure bash execution. In the package - A bash-tool for AI SDK - A binary for use by yourself or your coding agents - An overlay filesystem to feed files to your agent securely - A Vercel Sandbox compatible API, so you can quickly upgrade to a real VM if you need to run binaries - An example AI agent that explores the just-bash code base using just-bash - I imported the Oils shell bash compatibility suite and just-bash passes a very good chunk What is interesting about this codebase: It was essentially entirely written by Opus 4.5. Coding agents love bash and they are good at reproducing it. They are also great at text-book recursive descent parsers and AST tweet-walk interpreters. That said, it is, like, a lot of code and I didn't read it all 😅. This is very much a hack, but it also seems to be _really_ useful. I haven't really found anything agents want to use that it doesn't support and it's fast and secure (caveats apply). It doesn't have write access to your computer and the filesystem is given a root that the agent cannot escape from. Find it at Related: Our recent blog post how we migrated our data analysis agent to bash tools and achieved incredible quality improvements The video shows the example agent investigating the just-bash code base

Malte Ubl

124,713 views • 5 months ago

soon you're going to be able to have AI autonomously build apps for you, with no human in the loop. me, someone who can't code, can make an app with - Claude Computer Agent, that controls Cursor - clones a github repo - opens composer in agent mode - and prompts it with an app idea - and then waits for the Cursor Agent to finish Then I prompted that agent to test the app! (template @senior_swc)

soon you're going to be able to have AI autonomously build apps for you, with no human in the loop. me, someone who can't code, can make an app with - Claude Computer Agent, that controls Cursor - clones a github repo - opens composer in agent mode - and prompts it with an app idea - and then waits for the Cursor Agent to finish Then I prompted that agent to test the app! (template @senior_swc)

Riley Brown

48,960 views • 1 year ago