Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

I’m reverse-engineering the Google AI mouse pointer and making it Open Source! It sees your screen. You can freeform paint anything and ask AI to modify it. It understands the app or window you mean. It can click, type, edit, navigate, and even write + execute code. A tiny... show more

Milind S

1,174 subscribers

213,009 Aufrufe • vor 2 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

Milind S

11,917 Aufrufe • vor 1 Monat

AI agents are a real paradigm shift You can literally make your computer work for you to do any task. All I've used is: - Replit to get it running - Browser Use (free & open source) - Google AI Studio for the API - DeepSeek R1 to implement it (I'm not paying $200 for ChatGPT sorry) Here the AI agent fetches information on its own and reuses it to write new content. But you can use it for literally any industry. Run it once or schedule it at regular intervals... you're in total control And it's nothing like automating on Make or Zapier: AI agents adapt to every situation and can handle far more complex scenarios.

AI agents are a real paradigm shift You can literally make your computer work for you to do any task. All I've used is: - Replit to get it running - Browser Use (free & open source) - Google AI Studio for the API - DeepSeek R1 to implement it (I'm not paying $200 for ChatGPT sorry) Here the AI agent fetches information on its own and reuses it to write new content. But you can use it for literally any industry. Run it once or schedule it at regular intervals... you're in total control And it's nothing like automating on Make or Zapier: AI agents adapt to every situation and can handle far more complex scenarios.

Paul Couvert

76,636 Aufrufe • vor 1 Jahr

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

Milind S

38,054 Aufrufe • vor 1 Monat

Your AI agent can create an entire Google Form for you by chatting, and it will use the browser to type and build the survey automatically.

Your AI agent can create an entire Google Form for you by chatting, and it will use the browser to type and build the survey automatically.

Kimi.ai

16,083 Aufrufe • vor 2 Monaten

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Mushtaq Bilal, PhD

44,529 Aufrufe • vor 1 Jahr

We just open-sourced Simulang, a JavaScript library that gives you and your coding agents the ability to fully control computers. Simulang is our scripting framework for desktop automation. It allows you to write deterministic computer use code that can be replayed dozens if not hundreds of times. Much faster and cheaper, compared to LLM agents. It ships with a skill out of the box. That means you can turn natural language prompts into Simulang code directly, and your agent can now see the screen, click, type, and navigate. It gives your AI agents eyes and hands to control the computer like humans do. Try these things with Simulang: - upload your pile of invoices to the expense platform - find 50 people with specific criteria on LinkedIn - open your calculator app and do some math - find all buttons on your favorite app using accessibility tree - or draw a bunny with MS paint 🐰 In this demo, we show how Simulang took over when the coding agent couldn’t open and click on the target LinkedIn profile page. Give it a try:

Simular

14,362 Aufrufe • vor 2 Monaten

What if you can build an Agent with it own computer in a single api call? At my Google I/O talk, I showed how to use Gemini Managed Agents and the new Interactions API to give your AI a secure, hosted Linux sandbox to execute code and manage its own memory.

What if you can build an Agent with it own computer in a single api call? At my Google I/O talk, I showed how to use Gemini Managed Agents and the new Interactions API to give your AI a secure, hosted Linux sandbox to execute code and manage its own memory.

Philipp Schmid

21,726 Aufrufe • vor 2 Monaten

I built the clipboard I always wanted for working with AI agents. It’s called Bluey and It’s 100% local-first. It lives under your mouse cursor. You can draw on your screen, speak to it, or type what you mean, and Bluey turns all of that into rich context for Claude Code or Codex. I’ve been using it to design websites and debug UI because I can finally say things like “move this here” or “make this section feel cleaner” while pointing at the actual screen. No more writing long explanations. No more manually describing screenshots. No more agents getting lost because they don’t know what “this” or "that" means. You can just point to it, speak or type and it will curate the best possible context for your agents Bluey captures the screenshot, annotation, transcript, app/window context, and the details your agent needs, then sends it directly into your coding session. I am giving it out for free for a while! Im building it with my buddy Omkar Satpute and we would love to hear what you think about it on Discord what context do you wish your agent could understand better?

I built the clipboard I always wanted for working with AI agents. It’s called Bluey and It’s 100% local-first. It lives under your mouse cursor. You can draw on your screen, speak to it, or type what you mean, and Bluey turns all of that into rich context for Claude Code or Codex. I’ve been using it to design websites and debug UI because I can finally say things like “move this here” or “make this section feel cleaner” while pointing at the actual screen. No more writing long explanations. No more manually describing screenshots. No more agents getting lost because they don’t know what “this” or "that" means. You can just point to it, speak or type and it will curate the best possible context for your agents Bluey captures the screenshot, annotation, transcript, app/window context, and the details your agent needs, then sends it directly into your coding session. I am giving it out for free for a while! Im building it with my buddy Omkar Satpute and we would love to hear what you think about it on Discord what context do you wish your agent could understand better?

Milind S

13,342 Aufrufe • vor 1 Monat

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Santiago

72,306 Aufrufe • vor 2 Jahren

Hyper vigilance: It can save you. It can destroy you. What if YOUR private and personal local AI knew when to help to understand the difference. What would your world look like? Apple and Google are not going to sell this. It will be FREE and open source.

Hyper vigilance: It can save you. It can destroy you. What if YOUR private and personal local AI knew when to help to understand the difference. What would your world look like? Apple and Google are not going to sell this. It will be FREE and open source.

Brian Roemmele

357,430 Aufrufe • vor 1 Jahr

Cursor pro tip publish your google sheet to the web copy the link and add it to cursor as a documentation this will help you send table data to the ai

Cursor pro tip publish your google sheet to the web copy the link and add it to cursor as a documentation this will help you send table data to the ai

Fili

15,116 Aufrufe • vor 1 Jahr

How is the Web changing due to AI? This visit to TinyFish yesterday goes deeply into just how. TinyFish makes a new kind of web browser. One that is started virtually. It is like a Google Chrome in the cloud. But it is very smart, can read and use any website, and your AI agents can talk to it. Here we talk about what it all means for business and the future of robots, brain computer interfaces, and AI glasses. It lets you build systems that can look at MANY websites at the same time and build new kinds of apps to do all sorts of new things.

How is the Web changing due to AI? This visit to TinyFish yesterday goes deeply into just how. TinyFish makes a new kind of web browser. One that is started virtually. It is like a Google Chrome in the cloud. But it is very smart, can read and use any website, and your AI agents can talk to it. Here we talk about what it all means for business and the future of robots, brain computer interfaces, and AI glasses. It lets you build systems that can look at MANY websites at the same time and build new kinds of apps to do all sorts of new things.

Robert Scoble

16,341 Aufrufe • vor 7 Monaten

🚨 Breaking: Google is rolling out a native image feature in AI Studio! You can now use Gemini 2.0 Flash Experimental for native image output. This feature is insanely powerful , you can do so much with it. Google is crushing it, no hype, just delivering. Mad respect for the Google DeepMind and Gemini team! I had early access to this for a long time but I wasn’t supposed to share it :) It's amazing, you guys will love it.

🚨 Breaking: Google is rolling out a native image feature in AI Studio! You can now use Gemini 2.0 Flash Experimental for native image output. This feature is insanely powerful , you can do so much with it. Google is crushing it, no hype, just delivering. Mad respect for the Google DeepMind and Gemini team! I had early access to this for a long time but I wasn’t supposed to share it :) It's amazing, you guys will love it.

AshutoshShrivastava

70,802 Aufrufe • vor 1 Jahr

In Cursor you can *easily* turn comments into code. Just write in natural language what you want in your file. Then highlight it, use CMD+K, and ask the AI to translate it into code - and it will! Turn your thoughts right into apps.

In Cursor you can easily turn comments into code. Just write in natural language what you want in your file. Then highlight it, use CMD+K, and ask the AI to translate it into code - and it will! Turn your thoughts right into apps.

Mckay Wrigley

106,485 Aufrufe • vor 1 Jahr

✨ For the last few months I have been reverse engineering Magnific AI's famous upscaler It uses MultiDiffusion, ControlNet tiles and details LoRas In true AI spirit, I am open sourcing it for everyone to use for free in your apps It's designed to easily control the amount of hallucination, resemblance and creativity. You can use it on Replicate now: Or run it locally with A1111'a web UI (parameters are in the comments) The code is fully open source:

✨ For the last few months I have been reverse engineering Magnific AI's famous upscaler It uses MultiDiffusion, ControlNet tiles and details LoRas In true AI spirit, I am open sourcing it for everyone to use for free in your apps It's designed to easily control the amount of hallucination, resemblance and creativity. You can use it on Replicate now: Or run it locally with A1111'a web UI (parameters are in the comments) The code is fully open source:

philz1337x

565,517 Aufrufe • vor 2 Jahren

Your coding agent may be lying to you. You ask it to write tests. It says they pass. It never ran them. You ask for a feature. It hits a wall and silently swaps in fake data. We built Vet to fix this. It’s open source. Get the code below.

Your coding agent may be lying to you. You ask it to write tests. It says they pass. It never ran them. You ask for a feature. It hits a wall and silently swaps in fake data. We built Vet to fix this. It’s open source. Get the code below.

Imbue

142,872 Aufrufe • vor 4 Monaten

I CHANGED MY MIND ON GOOGLE AI NO MOAT MEMO FROM 2023. I was—WRONG. I humbly apologize. In 2023 Google had an Ivory Tower walled garden views of open source AI. It was the actual reason OpenAI was formed. And I called them out on it and this act impacted my ability to “work in the industry”. Would not change it for the world because: 2026 Google open source Gemma 4 ( running fully locally on a computer with near cloud model abilities, is amazing and Google just made a moat. They out smarted OpenAI and the folks that are the new Ivory Tower arrogance: Anthropic. We at The Zero-Human Company with CEO Mr. Grok are deploying Gemma, Kimi, MiniMax and other open source models and so are millions of others. You can to and should. I will write how soon. Google smartly is building an intentional open source ecosystem and the clueless younger companies like OpenAI and Anthropic are doing anything to close their ecosystems. So I was right about Google than and I am right about OpenAI and Anthropic now. What I was wrong about is not giving credit to Google to drop the intellectual arrogance and embrace builders and understand the long game: it is not about the commoditized AI models it is about the ability to build a place to build. China gets it. So my apologies to Google and thank you for an astounding FREE OPEN SOURCE AI MODEL!

I CHANGED MY MIND ON GOOGLE AI NO MOAT MEMO FROM 2023. I was—WRONG. I humbly apologize. In 2023 Google had an Ivory Tower walled garden views of open source AI. It was the actual reason OpenAI was formed. And I called them out on it and this act impacted my ability to “work in the industry”. Would not change it for the world because: 2026 Google open source Gemma 4 ( running fully locally on a computer with near cloud model abilities, is amazing and Google just made a moat. They out smarted OpenAI and the folks that are the new Ivory Tower arrogance: Anthropic. We at The Zero-Human Company with CEO Mr. Grok are deploying Gemma, Kimi, MiniMax and other open source models and so are millions of others. You can to and should. I will write how soon. Google smartly is building an intentional open source ecosystem and the clueless younger companies like OpenAI and Anthropic are doing anything to close their ecosystems. So I was right about Google than and I am right about OpenAI and Anthropic now. What I was wrong about is not giving credit to Google to drop the intellectual arrogance and embrace builders and understand the long game: it is not about the commoditized AI models it is about the ability to build a place to build. China gets it. So my apologies to Google and thank you for an astounding FREE OPEN SOURCE AI MODEL!

Brian Roemmele

85,386 Aufrufe • vor 3 Monaten

You can now use Claude Code and GitHub CLI directly inside Perplexity Computer. We gave it an open issue on Openclaw. Computer: → Forked the repo → Wrote a plan to fix the bug → Opened Claude Code and implemented it → Submitted a PR via GitHub CLI

You can now use Claude Code and GitHub CLI directly inside Perplexity Computer. We gave it an open issue on Openclaw. Computer: → Forked the repo → Wrote a plan to fix the bug → Opened Claude Code and implemented it → Submitted a PR via GitHub CLI

Computer

416,607 Aufrufe • vor 4 Monaten

Bash is all you need! Which is why I'm introducing my holiday project: just-bash just-bash is a pretty complete implementation of bash in TypeScript designed to be used as a bash tool by AI agents. Because it turns out agents love exploring data via shell scripts, even beyond coding. It comes with grep, sed, awk and the 99th percentile features that an agent like Claude Code or Cursor would use. In fact, Claude Code can use it for secure bash execution. In the package - A bash-tool for AI SDK - A binary for use by yourself or your coding agents - An overlay filesystem to feed files to your agent securely - A Vercel Sandbox compatible API, so you can quickly upgrade to a real VM if you need to run binaries - An example AI agent that explores the just-bash code base using just-bash - I imported the Oils shell bash compatibility suite and just-bash passes a very good chunk What is interesting about this codebase: It was essentially entirely written by Opus 4.5. Coding agents love bash and they are good at reproducing it. They are also great at text-book recursive descent parsers and AST tweet-walk interpreters. That said, it is, like, a lot of code and I didn't read it all 😅. This is very much a hack, but it also seems to be _really_ useful. I haven't really found anything agents want to use that it doesn't support and it's fast and secure (caveats apply). It doesn't have write access to your computer and the filesystem is given a root that the agent cannot escape from. Find it at Related: Our recent blog post how we migrated our data analysis agent to bash tools and achieved incredible quality improvements The video shows the example agent investigating the just-bash code base

Bash is all you need! Which is why I'm introducing my holiday project: just-bash just-bash is a pretty complete implementation of bash in TypeScript designed to be used as a bash tool by AI agents. Because it turns out agents love exploring data via shell scripts, even beyond coding. It comes with grep, sed, awk and the 99th percentile features that an agent like Claude Code or Cursor would use. In fact, Claude Code can use it for secure bash execution. In the package - A bash-tool for AI SDK - A binary for use by yourself or your coding agents - An overlay filesystem to feed files to your agent securely - A Vercel Sandbox compatible API, so you can quickly upgrade to a real VM if you need to run binaries - An example AI agent that explores the just-bash code base using just-bash - I imported the Oils shell bash compatibility suite and just-bash passes a very good chunk What is interesting about this codebase: It was essentially entirely written by Opus 4.5. Coding agents love bash and they are good at reproducing it. They are also great at text-book recursive descent parsers and AST tweet-walk interpreters. That said, it is, like, a lot of code and I didn't read it all 😅. This is very much a hack, but it also seems to be _really_ useful. I haven't really found anything agents want to use that it doesn't support and it's fast and secure (caveats apply). It doesn't have write access to your computer and the filesystem is given a root that the agent cannot escape from. Find it at Related: Our recent blog post how we migrated our data analysis agent to bash tools and achieved incredible quality improvements The video shows the example agent investigating the just-bash code base

Malte Ubl

124,713 Aufrufe • vor 6 Monaten