Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works... show more

Milind S

1,174 subscribers

38,013 views • 1 month ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

what if i told you... computer use can be faster on local models moondream3 with its photon update today that gives it mac support can see your screen and use it with 1s latency, ty vik here we have whisper+qwen+moondream triple model pipeline working offline flawlessly

what if i told you... computer use can be faster on local models moondream3 with its photon update today that gives it mac support can see your screen and use it with 1s latency, ty vik here we have whisper+qwen+moondream triple model pipeline working offline flawlessly

murat 🍥

389,922 views • 2 months ago

I’m reverse-engineering the Google AI mouse pointer and making it Open Source! It sees your screen. You can freeform paint anything and ask AI to modify it. It understands the app or window you mean. It can click, type, edit, navigate, and even write + execute code. A tiny cursor becoming a real computer-use agent. Made with the help of Cua which is an AMAZING reference for computer use Sorry Google DeepMind but I had to do this heyclicky Started a revolution and we will be the ones to win Also credits to Omkar Satpute for the help Feel free to modify it , update it and make it your own, we gotta beat them at this. Link to the repo in the comments:

I’m reverse-engineering the Google AI mouse pointer and making it Open Source! It sees your screen. You can freeform paint anything and ask AI to modify it. It understands the app or window you mean. It can click, type, edit, navigate, and even write + execute code. A tiny cursor becoming a real computer-use agent. Made with the help of Cua which is an AMAZING reference for computer use Sorry Google DeepMind but I had to do this heyclicky Started a revolution and we will be the ones to win Also credits to Omkar Satpute for the help Feel free to modify it , update it and make it your own, we gotta beat them at this. Link to the repo in the comments:

Milind S

213,009 views • 1 month ago

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

Milind S

11,917 views • 1 month ago

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Santiago

72,306 views • 1 year ago

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Mushtaq Bilal, PhD

44,529 views • 1 year ago

I built the clipboard I always wanted for working with AI agents. It’s called Bluey and It’s 100% local-first. It lives under your mouse cursor. You can draw on your screen, speak to it, or type what you mean, and Bluey turns all of that into rich context for Claude Code or Codex. I’ve been using it to design websites and debug UI because I can finally say things like “move this here” or “make this section feel cleaner” while pointing at the actual screen. No more writing long explanations. No more manually describing screenshots. No more agents getting lost because they don’t know what “this” or "that" means. You can just point to it, speak or type and it will curate the best possible context for your agents Bluey captures the screenshot, annotation, transcript, app/window context, and the details your agent needs, then sends it directly into your coding session. I am giving it out for free for a while! Im building it with my buddy Omkar Satpute and we would love to hear what you think about it on Discord what context do you wish your agent could understand better?

I built the clipboard I always wanted for working with AI agents. It’s called Bluey and It’s 100% local-first. It lives under your mouse cursor. You can draw on your screen, speak to it, or type what you mean, and Bluey turns all of that into rich context for Claude Code or Codex. I’ve been using it to design websites and debug UI because I can finally say things like “move this here” or “make this section feel cleaner” while pointing at the actual screen. No more writing long explanations. No more manually describing screenshots. No more agents getting lost because they don’t know what “this” or "that" means. You can just point to it, speak or type and it will curate the best possible context for your agents Bluey captures the screenshot, annotation, transcript, app/window context, and the details your agent needs, then sends it directly into your coding session. I am giving it out for free for a while! Im building it with my buddy Omkar Satpute and we would love to hear what you think about it on Discord what context do you wish your agent could understand better?

Milind S

13,302 views • 1 month ago

holy shit, i built a better Cluely. introducing: Obscura, a better, more performant Cluely. Obscura helps you during your meeting and interview calls, in real time, with more context and a built-in agent to websearch in less than a second. it is completely INVISIBLE to your screen recorder, screen share and cannot be detected during your google calls or zoom calls no matter what. some functionalities: 1. toggle visibility: so that only you can see it 2. panic kill: if you wanna kill it in a second 3. screenshot ocr: for screenshot capabilities 4. context window: pass any context in .md or .docx file for it to read before your calls and Obscura will understand it 5. performative built-in agent: to search for anything in less than a second. it uses Deepgram for STT model and Anthropic / OpenAI api keys to generate answers. it takes $1 for 20 hours of your calls as compared to cluely's $20/mo or $75/mo plan (but that depends on tokens consumed on both models). releasing a .dmg for people for people to use. access will be given to people who share this with their friends. I will roll out a referral program so that people can share it with others. feel free to give some feedback. thanks to rahat for being on call with me and testing it.

holy shit, i built a better Cluely. introducing: Obscura, a better, more performant Cluely. Obscura helps you during your meeting and interview calls, in real time, with more context and a built-in agent to websearch in less than a second. it is completely INVISIBLE to your screen recorder, screen share and cannot be detected during your google calls or zoom calls no matter what. some functionalities: 1. toggle visibility: so that only you can see it 2. panic kill: if you wanna kill it in a second 3. screenshot ocr: for screenshot capabilities 4. context window: pass any context in .md or .docx file for it to read before your calls and Obscura will understand it 5. performative built-in agent: to search for anything in less than a second. it uses Deepgram for STT model and Anthropic / OpenAI api keys to generate answers. it takes $1 for 20 hours of your calls as compared to cluely's $20/mo or $75/mo plan (but that depends on tokens consumed on both models). releasing a .dmg for people for people to use. access will be given to people who share this with their friends. I will roll out a referral program so that people can share it with others. feel free to give some feedback. thanks to rahat for being on call with me and testing it.

Naruto11.eth

15,745 views • 3 months ago

We just open-sourced Simulang, a JavaScript library that gives you and your coding agents the ability to fully control computers. Simulang is our scripting framework for desktop automation. It allows you to write deterministic computer use code that can be replayed dozens if not hundreds of times. Much faster and cheaper, compared to LLM agents. It ships with a skill out of the box. That means you can turn natural language prompts into Simulang code directly, and your agent can now see the screen, click, type, and navigate. It gives your AI agents eyes and hands to control the computer like humans do. Try these things with Simulang: - upload your pile of invoices to the expense platform - find 50 people with specific criteria on LinkedIn - open your calculator app and do some math - find all buttons on your favorite app using accessibility tree - or draw a bunny with MS paint 🐰 In this demo, we show how Simulang took over when the coding agent couldn’t open and click on the target LinkedIn profile page. Give it a try:

Simular

14,362 views • 1 month ago

Been working on a new UX for agents. It will understand your workflow as you use your computer and proactively nudge you when it can be helpful. I feel like agents are these super powerful beings trapped in terminals and chat interfaces. Now, just use your voice. Demo:

Been working on a new UX for agents. It will understand your workflow as you use your computer and proactively nudge you when it can be helpful. I feel like agents are these super powerful beings trapped in terminals and chat interfaces. Now, just use your voice. Demo:

Farza 🇵🇰🇺🇸

225,029 views • 1 month ago

Anthropic just announced Computer Use It allows Claude to control your computer screen based on a prompt and take actions on your behalf The use cases in agentic coding with automated debugging, customer support, and education are going to be INSANE

Anthropic just announced Computer Use It allows Claude to control your computer screen based on a prompt and take actions on your behalf The use cases in agentic coding with automated debugging, customer support, and education are going to be INSANE

Rowan Cheung

1,024,377 views • 1 year ago

I took Clicky and I made it 5x Faster So I saw that Farza 🇵🇰🇺🇸 was using uses Claude's vision to find UI elements on screen, send a screenshot, wait for coordinates back. It works, but it's slow. I replaced that with OmniParser V2 by Microsoft which is a local Object detection model trained specifically on UI elements. It runs on-device, detects every button, menu, and icon in 400ms, and gives me pixel-precise coordinates. No API call, no latency, no cost. The green highlights you see around the UI elements is the detection overlay and you can see as I am switching the screen it takes no time to detect and highlight which is pretty neat! With Local models improving by the day, Its the right direction for applications like these. Next up: video-synced tutorials where a YouTube tutorial pauses and waits for you to perform each action in the real app. I am not stopping!

I took Clicky and I made it 5x Faster So I saw that Farza 🇵🇰🇺🇸 was using uses Claude's vision to find UI elements on screen, send a screenshot, wait for coordinates back. It works, but it's slow. I replaced that with OmniParser V2 by Microsoft which is a local Object detection model trained specifically on UI elements. It runs on-device, detects every button, menu, and icon in 400ms, and gives me pixel-precise coordinates. No API call, no latency, no cost. The green highlights you see around the UI elements is the detection overlay and you can see as I am switching the screen it takes no time to detect and highlight which is pretty neat! With Local models improving by the day, Its the right direction for applications like these. Next up: video-synced tutorials where a YouTube tutorial pauses and waits for you to perform each action in the real app. I am not stopping!

Milind S

73,026 views • 2 months ago

🤯 with OpenAI Whisper + TTS + GPT4V, you can just permanently share your screen with an AI and talk to it to get help on anything on screen. a generally intelligent assistant for your digital life. with a bit more engineering you could probably feed it extra docs and context based on the currently active tab/app.

🤯 with OpenAI Whisper + TTS + GPT4V, you can just permanently share your screen with an AI and talk to it to get help on anything on screen. a generally intelligent assistant for your digital life. with a bit more engineering you could probably feed it extra docs and context based on the currently active tab/app.

swyx

1,051,687 views • 2 years ago

At #GoogleIO, we introduced Google Gemini Spark, your 24/7 personal AI agent that helps you navigate your digital life, taking action on your behalf and under your direction — and you can use it across mobile and desktop, even when your laptop is closed. It's available now to all Google AI Ultra subscribers in the U.S. See it in action ⬇️

At #GoogleIO, we introduced Google Gemini Spark, your 24/7 personal AI agent that helps you navigate your digital life, taking action on your behalf and under your direction — and you can use it across mobile and desktop, even when your laptop is closed. It's available now to all Google AI Ultra subscribers in the U.S. See it in action ⬇️

Google

60,940 views • 1 month ago

How I have Claude automate anything I do on my computer just by showing it: I use the free Clips app and hit record to start recording. Then I just do the action - like navigating to Rippling and approving any unapproved time off - and I’ll often just talk out loud as I do it explaining any add’l details like I would when showing a human. The Clips app captures everything on your screen and everything you say, and transforms it into a format Claude can “see and hear”. You'll then get a link that you can easily share with Claude (or humans too). I’ll paste it to Claude and say something like, "automate this for me every day at 8:00 AM." The link has special metadata that agents like Claude can discover. It exposes APIs to see the full transcript with timestamps and grab screenshots of what was on your screen at any of those timestamps. Once it's created the routine, you can go to the Routines tab in Claude Code desktop app. This also works with the CLI, Claude Cowork, Codex etc. You’ll see your new routine there, and you can just hit Run Now to have it start running it. Video attached shows this step by step. The Clips apps is free and open source - I’ll link to it and the source code in the thread.

How I have Claude automate anything I do on my computer just by showing it: I use the free Clips app and hit record to start recording. Then I just do the action - like navigating to Rippling and approving any unapproved time off - and I’ll often just talk out loud as I do it explaining any add’l details like I would when showing a human. The Clips app captures everything on your screen and everything you say, and transforms it into a format Claude can “see and hear”. You'll then get a link that you can easily share with Claude (or humans too). I’ll paste it to Claude and say something like, "automate this for me every day at 8:00 AM." The link has special metadata that agents like Claude can discover. It exposes APIs to see the full transcript with timestamps and grab screenshots of what was on your screen at any of those timestamps. Once it's created the routine, you can go to the Routines tab in Claude Code desktop app. This also works with the CLI, Claude Cowork, Codex etc. You’ll see your new routine there, and you can just hit Run Now to have it start running it. Video attached shows this step by step. The Clips apps is free and open source - I’ll link to it and the source code in the thread.

Steve (Builder.io)

24,787 views • 8 days ago

Run FacePoke on Your Local Computer I wrote a 1-click launcher for Julian Bilcke's FacePoke so you can play with it on your own computer (NVIDIA ONLY). Here's a 1X (Not sped up) video recording of me playing with it on NVIDIA RTX 4090.

Run FacePoke on Your Local Computer I wrote a 1-click launcher for Julian Bilcke's FacePoke so you can play with it on your own computer (NVIDIA ONLY). Here's a 1X (Not sped up) video recording of me playing with it on NVIDIA RTX 4090.

cocktail peanut

42,915 views • 1 year ago

Today we introduced Gemini Enterprise, built with our most advanced Gemini models. It allows you to chat with your company’s documents, data and apps as well as build and deploy AI agents, all grounded in your information and context. Have a look at how it helps you build an agent for almost anything:

Today we introduced Gemini Enterprise, built with our most advanced Gemini models. It allows you to chat with your company’s documents, data and apps as well as build and deploy AI agents, all grounded in your information and context. Have a look at how it helps you build an agent for almost anything:

Sundar Pichai

1,105,516 views • 9 months ago

Run state-of-the-art RAG applications locally on your computer with ollama and use all the fantastic open-source models like llama3, msk's awesome models, or Command R from Cohere With Verba 1.0, we put it all in your hands 🙌 Get on board for a wild open-source ride, we're bridging any moat as open-source is here to win

Run state-of-the-art RAG applications locally on your computer with ollama and use all the fantastic open-source models like llama3, msk's awesome models, or Command R from Cohere With Verba 1.0, we put it all in your hands 🙌 Get on board for a wild open-source ride, we're bridging any moat as open-source is here to win

Philip Vollet

41,450 views • 2 years ago

Today we're excited to introduce Vy, our AI that sees and acts on your computer. At Vercept, our mission is to reinvent how humans use computers–enabling you to accomplish orders of magnitude more than what you can do today. Vy is a first glimpse at AI that sees and uses your computer just like you do. It runs natively on your computer, with access to all your applications and accounts that you’re already signed into. It is available for download starting today! As part of the release, we’ve built VyUI, an AI model bridging the gap between language and your screen. It sets a new state-of-the-art on UI grounding benchmarks like ScreenSpot v1, ScreenSpot v2, and GroundUI Web–beating leading models from OpenAI, Google, and Anthropic. Check out what Vy can do in the thread below: 🧵👇

Today we're excited to introduce Vy, our AI that sees and acts on your computer. At Vercept, our mission is to reinvent how humans use computers–enabling you to accomplish orders of magnitude more than what you can do today. Vy is a first glimpse at AI that sees and uses your computer just like you do. It runs natively on your computer, with access to all your applications and accounts that you’re already signed into. It is available for download starting today! As part of the release, we’ve built VyUI, an AI model bridging the gap between language and your screen. It sets a new state-of-the-art on UI grounding benchmarks like ScreenSpot v1, ScreenSpot v2, and GroundUI Web–beating leading models from OpenAI, Google, and Anthropic. Check out what Vy can do in the thread below: 🧵👇

Vercept

123,728 views • 1 year ago

BREAKING 🚨: Google introduced Gemini Enterprise with Agents, Connectors and Agent Builder! “It allows you to chat with your company’s documents, data and apps as well as build and deploy AI agents, all grounded in your information and context.“

BREAKING 🚨: Google introduced Gemini Enterprise with Agents, Connectors and Agent Builder! “It allows you to chat with your company’s documents, data and apps as well as build and deploy AI agents, all grounded in your information and context.“

TestingCatalog News 🗞

104,349 views • 9 months ago