Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works... show more

Milind S

1,125 subscribers

37,681 Aufrufe • vor 12 Tagen •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

what if i told you... computer use can be faster on local models moondream3 with its photon update today that gives it mac support can see your screen and use it with 1s latency, ty vik here we have whisper+qwen+moondream triple model pipeline working offline flawlessly

what if i told you... computer use can be faster on local models moondream3 with its photon update today that gives it mac support can see your screen and use it with 1s latency, ty vik here we have whisper+qwen+moondream triple model pipeline working offline flawlessly

murat 🍥

388,479 Aufrufe • vor 1 Monat

I’m reverse-engineering the Google AI mouse pointer and making it Open Source! It sees your screen. You can freeform paint anything and ask AI to modify it. It understands the app or window you mean. It can click, type, edit, navigate, and even write + execute code. A tiny cursor becoming a real computer-use agent. Made with the help of Cua which is an AMAZING reference for computer use Sorry Google DeepMind but I had to do this heyclicky Started a revolution and we will be the ones to win Also credits to Omkar Satpute for the help Feel free to modify it , update it and make it your own, we gotta beat them at this. Link to the repo in the comments:

I’m reverse-engineering the Google AI mouse pointer and making it Open Source! It sees your screen. You can freeform paint anything and ask AI to modify it. It understands the app or window you mean. It can click, type, edit, navigate, and even write + execute code. A tiny cursor becoming a real computer-use agent. Made with the help of Cua which is an AMAZING reference for computer use Sorry Google DeepMind but I had to do this heyclicky Started a revolution and we will be the ones to win Also credits to Omkar Satpute for the help Feel free to modify it , update it and make it your own, we gotta beat them at this. Link to the repo in the comments:

Milind S

210,879 Aufrufe • vor 1 Monat

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

Milind S

11,830 Aufrufe • vor 9 Tagen

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Imagine an AI application that can type anywhere you can and use the full context of what's on your screen. This is the application we all deserve (at least if you have macOS.) Check out Omnipilot. It's an app that works with every other macOS application and uses Claude Sonet 3.5 in the background—it also supports Gemini and GPT-4o. Here is the idea: You can use the tool to ask questions about anything on your screen. Or you can use it to autocomplete the text you are typing. You don't need to copy and paste anymore or waste your time providing context to a model. It sees what you see. It works right where you are. That's pretty cool. Here are a couple of cool examples: • Use it to reply to an email • Use it in the terminal to autocomplete a command • Use it to finish a document • Use to send a message on Slack AI at the system level is bonkers. You can read a ton more on their Product Hunt launch page: Thanks to the Omnipilot team for collaborating with me on this post!

Santiago

72,306 Aufrufe • vor 1 Jahr

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Google offers a AI-powered personal assistant that you can use for free. You can talk to it in real time, share your screen with it, and ask it to help you with any given task (sound on 👇). Surprisingly, not many people are using it. Here's how to use it:

Mushtaq Bilal, PhD

44,529 Aufrufe • vor 1 Jahr

holy shit, i built a better Cluely. introducing: Obscura, a better, more performant Cluely. Obscura helps you during your meeting and interview calls, in real time, with more context and a built-in agent to websearch in less than a second. it is completely INVISIBLE to your screen recorder, screen share and cannot be detected during your google calls or zoom calls no matter what. some functionalities: 1. toggle visibility: so that only you can see it 2. panic kill: if you wanna kill it in a second 3. screenshot ocr: for screenshot capabilities 4. context window: pass any context in .md or .docx file for it to read before your calls and Obscura will understand it 5. performative built-in agent: to search for anything in less than a second. it uses Deepgram for STT model and Anthropic / OpenAI api keys to generate answers. it takes $1 for 20 hours of your calls as compared to cluely's $20/mo or $75/mo plan (but that depends on tokens consumed on both models). releasing a .dmg for people for people to use. access will be given to people who share this with their friends. I will roll out a referral program so that people can share it with others. feel free to give some feedback. thanks to rahat for being on call with me and testing it.

holy shit, i built a better Cluely. introducing: Obscura, a better, more performant Cluely. Obscura helps you during your meeting and interview calls, in real time, with more context and a built-in agent to websearch in less than a second. it is completely INVISIBLE to your screen recorder, screen share and cannot be detected during your google calls or zoom calls no matter what. some functionalities: 1. toggle visibility: so that only you can see it 2. panic kill: if you wanna kill it in a second 3. screenshot ocr: for screenshot capabilities 4. context window: pass any context in .md or .docx file for it to read before your calls and Obscura will understand it 5. performative built-in agent: to search for anything in less than a second. it uses Deepgram for STT model and Anthropic / OpenAI api keys to generate answers. it takes $1 for 20 hours of your calls as compared to cluely's $20/mo or $75/mo plan (but that depends on tokens consumed on both models). releasing a .dmg for people for people to use. access will be given to people who share this with their friends. I will roll out a referral program so that people can share it with others. feel free to give some feedback. thanks to rahat for being on call with me and testing it.

Naruto11.eth

15,682 Aufrufe • vor 2 Monaten

We just open-sourced Simulang, a JavaScript library that gives you and your coding agents the ability to fully control computers. Simulang is our scripting framework for desktop automation. It allows you to write deterministic computer use code that can be replayed dozens if not hundreds of times. Much faster and cheaper, compared to LLM agents. It ships with a skill out of the box. That means you can turn natural language prompts into Simulang code directly, and your agent can now see the screen, click, type, and navigate. It gives your AI agents eyes and hands to control the computer like humans do. Try these things with Simulang: - upload your pile of invoices to the expense platform - find 50 people with specific criteria on LinkedIn - open your calculator app and do some math - find all buttons on your favorite app using accessibility tree - or draw a bunny with MS paint 🐰 In this demo, we show how Simulang took over when the coding agent couldn’t open and click on the target LinkedIn profile page. Give it a try:

Simular

13,816 Aufrufe • vor 28 Tagen

Been working on a new UX for agents. It will understand your workflow as you use your computer and proactively nudge you when it can be helpful. I feel like agents are these super powerful beings trapped in terminals and chat interfaces. Now, just use your voice. Demo:

Been working on a new UX for agents. It will understand your workflow as you use your computer and proactively nudge you when it can be helpful. I feel like agents are these super powerful beings trapped in terminals and chat interfaces. Now, just use your voice. Demo:

Farza 🇵🇰🇺🇸

221,815 Aufrufe • vor 26 Tagen

Introducing Ambient Context Aware AI on your computer. I have just scraped the whole Discord with it and made a summary of pain points ppl have Features: - Sees the whole screen ( even parts you need to scroll to) - long context window - Has memory Link:

Introducing Ambient Context Aware AI on your computer. I have just scraped the whole Discord with it and made a summary of pain points ppl have Features: - Sees the whole screen ( even parts you need to scroll to) - long context window - Has memory Link:

Robert Lukoszko

15,383 Aufrufe • vor 1 Jahr

Anthropic just announced Computer Use It allows Claude to control your computer screen based on a prompt and take actions on your behalf The use cases in agentic coding with automated debugging, customer support, and education are going to be INSANE

Anthropic just announced Computer Use It allows Claude to control your computer screen based on a prompt and take actions on your behalf The use cases in agentic coding with automated debugging, customer support, and education are going to be INSANE

Rowan Cheung

1,024,238 Aufrufe • vor 1 Jahr

I took Clicky and I made it 5x Faster So I saw that Farza 🇵🇰🇺🇸 was using uses Claude's vision to find UI elements on screen, send a screenshot, wait for coordinates back. It works, but it's slow. I replaced that with OmniParser V2 by Microsoft which is a local Object detection model trained specifically on UI elements. It runs on-device, detects every button, menu, and icon in 400ms, and gives me pixel-precise coordinates. No API call, no latency, no cost. The green highlights you see around the UI elements is the detection overlay and you can see as I am switching the screen it takes no time to detect and highlight which is pretty neat! With Local models improving by the day, Its the right direction for applications like these. Next up: video-synced tutorials where a YouTube tutorial pauses and waits for you to perform each action in the real app. I am not stopping!

I took Clicky and I made it 5x Faster So I saw that Farza 🇵🇰🇺🇸 was using uses Claude's vision to find UI elements on screen, send a screenshot, wait for coordinates back. It works, but it's slow. I replaced that with OmniParser V2 by Microsoft which is a local Object detection model trained specifically on UI elements. It runs on-device, detects every button, menu, and icon in 400ms, and gives me pixel-precise coordinates. No API call, no latency, no cost. The green highlights you see around the UI elements is the detection overlay and you can see as I am switching the screen it takes no time to detect and highlight which is pretty neat! With Local models improving by the day, Its the right direction for applications like these. Next up: video-synced tutorials where a YouTube tutorial pauses and waits for you to perform each action in the real app. I am not stopping!

Milind S

72,871 Aufrufe • vor 1 Monat

🤯 with OpenAI Whisper + TTS + GPT4V, you can just permanently share your screen with an AI and talk to it to get help on anything on screen. a generally intelligent assistant for your digital life. with a bit more engineering you could probably feed it extra docs and context based on the currently active tab/app.

🤯 with OpenAI Whisper + TTS + GPT4V, you can just permanently share your screen with an AI and talk to it to get help on anything on screen. a generally intelligent assistant for your digital life. with a bit more engineering you could probably feed it extra docs and context based on the currently active tab/app.

swyx

1,051,669 Aufrufe • vor 2 Jahren

At #GoogleIO, we introduced Google Gemini Spark, your 24/7 personal AI agent that helps you navigate your digital life, taking action on your behalf and under your direction — and you can use it across mobile and desktop, even when your laptop is closed. It's available now to all Google AI Ultra subscribers in the U.S. See it in action ⬇️

At #GoogleIO, we introduced Google Gemini Spark, your 24/7 personal AI agent that helps you navigate your digital life, taking action on your behalf and under your direction — and you can use it across mobile and desktop, even when your laptop is closed. It's available now to all Google AI Ultra subscribers in the U.S. See it in action ⬇️

Google

60,425 Aufrufe • vor 14 Tagen

Run FacePoke on Your Local Computer I wrote a 1-click launcher for Julian Bilcke's FacePoke so you can play with it on your own computer (NVIDIA ONLY). Here's a 1X (Not sped up) video recording of me playing with it on NVIDIA RTX 4090.

Run FacePoke on Your Local Computer I wrote a 1-click launcher for Julian Bilcke's FacePoke so you can play with it on your own computer (NVIDIA ONLY). Here's a 1X (Not sped up) video recording of me playing with it on NVIDIA RTX 4090.

cocktail peanut

42,765 Aufrufe • vor 1 Jahr

Today we introduced Gemini Enterprise, built with our most advanced Gemini models. It allows you to chat with your company’s documents, data and apps as well as build and deploy AI agents, all grounded in your information and context. Have a look at how it helps you build an agent for almost anything:

Today we introduced Gemini Enterprise, built with our most advanced Gemini models. It allows you to chat with your company’s documents, data and apps as well as build and deploy AI agents, all grounded in your information and context. Have a look at how it helps you build an agent for almost anything:

Sundar Pichai

1,105,348 Aufrufe • vor 8 Monaten

Run state-of-the-art RAG applications locally on your computer with ollama and use all the fantastic open-source models like llama3, msk's awesome models, or Command R from cohere With Verba 1.0, we put it all in your hands 🙌 Get on board for a wild open-source ride, we're bridging any moat as open-source is here to win

Run state-of-the-art RAG applications locally on your computer with ollama and use all the fantastic open-source models like llama3, msk's awesome models, or Command R from cohere With Verba 1.0, we put it all in your hands 🙌 Get on board for a wild open-source ride, we're bridging any moat as open-source is here to win

Philip Vollet

41,375 Aufrufe • vor 2 Jahren

Today we're excited to introduce Vy, our AI that sees and acts on your computer. At Vercept, our mission is to reinvent how humans use computers–enabling you to accomplish orders of magnitude more than what you can do today. Vy is a first glimpse at AI that sees and uses your computer just like you do. It runs natively on your computer, with access to all your applications and accounts that you’re already signed into. It is available for download starting today! As part of the release, we’ve built VyUI, an AI model bridging the gap between language and your screen. It sets a new state-of-the-art on UI grounding benchmarks like ScreenSpot v1, ScreenSpot v2, and GroundUI Web–beating leading models from OpenAI, Google, and Anthropic. Check out what Vy can do in the thread below: 🧵👇

Today we're excited to introduce Vy, our AI that sees and acts on your computer. At Vercept, our mission is to reinvent how humans use computers–enabling you to accomplish orders of magnitude more than what you can do today. Vy is a first glimpse at AI that sees and uses your computer just like you do. It runs natively on your computer, with access to all your applications and accounts that you’re already signed into. It is available for download starting today! As part of the release, we’ve built VyUI, an AI model bridging the gap between language and your screen. It sets a new state-of-the-art on UI grounding benchmarks like ScreenSpot v1, ScreenSpot v2, and GroundUI Web–beating leading models from OpenAI, Google, and Anthropic. Check out what Vy can do in the thread below: 🧵👇

Vercept

123,683 Aufrufe • vor 1 Jahr

BREAKING 🚨: Google introduced Gemini Enterprise with Agents, Connectors and Agent Builder! “It allows you to chat with your company’s documents, data and apps as well as build and deploy AI agents, all grounded in your information and context.“

BREAKING 🚨: Google introduced Gemini Enterprise with Agents, Connectors and Agent Builder! “It allows you to chat with your company’s documents, data and apps as well as build and deploy AI agents, all grounded in your information and context.“

TestingCatalog News 🗞

104,349 Aufrufe • vor 8 Monaten

every AI product made you go to it. open a tab. paste your screen. explain your context. wait for an answer. go back to what you were doing. we got tired of that loop. so we built Nino — a Windows AI companion that lives on your desktop. hold Ctrl + Alt. it already sees your screen. just ask. it can see your screen, point at stuff, interact with apps, and run background agents while you keep working. launching today.

every AI product made you go to it. open a tab. paste your screen. explain your context. wait for an answer. go back to what you were doing. we got tired of that loop. so we built Nino — a Windows AI companion that lives on your desktop. hold Ctrl + Alt. it already sees your screen. just ask. it can see your screen, point at stuff, interact with apps, and run background agents while you keep working. launching today.

Karan

188,671 Aufrufe • vor 1 Monat

🚨 Announcing OpenClaw As A Service For Normies Stop installing OpenClaw on your personal computer or Mac Mini, it can be super in-secure. You also need to be fairly technical to get it right! We are excited for you to deploy Open Claw on our AI-native cloud in seconds! - deploy in 1-click - super use-friendly. You can just chat with it and set it up - It only costs a few cents to get started You can literally connect to all your services and get started immediately and pay as you

🚨 Announcing OpenClaw As A Service For Normies Stop installing OpenClaw on your personal computer or Mac Mini, it can be super in-secure. You also need to be fairly technical to get it right! We are excited for you to deploy Open Claw on our AI-native cloud in seconds! - deploy in 1-click - super use-friendly. You can just chat with it and set it up - It only costs a few cents to get started You can literally connect to all your services and get started immediately and pay as you

Bindu Reddy

42,249 Aufrufe • vor 2 Monaten