Milind S
@milindlabs • 4,848 subscribers
Check out https://t.co/TpQZ5Wcsjc and https://t.co/Lw6IDgUYrp
Shorts
Videos
Okay so Jev can actually do computer use really well Without any screenshots, or LLMs and no Pixels leave my mac I dont even read the Dom elements A local CoreML model segments every button and UI element on screen. On-device OCR reads the labels. That text is all Jev gets. It returns a probability across those elements and tells me the best one to click. Then it clicks, re-runs detection, and decides again. In a loop until the goal is done. ~90ms per decision. Faster than any LLM computer use I've tried. Blazing fast computer use, without any latency TypeSafe AI is building something really interesting
Milind S247,483 görüntüleme • 1 ay önce
I made an Open Source version of the Grok Bot complete with all the features It does not need any subscriptions at all and uses the existing subscriptions you already have It can spin up virtual machines from Ascii - Boat It uses Cua for computer use It also has plugin support supporting all the integrations by Composio Routines coming soon :D Link to the repo in the comments
Milind S487,866 görüntüleme • 2 ay önce
I’m reverse-engineering the Google AI mouse pointer and making it Open Source! It sees your screen. You can freeform paint anything and ask AI to modify it. It understands the app or window you mean. It can click, type, edit, navigate, and even write + execute code. A tiny cursor becoming a real computer-use agent. Made with the help of Cua which is an AMAZING reference for computer use Sorry Google DeepMind but I had to do this heyclicky Started a revolution and we will be the ones to win Also credits to Omkar Satpute for the help Feel free to modify it , update it and make it your own, we gotta beat them at this. Link to the repo in the comments:
Milind S216,046 görüntüleme • 5 ay önce
We made an open-source Grok Bot for iOS. Actually… we didn’t make it. The community did. OpenMausBot has now crossed 1.7K stars on GitHub, and some genuinely incredible people have been contributing to it this beautiful iOS app is a direct result of that. we’re shipping at an absolutely ridiculous pace It was a pretty overwhelming week with all the PRs and issues coming in but the app is so good now I’m dropping the TestFlight link in our Discord channel Come try it, break it, share your feedback, and help us shape what comes next. See you there. I genuinely cannot wait for you to use it. Android coming soon All open source, including the app, built by the community
Milind S72,493 görüntüleme • 1 ay önce
This is how you can deploy the open-source version of grok bot on your own servers or that Mac mini sitting on your desk. "npx openmausbot@latest" → Connect your AI provider → Choose a model → Scan a QR code to pair your phone Close the laptop. Your bots keep working. Check in, send tasks, and answer approvals from your phone. We have crossed 2.3k GitHub stars, and this community keeps finding use cases I never imagined. This feature now takes it to the next level. I am sharing everything in the discord channel including the test-flight link for the app More than happy to walk each one of you through the process. I dont care if you have technical knowledge or not. Its for everyone.
Milind S36,992 görüntüleme • 1 ay önce
I took Clicky and I made it 5x Faster So I saw that Farza 🇵🇰🇺🇸 was using uses Claude's vision to find UI elements on screen, send a screenshot, wait for coordinates back. It works, but it's slow. I replaced that with OmniParser V2 by Microsoft which is a local Object detection model trained specifically on UI elements. It runs on-device, detects every button, menu, and icon in 400ms, and gives me pixel-precise coordinates. No API call, no latency, no cost. The green highlights you see around the UI elements is the detection overlay and you can see as I am switching the screen it takes no time to detect and highlight which is pretty neat! With Local models improving by the day, Its the right direction for applications like these. Next up: video-synced tutorials where a YouTube tutorial pauses and waits for you to perform each action in the real app. I am not stopping!
Milind S73,628 görüntüleme • 6 ay önce
We now have webhook support. That means OpenMausBot can now react when something happens in the outside world This was a suggestion from Artur Kre, and honestly, it makes a lot of sense. You want OpenMaus to work with your other agents and apps. For example, connect your website contact form to a Sales Bot. The moment someone requests a demo, the webhook sends their details to the Bot. It can research the company, understand what the person needs, and draft a personalized response directly in your chat. The same idea can work with Shopify orders, Stripe payments, customer reviews, support requests, GitHub events, Zapier workflows, or anything else that can send a webhook. Something happens. The right Bot wakes up. The work begins. Loving the feature requests. Keep them coming Its the community thats driving the product now and I couldnt be more grateful for the feature requests and PRs that are coming up Spoiler: IOS app coming soon There is so much more coming up next. Working full-time to get everything in and shipping like a madman
Milind S21,668 görüntüleme • 2 ay önce
People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini
Milind S38,361 görüntüleme • 4 ay önce
I made a stick shift claude code app Its very fun, suppose you are working on a complex project with fable and suddenly you wanna change the model to sonnet to do a simple "can you git push this" Or "can you quickly change the color of this button" Its for everyone who loves driving a manual automatic mode coming soon. for the rest of you.
Milind S19,843 görüntüleme • 3 ay önce
computer use agents are slow because they think too much they make round trips to an LLM. every single time. even for a command you've said a hundred times before. that's why it always feels slow. my mac now remembers. the first time i say something new, like "search youtube for lofi beats", a big model figures it out once, does it, and saves the recipe as a json spec: {"op":"act","do":"open","target":"Safari"} {"op":"act","do":"type","text":"{query}"} every time after that, a tiny model on my laptop recognizes the command, fills in the parameters, and replays the recipe with CUA say it once, it's slow. say it again, it's instant. the fine tuned tiny model is 350 million parameters. its only job is recognizing which of my commands i just said. and when the big model does plan, it streams actions line by line. each line runs the moment it's finished, so the first action is executing while the model is still writing the second. everything it learns lives in one plain text file on my disk. i've been playing around a lot with Cua by Francesco and peekaboo by Peter Steinberger 🦞 . curious whether a standard computer use spec makes sense, one that streaming models can drive to make computer use actually fast.
Milind S14,235 görüntüleme • 3 ay önce
A screen recording is the best context you can give an AI agent. It's also the worst. A few seconds of video is thousands of images. Feed that to an agent and you instantly blow up its memory. So I built a tool that fixes it. How it works: Hit record and just do your thing. Click around, type, and talk out loud about what you're doing. While you record, it quietly captures the stuff that actually matters: your voice transcription, every click and the exact thing you clicked, what you typed, and the shortcuts you press. When you stop, it doesn't hand the agent the whole video. It writes a short, readable summary: a timeline of what you said and did. The agent reads that tiny summary first. Then, only when it needs to actually see a moment, it pulls that one exact frame from the video, like flipping to a single page instead of reading the whole book. The result: the agent understands everything you did! Full context. Less tokens.
Milind S12,486 görüntüleme • 3 ay önce
I built the clipboard I always wanted for working with AI agents. It’s called Bluey and It’s 100% local-first. It lives under your mouse cursor. You can draw on your screen, speak to it, or type what you mean, and Bluey turns all of that into rich context for Claude Code or Codex. I’ve been using it to design websites and debug UI because I can finally say things like “move this here” or “make this section feel cleaner” while pointing at the actual screen. No more writing long explanations. No more manually describing screenshots. No more agents getting lost because they don’t know what “this” or "that" means. You can just point to it, speak or type and it will curate the best possible context for your agents Bluey captures the screenshot, annotation, transcript, app/window context, and the details your agent needs, then sends it directly into your coding session. I am giving it out for free for a while! Im building it with my buddy Omkar Satpute and we would love to hear what you think about it on Discord what context do you wish your agent could understand better?
Milind S13,342 görüntüleme • 4 ay önce
I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!
Milind S12,085 görüntüleme • 4 ay önce
Daha fazla içerik yok.