Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Okay so Jev can actually do computer use really well Without any screenshots, or LLMs and no Pixels leave my mac I dont even read the Dom elements A local CoreML model segments every button and UI element on screen. On-device OCR reads the labels. That text is all... show more

Milind S

3,902 subscribers

116,719 Aufrufe • vor 1 Tag •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

62 Kommentare

Profilbild von Milind S

Milind Svor 1 Tag

If this gets enough attention I’ll make it open source. Meanwhile check out my other stuff

Profilbild von Vedant Bhayani

Vedant Bhayanivor 1 Tag

I think the future is multiple other companies building new kind of llm models for individual tasks with which you just plug it in with frontiers model for steering and thinking Also domain specific harness and custom trained models with your data for specific use cases are going be more relevant and we are going to see even more than before

Profilbild von Milind S

Milind Svor 1 Tag

I agree that’s where things are headed

$Profilbild von /h(elton)?Fabio\.(t|j)s/gi$

/h(elton)?Fabio\.(t|j)s/givor 1 Tag

hey man, do you mind doing a test with a local model that uses the same architecture? like this one based on qwen-2.5-1b?

Profilbild von Milind S

Milind Svor 1 Tag

I’ll check it out

Profilbild von Valentin Pletzer

Valentin Pletzervor 1 Tag

very cool!

Profilbild von Milind S

Milind Svor 1 Tag

Thank you!

Profilbild von Artur Kre

Artur Krevor 1 Tag

wow insane, which local model does the window UI segmenting ?

Profilbild von Milind S

Milind Svor 1 Tag

Omniparser!

Profilbild von Deep

Deepvor 1 Tag

local segmentation plus OCR instead of a vision model. cuts what you send to the model, and it runs offline too.

Profilbild von Matt Ronge

Matt Rongevor 1 Tag

Thats very slick. Does it use a larger model to know where to go? So if you ask it something complex to do in Blender for example it can break it down into steps?

Profilbild von Milind S

Milind Svor 1 Tag

There is literally no other larger models being used in this. Its as simple as it gets

Profilbild von Matt Ronge

Matt Rongevor 1 Tag

Open source when lol

Profilbild von Milind S

Milind Svor 1 Tag

Haha let me clean it up and then I’ll do it promise :D Its based on my other open source project already called tiptour macos

Profilbild von Matt Ronge

Matt Rongevor 1 Tag

Yea love it, I've been following your work for a while. Thanks for sharing it!

Profilbild von Varik Verilion

Varik Verilionvor 1 Tag

The interesting part is the interface boundary. Segmentation and OCR turn pixels into a compact, structured state before Jev plans an action. That makes local inference practical and keeps the sensitive screen data on-device.

Profilbild von Justin

Justinvor 1 Tag

Bro, you build way too fast :D

Profilbild von Milind S

Milind Svor 1 Tag

Thats how I roll :D

Profilbild von Bruno Skvorc

Bruno Skvorcvor 1 Tag

chat too

Profilbild von bikram

bikramvor 1 Tag

Yes. Share. Also, should go into OMB, will create a really good setup.

Profilbild von Milind S

Milind Svor 1 Tag

Great idea

Profilbild von B⭕bby

B⭕bbyvor 1 Tag

Can you add voice to text on top so you can control it with your voice

Profilbild von Milind S

Milind Svor 1 Tag

Totally i can

Profilbild von Omkar Satpute

Omkar Satputevor 1 Tag

SupaaaaFastttt computer use

Profilbild von ethereumdegen.eth 🕶️ᵍᵐ

ethereumdegen.eth 🕶️ᵍᵐvor 1 Tag

I would love to use this and test it out

Profilbild von bikram

bikramvor 1 Tag

What are you using for on-device OCR

Profilbild von Milind S

Milind Svor 1 Tag

Apple ships it natively on all devices

Profilbild von Tacdel

Tacdelvor 1 Tag

90ms local loops make 5-second cloud VLM latency look unplayable. How does it handle icon-only buttons where OCR finds zero text?

Profilbild von Carlos Ziegler

Carlos Zieglervor 1 Tag

This is really nice!!!!

Profilbild von Milind S

Milind Svor 1 Tag

I know right!!

Profilbild von Doomshade

Doomshadevor 1 Tag

Dude this is sick, would really love to try this out with some desktop applications!

Profilbild von Milind S

Milind Svor 1 Tag

Alright man. I’ll send it to you when its ready

Profilbild von Doomshade

Doomshadevor 1 Tag

🙏 much appreciated! btw - love openmaus - got it configured on my machine and made a few tweaks to my system - need to pull some of the latest features you've shipped !

Profilbild von Milind S

Milind Svor 1 Tag

Lfgg

Profilbild von Sahibzada Allahyar

Sahibzada Allahyarvor 1 Tag

this one can do it too and it's local

Profilbild von Joey van Koningsbruggen

Joey van Koningsbruggenvor 1 Tag

Very cool

Profilbild von up

upvor 1 Tag

Loved it, crazy capabilities

Profilbild von Milind S

Milind Svor 1 Tag

Indeed

Profilbild von shaik arbaz

shaik arbazvor 1 Tag

Make it open source.

Profilbild von Milind S

Milind Svor 1 Tag

Will do

Profilbild von Arpit

Arpitvor 1 Tag

How can it be used to do computer use in bg?

Profilbild von Milind S

Milind Svor 1 Tag

I’m sure @trycua can

Profilbild von Hugo Catarino

Hugo Catarinovor 1 Tag

Que brutalidade!!! 🤯

Profilbild von Milind S

Milind Svor 1 Tag

Sí!!!

Profilbild von Christian Giangrande

Christian Giangrandevor 1 Tag

Impliment into maus 🙏🙏🙏

Profilbild von Milind S

Milind Svor 1 Tag

Hell yeah

Profilbild von Konrad

Konradvor 1 Tag

how does the LM come up with more complex interactions, like in Blender, dragging while holding down buttons?

Profilbild von Milind S

Milind Svor 1 Tag

Gotta pair this with an llm to do that

Profilbild von Konrad

Konradvor 1 Tag

Message me the details on WhatsApp, I have new investors for the stuff, doing a seed round the coming weeks. Getting the production cost from ~48ct to ~20ct could be realistic with this new tooling.

Profilbild von Mar

Marvor 1 Tag

Dude, I need this!

Profilbild von Milind S

Milind Svor 1 Tag

Releasing soon

Profilbild von Nikhil Gangaraju

Nikhil Gangarajuvor 1 Tag

+1 for the open source version. Great demo!

Profilbild von Milind S

Milind Svor 1 Tag

Thanks! Repo coming shortly

Profilbild von Shahn

Shahnvor 1 Tag

this is really cool @milindlabs I’ve been trying out Jev for some browser use features as well, here you runs loop with ocr model right ?

Profilbild von Milind S

Milind Svor 1 Tag

I do yes along with the segmentation model

Profilbild von Guilherme

Guilhermevor 1 Tag

Does it support multi-screen ?

Profilbild von Milind S

Milind Svor 1 Tag

It does

Profilbild von D Row Kavi

D Row Kavivor 1 Tag

@bot @poteto 😅🤘🏽

Profilbild von AskMeHow

AskMeHowvor 1 Tag

open source pls

Profilbild von Milind S

Milind Svor 1 Tag

If you say so

Profilbild von Aditya Sinha

Aditya Sinhavor 1 Tag

@iamgingertrash how did you know

Profilbild von Dominik

Dominikvor 1 Tag

a click loop without an llm can still own the machine. i am building refuse rules on the write surface so speed never becomes silent permission.

Ähnliche Videos

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

Milind S

38,361 Aufrufe • vor 3 Monaten

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

MIKE

14,387 Aufrufe • vor 6 Tagen

I took Clicky and I made it 5x Faster So I saw that Farza 🇵🇰🇺🇸 was using uses Claude's vision to find UI elements on screen, send a screenshot, wait for coordinates back. It works, but it's slow. I replaced that with OmniParser V2 by Microsoft which is a local Object detection model trained specifically on UI elements. It runs on-device, detects every button, menu, and icon in 400ms, and gives me pixel-precise coordinates. No API call, no latency, no cost. The green highlights you see around the UI elements is the detection overlay and you can see as I am switching the screen it takes no time to detect and highlight which is pretty neat! With Local models improving by the day, Its the right direction for applications like these. Next up: video-synced tutorials where a YouTube tutorial pauses and waits for you to perform each action in the real app. I am not stopping!

I took Clicky and I made it 5x Faster So I saw that Farza 🇵🇰🇺🇸 was using uses Claude's vision to find UI elements on screen, send a screenshot, wait for coordinates back. It works, but it's slow. I replaced that with OmniParser V2 by Microsoft which is a local Object detection model trained specifically on UI elements. It runs on-device, detects every button, menu, and icon in 400ms, and gives me pixel-precise coordinates. No API call, no latency, no cost. The green highlights you see around the UI elements is the detection overlay and you can see as I am switching the screen it takes no time to detect and highlight which is pretty neat! With Local models improving by the day, Its the right direction for applications like these. Next up: video-synced tutorials where a YouTube tutorial pauses and waits for you to perform each action in the real app. I am not stopping!

Milind S

73,349 Aufrufe • vor 5 Monaten

Our first short course with Anthropic! Building Towards Computer Use with Anthropic. This teaches you to build an LLM-based agent that uses a computer interface by generating mouse clicks and keystrokes. Computer Use is an important, emerging capability for LLMs that will let AI agents do many more tasks than were possible before, since it lets them interact with interfaces designed for humans to use, rather than only tools that provide explicit API access. I hope you will enjoy learning about it! This course is taught by Anthropic's Head of Curriculum, Colt_Steele. You'll learn to apply image reasoning and tool use to "use" a computer as follows: a model processes an image of the screen, analyzes it to understand what's going on, and navigates the computer via mouse clicks and keystrokes. This course goes through the key building blocks, and culminates in a demo of an AI assistant that uses a web browser to search for a research paper, downloads the PDF, and finally summarizes the paper for you. In detail, you’ll: - Learn about Anthropic's family of models, when to use which one, and make API requests to Claude - Use multi-modal prompts that combine text and image content blocks, and also work with streaming responses - Improve your prompting by using prompt templates, using XML to structure prompts, and providing examples - Implement prompt caching to reduce cost and latency - Apply tool-use to build a chatbot that can call different tools to respond to queries - See all these building blocks come together in Computer Use demo Please sign up here:

Our first short course with Anthropic! Building Towards Computer Use with Anthropic. This teaches you to build an LLM-based agent that uses a computer interface by generating mouse clicks and keystrokes. Computer Use is an important, emerging capability for LLMs that will let AI agents do many more tasks than were possible before, since it lets them interact with interfaces designed for humans to use, rather than only tools that provide explicit API access. I hope you will enjoy learning about it! This course is taught by Anthropic's Head of Curriculum, Colt_Steele. You'll learn to apply image reasoning and tool use to "use" a computer as follows: a model processes an image of the screen, analyzes it to understand what's going on, and navigates the computer via mouse clicks and keystrokes. This course goes through the key building blocks, and culminates in a demo of an AI assistant that uses a web browser to search for a research paper, downloads the PDF, and finally summarizes the paper for you. In detail, you’ll: - Learn about Anthropic's family of models, when to use which one, and make API requests to Claude - Use multi-modal prompts that combine text and image content blocks, and also work with streaming responses - Improve your prompting by using prompt templates, using XML to structure prompts, and providing examples - Implement prompt caching to reduce cost and latency - Apply tool-use to build a chatbot that can call different tools to respond to queries - See all these building blocks come together in Computer Use demo Please sign up here:

Andrew Ng

170,541 Aufrufe • vor 1 Jahr

Jev is a new model from the co-inventor of ChatGPT that can use computer tools in realtime right now it's playing Minecraft on its own computer in the cloud on Orgo this is what it looks like with just one Jev, but imagine: what about a thousand? or what about a billion of Jevs? each working together on their own computers in the cloud, discovering new science, running experiments, and designing new drugs 24/7. this is going to be bigger than ChatGPT

Jev is a new model from the co-inventor of ChatGPT that can use computer tools in realtime right now it's playing Minecraft on its own computer in the cloud on Orgo this is what it looks like with just one Jev, but imagine: what about a thousand? or what about a billion of Jevs? each working together on their own computers in the cloud, discovering new science, running experiments, and designing new drugs 24/7. this is going to be bigger than ChatGPT

Nick Vasilescu

14,120 Aufrufe • vor 1 Tag

This is actually cool - I tried the same prompt for the new Interactive Playwright skill in Codex & GPT-5.4 xHigh - the one above is with the skill and the one below is without. What the skill does is uses the computer use capability of GPT-5.4 to look and navigate the UI. This never worked for me before, but with GPT-5.4 this is the first time I can actually see a massive difference. You can see how the first scene is much more coherent, higher fidelity and complete. The one below is missing a lot of elements and isn't as rich in detail. I'll keep using it for any UI work now.

This is actually cool - I tried the same prompt for the new Interactive Playwright skill in Codex & GPT-5.4 xHigh - the one above is with the skill and the one below is without. What the skill does is uses the computer use capability of GPT-5.4 to look and navigate the UI. This never worked for me before, but with GPT-5.4 this is the first time I can actually see a massive difference. You can see how the first scene is much more coherent, higher fidelity and complete. The one below is missing a lot of elements and isn't as rich in detail. I'll keep using it for any UI work now.

Peter Gostev (SF 24-28 August)

257,389 Aufrufe • vor 6 Monaten

computer use agents are slow because they think too much they make round trips to an LLM. every single time. even for a command you've said a hundred times before. that's why it always feels slow. my mac now remembers. the first time i say something new, like "search youtube for lofi beats", a big model figures it out once, does it, and saves the recipe as a json spec: {"op":"act","do":"open","target":"Safari"} {"op":"act","do":"type","text":"{query}"} every time after that, a tiny model on my laptop recognizes the command, fills in the parameters, and replays the recipe with CUA say it once, it's slow. say it again, it's instant. the fine tuned tiny model is 350 million parameters. its only job is recognizing which of my commands i just said. and when the big model does plan, it streams actions line by line. each line runs the moment it's finished, so the first action is executing while the model is still writing the second. everything it learns lives in one plain text file on my disk. i've been playing around a lot with Cua by Francesco and peekaboo by Peter Steinberger 🦞 . curious whether a standard computer use spec makes sense, one that streaming models can drive to make computer use actually fast.

computer use agents are slow because they think too much they make round trips to an LLM. every single time. even for a command you've said a hundred times before. that's why it always feels slow. my mac now remembers. the first time i say something new, like "search youtube for lofi beats", a big model figures it out once, does it, and saves the recipe as a json spec: {"op":"act","do":"open","target":"Safari"} {"op":"act","do":"type","text":"{query}"} every time after that, a tiny model on my laptop recognizes the command, fills in the parameters, and replays the recipe with CUA say it once, it's slow. say it again, it's instant. the fine tuned tiny model is 350 million parameters. its only job is recognizing which of my commands i just said. and when the big model does plan, it streams actions line by line. each line runs the moment it's finished, so the first action is executing while the model is still writing the second. everything it learns lives in one plain text file on my disk. i've been playing around a lot with Cua by Francesco and peekaboo by Peter Steinberger 🦞 . curious whether a standard computer use spec makes sense, one that streaming models can drive to make computer use actually fast.

Milind S

14,235 Aufrufe • vor 2 Monaten

One of our biggest engineering problems is that Conductor is Mac only, which means most of our testing is manual and local. So I asked Fable to: - port Conductor to Linux - open the build inside a Conductor Cloud sandbox - use Codex computer use to QA - send me a screen recording of the whole thing And it worked! The dream is that a user can report a bug, and it gets triaged, fixed, and QA'd in the background. We're getting very close!

One of our biggest engineering problems is that Conductor is Mac only, which means most of our testing is manual and local. So I asked Fable to: - port Conductor to Linux - open the build inside a Conductor Cloud sandbox - use Codex computer use to QA - send me a screen recording of the whole thing And it worked! The dream is that a user can report a bug, and it gets triaged, fixed, and QA'd in the background. We're getting very close!

Charlie Holtz

47,729 Aufrufe • vor 2 Monaten

Passport Prime gives you 50GB of encrypted storage for your most important files. So how do you get a file from your computer onto the device without exposing everything already there? Normally, plugging in a drive works both ways: the computer can read everything and write anything to it. So can any malware on that computer. Plug in Passport Prime, and the computer sees only one small, separate area: the Airlock. Your 50GB never appears. The computer can't open it, scan it, or tell what's inside. Drop your file into the Airlock folder on your desktop, like you would with any drive. Then unplug. Only now, on Passport Prime's screen, do you move the file from the Airlock into your encrypted storage. The last step happens after the computer is out of the picture. Nothing travels straight from a computer into your vault. Everything passes through an airlock first. That's why it's called Airlock🔐

Passport Prime gives you 50GB of encrypted storage for your most important files. So how do you get a file from your computer onto the device without exposing everything already there? Normally, plugging in a drive works both ways: the computer can read everything and write anything to it. So can any malware on that computer. Plug in Passport Prime, and the computer sees only one small, separate area: the Airlock. Your 50GB never appears. The computer can't open it, scan it, or tell what's inside. Drop your file into the Airlock folder on your desktop, like you would with any drive. Then unplug. Only now, on Passport Prime's screen, do you move the file from the Airlock into your encrypted storage. The last step happens after the computer is out of the picture. Nothing travels straight from a computer into your vault. Everything passes through an airlock first. That's why it's called Airlock🔐

FOUNDATION

43,395 Aufrufe • vor 1 Monat

THIS GUY ADDED A LIVE BROWSER TO CLAUDE CODE SO YOU CAN CLICK ANY ELEMENT AND EDIT IT INSTANTLY biggest issue with vibe coding UI is that you have to describe what you want to change. if you prompt it the wrong selector or wrong component, Claude can't find it. now you just click it. your app runs in an embedded browser with Claude Code. you can click any button, any text, any div. Claude instantly knows exactly what you're pointing at. click. instruct. done. no more "change the button in the top right corner of the second card component." all you have to do now is just click the button. AND its open source

THIS GUY ADDED A LIVE BROWSER TO CLAUDE CODE SO YOU CAN CLICK ANY ELEMENT AND EDIT IT INSTANTLY biggest issue with vibe coding UI is that you have to describe what you want to change. if you prompt it the wrong selector or wrong component, Claude can't find it. now you just click it. your app runs in an embedded browser with Claude Code. you can click any button, any text, any div. Claude instantly knows exactly what you're pointing at. click. instruct. done. no more "change the button in the top right corner of the second card component." all you have to do now is just click the button. AND its open source

Om Patel

153,892 Aufrufe • vor 5 Monaten

Today we're excited to introduce Vy, our AI that sees and acts on your computer. At Vercept, our mission is to reinvent how humans use computers–enabling you to accomplish orders of magnitude more than what you can do today. Vy is a first glimpse at AI that sees and uses your computer just like you do. It runs natively on your computer, with access to all your applications and accounts that you’re already signed into. It is available for download starting today! As part of the release, we’ve built VyUI, an AI model bridging the gap between language and your screen. It sets a new state-of-the-art on UI grounding benchmarks like ScreenSpot v1, ScreenSpot v2, and GroundUI Web–beating leading models from OpenAI, Google, and Anthropic. Check out what Vy can do in the thread below: 🧵👇

Today we're excited to introduce Vy, our AI that sees and acts on your computer. At Vercept, our mission is to reinvent how humans use computers–enabling you to accomplish orders of magnitude more than what you can do today. Vy is a first glimpse at AI that sees and uses your computer just like you do. It runs natively on your computer, with access to all your applications and accounts that you’re already signed into. It is available for download starting today! As part of the release, we’ve built VyUI, an AI model bridging the gap between language and your screen. It sets a new state-of-the-art on UI grounding benchmarks like ScreenSpot v1, ScreenSpot v2, and GroundUI Web–beating leading models from OpenAI, Google, and Anthropic. Check out what Vy can do in the thread below: 🧵👇

Vercept

123,748 Aufrufe • vor 1 Jahr

WOW! I'm so excited about this. OpenAI Developers said Codex was good at Computer Use, but I wasn't prepared for this. For the last two weeks I've been working on a Computer Use skill to work with Linux. And while I had some success, it was a pretty frustrating experience. That is...until the breakthrough. Using accessibility tools, Codex can now control my entire computer, not just the browser. There are limits to this, of course, but what a time to be alive. This Computer Use skill will unlock and entirely new set of automations, all powered by Codex. Demonstration below 👇

WOW! I'm so excited about this. OpenAI Developers said Codex was good at Computer Use, but I wasn't prepared for this. For the last two weeks I've been working on a Computer Use skill to work with Linux. And while I had some success, it was a pretty frustrating experience. That is...until the breakthrough. Using accessibility tools, Codex can now control my entire computer, not just the browser. There are limits to this, of course, but what a time to be alive. This Computer Use skill will unlock and entirely new set of automations, all powered by Codex. Demonstration below 👇

am.will

79,698 Aufrufe • vor 5 Monaten

APPLE BUILT A $10,000 COMPUTER THAT RUNS WHAT USED TO REQUIRE AN ENTIRE SERVER ROOM 06:40 he shows the number that breaks everything - one Mac Studio is more powerful than four Dell or AMD machines combined and uses half the energy DeepSeek R1 doesn't run on even two competitors - you need four - one Mac Studio does it alone runs quieter than a fan in the room and uses less energy than a kettle in the kitchen four Mac Studios on a desk connected through a regular cable run a 1 trillion parameter model - what used to require an entire data center anyone can put this at home and Jeff Geerling proved it - run the same models that big companies use without cloud, without subscriptions, without data leaving the room

APPLE BUILT A $10,000 COMPUTER THAT RUNS WHAT USED TO REQUIRE AN ENTIRE SERVER ROOM 06:40 he shows the number that breaks everything - one Mac Studio is more powerful than four Dell or AMD machines combined and uses half the energy DeepSeek R1 doesn't run on even two competitors - you need four - one Mac Studio does it alone runs quieter than a fan in the room and uses less energy than a kettle in the kitchen four Mac Studios on a desk connected through a regular cable run a 1 trillion parameter model - what used to require an entire data center anyone can put this at home and Jeff Geerling proved it - run the same models that big companies use without cloud, without subscriptions, without data leaving the room

Noisy

26,305 Aufrufe • vor 2 Monaten

Mac Mini is super nice because you can use native iMessage for talking with OpenClaw. The only problem is that it's very expensive and you need to always have your Mac Mini on to run this setup. So I made a plugin to allow anyone on any computer (either Linux, Windows, or Mac) to be able to have access to iMessage natively. Even better, it has access to all of the reactions, triple dot indicator, etc. Check it out and let me know what you think.

Mac Mini is super nice because you can use native iMessage for talking with OpenClaw. The only problem is that it's very expensive and you need to always have your Mac Mini on to run this setup. So I made a plugin to allow anyone on any computer (either Linux, Windows, or Mac) to be able to have access to iMessage natively. Even better, it has access to all of the reactions, triple dot indicator, etc. Check it out and let me know what you think.

Nick Vasilescu

265,845 Aufrufe • vor 7 Monaten

Nobody seems to know how insane GPT-5.4 is with computer use. I asked GPT-5.4 to draw the OpenAI logo in Microsoft Paint. No computer use API. Just a screenshot and basic tool calls (click, drag, press_key) all coordinate-based. The first drawing was awful. And GPT knew it. It looked at its own result and essentially went "yeah, no." What happened next is what broke my brain: It opened a browser. Went to Bing Images. Searched for the OpenAI logo. Found one. Then (and I cannot stress this enough) it used the Windows area screenshot shortcut (Win+Shift+S) to snip just the logo off the screen. Went back to Paint. Imported it. Centered it. All on its own. No instructions to do any of that. It just improvised a better strategy when the first one failed. My prompt was "Draw the OpenAI logo" with Paint already opened on the computer. Sure, it's "cheating." But honestly? That's exactly what I'd do too. And the fact that it came up with this plan from nothing but a screenshot and a coordinate system is wild.

Clad3815

1,069,968 Aufrufe • vor 6 Monaten

David Friedberg: Frontier Models are Training on Your Novel Insights as “De-Identified Data” @jason: “Should they trust any of these LLMs with their proprietary knowledge for fear of having it cribbed into a core LLM?” david friedberg: “I have had experiences where we've asked some fairly novel scientific questions, and (the AI model) identifies it as a novel insight. It's like, ‘Oh, never thought about that, interesting, blah, blah, blah.’ And then using a different account, asking the next version (of the model) later, I've now experienced this. It's like, ‘Oh, well, you could do this,’ and it actually just describes this exact thing that we had in our chat in the previous version. Now, these are a handful of anecdotal experiences, but I know the domain that we work in, and the niche of it, and the ideation of this stuff, and the novelty of this stuff, and the lack of papers being published, and so on. So I know that there isn't some new corpus of information out there that's training the new model. So all I can say at that point is that my conversation or our analyses have been used for training.” David Sacks: “Okay, this does raise a really good question. What does it mean that the model is allowed to train on unidentifiable data?” Friedberg: “Well, that's my point. So it doesn't use any of my personal information, but it can use an insight derived from our chat, which it can then say is some training data that is unrelated. But the truth is, it's actually a piece of IP that's our organization’s IP, and our engagement back and forth. We don't have any NDA or confidentiality provisions or protections with them being a service provider back to us. This is why I care a lot about open source because I don't want them having my chat logs because they can use it for training to create an IP advantage that is now diffused to the rest of the market.”

David Friedberg: Frontier Models are Training on Your Novel Insights as “De-Identified Data” @jason: “Should they trust any of these LLMs with their proprietary knowledge for fear of having it cribbed into a core LLM?” david friedberg: “I have had experiences where we've asked some fairly novel scientific questions, and (the AI model) identifies it as a novel insight. It's like, ‘Oh, never thought about that, interesting, blah, blah, blah.’ And then using a different account, asking the next version (of the model) later, I've now experienced this. It's like, ‘Oh, well, you could do this,’ and it actually just describes this exact thing that we had in our chat in the previous version. Now, these are a handful of anecdotal experiences, but I know the domain that we work in, and the niche of it, and the ideation of this stuff, and the novelty of this stuff, and the lack of papers being published, and so on. So I know that there isn't some new corpus of information out there that's training the new model. So all I can say at that point is that my conversation or our analyses have been used for training.” David Sacks: “Okay, this does raise a really good question. What does it mean that the model is allowed to train on unidentifiable data?” Friedberg: “Well, that's my point. So it doesn't use any of my personal information, but it can use an insight derived from our chat, which it can then say is some training data that is unrelated. But the truth is, it's actually a piece of IP that's our organization’s IP, and our engagement back and forth. We don't have any NDA or confidentiality provisions or protections with them being a service provider back to us. This is why I care a lot about open source because I don't want them having my chat logs because they can use it for training to create an IP advantage that is now diffused to the rest of the market.”

The All-In Podcast

54,050 Aufrufe • vor 7 Tagen

Corporations that are purely AI and robotics will vastly outperform any corporations that have people in the loop You can think of it like how 'computer' used to be a job that humans had. You would go and get a job as a computer where you would do calculations. They had entire skyscrapers full of humans....20, 30 floors of humans...just doing calculations. Now, that entire skyscraper of humans doing calculations can be replaced by a laptop with a spreadsheet. That spreadsheet can do vastly more calculations than an entire building full of human computers So, you think about it: what if only some of the cells in your spreadsheet were calculated by humans? That would be much worse than if all of the cells in your spreadsheet were calculated by the computer. And so, really what will happen is the pure AI, pure robotics corporations or collectives will far outperform any corporations that have humans in the loop. It will happen very quickly

Corporations that are purely AI and robotics will vastly outperform any corporations that have people in the loop You can think of it like how 'computer' used to be a job that humans had. You would go and get a job as a computer where you would do calculations. They had entire skyscrapers full of humans....20, 30 floors of humans...just doing calculations. Now, that entire skyscraper of humans doing calculations can be replaced by a laptop with a spreadsheet. That spreadsheet can do vastly more calculations than an entire building full of human computers So, you think about it: what if only some of the cells in your spreadsheet were calculated by humans? That would be much worse than if all of the cells in your spreadsheet were calculated by the computer. And so, really what will happen is the pure AI, pure robotics corporations or collectives will far outperform any corporations that have humans in the loop. It will happen very quickly

X Freeze

16,850,188 Aufrufe • vor 7 Monaten

Astra began to saturate a computer-use eval we haven’t even released yet. Astra was not only able to kill several hostile mobs in Minecraft, but it made it all the way to building a Nether portal in less than three hours. We believe that if we ran it longer, it would beat the game. Astra was able to do things like tower up to kill a horde of zombies, trap a skeleton inside a hole to dodge its arrows, craft a boat, and navigate back to its base. Here is the full 3 hour play through. All of this was without any specialized harness: pure computer use.

Astra began to saturate a computer-use eval we haven’t even released yet. Astra was not only able to kill several hostile mobs in Minecraft, but it made it all the way to building a Nether portal in less than three hours. We believe that if we ran it longer, it would beat the game. Astra was able to do things like tower up to kill a horde of zombies, trap a skeleton inside a hole to dodge its arrows, craft a boat, and navigate back to its base. Here is the full 3 hour play through. All of this was without any specialized harness: pure computer use.

Vals AI

31,057 Aufrufe • vor 10 Tagen