Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Okay so Jev can actually do computer use really well Without any screenshots, or LLMs and no Pixels leave my mac I dont even read the Dom elements A local CoreML model segments every button and UI element on screen. On-device OCR reads the labels. That text is all... show more

Milind S

3,902 subscribers

116,719 views • 1 day ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

62 Comments

Milind S1 day ago

If this gets enough attention I’ll make it open source. Meanwhile check out my other stuff

Vedant Bhayani1 day ago

I think the future is multiple other companies building new kind of llm models for individual tasks with which you just plug it in with frontiers model for steering and thinking Also domain specific harness and custom trained models with your data for specific use cases are going be more relevant and we are going to see even more than before

Milind S1 day ago

I agree that’s where things are headed

$/h(elton)?Fabio\.(t|j)s/gi's profile picture$

/h(elton)?Fabio\.(t|j)s/gi1 day ago

hey man, do you mind doing a test with a local model that uses the same architecture? like this one based on qwen-2.5-1b?

Milind S1 day ago

I’ll check it out

Valentin Pletzer1 day ago

very cool!

Milind S1 day ago

Thank you!

Artur Kre1 day ago

wow insane, which local model does the window UI segmenting ?

Milind S1 day ago

Omniparser!

Deep1 day ago

local segmentation plus OCR instead of a vision model. cuts what you send to the model, and it runs offline too.

Matt Ronge1 day ago

Thats very slick. Does it use a larger model to know where to go? So if you ask it something complex to do in Blender for example it can break it down into steps?

Milind S1 day ago

There is literally no other larger models being used in this. Its as simple as it gets

Matt Ronge1 day ago

Open source when lol

Milind S1 day ago

Haha let me clean it up and then I’ll do it promise :D Its based on my other open source project already called tiptour macos

Matt Ronge1 day ago

Yea love it, I've been following your work for a while. Thanks for sharing it!

Varik Verilion1 day ago

The interesting part is the interface boundary. Segmentation and OCR turn pixels into a compact, structured state before Jev plans an action. That makes local inference practical and keeps the sensitive screen data on-device.

Justin1 day ago

Bro, you build way too fast :D

Milind S1 day ago

Thats how I roll :D

Bruno Skvorc1 day ago

chat too

bikram1 day ago

Yes. Share. Also, should go into OMB, will create a really good setup.

Milind S1 day ago

Great idea

B⭕bby1 day ago

Can you add voice to text on top so you can control it with your voice

Milind S1 day ago

Totally i can

Omkar Satpute1 day ago

SupaaaaFastttt computer use

ethereumdegen.eth 🕶️ᵍᵐ1 day ago

I would love to use this and test it out

bikram1 day ago

What are you using for on-device OCR

Milind S1 day ago

Apple ships it natively on all devices

Tacdel1 day ago

90ms local loops make 5-second cloud VLM latency look unplayable. How does it handle icon-only buttons where OCR finds zero text?

Carlos Ziegler1 day ago

This is really nice!!!!

Milind S1 day ago

I know right!!

Doomshade1 day ago

Dude this is sick, would really love to try this out with some desktop applications!

Milind S1 day ago

Alright man. I’ll send it to you when its ready

Doomshade1 day ago

🙏 much appreciated! btw - love openmaus - got it configured on my machine and made a few tweaks to my system - need to pull some of the latest features you've shipped !

Milind S1 day ago

Lfgg

Sahibzada Allahyar1 day ago

this one can do it too and it's local

Joey van Koningsbruggen1 day ago

Very cool

up1 day ago

Loved it, crazy capabilities

Milind S1 day ago

Indeed

shaik arbaz1 day ago

Make it open source.

Milind S1 day ago

Will do

Arpit1 day ago

How can it be used to do computer use in bg?

Milind S1 day ago

I’m sure @trycua can

Hugo Catarino1 day ago

Que brutalidade!!! 🤯

Milind S1 day ago

Sí!!!

Christian Giangrande1 day ago

Impliment into maus 🙏🙏🙏

Milind S1 day ago

Hell yeah

Konrad1 day ago

how does the LM come up with more complex interactions, like in Blender, dragging while holding down buttons?

Milind S1 day ago

Gotta pair this with an llm to do that

Konrad1 day ago

Message me the details on WhatsApp, I have new investors for the stuff, doing a seed round the coming weeks. Getting the production cost from ~48ct to ~20ct could be realistic with this new tooling.

Mar1 day ago

Dude, I need this!

Milind S1 day ago

Releasing soon

Nikhil Gangaraju1 day ago

+1 for the open source version. Great demo!

Milind S1 day ago

Thanks! Repo coming shortly

Shahn1 day ago

this is really cool @milindlabs I’ve been trying out Jev for some browser use features as well, here you runs loop with ocr model right ?

Milind S1 day ago

I do yes along with the segmentation model

Guilherme1 day ago

Does it support multi-screen ?

Milind S1 day ago

It does

D Row Kavi1 day ago

@bot @poteto 😅🤘🏽

AskMeHow1 day ago

open source pls

Milind S1 day ago

If you say so

Aditya Sinha1 day ago

@iamgingertrash how did you know

Dominik1 day ago

a click loop without an llm can still own the machine. i am building refuse rules on the write surface so speed never becomes silent permission.

Related Videos

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

Milind S

38,361 views • 3 months ago

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

MIKE

14,387 views • 6 days ago

I took Clicky and I made it 5x Faster So I saw that Farza 🇵🇰🇺🇸 was using uses Claude's vision to find UI elements on screen, send a screenshot, wait for coordinates back. It works, but it's slow. I replaced that with OmniParser V2 by Microsoft which is a local Object detection model trained specifically on UI elements. It runs on-device, detects every button, menu, and icon in 400ms, and gives me pixel-precise coordinates. No API call, no latency, no cost. The green highlights you see around the UI elements is the detection overlay and you can see as I am switching the screen it takes no time to detect and highlight which is pretty neat! With Local models improving by the day, Its the right direction for applications like these. Next up: video-synced tutorials where a YouTube tutorial pauses and waits for you to perform each action in the real app. I am not stopping!

I took Clicky and I made it 5x Faster So I saw that Farza 🇵🇰🇺🇸 was using uses Claude's vision to find UI elements on screen, send a screenshot, wait for coordinates back. It works, but it's slow. I replaced that with OmniParser V2 by Microsoft which is a local Object detection model trained specifically on UI elements. It runs on-device, detects every button, menu, and icon in 400ms, and gives me pixel-precise coordinates. No API call, no latency, no cost. The green highlights you see around the UI elements is the detection overlay and you can see as I am switching the screen it takes no time to detect and highlight which is pretty neat! With Local models improving by the day, Its the right direction for applications like these. Next up: video-synced tutorials where a YouTube tutorial pauses and waits for you to perform each action in the real app. I am not stopping!

Milind S

73,349 views • 5 months ago

Our first short course with Anthropic! Building Towards Computer Use with Anthropic. This teaches you to build an LLM-based agent that uses a computer interface by generating mouse clicks and keystrokes. Computer Use is an important, emerging capability for LLMs that will let AI agents do many more tasks than were possible before, since it lets them interact with interfaces designed for humans to use, rather than only tools that provide explicit API access. I hope you will enjoy learning about it! This course is taught by Anthropic's Head of Curriculum, Colt_Steele. You'll learn to apply image reasoning and tool use to "use" a computer as follows: a model processes an image of the screen, analyzes it to understand what's going on, and navigates the computer via mouse clicks and keystrokes. This course goes through the key building blocks, and culminates in a demo of an AI assistant that uses a web browser to search for a research paper, downloads the PDF, and finally summarizes the paper for you. In detail, you’ll: - Learn about Anthropic's family of models, when to use which one, and make API requests to Claude - Use multi-modal prompts that combine text and image content blocks, and also work with streaming responses - Improve your prompting by using prompt templates, using XML to structure prompts, and providing examples - Implement prompt caching to reduce cost and latency - Apply tool-use to build a chatbot that can call different tools to respond to queries - See all these building blocks come together in Computer Use demo Please sign up here:

Our first short course with Anthropic! Building Towards Computer Use with Anthropic. This teaches you to build an LLM-based agent that uses a computer interface by generating mouse clicks and keystrokes. Computer Use is an important, emerging capability for LLMs that will let AI agents do many more tasks than were possible before, since it lets them interact with interfaces designed for humans to use, rather than only tools that provide explicit API access. I hope you will enjoy learning about it! This course is taught by Anthropic's Head of Curriculum, Colt_Steele. You'll learn to apply image reasoning and tool use to "use" a computer as follows: a model processes an image of the screen, analyzes it to understand what's going on, and navigates the computer via mouse clicks and keystrokes. This course goes through the key building blocks, and culminates in a demo of an AI assistant that uses a web browser to search for a research paper, downloads the PDF, and finally summarizes the paper for you. In detail, you’ll: - Learn about Anthropic's family of models, when to use which one, and make API requests to Claude - Use multi-modal prompts that combine text and image content blocks, and also work with streaming responses - Improve your prompting by using prompt templates, using XML to structure prompts, and providing examples - Implement prompt caching to reduce cost and latency - Apply tool-use to build a chatbot that can call different tools to respond to queries - See all these building blocks come together in Computer Use demo Please sign up here:

Andrew Ng

170,541 views • 1 year ago

Jev is a new model from the co-inventor of ChatGPT that can use computer tools in realtime right now it's playing Minecraft on its own computer in the cloud on Orgo this is what it looks like with just one Jev, but imagine: what about a thousand? or what about a billion of Jevs? each working together on their own computers in the cloud, discovering new science, running experiments, and designing new drugs 24/7. this is going to be bigger than ChatGPT

Jev is a new model from the co-inventor of ChatGPT that can use computer tools in realtime right now it's playing Minecraft on its own computer in the cloud on Orgo this is what it looks like with just one Jev, but imagine: what about a thousand? or what about a billion of Jevs? each working together on their own computers in the cloud, discovering new science, running experiments, and designing new drugs 24/7. this is going to be bigger than ChatGPT

Nick Vasilescu

14,120 views • 1 day ago

This is actually cool - I tried the same prompt for the new Interactive Playwright skill in Codex & GPT-5.4 xHigh - the one above is with the skill and the one below is without. What the skill does is uses the computer use capability of GPT-5.4 to look and navigate the UI. This never worked for me before, but with GPT-5.4 this is the first time I can actually see a massive difference. You can see how the first scene is much more coherent, higher fidelity and complete. The one below is missing a lot of elements and isn't as rich in detail. I'll keep using it for any UI work now.

This is actually cool - I tried the same prompt for the new Interactive Playwright skill in Codex & GPT-5.4 xHigh - the one above is with the skill and the one below is without. What the skill does is uses the computer use capability of GPT-5.4 to look and navigate the UI. This never worked for me before, but with GPT-5.4 this is the first time I can actually see a massive difference. You can see how the first scene is much more coherent, higher fidelity and complete. The one below is missing a lot of elements and isn't as rich in detail. I'll keep using it for any UI work now.

Peter Gostev (SF 24-28 August)

257,389 views • 6 months ago

computer use agents are slow because they think too much they make round trips to an LLM. every single time. even for a command you've said a hundred times before. that's why it always feels slow. my mac now remembers. the first time i say something new, like "search youtube for lofi beats", a big model figures it out once, does it, and saves the recipe as a json spec: {"op":"act","do":"open","target":"Safari"} {"op":"act","do":"type","text":"{query}"} every time after that, a tiny model on my laptop recognizes the command, fills in the parameters, and replays the recipe with CUA say it once, it's slow. say it again, it's instant. the fine tuned tiny model is 350 million parameters. its only job is recognizing which of my commands i just said. and when the big model does plan, it streams actions line by line. each line runs the moment it's finished, so the first action is executing while the model is still writing the second. everything it learns lives in one plain text file on my disk. i've been playing around a lot with Cua by Francesco and peekaboo by Peter Steinberger 🦞 . curious whether a standard computer use spec makes sense, one that streaming models can drive to make computer use actually fast.

computer use agents are slow because they think too much they make round trips to an LLM. every single time. even for a command you've said a hundred times before. that's why it always feels slow. my mac now remembers. the first time i say something new, like "search youtube for lofi beats", a big model figures it out once, does it, and saves the recipe as a json spec: {"op":"act","do":"open","target":"Safari"} {"op":"act","do":"type","text":"{query}"} every time after that, a tiny model on my laptop recognizes the command, fills in the parameters, and replays the recipe with CUA say it once, it's slow. say it again, it's instant. the fine tuned tiny model is 350 million parameters. its only job is recognizing which of my commands i just said. and when the big model does plan, it streams actions line by line. each line runs the moment it's finished, so the first action is executing while the model is still writing the second. everything it learns lives in one plain text file on my disk. i've been playing around a lot with Cua by Francesco and peekaboo by Peter Steinberger 🦞 . curious whether a standard computer use spec makes sense, one that streaming models can drive to make computer use actually fast.

Milind S

14,235 views • 2 months ago

One of our biggest engineering problems is that Conductor is Mac only, which means most of our testing is manual and local. So I asked Fable to: - port Conductor to Linux - open the build inside a Conductor Cloud sandbox - use Codex computer use to QA - send me a screen recording of the whole thing And it worked! The dream is that a user can report a bug, and it gets triaged, fixed, and QA'd in the background. We're getting very close!

One of our biggest engineering problems is that Conductor is Mac only, which means most of our testing is manual and local. So I asked Fable to: - port Conductor to Linux - open the build inside a Conductor Cloud sandbox - use Codex computer use to QA - send me a screen recording of the whole thing And it worked! The dream is that a user can report a bug, and it gets triaged, fixed, and QA'd in the background. We're getting very close!

Charlie Holtz

47,729 views • 2 months ago

Passport Prime gives you 50GB of encrypted storage for your most important files. So how do you get a file from your computer onto the device without exposing everything already there? Normally, plugging in a drive works both ways: the computer can read everything and write anything to it. So can any malware on that computer. Plug in Passport Prime, and the computer sees only one small, separate area: the Airlock. Your 50GB never appears. The computer can't open it, scan it, or tell what's inside. Drop your file into the Airlock folder on your desktop, like you would with any drive. Then unplug. Only now, on Passport Prime's screen, do you move the file from the Airlock into your encrypted storage. The last step happens after the computer is out of the picture. Nothing travels straight from a computer into your vault. Everything passes through an airlock first. That's why it's called Airlock🔐

Passport Prime gives you 50GB of encrypted storage for your most important files. So how do you get a file from your computer onto the device without exposing everything already there? Normally, plugging in a drive works both ways: the computer can read everything and write anything to it. So can any malware on that computer. Plug in Passport Prime, and the computer sees only one small, separate area: the Airlock. Your 50GB never appears. The computer can't open it, scan it, or tell what's inside. Drop your file into the Airlock folder on your desktop, like you would with any drive. Then unplug. Only now, on Passport Prime's screen, do you move the file from the Airlock into your encrypted storage. The last step happens after the computer is out of the picture. Nothing travels straight from a computer into your vault. Everything passes through an airlock first. That's why it's called Airlock🔐

FOUNDATION

43,395 views • 1 month ago

THIS GUY ADDED A LIVE BROWSER TO CLAUDE CODE SO YOU CAN CLICK ANY ELEMENT AND EDIT IT INSTANTLY biggest issue with vibe coding UI is that you have to describe what you want to change. if you prompt it the wrong selector or wrong component, Claude can't find it. now you just click it. your app runs in an embedded browser with Claude Code. you can click any button, any text, any div. Claude instantly knows exactly what you're pointing at. click. instruct. done. no more "change the button in the top right corner of the second card component." all you have to do now is just click the button. AND its open source

THIS GUY ADDED A LIVE BROWSER TO CLAUDE CODE SO YOU CAN CLICK ANY ELEMENT AND EDIT IT INSTANTLY biggest issue with vibe coding UI is that you have to describe what you want to change. if you prompt it the wrong selector or wrong component, Claude can't find it. now you just click it. your app runs in an embedded browser with Claude Code. you can click any button, any text, any div. Claude instantly knows exactly what you're pointing at. click. instruct. done. no more "change the button in the top right corner of the second card component." all you have to do now is just click the button. AND its open source

Om Patel

153,892 views • 5 months ago

Today we're excited to introduce Vy, our AI that sees and acts on your computer. At Vercept, our mission is to reinvent how humans use computers–enabling you to accomplish orders of magnitude more than what you can do today. Vy is a first glimpse at AI that sees and uses your computer just like you do. It runs natively on your computer, with access to all your applications and accounts that you’re already signed into. It is available for download starting today! As part of the release, we’ve built VyUI, an AI model bridging the gap between language and your screen. It sets a new state-of-the-art on UI grounding benchmarks like ScreenSpot v1, ScreenSpot v2, and GroundUI Web–beating leading models from OpenAI, Google, and Anthropic. Check out what Vy can do in the thread below: 🧵👇

Today we're excited to introduce Vy, our AI that sees and acts on your computer. At Vercept, our mission is to reinvent how humans use computers–enabling you to accomplish orders of magnitude more than what you can do today. Vy is a first glimpse at AI that sees and uses your computer just like you do. It runs natively on your computer, with access to all your applications and accounts that you’re already signed into. It is available for download starting today! As part of the release, we’ve built VyUI, an AI model bridging the gap between language and your screen. It sets a new state-of-the-art on UI grounding benchmarks like ScreenSpot v1, ScreenSpot v2, and GroundUI Web–beating leading models from OpenAI, Google, and Anthropic. Check out what Vy can do in the thread below: 🧵👇

Vercept

123,748 views • 1 year ago

WOW! I'm so excited about this. OpenAI Developers said Codex was good at Computer Use, but I wasn't prepared for this. For the last two weeks I've been working on a Computer Use skill to work with Linux. And while I had some success, it was a pretty frustrating experience. That is...until the breakthrough. Using accessibility tools, Codex can now control my entire computer, not just the browser. There are limits to this, of course, but what a time to be alive. This Computer Use skill will unlock and entirely new set of automations, all powered by Codex. Demonstration below 👇

WOW! I'm so excited about this. OpenAI Developers said Codex was good at Computer Use, but I wasn't prepared for this. For the last two weeks I've been working on a Computer Use skill to work with Linux. And while I had some success, it was a pretty frustrating experience. That is...until the breakthrough. Using accessibility tools, Codex can now control my entire computer, not just the browser. There are limits to this, of course, but what a time to be alive. This Computer Use skill will unlock and entirely new set of automations, all powered by Codex. Demonstration below 👇

am.will

79,698 views • 5 months ago

APPLE BUILT A $10,000 COMPUTER THAT RUNS WHAT USED TO REQUIRE AN ENTIRE SERVER ROOM 06:40 he shows the number that breaks everything - one Mac Studio is more powerful than four Dell or AMD machines combined and uses half the energy DeepSeek R1 doesn't run on even two competitors - you need four - one Mac Studio does it alone runs quieter than a fan in the room and uses less energy than a kettle in the kitchen four Mac Studios on a desk connected through a regular cable run a 1 trillion parameter model - what used to require an entire data center anyone can put this at home and Jeff Geerling proved it - run the same models that big companies use without cloud, without subscriptions, without data leaving the room

APPLE BUILT A $10,000 COMPUTER THAT RUNS WHAT USED TO REQUIRE AN ENTIRE SERVER ROOM 06:40 he shows the number that breaks everything - one Mac Studio is more powerful than four Dell or AMD machines combined and uses half the energy DeepSeek R1 doesn't run on even two competitors - you need four - one Mac Studio does it alone runs quieter than a fan in the room and uses less energy than a kettle in the kitchen four Mac Studios on a desk connected through a regular cable run a 1 trillion parameter model - what used to require an entire data center anyone can put this at home and Jeff Geerling proved it - run the same models that big companies use without cloud, without subscriptions, without data leaving the room

Noisy

26,305 views • 2 months ago

Mac Mini is super nice because you can use native iMessage for talking with OpenClaw. The only problem is that it's very expensive and you need to always have your Mac Mini on to run this setup. So I made a plugin to allow anyone on any computer (either Linux, Windows, or Mac) to be able to have access to iMessage natively. Even better, it has access to all of the reactions, triple dot indicator, etc. Check it out and let me know what you think.

Mac Mini is super nice because you can use native iMessage for talking with OpenClaw. The only problem is that it's very expensive and you need to always have your Mac Mini on to run this setup. So I made a plugin to allow anyone on any computer (either Linux, Windows, or Mac) to be able to have access to iMessage natively. Even better, it has access to all of the reactions, triple dot indicator, etc. Check it out and let me know what you think.

Nick Vasilescu

265,845 views • 7 months ago

Nobody seems to know how insane GPT-5.4 is with computer use. I asked GPT-5.4 to draw the OpenAI logo in Microsoft Paint. No computer use API. Just a screenshot and basic tool calls (click, drag, press_key) all coordinate-based. The first drawing was awful. And GPT knew it. It looked at its own result and essentially went "yeah, no." What happened next is what broke my brain: It opened a browser. Went to Bing Images. Searched for the OpenAI logo. Found one. Then (and I cannot stress this enough) it used the Windows area screenshot shortcut (Win+Shift+S) to snip just the logo off the screen. Went back to Paint. Imported it. Centered it. All on its own. No instructions to do any of that. It just improvised a better strategy when the first one failed. My prompt was "Draw the OpenAI logo" with Paint already opened on the computer. Sure, it's "cheating." But honestly? That's exactly what I'd do too. And the fact that it came up with this plan from nothing but a screenshot and a coordinate system is wild.

Clad3815

1,069,968 views • 6 months ago

David Friedberg: Frontier Models are Training on Your Novel Insights as “De-Identified Data” @jason: “Should they trust any of these LLMs with their proprietary knowledge for fear of having it cribbed into a core LLM?” david friedberg: “I have had experiences where we've asked some fairly novel scientific questions, and (the AI model) identifies it as a novel insight. It's like, ‘Oh, never thought about that, interesting, blah, blah, blah.’ And then using a different account, asking the next version (of the model) later, I've now experienced this. It's like, ‘Oh, well, you could do this,’ and it actually just describes this exact thing that we had in our chat in the previous version. Now, these are a handful of anecdotal experiences, but I know the domain that we work in, and the niche of it, and the ideation of this stuff, and the novelty of this stuff, and the lack of papers being published, and so on. So I know that there isn't some new corpus of information out there that's training the new model. So all I can say at that point is that my conversation or our analyses have been used for training.” David Sacks: “Okay, this does raise a really good question. What does it mean that the model is allowed to train on unidentifiable data?” Friedberg: “Well, that's my point. So it doesn't use any of my personal information, but it can use an insight derived from our chat, which it can then say is some training data that is unrelated. But the truth is, it's actually a piece of IP that's our organization’s IP, and our engagement back and forth. We don't have any NDA or confidentiality provisions or protections with them being a service provider back to us. This is why I care a lot about open source because I don't want them having my chat logs because they can use it for training to create an IP advantage that is now diffused to the rest of the market.”

David Friedberg: Frontier Models are Training on Your Novel Insights as “De-Identified Data” @jason: “Should they trust any of these LLMs with their proprietary knowledge for fear of having it cribbed into a core LLM?” david friedberg: “I have had experiences where we've asked some fairly novel scientific questions, and (the AI model) identifies it as a novel insight. It's like, ‘Oh, never thought about that, interesting, blah, blah, blah.’ And then using a different account, asking the next version (of the model) later, I've now experienced this. It's like, ‘Oh, well, you could do this,’ and it actually just describes this exact thing that we had in our chat in the previous version. Now, these are a handful of anecdotal experiences, but I know the domain that we work in, and the niche of it, and the ideation of this stuff, and the novelty of this stuff, and the lack of papers being published, and so on. So I know that there isn't some new corpus of information out there that's training the new model. So all I can say at that point is that my conversation or our analyses have been used for training.” David Sacks: “Okay, this does raise a really good question. What does it mean that the model is allowed to train on unidentifiable data?” Friedberg: “Well, that's my point. So it doesn't use any of my personal information, but it can use an insight derived from our chat, which it can then say is some training data that is unrelated. But the truth is, it's actually a piece of IP that's our organization’s IP, and our engagement back and forth. We don't have any NDA or confidentiality provisions or protections with them being a service provider back to us. This is why I care a lot about open source because I don't want them having my chat logs because they can use it for training to create an IP advantage that is now diffused to the rest of the market.”

The All-In Podcast

54,050 views • 7 days ago

Corporations that are purely AI and robotics will vastly outperform any corporations that have people in the loop You can think of it like how 'computer' used to be a job that humans had. You would go and get a job as a computer where you would do calculations. They had entire skyscrapers full of humans....20, 30 floors of humans...just doing calculations. Now, that entire skyscraper of humans doing calculations can be replaced by a laptop with a spreadsheet. That spreadsheet can do vastly more calculations than an entire building full of human computers So, you think about it: what if only some of the cells in your spreadsheet were calculated by humans? That would be much worse than if all of the cells in your spreadsheet were calculated by the computer. And so, really what will happen is the pure AI, pure robotics corporations or collectives will far outperform any corporations that have humans in the loop. It will happen very quickly

Corporations that are purely AI and robotics will vastly outperform any corporations that have people in the loop You can think of it like how 'computer' used to be a job that humans had. You would go and get a job as a computer where you would do calculations. They had entire skyscrapers full of humans....20, 30 floors of humans...just doing calculations. Now, that entire skyscraper of humans doing calculations can be replaced by a laptop with a spreadsheet. That spreadsheet can do vastly more calculations than an entire building full of human computers So, you think about it: what if only some of the cells in your spreadsheet were calculated by humans? That would be much worse than if all of the cells in your spreadsheet were calculated by the computer. And so, really what will happen is the pure AI, pure robotics corporations or collectives will far outperform any corporations that have humans in the loop. It will happen very quickly

X Freeze

16,850,188 views • 7 months ago

Astra began to saturate a computer-use eval we haven’t even released yet. Astra was not only able to kill several hostile mobs in Minecraft, but it made it all the way to building a Nether portal in less than three hours. We believe that if we ran it longer, it would beat the game. Astra was able to do things like tower up to kill a horde of zombies, trap a skeleton inside a hole to dodge its arrows, craft a boat, and navigate back to its base. Here is the full 3 hour play through. All of this was without any specialized harness: pure computer use.

Astra began to saturate a computer-use eval we haven’t even released yet. Astra was not only able to kill several hostile mobs in Minecraft, but it made it all the way to building a Nether portal in less than three hours. We believe that if we ran it longer, it would beat the game. Astra was able to do things like tower up to kill a horde of zombies, trap a skeleton inside a hole to dodge its arrows, craft a boat, and navigate back to its base. Here is the full 3 hour play through. All of this was without any specialized harness: pure computer use.

Vals AI

31,057 views • 10 days ago