正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Okay so Jev can actually do computer use really well Without any screenshots, or LLMs and no Pixels leave my mac I dont even read the Dom elements A local CoreML model segments every button and UI element on screen. On-device OCR reads the labels. That text is all... show more

Milind S

3,902 subscribers

116,719 次观看 • 1 天前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

62 条评论

Milind S 的头像

Milind S1 天前

If this gets enough attention I’ll make it open source. Meanwhile check out my other stuff

Vedant Bhayani 的头像

Vedant Bhayani1 天前

I think the future is multiple other companies building new kind of llm models for individual tasks with which you just plug it in with frontiers model for steering and thinking Also domain specific harness and custom trained models with your data for specific use cases are going be more relevant and we are going to see even more than before

Milind S 的头像

Milind S1 天前

I agree that’s where things are headed

$/h(elton)?Fabio\.(t|j)s/gi 的头像$

/h(elton)?Fabio\.(t|j)s/gi1 天前

hey man, do you mind doing a test with a local model that uses the same architecture? like this one based on qwen-2.5-1b?

Milind S 的头像

Milind S1 天前

I’ll check it out

Valentin Pletzer 的头像

Valentin Pletzer1 天前

very cool!

Milind S 的头像

Milind S1 天前

Thank you!

Artur Kre 的头像

Artur Kre1 天前

wow insane, which local model does the window UI segmenting ?

Milind S 的头像

Milind S1 天前

Omniparser!

Deep 的头像

Deep1 天前

local segmentation plus OCR instead of a vision model. cuts what you send to the model, and it runs offline too.

Matt Ronge 的头像

Matt Ronge1 天前

Thats very slick. Does it use a larger model to know where to go? So if you ask it something complex to do in Blender for example it can break it down into steps?

Milind S 的头像

Milind S1 天前

There is literally no other larger models being used in this. Its as simple as it gets

Matt Ronge 的头像

Matt Ronge1 天前

Open source when lol

Milind S 的头像

Milind S1 天前

Haha let me clean it up and then I’ll do it promise :D Its based on my other open source project already called tiptour macos

Matt Ronge 的头像

Matt Ronge1 天前

Yea love it, I've been following your work for a while. Thanks for sharing it!

Varik Verilion 的头像

Varik Verilion1 天前

The interesting part is the interface boundary. Segmentation and OCR turn pixels into a compact, structured state before Jev plans an action. That makes local inference practical and keeps the sensitive screen data on-device.

Justin 的头像

Justin1 天前

Bro, you build way too fast :D

Milind S 的头像

Milind S1 天前

Thats how I roll :D

Bruno Skvorc 的头像

Bruno Skvorc1 天前

chat too

bikram 的头像

bikram1 天前

Yes. Share. Also, should go into OMB, will create a really good setup.

Milind S 的头像

Milind S1 天前

Great idea

B⭕bby 的头像

B⭕bby1 天前

Can you add voice to text on top so you can control it with your voice

Milind S 的头像

Milind S1 天前

Totally i can

Omkar Satpute 的头像

Omkar Satpute1 天前

SupaaaaFastttt computer use

ethereumdegen.eth 🕶️ᵍᵐ 的头像

ethereumdegen.eth 🕶️ᵍᵐ1 天前

I would love to use this and test it out

bikram 的头像

bikram1 天前

What are you using for on-device OCR

Milind S 的头像

Milind S1 天前

Apple ships it natively on all devices

Tacdel 的头像

Tacdel1 天前

90ms local loops make 5-second cloud VLM latency look unplayable. How does it handle icon-only buttons where OCR finds zero text?

Carlos Ziegler 的头像

Carlos Ziegler1 天前

This is really nice!!!!

Milind S 的头像

Milind S1 天前

I know right!!

Doomshade 的头像

Doomshade1 天前

Dude this is sick, would really love to try this out with some desktop applications!

Milind S 的头像

Milind S1 天前

Alright man. I’ll send it to you when its ready

Doomshade 的头像

Doomshade1 天前

🙏 much appreciated! btw - love openmaus - got it configured on my machine and made a few tweaks to my system - need to pull some of the latest features you've shipped !

Milind S 的头像

Milind S1 天前

Lfgg

Sahibzada Allahyar 的头像

Sahibzada Allahyar1 天前

this one can do it too and it's local

Joey van Koningsbruggen 的头像

Joey van Koningsbruggen1 天前

Very cool

up 的头像

up1 天前

Loved it, crazy capabilities

Milind S 的头像

Milind S1 天前

Indeed

shaik arbaz 的头像

shaik arbaz1 天前

Make it open source.

Milind S 的头像

Milind S1 天前

Will do

Arpit 的头像

Arpit1 天前

How can it be used to do computer use in bg?

Milind S 的头像

Milind S1 天前

I’m sure @trycua can

Hugo Catarino 的头像

Hugo Catarino1 天前

Que brutalidade!!! 🤯

Milind S 的头像

Milind S1 天前

Sí!!!

Christian Giangrande 的头像

Christian Giangrande1 天前

Impliment into maus 🙏🙏🙏

Milind S 的头像

Milind S1 天前

Hell yeah

Konrad 的头像

Konrad1 天前

how does the LM come up with more complex interactions, like in Blender, dragging while holding down buttons?

Milind S 的头像

Milind S1 天前

Gotta pair this with an llm to do that

Konrad 的头像

Konrad1 天前

Message me the details on WhatsApp, I have new investors for the stuff, doing a seed round the coming weeks. Getting the production cost from ~48ct to ~20ct could be realistic with this new tooling.

Mar 的头像

Mar1 天前

Dude, I need this!

Milind S 的头像

Milind S1 天前

Releasing soon

Nikhil Gangaraju 的头像

Nikhil Gangaraju1 天前

+1 for the open source version. Great demo!

Milind S 的头像

Milind S1 天前

Thanks! Repo coming shortly

Shahn 的头像

Shahn1 天前

this is really cool @milindlabs I’ve been trying out Jev for some browser use features as well, here you runs loop with ocr model right ?

Milind S 的头像

Milind S1 天前

I do yes along with the segmentation model

Guilherme 的头像

Guilherme1 天前

Does it support multi-screen ?

Milind S 的头像

Milind S1 天前

It does

D Row Kavi 的头像

D Row Kavi1 天前

@bot @poteto 😅🤘🏽

AskMeHow 的头像

AskMeHow1 天前

open source pls

Milind S 的头像

Milind S1 天前

If you say so

Aditya Sinha 的头像

Aditya Sinha1 天前

@iamgingertrash how did you know

Dominik 的头像

Dominik1 天前

a click loop without an llm can still own the machine. i am building refuse rules on the write surface so speed never becomes silent permission.

相关视频

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

Milind S

38,361 次观看 • 3 个月前

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

MIKE

14,387 次观看 • 6 天前

I took Clicky and I made it 5x Faster So I saw that Farza 🇵🇰🇺🇸 was using uses Claude's vision to find UI elements on screen, send a screenshot, wait for coordinates back. It works, but it's slow. I replaced that with OmniParser V2 by Microsoft which is a local Object detection model trained specifically on UI elements. It runs on-device, detects every button, menu, and icon in 400ms, and gives me pixel-precise coordinates. No API call, no latency, no cost. The green highlights you see around the UI elements is the detection overlay and you can see as I am switching the screen it takes no time to detect and highlight which is pretty neat! With Local models improving by the day, Its the right direction for applications like these. Next up: video-synced tutorials where a YouTube tutorial pauses and waits for you to perform each action in the real app. I am not stopping!

I took Clicky and I made it 5x Faster So I saw that Farza 🇵🇰🇺🇸 was using uses Claude's vision to find UI elements on screen, send a screenshot, wait for coordinates back. It works, but it's slow. I replaced that with OmniParser V2 by Microsoft which is a local Object detection model trained specifically on UI elements. It runs on-device, detects every button, menu, and icon in 400ms, and gives me pixel-precise coordinates. No API call, no latency, no cost. The green highlights you see around the UI elements is the detection overlay and you can see as I am switching the screen it takes no time to detect and highlight which is pretty neat! With Local models improving by the day, Its the right direction for applications like these. Next up: video-synced tutorials where a YouTube tutorial pauses and waits for you to perform each action in the real app. I am not stopping!

Milind S

73,349 次观看 • 5 个月前

Our first short course with Anthropic! Building Towards Computer Use with Anthropic. This teaches you to build an LLM-based agent that uses a computer interface by generating mouse clicks and keystrokes. Computer Use is an important, emerging capability for LLMs that will let AI agents do many more tasks than were possible before, since it lets them interact with interfaces designed for humans to use, rather than only tools that provide explicit API access. I hope you will enjoy learning about it! This course is taught by Anthropic's Head of Curriculum, Colt_Steele. You'll learn to apply image reasoning and tool use to "use" a computer as follows: a model processes an image of the screen, analyzes it to understand what's going on, and navigates the computer via mouse clicks and keystrokes. This course goes through the key building blocks, and culminates in a demo of an AI assistant that uses a web browser to search for a research paper, downloads the PDF, and finally summarizes the paper for you. In detail, you’ll: - Learn about Anthropic's family of models, when to use which one, and make API requests to Claude - Use multi-modal prompts that combine text and image content blocks, and also work with streaming responses - Improve your prompting by using prompt templates, using XML to structure prompts, and providing examples - Implement prompt caching to reduce cost and latency - Apply tool-use to build a chatbot that can call different tools to respond to queries - See all these building blocks come together in Computer Use demo Please sign up here:

Our first short course with Anthropic! Building Towards Computer Use with Anthropic. This teaches you to build an LLM-based agent that uses a computer interface by generating mouse clicks and keystrokes. Computer Use is an important, emerging capability for LLMs that will let AI agents do many more tasks than were possible before, since it lets them interact with interfaces designed for humans to use, rather than only tools that provide explicit API access. I hope you will enjoy learning about it! This course is taught by Anthropic's Head of Curriculum, Colt_Steele. You'll learn to apply image reasoning and tool use to "use" a computer as follows: a model processes an image of the screen, analyzes it to understand what's going on, and navigates the computer via mouse clicks and keystrokes. This course goes through the key building blocks, and culminates in a demo of an AI assistant that uses a web browser to search for a research paper, downloads the PDF, and finally summarizes the paper for you. In detail, you’ll: - Learn about Anthropic's family of models, when to use which one, and make API requests to Claude - Use multi-modal prompts that combine text and image content blocks, and also work with streaming responses - Improve your prompting by using prompt templates, using XML to structure prompts, and providing examples - Implement prompt caching to reduce cost and latency - Apply tool-use to build a chatbot that can call different tools to respond to queries - See all these building blocks come together in Computer Use demo Please sign up here:

Andrew Ng

170,541 次观看 • 1 年前

Jev is a new model from the co-inventor of ChatGPT that can use computer tools in realtime right now it's playing Minecraft on its own computer in the cloud on Orgo this is what it looks like with just one Jev, but imagine: what about a thousand? or what about a billion of Jevs? each working together on their own computers in the cloud, discovering new science, running experiments, and designing new drugs 24/7. this is going to be bigger than ChatGPT

Jev is a new model from the co-inventor of ChatGPT that can use computer tools in realtime right now it's playing Minecraft on its own computer in the cloud on Orgo this is what it looks like with just one Jev, but imagine: what about a thousand? or what about a billion of Jevs? each working together on their own computers in the cloud, discovering new science, running experiments, and designing new drugs 24/7. this is going to be bigger than ChatGPT

Nick Vasilescu

14,120 次观看 • 1 天前

This is actually cool - I tried the same prompt for the new Interactive Playwright skill in Codex & GPT-5.4 xHigh - the one above is with the skill and the one below is without. What the skill does is uses the computer use capability of GPT-5.4 to look and navigate the UI. This never worked for me before, but with GPT-5.4 this is the first time I can actually see a massive difference. You can see how the first scene is much more coherent, higher fidelity and complete. The one below is missing a lot of elements and isn't as rich in detail. I'll keep using it for any UI work now.

This is actually cool - I tried the same prompt for the new Interactive Playwright skill in Codex & GPT-5.4 xHigh - the one above is with the skill and the one below is without. What the skill does is uses the computer use capability of GPT-5.4 to look and navigate the UI. This never worked for me before, but with GPT-5.4 this is the first time I can actually see a massive difference. You can see how the first scene is much more coherent, higher fidelity and complete. The one below is missing a lot of elements and isn't as rich in detail. I'll keep using it for any UI work now.

Peter Gostev (SF 24-28 August)

257,389 次观看 • 6 个月前

computer use agents are slow because they think too much they make round trips to an LLM. every single time. even for a command you've said a hundred times before. that's why it always feels slow. my mac now remembers. the first time i say something new, like "search youtube for lofi beats", a big model figures it out once, does it, and saves the recipe as a json spec: {"op":"act","do":"open","target":"Safari"} {"op":"act","do":"type","text":"{query}"} every time after that, a tiny model on my laptop recognizes the command, fills in the parameters, and replays the recipe with CUA say it once, it's slow. say it again, it's instant. the fine tuned tiny model is 350 million parameters. its only job is recognizing which of my commands i just said. and when the big model does plan, it streams actions line by line. each line runs the moment it's finished, so the first action is executing while the model is still writing the second. everything it learns lives in one plain text file on my disk. i've been playing around a lot with Cua by Francesco and peekaboo by Peter Steinberger 🦞 . curious whether a standard computer use spec makes sense, one that streaming models can drive to make computer use actually fast.

computer use agents are slow because they think too much they make round trips to an LLM. every single time. even for a command you've said a hundred times before. that's why it always feels slow. my mac now remembers. the first time i say something new, like "search youtube for lofi beats", a big model figures it out once, does it, and saves the recipe as a json spec: {"op":"act","do":"open","target":"Safari"} {"op":"act","do":"type","text":"{query}"} every time after that, a tiny model on my laptop recognizes the command, fills in the parameters, and replays the recipe with CUA say it once, it's slow. say it again, it's instant. the fine tuned tiny model is 350 million parameters. its only job is recognizing which of my commands i just said. and when the big model does plan, it streams actions line by line. each line runs the moment it's finished, so the first action is executing while the model is still writing the second. everything it learns lives in one plain text file on my disk. i've been playing around a lot with Cua by Francesco and peekaboo by Peter Steinberger 🦞 . curious whether a standard computer use spec makes sense, one that streaming models can drive to make computer use actually fast.

Milind S

14,235 次观看 • 2 个月前

One of our biggest engineering problems is that Conductor is Mac only, which means most of our testing is manual and local. So I asked Fable to: - port Conductor to Linux - open the build inside a Conductor Cloud sandbox - use Codex computer use to QA - send me a screen recording of the whole thing And it worked! The dream is that a user can report a bug, and it gets triaged, fixed, and QA'd in the background. We're getting very close!

One of our biggest engineering problems is that Conductor is Mac only, which means most of our testing is manual and local. So I asked Fable to: - port Conductor to Linux - open the build inside a Conductor Cloud sandbox - use Codex computer use to QA - send me a screen recording of the whole thing And it worked! The dream is that a user can report a bug, and it gets triaged, fixed, and QA'd in the background. We're getting very close!

Charlie Holtz

47,729 次观看 • 2 个月前

Passport Prime gives you 50GB of encrypted storage for your most important files. So how do you get a file from your computer onto the device without exposing everything already there? Normally, plugging in a drive works both ways: the computer can read everything and write anything to it. So can any malware on that computer. Plug in Passport Prime, and the computer sees only one small, separate area: the Airlock. Your 50GB never appears. The computer can't open it, scan it, or tell what's inside. Drop your file into the Airlock folder on your desktop, like you would with any drive. Then unplug. Only now, on Passport Prime's screen, do you move the file from the Airlock into your encrypted storage. The last step happens after the computer is out of the picture. Nothing travels straight from a computer into your vault. Everything passes through an airlock first. That's why it's called Airlock🔐

Passport Prime gives you 50GB of encrypted storage for your most important files. So how do you get a file from your computer onto the device without exposing everything already there? Normally, plugging in a drive works both ways: the computer can read everything and write anything to it. So can any malware on that computer. Plug in Passport Prime, and the computer sees only one small, separate area: the Airlock. Your 50GB never appears. The computer can't open it, scan it, or tell what's inside. Drop your file into the Airlock folder on your desktop, like you would with any drive. Then unplug. Only now, on Passport Prime's screen, do you move the file from the Airlock into your encrypted storage. The last step happens after the computer is out of the picture. Nothing travels straight from a computer into your vault. Everything passes through an airlock first. That's why it's called Airlock🔐

FOUNDATION

43,395 次观看 • 1 个月前

THIS GUY ADDED A LIVE BROWSER TO CLAUDE CODE SO YOU CAN CLICK ANY ELEMENT AND EDIT IT INSTANTLY biggest issue with vibe coding UI is that you have to describe what you want to change. if you prompt it the wrong selector or wrong component, Claude can't find it. now you just click it. your app runs in an embedded browser with Claude Code. you can click any button, any text, any div. Claude instantly knows exactly what you're pointing at. click. instruct. done. no more "change the button in the top right corner of the second card component." all you have to do now is just click the button. AND its open source

THIS GUY ADDED A LIVE BROWSER TO CLAUDE CODE SO YOU CAN CLICK ANY ELEMENT AND EDIT IT INSTANTLY biggest issue with vibe coding UI is that you have to describe what you want to change. if you prompt it the wrong selector or wrong component, Claude can't find it. now you just click it. your app runs in an embedded browser with Claude Code. you can click any button, any text, any div. Claude instantly knows exactly what you're pointing at. click. instruct. done. no more "change the button in the top right corner of the second card component." all you have to do now is just click the button. AND its open source

Om Patel

153,892 次观看 • 5 个月前

Today we're excited to introduce Vy, our AI that sees and acts on your computer. At Vercept, our mission is to reinvent how humans use computers–enabling you to accomplish orders of magnitude more than what you can do today. Vy is a first glimpse at AI that sees and uses your computer just like you do. It runs natively on your computer, with access to all your applications and accounts that you’re already signed into. It is available for download starting today! As part of the release, we’ve built VyUI, an AI model bridging the gap between language and your screen. It sets a new state-of-the-art on UI grounding benchmarks like ScreenSpot v1, ScreenSpot v2, and GroundUI Web–beating leading models from OpenAI, Google, and Anthropic. Check out what Vy can do in the thread below: 🧵👇

Today we're excited to introduce Vy, our AI that sees and acts on your computer. At Vercept, our mission is to reinvent how humans use computers–enabling you to accomplish orders of magnitude more than what you can do today. Vy is a first glimpse at AI that sees and uses your computer just like you do. It runs natively on your computer, with access to all your applications and accounts that you’re already signed into. It is available for download starting today! As part of the release, we’ve built VyUI, an AI model bridging the gap between language and your screen. It sets a new state-of-the-art on UI grounding benchmarks like ScreenSpot v1, ScreenSpot v2, and GroundUI Web–beating leading models from OpenAI, Google, and Anthropic. Check out what Vy can do in the thread below: 🧵👇

Vercept

123,748 次观看 • 1 年前

WOW! I'm so excited about this. OpenAI Developers said Codex was good at Computer Use, but I wasn't prepared for this. For the last two weeks I've been working on a Computer Use skill to work with Linux. And while I had some success, it was a pretty frustrating experience. That is...until the breakthrough. Using accessibility tools, Codex can now control my entire computer, not just the browser. There are limits to this, of course, but what a time to be alive. This Computer Use skill will unlock and entirely new set of automations, all powered by Codex. Demonstration below 👇

WOW! I'm so excited about this. OpenAI Developers said Codex was good at Computer Use, but I wasn't prepared for this. For the last two weeks I've been working on a Computer Use skill to work with Linux. And while I had some success, it was a pretty frustrating experience. That is...until the breakthrough. Using accessibility tools, Codex can now control my entire computer, not just the browser. There are limits to this, of course, but what a time to be alive. This Computer Use skill will unlock and entirely new set of automations, all powered by Codex. Demonstration below 👇

am.will

79,698 次观看 • 5 个月前

APPLE BUILT A $10,000 COMPUTER THAT RUNS WHAT USED TO REQUIRE AN ENTIRE SERVER ROOM 06:40 he shows the number that breaks everything - one Mac Studio is more powerful than four Dell or AMD machines combined and uses half the energy DeepSeek R1 doesn't run on even two competitors - you need four - one Mac Studio does it alone runs quieter than a fan in the room and uses less energy than a kettle in the kitchen four Mac Studios on a desk connected through a regular cable run a 1 trillion parameter model - what used to require an entire data center anyone can put this at home and Jeff Geerling proved it - run the same models that big companies use without cloud, without subscriptions, without data leaving the room

APPLE BUILT A $10,000 COMPUTER THAT RUNS WHAT USED TO REQUIRE AN ENTIRE SERVER ROOM 06:40 he shows the number that breaks everything - one Mac Studio is more powerful than four Dell or AMD machines combined and uses half the energy DeepSeek R1 doesn't run on even two competitors - you need four - one Mac Studio does it alone runs quieter than a fan in the room and uses less energy than a kettle in the kitchen four Mac Studios on a desk connected through a regular cable run a 1 trillion parameter model - what used to require an entire data center anyone can put this at home and Jeff Geerling proved it - run the same models that big companies use without cloud, without subscriptions, without data leaving the room

Noisy

26,305 次观看 • 2 个月前

Mac Mini is super nice because you can use native iMessage for talking with OpenClaw. The only problem is that it's very expensive and you need to always have your Mac Mini on to run this setup. So I made a plugin to allow anyone on any computer (either Linux, Windows, or Mac) to be able to have access to iMessage natively. Even better, it has access to all of the reactions, triple dot indicator, etc. Check it out and let me know what you think.

Mac Mini is super nice because you can use native iMessage for talking with OpenClaw. The only problem is that it's very expensive and you need to always have your Mac Mini on to run this setup. So I made a plugin to allow anyone on any computer (either Linux, Windows, or Mac) to be able to have access to iMessage natively. Even better, it has access to all of the reactions, triple dot indicator, etc. Check it out and let me know what you think.

Nick Vasilescu

265,845 次观看 • 7 个月前

Nobody seems to know how insane GPT-5.4 is with computer use. I asked GPT-5.4 to draw the OpenAI logo in Microsoft Paint. No computer use API. Just a screenshot and basic tool calls (click, drag, press_key) all coordinate-based. The first drawing was awful. And GPT knew it. It looked at its own result and essentially went "yeah, no." What happened next is what broke my brain: It opened a browser. Went to Bing Images. Searched for the OpenAI logo. Found one. Then (and I cannot stress this enough) it used the Windows area screenshot shortcut (Win+Shift+S) to snip just the logo off the screen. Went back to Paint. Imported it. Centered it. All on its own. No instructions to do any of that. It just improvised a better strategy when the first one failed. My prompt was "Draw the OpenAI logo" with Paint already opened on the computer. Sure, it's "cheating." But honestly? That's exactly what I'd do too. And the fact that it came up with this plan from nothing but a screenshot and a coordinate system is wild.

Clad3815

1,069,968 次观看 • 6 个月前

David Friedberg: Frontier Models are Training on Your Novel Insights as “De-Identified Data” @jason: “Should they trust any of these LLMs with their proprietary knowledge for fear of having it cribbed into a core LLM?” david friedberg: “I have had experiences where we've asked some fairly novel scientific questions, and (the AI model) identifies it as a novel insight. It's like, ‘Oh, never thought about that, interesting, blah, blah, blah.’ And then using a different account, asking the next version (of the model) later, I've now experienced this. It's like, ‘Oh, well, you could do this,’ and it actually just describes this exact thing that we had in our chat in the previous version. Now, these are a handful of anecdotal experiences, but I know the domain that we work in, and the niche of it, and the ideation of this stuff, and the novelty of this stuff, and the lack of papers being published, and so on. So I know that there isn't some new corpus of information out there that's training the new model. So all I can say at that point is that my conversation or our analyses have been used for training.” David Sacks: “Okay, this does raise a really good question. What does it mean that the model is allowed to train on unidentifiable data?” Friedberg: “Well, that's my point. So it doesn't use any of my personal information, but it can use an insight derived from our chat, which it can then say is some training data that is unrelated. But the truth is, it's actually a piece of IP that's our organization’s IP, and our engagement back and forth. We don't have any NDA or confidentiality provisions or protections with them being a service provider back to us. This is why I care a lot about open source because I don't want them having my chat logs because they can use it for training to create an IP advantage that is now diffused to the rest of the market.”

David Friedberg: Frontier Models are Training on Your Novel Insights as “De-Identified Data” @jason: “Should they trust any of these LLMs with their proprietary knowledge for fear of having it cribbed into a core LLM?” david friedberg: “I have had experiences where we've asked some fairly novel scientific questions, and (the AI model) identifies it as a novel insight. It's like, ‘Oh, never thought about that, interesting, blah, blah, blah.’ And then using a different account, asking the next version (of the model) later, I've now experienced this. It's like, ‘Oh, well, you could do this,’ and it actually just describes this exact thing that we had in our chat in the previous version. Now, these are a handful of anecdotal experiences, but I know the domain that we work in, and the niche of it, and the ideation of this stuff, and the novelty of this stuff, and the lack of papers being published, and so on. So I know that there isn't some new corpus of information out there that's training the new model. So all I can say at that point is that my conversation or our analyses have been used for training.” David Sacks: “Okay, this does raise a really good question. What does it mean that the model is allowed to train on unidentifiable data?” Friedberg: “Well, that's my point. So it doesn't use any of my personal information, but it can use an insight derived from our chat, which it can then say is some training data that is unrelated. But the truth is, it's actually a piece of IP that's our organization’s IP, and our engagement back and forth. We don't have any NDA or confidentiality provisions or protections with them being a service provider back to us. This is why I care a lot about open source because I don't want them having my chat logs because they can use it for training to create an IP advantage that is now diffused to the rest of the market.”

The All-In Podcast

54,050 次观看 • 7 天前

Corporations that are purely AI and robotics will vastly outperform any corporations that have people in the loop You can think of it like how 'computer' used to be a job that humans had. You would go and get a job as a computer where you would do calculations. They had entire skyscrapers full of humans....20, 30 floors of humans...just doing calculations. Now, that entire skyscraper of humans doing calculations can be replaced by a laptop with a spreadsheet. That spreadsheet can do vastly more calculations than an entire building full of human computers So, you think about it: what if only some of the cells in your spreadsheet were calculated by humans? That would be much worse than if all of the cells in your spreadsheet were calculated by the computer. And so, really what will happen is the pure AI, pure robotics corporations or collectives will far outperform any corporations that have humans in the loop. It will happen very quickly

Corporations that are purely AI and robotics will vastly outperform any corporations that have people in the loop You can think of it like how 'computer' used to be a job that humans had. You would go and get a job as a computer where you would do calculations. They had entire skyscrapers full of humans....20, 30 floors of humans...just doing calculations. Now, that entire skyscraper of humans doing calculations can be replaced by a laptop with a spreadsheet. That spreadsheet can do vastly more calculations than an entire building full of human computers So, you think about it: what if only some of the cells in your spreadsheet were calculated by humans? That would be much worse than if all of the cells in your spreadsheet were calculated by the computer. And so, really what will happen is the pure AI, pure robotics corporations or collectives will far outperform any corporations that have humans in the loop. It will happen very quickly

X Freeze

16,850,188 次观看 • 7 个月前

Astra began to saturate a computer-use eval we haven’t even released yet. Astra was not only able to kill several hostile mobs in Minecraft, but it made it all the way to building a Nether portal in less than three hours. We believe that if we ran it longer, it would beat the game. Astra was able to do things like tower up to kill a horde of zombies, trap a skeleton inside a hole to dodge its arrows, craft a boat, and navigate back to its base. Here is the full 3 hour play through. All of this was without any specialized harness: pure computer use.

Astra began to saturate a computer-use eval we haven’t even released yet. Astra was not only able to kill several hostile mobs in Minecraft, but it made it all the way to building a Nether portal in less than three hours. We believe that if we ran it longer, it would beat the game. Astra was able to do things like tower up to kill a horde of zombies, trap a skeleton inside a hole to dodge its arrows, craft a boat, and navigate back to its base. Here is the full 3 hour play through. All of this was without any specialized harness: pure computer use.

Vals AI

31,057 次观看 • 10 天前