Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Claim: Computer-Use agents are embodied generalists Back in August, we used a single computer-use agent across 4 different embodied AI-related competitions at ECCV, and we ranked 1st, 2nd, 3rd, and 6th places. How? We constructed simple GUI software for each competition that connects to the benchmark tasks (e.g., using... a keyboard to navigate inside a 3D house rendered in a web interface), and that's it. The agent itself stays the same. I have been meaning to write about it for a long time, but here it is now: Now to the position: we believe that computer-use agents are embodied generalists. Why? TLDR: We believe computer-use agents share striking similarities with models for embodied AI, and embodied tasks can be formulated as interactions with digital software. Once we have that, we can learn from what worked for computer use over the past 2 years, focus on digital scaling, and avoid mistakes or local minima in the grand scheme of things. I would urge anyone working in computer use or embodied AI to take a quick read of the paper, or at least go through the website. We do not think many people would agree now, and only time will tell. But we do think this is an important line of thought that everyone should consider. We look forward to feedback and constructive criticism!show more

Pranjal Aggarwal

1,236 subscribers

11,392 views • 4 days ago •via X (Twitter)

Science & Technology Education

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

🎮 Computer Use Agent Arena is LIVE! 🚀 🔥 Easiest way to test computer-use agents in the wild without any setup 🌟 Compare top VLMs: OpenAI Operator, Claude 3.7, Gemini 2.5 Pro, Qwen 2.5 vl and more 🕹️ Test agents on 100+ real apps & webs with one-click config 🔒 Safe & free access on cloud-hosted machines Page: Leaderboard (tentative): Blog: Data & Code (coming soon): ⭐️Why Computer Agent Arena? 1️⃣Beyond Static Benchmarks: We use computers to perform enormous tasks and workflows every day, and AI agents have the potential to automate these tasks. However, existing benchmarks are very limited (e.g., only 369 tasks in OSWorld and 812 tasks in WebArena). To better measure their capabilities, we introduce Computer Agent Arena for users to easily compare & test AI agents on all kinds of crowdsourced real-world computer use tasks. 2️⃣Cloud Testing, Simplified: As agents like OpenAI’s Operator and Claude 3.7 sonnet release, users face configuration challenges and privacy hurdles to deploy on their own computers. Our platform integrates these agents with cloud-hosted machines, providing users with quick and secure access. 3️⃣Unified Embodied Digital Environment: Unlike Chatbot Arena, we provide users with a real embodied environment—computers—where all agents are grounded in real computer tasks and environments. Led by @XLANG_Lab [1/🧵]

🎮 Computer Use Agent Arena is LIVE! 🚀 🔥 Easiest way to test computer-use agents in the wild without any setup 🌟 Compare top VLMs: OpenAI Operator, Claude 3.7, Gemini 2.5 Pro, Qwen 2.5 vl and more 🕹️ Test agents on 100+ real apps & webs with one-click config 🔒 Safe & free access on cloud-hosted machines Page: Leaderboard (tentative): Blog: Data & Code (coming soon): ⭐️Why Computer Agent Arena? 1️⃣Beyond Static Benchmarks: We use computers to perform enormous tasks and workflows every day, and AI agents have the potential to automate these tasks. However, existing benchmarks are very limited (e.g., only 369 tasks in OSWorld and 812 tasks in WebArena). To better measure their capabilities, we introduce Computer Agent Arena for users to easily compare & test AI agents on all kinds of crowdsourced real-world computer use tasks. 2️⃣Cloud Testing, Simplified: As agents like OpenAI’s Operator and Claude 3.7 sonnet release, users face configuration challenges and privacy hurdles to deploy on their own computers. Our platform integrates these agents with cloud-hosted machines, providing users with quick and secure access. 3️⃣Unified Embodied Digital Environment: Unlike Chatbot Arena, we provide users with a real embodied environment—computers—where all agents are grounded in real computer tasks and environments. Led by @XLANG_Lab [1/🧵]

Bowen Wang

93,360 views • 1 year ago

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

MIKE

14,387 views • 1 month ago

Worktrace AI and computer use complete each other. 🧡 We recently made a big update to Worktrace AI . It not only FINDS repetitive work in your company, but it now AUTOMATES it for you too. Advancements in computer use have unlocked a wave of personal agents. It will unlock enterprise agents too. The missing piece in enterprise is understanding how work happens. Worktrace records your screen and maps out every workflow to surface top automation opportunities. It takes what it has observed, often hundreds of times over, and turns it into an agent that uses the computer exactly the way your team does. We'd love to demo it to you!

Worktrace AI and computer use complete each other. 🧡 We recently made a big update to Worktrace AI . It not only FINDS repetitive work in your company, but it now AUTOMATES it for you too. Advancements in computer use have unlocked a wave of personal agents. It will unlock enterprise agents too. The missing piece in enterprise is understanding how work happens. Worktrace records your screen and maps out every workflow to surface top automation opportunities. It takes what it has observed, often hundreds of times over, and turns it into an agent that uses the computer exactly the way your team does. We'd love to demo it to you!

Angela Jiang

18,638 views • 1 month ago

Google just unveiled a new project that shows how AI Agents are coming for full browser and computer use. This opens up AI to any of the laborious knowledge tasks that we do today. Most importantly, it lets us automate work that we never had even contemplated before.

Google just unveiled a new project that shows how AI Agents are coming for full browser and computer use. This opens up AI to any of the laborious knowledge tasks that we do today. Most importantly, it lets us automate work that we never had even contemplated before.

Aaron Levie

145,464 views • 1 year ago

AI power users are here and the case for agent-friendly software is materializing. Aaron Levie on agents outnumbering people 1000x: "If you start to imagine that we all have to build software for agents… we spend as much time now thinking about the agent interface to our tool as we do the human interface." "If you have a hundred or a thousand times more agents than people, then your software has to be built for agents… it's going to be through an API or a CLI or MCP." "What if you give a coding agent access to your SaaS tools and a coding agent access to your knowledge work, workflows and context?" "It can actually code its way or use APIs through whatever task it's trying to achieve." "That appears to be like a paradigm that is starting to compound… and I actually think it kind of makes sense as the ultimate manifestation of this stuff." Aaron Levie

AI power users are here and the case for agent-friendly software is materializing. Aaron Levie on agents outnumbering people 1000x: "If you start to imagine that we all have to build software for agents… we spend as much time now thinking about the agent interface to our tool as we do the human interface." "If you have a hundred or a thousand times more agents than people, then your software has to be built for agents… it's going to be through an API or a CLI or MCP." "What if you give a coding agent access to your SaaS tools and a coding agent access to your knowledge work, workflows and context?" "It can actually code its way or use APIs through whatever task it's trying to achieve." "That appears to be like a paradigm that is starting to compound… and I actually think it kind of makes sense as the ultimate manifestation of this stuff." Aaron Levie

a16z

42,152 views • 2 months ago

Standard Intelligence's devansh responds to roon's tweet that "text is the universal interface," and explains why their new foundation model is trained on video: "At some point in the arbitrarily long future, if we only use text models, we could force most things to be text. But I think there are just a lot of things that are much more native when done from a computer-use [perspective]." "GUIs are designed for humans to use. We have this massive long tail of things on the internet that are entirely undoable by LLMs." "For example, when I do ML engineering most of my time is spent doing the grunt work of engineering. It's a lot of looking at graphs, analyzing, and comparing loss curves. You can do this in text, but it's a much larger pain than doing it in the native interface." "There's a reason humans don't interact with a computer purely through text, it would kind of suck."

Standard Intelligence's devansh responds to roon's tweet that "text is the universal interface," and explains why their new foundation model is trained on video: "At some point in the arbitrarily long future, if we only use text models, we could force most things to be text. But I think there are just a lot of things that are much more native when done from a computer-use [perspective]." "GUIs are designed for humans to use. We have this massive long tail of things on the internet that are entirely undoable by LLMs." "For example, when I do ML engineering most of my time is spent doing the grunt work of engineering. It's a lot of looking at graphs, analyzing, and comparing loss curves. You can do this in text, but it's a much larger pain than doing it in the native interface." "There's a reason humans don't interact with a computer purely through text, it would kind of suck."

TBPN

64,699 views • 8 months ago

autoMate Like Manus, Computer Use Agent(CUA) and Omniparser, we are computer-using agents. AI-driven local automation assistant that uses natural language to make computers work by themselves

autoMate Like Manus, Computer Use Agent(CUA) and Omniparser, we are computer-using agents. AI-driven local automation assistant that uses natural language to make computers work by themselves

AK

14,288 views • 1 year ago

Sam is absolutely right. The moment AI agents get full access to a computer and the web, we cross a threshold from “tools we use” to “systems that act.” This is not just another upgrade. It is the beginning of autonomous productivity at scale. Research, execution, iteration, all happening in parallel while we focus on higher level thinking. The interface era is ending. The agent era is beginning.

Sam is absolutely right. The moment AI agents get full access to a computer and the web, we cross a threshold from “tools we use” to “systems that act.” This is not just another upgrade. It is the beginning of autonomous productivity at scale. Research, execution, iteration, all happening in parallel while we focus on higher level thinking. The interface era is ending. The agent era is beginning.

VraserX e/acc

11,102 views • 9 months ago

Jensen Huang just said this is the greatest era in history to build software. AI agents will not kill software. They will do the exact opposite: create a massive new wave of software demand. at NVIDIA GTC Taipei 2026 "Click and type. We now replace that with explaining to the AI what we want, our intent, and the AI generates the code or uses tools to produce the necessary output. This is how computers are going to work in the future. This is Agentic AI. For two years, we've been building toward this, and now it has arrived. One of the big breakthroughs, of course, is tool use. A lot of people have said, “Jensen, AI is coming. Agentic AI is coming. Therefore, all the software companies are going to go out of business.” This is exactly the opposite. Because there are going to be so many agents, the world is no longer limited by the number of people. Therefore, those agents are going to use more tools than ever. This is actually an incredible time to be a software company. But the software has to be presented to the agent in a way that the agent can use it. This is a big breakthrough. And in fact, what we have done, as you know, what Nvidia’s treasure..." ---- From 'NVIDIA' YT channel (link in comment)

Jensen Huang just said this is the greatest era in history to build software. AI agents will not kill software. They will do the exact opposite: create a massive new wave of software demand. at NVIDIA GTC Taipei 2026 "Click and type. We now replace that with explaining to the AI what we want, our intent, and the AI generates the code or uses tools to produce the necessary output. This is how computers are going to work in the future. This is Agentic AI. For two years, we've been building toward this, and now it has arrived. One of the big breakthroughs, of course, is tool use. A lot of people have said, “Jensen, AI is coming. Agentic AI is coming. Therefore, all the software companies are going to go out of business.” This is exactly the opposite. Because there are going to be so many agents, the world is no longer limited by the number of people. Therefore, those agents are going to use more tools than ever. This is actually an incredible time to be a software company. But the software has to be presented to the agent in a way that the agent can use it. This is a big breakthrough. And in fact, what we have done, as you know, what Nvidia’s treasure..." ---- From 'NVIDIA' YT channel (link in comment)

Rohan Paul

16,584 views • 5 months ago

We spent decades putting business processes into software. But what connects those systems is still a person with 20 browser tabs open, copying and pasting. A human API. My request for startups: computer use agents that actually finish the job. Agents that companies can trust to do real work at scale, and that work in the software companies already use. If you're building this, we want to hear from you! More below. cc a16z speedrun

We spent decades putting business processes into software. But what connects those systems is still a person with 20 browser tabs open, copying and pasting. A human API. My request for startups: computer use agents that actually finish the job. Agents that companies can trust to do real work at scale, and that work in the software companies already use. If you're building this, we want to hear from you! More below. cc a16z speedrun

Fabrizio Serafini

38,597 views • 26 days ago

Today Google DeepMind released a state of the art computer use model, in partnership with Browserbase. Computer use is hard to evaluate. You need reliable browser infrastructure and realistic tasks. Below, we cover how we ran these benchmarks and how you can try yourself!

Today Google DeepMind released a state of the art computer use model, in partnership with Browserbase. Computer use is hard to evaluate. You need reliable browser infrastructure and realistic tasks. Below, we cover how we ran these benchmarks and how you can try yourself!

Browserbase

155,224 views • 1 year ago

OpenAI just recreated Samantha from the movie "Her" This means we can finally have agents that do computer work as we talk to them in real time. Imagine the world when your agent creates thousands of versions of itself, each with its own Orgo computer... That's the future, and we're living it today.

OpenAI just recreated Samantha from the movie "Her" This means we can finally have agents that do computer work as we talk to them in real time. Imagine the world when your agent creates thousands of versions of itself, each with its own Orgo computer... That's the future, and we're living it today.

Nick Vasilescu

198,638 views • 6 months ago

Our first short course with Anthropic! Building Towards Computer Use with Anthropic. This teaches you to build an LLM-based agent that uses a computer interface by generating mouse clicks and keystrokes. Computer Use is an important, emerging capability for LLMs that will let AI agents do many more tasks than were possible before, since it lets them interact with interfaces designed for humans to use, rather than only tools that provide explicit API access. I hope you will enjoy learning about it! This course is taught by Anthropic's Head of Curriculum, Colt_Steele. You'll learn to apply image reasoning and tool use to "use" a computer as follows: a model processes an image of the screen, analyzes it to understand what's going on, and navigates the computer via mouse clicks and keystrokes. This course goes through the key building blocks, and culminates in a demo of an AI assistant that uses a web browser to search for a research paper, downloads the PDF, and finally summarizes the paper for you. In detail, you’ll: - Learn about Anthropic's family of models, when to use which one, and make API requests to Claude - Use multi-modal prompts that combine text and image content blocks, and also work with streaming responses - Improve your prompting by using prompt templates, using XML to structure prompts, and providing examples - Implement prompt caching to reduce cost and latency - Apply tool-use to build a chatbot that can call different tools to respond to queries - See all these building blocks come together in Computer Use demo Please sign up here:

Our first short course with Anthropic! Building Towards Computer Use with Anthropic. This teaches you to build an LLM-based agent that uses a computer interface by generating mouse clicks and keystrokes. Computer Use is an important, emerging capability for LLMs that will let AI agents do many more tasks than were possible before, since it lets them interact with interfaces designed for humans to use, rather than only tools that provide explicit API access. I hope you will enjoy learning about it! This course is taught by Anthropic's Head of Curriculum, Colt_Steele. You'll learn to apply image reasoning and tool use to "use" a computer as follows: a model processes an image of the screen, analyzes it to understand what's going on, and navigates the computer via mouse clicks and keystrokes. This course goes through the key building blocks, and culminates in a demo of an AI assistant that uses a web browser to search for a research paper, downloads the PDF, and finally summarizes the paper for you. In detail, you’ll: - Learn about Anthropic's family of models, when to use which one, and make API requests to Claude - Use multi-modal prompts that combine text and image content blocks, and also work with streaming responses - Improve your prompting by using prompt templates, using XML to structure prompts, and providing examples - Implement prompt caching to reduce cost and latency - Apply tool-use to build a chatbot that can call different tools to respond to queries - See all these building blocks come together in Computer Use demo Please sign up here:

Andrew Ng

170,689 views • 1 year ago

At Texas A&M University, we serve with purpose, lead with character and take action for the good of our communities. This is what it means to be an Aggie: to believe that what we can do together is far greater than what any one of us can do alone. For 150 years, Aggies have embodied that belief for Texas and for America. Together, we stand. Together, we serve. Together, we are a force for good. Learn more:

At Texas A&M University, we serve with purpose, lead with character and take action for the good of our communities. This is what it means to be an Aggie: to believe that what we can do together is far greater than what any one of us can do alone. For 150 years, Aggies have embodied that belief for Texas and for America. Together, we stand. Together, we serve. Together, we are a force for good. Learn more:

Texas A&M University

278,698 views • 2 months ago

Excited to share the multiplayer features we built to make running computer use agents easy. Here's a quick video of Claude ordering pizza and setting up Claude Code on the same computer.

Excited to share the multiplayer features we built to make running computer use agents easy. Here's a quick video of Claude ordering pizza and setting up Claude Code on the same computer.

Spencer Kinney

14,562 views • 6 months ago

NVIDIA CEO Jensen Huang on the future of artificial intelligence: "It is going to be largely optimistic, largely wonderful. There are some use cases that we have to be extremely careful about—as in all great technology. If you look at the progress of technology over the years, electricity, the internet, computers, the automobile, I think we can all, all accept that in each one of those cases, there are situations and cases where we have to be very careful about. So as the creators of this technology, the industry has to be mindful, thoughtful, rigorous about building and advancing this technology properly. This is the first technology in my generation where we invented a new computer technology where it's easier to access than not. For example the computer until now has been programmable for about 30 million people in the world. About 30 million of us know how to program it and apply it to some useful career. Now everybody in the world can program an AI. You just go up to it and just say, 'I'd like to, you know, ask you to do something.' If you're not sure how to use AI, you tell the AI, 'I don't know how to use AI. Tell me how to use AI. How do I, how do I write a program? How do I book a trip? How do I do a research?' And so we've closed the technology divide tremendously, and that's something that's really important to recognize."

NVIDIA CEO Jensen Huang on the future of artificial intelligence: "It is going to be largely optimistic, largely wonderful. There are some use cases that we have to be extremely careful about—as in all great technology. If you look at the progress of technology over the years, electricity, the internet, computers, the automobile, I think we can all, all accept that in each one of those cases, there are situations and cases where we have to be very careful about. So as the creators of this technology, the industry has to be mindful, thoughtful, rigorous about building and advancing this technology properly. This is the first technology in my generation where we invented a new computer technology where it's easier to access than not. For example the computer until now has been programmable for about 30 million people in the world. About 30 million of us know how to program it and apply it to some useful career. Now everybody in the world can program an AI. You just go up to it and just say, 'I'd like to, you know, ask you to do something.' If you're not sure how to use AI, you tell the AI, 'I don't know how to use AI. Tell me how to use AI. How do I, how do I write a program? How do I book a trip? How do I do a research?' And so we've closed the technology divide tremendously, and that's something that's really important to recognize."

Ryan Saavedra

20,105 views • 4 months ago

Browser/Computer Automation Jared Friedman AI agents can now browse the web and use desktop applications. It means that every website and every app now effectively has an API, and any workflow that people can do on a computer can be automated.

Browser/Computer Automation Jared Friedman AI agents can now browse the web and use desktop applications. It means that every website and every app now effectively has an API, and any workflow that people can do on a computer can be automated.

Y Combinator

62,191 views • 1 year ago

Jensen Huang and Michael Dell discussed the supply bottleneck: Michael - memory is the challenge. Jensen - we will have billions of agents, and we will need to give each agent essentially its own computer to use.

Jensen Huang and Michael Dell discussed the supply bottleneck: Michael - memory is the challenge. Jensen - we will have billions of agents, and we will need to give each agent essentially its own computer to use.

The AI Investor

249,135 views • 5 months ago