Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

OSWorld2.0 Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

_akhaliq's profile picture

AK

519,304 subscribers

29,294 görüntüleme • 4 ay önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

LongHorizon-Harness Advancing Long-Horizon Agents for Real-World Tasks paper:
1:12

LongHorizon-Harness Advancing Long-Horizon Agents for Real-World Tasks paper:

AK

26,033 görüntüleme • 2 ay önce

Two years ago, we built OSWorld 1.0 — the benchmark that became the standard for computer-use agents. Agents now score 83.5% on it. Problem solved? Not even close. 🚀Today we introduce OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks. What's new: 🎯 108 real-world workflows, each ~1.6 hours ⏱️ for a skilled human ⚙️ ~318 tool calls/task vs. ~30 in OSWorld 1.0 🌍 Grounded in authentic artifacts & stateful user profiles ⚡ Captures real phenomena: dynamic environments, streaming interaction, cross-source reasoning, implicit-state inference & more 📊 Best results: Claude Opus 4.8 reaches the highest accuracy at 20.6%, while GPT-5.5 is far more token-efficient but plateaus near 13%. No one is close to solving real computer use. 🏠 Homepage: 📄 Paper: 💻 Code: 🤗 Dataset: 🧵 [1/8]
0:22

Two years ago, we built OSWorld 1.0 — the benchmark that became the standard for computer-use agents. Agents now score 83.5% on it. Problem solved? Not even close. 🚀Today we introduce OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks. What's new: 🎯 108 real-world workflows, each ~1.6 hours ⏱️ for a skilled human ⚙️ ~318 tool calls/task vs. ~30 in OSWorld 1.0 🌍 Grounded in authentic artifacts & stateful user profiles ⚡ Captures real phenomena: dynamic environments, streaming interaction, cross-source reasoning, implicit-state inference & more 📊 Best results: Claude Opus 4.8 reaches the highest accuracy at 20.6%, while GPT-5.5 is far more token-efficient but plateaus near 13%. No one is close to solving real computer use. 🏠 Homepage: 📄 Paper: 💻 Code: 🤗 Dataset: 🧵 [1/8]

XLANG NLP Lab

173,634 görüntüleme • 4 ay önce

🤔Can we assess agents across various apps & OS w.o. crafting new envs? OSWorld🖥️: A unified, real computer env for multimodal agents to evaluate open-ended computer tasks with arbitrary apps and interfaces on Ubuntu, Windows, & macOS. + annotated 369 real-world computer tasks 👇
1:00

🤔Can we assess agents across various apps & OS w.o. crafting new envs? OSWorld🖥️: A unified, real computer env for multimodal agents to evaluate open-ended computer tasks with arbitrary apps and interfaces on Ubuntu, Windows, & macOS. + annotated 369 real-world computer tasks 👇

Tianbao Xie

68,974 görüntüleme • 2 yıl önce

Computer use agents are slow and brittle. The fix isn’t just stronger models, but also deploying them as multi-agent systems. MACU is a general Multi-Agent Computer Use framework that consistently lifts success rates by 3.4-25.5% and is up to 1.5x faster on long-horizon tasks.🧵
0:23

Computer use agents are slow and brittle. The fix isn’t just stronger models, but also deploying them as multi-agent systems. MACU is a general Multi-Agent Computer Use framework that consistently lifts success rates by 3.4-25.5% and is up to 1.5x faster on long-horizon tasks.🧵

Jing Yu Koh

32,664 görüntüleme • 4 ay önce

🎮 Computer Use Agent Arena is LIVE! 🚀 🔥 Easiest way to test computer-use agents in the wild without any setup 🌟 Compare top VLMs: OpenAI Operator, Claude 3.7, Gemini 2.5 Pro, Qwen 2.5 vl and more 🕹️ Test agents on 100+ real apps & webs with one-click config 🔒 Safe & free access on cloud-hosted machines Page: Leaderboard (tentative): Blog: Data & Code (coming soon): ⭐️Why Computer Agent Arena? 1️⃣Beyond Static Benchmarks: We use computers to perform enormous tasks and workflows every day, and AI agents have the potential to automate these tasks. However, existing benchmarks are very limited (e.g., only 369 tasks in OSWorld and 812 tasks in WebArena). To better measure their capabilities, we introduce Computer Agent Arena for users to easily compare & test AI agents on all kinds of crowdsourced real-world computer use tasks. 2️⃣Cloud Testing, Simplified: As agents like OpenAI’s Operator and Claude 3.7 sonnet release, users face configuration challenges and privacy hurdles to deploy on their own computers. Our platform integrates these agents with cloud-hosted machines, providing users with quick and secure access. 3️⃣Unified Embodied Digital Environment: Unlike Chatbot Arena, we provide users with a real embodied environment—computers—where all agents are grounded in real computer tasks and environments. Led by @XLANG_Lab [1/🧵]
0:32

🎮 Computer Use Agent Arena is LIVE! 🚀 🔥 Easiest way to test computer-use agents in the wild without any setup 🌟 Compare top VLMs: OpenAI Operator, Claude 3.7, Gemini 2.5 Pro, Qwen 2.5 vl and more 🕹️ Test agents on 100+ real apps & webs with one-click config 🔒 Safe & free access on cloud-hosted machines Page: Leaderboard (tentative): Blog: Data & Code (coming soon): ⭐️Why Computer Agent Arena? 1️⃣Beyond Static Benchmarks: We use computers to perform enormous tasks and workflows every day, and AI agents have the potential to automate these tasks. However, existing benchmarks are very limited (e.g., only 369 tasks in OSWorld and 812 tasks in WebArena). To better measure their capabilities, we introduce Computer Agent Arena for users to easily compare & test AI agents on all kinds of crowdsourced real-world computer use tasks. 2️⃣Cloud Testing, Simplified: As agents like OpenAI’s Operator and Claude 3.7 sonnet release, users face configuration challenges and privacy hurdles to deploy on their own computers. Our platform integrates these agents with cloud-hosted machines, providing users with quick and secure access. 3️⃣Unified Embodied Digital Environment: Unlike Chatbot Arena, we provide users with a real embodied environment—computers—where all agents are grounded in real computer tasks and environments. Led by @XLANG_Lab [1/🧵]

Bowen Wang

93,360 görüntüleme • 1 yıl önce

Need help in the real world? RoboVQA can guide robots and humans through long-horizon tasks on a phone via Google Meet. We release a dataset of 800k (video, question/answer) with robots & humans doing various long-horizon tasks. Data: Google DeepMind
0:49

Need help in the real world? RoboVQA can guide robots and humans through long-horizon tasks on a phone via Google Meet. We release a dataset of 800k (video, question/answer) with robots & humans doing various long-horizon tasks. Data: Google DeepMind

Pierre Sermanet

64,435 görüntüleme • 3 yıl önce

What if an AI could actually use your computer—not just tell you what to do? Nex-N2.5 Pro takes on long-horizon Computer Use tasks, understanding the screen, navigating the game, making decisions, and continuously adapting as the situation changes. This demo shows it playing Pokémon Platinum from a fresh game and progressing all the way to earning the first Gym Badge. 🎮 397B parameters • Multimodal Vision • Computer Use • Long-horizon tasks Try Nex-N2.5-Pro for FREE on OpenRouter for a limited time: #NexN25 #NexAGI #AI #ComputerUse #OpenRouter
6:53

What if an AI could actually use your computer—not just tell you what to do? Nex-N2.5 Pro takes on long-horizon Computer Use tasks, understanding the screen, navigating the game, making decisions, and continuously adapting as the situation changes. This demo shows it playing Pokémon Platinum from a fresh game and progressing all the way to earning the first Gym Badge. 🎮 397B parameters • Multimodal Vision • Computer Use • Long-horizon tasks Try Nex-N2.5-Pro for FREE on OpenRouter for a limited time: #NexN25 #NexAGI #AI #ComputerUse #OpenRouter

Red chili 🌶️

49,812 görüntüleme • 1 ay önce

How good is Opus 4.7 at long running computer use tasks?
0:40

How good is Opus 4.7 at long running computer use tasks?

AshutoshShrivastava

10,535 görüntüleme • 6 ay önce

World Model meets robot policy! Robbyant's LingBot-VA: unifies video world modeling and robotic policy learning. - A single model generates both future video and the actions to make it real. - Long-term memory enables long-horizon tasks. - Claims significant outperformance over π₀.₅ in real-world tasks. - It's open-source
1:33

World Model meets robot policy! Robbyant's LingBot-VA: unifies video world modeling and robotic policy learning. - A single model generates both future video and the actions to make it real. - Long-term memory enables long-horizon tasks. - Claims significant outperformance over π₀.₅ in real-world tasks. - It's open-source

The Humanoid Hub

17,721 görüntüleme • 9 ay önce

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.
1:12

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.

Russ Salakhutdinov

22,518 görüntüleme • 6 ay önce

Humanoid Robots can Think, Move, and Adapt in Real Time. LimX COSA (Cognitive OS of Agents), the first physical-world-native agentic OS, unifies cognitive reasoning and whole-body control, enabling robots to complete long-horizon, multi-step tasks.
4:29

Humanoid Robots can Think, Move, and Adapt in Real Time. LimX COSA (Cognitive OS of Agents), the first physical-world-native agentic OS, unifies cognitive reasoning and whole-body control, enabling robots to complete long-horizon, multi-step tasks.

LimX Dynamics

1,631,725 görüntüleme • 9 ay önce

Introducing Lux, the most powerful and fastest Computer Use model. Lux outperforms Google Gemini CUA, OpenAI Operator and Anthropic Claude on benchmark with 300 real-world tasks. Try our developer-friendly SDK to build powerful, real-world applications. 🧵
1:14

Introducing Lux, the most powerful and fastest Computer Use model. Lux outperforms Google Gemini CUA, OpenAI Operator and Anthropic Claude on benchmark with 300 real-world tasks. Try our developer-friendly SDK to build powerful, real-world applications. 🧵

OpenAGI Labs

398,704 görüntüleme • 11 ay önce

Today we're announcing cua-bench: a framework for benchmarking, training data, and RL environments for computer-use AI agents. Why? Current agents show 10x variance across minor UI changes. Here's how we're fixing it.
2:02

Today we're announcing cua-bench: a framework for benchmarking, training data, and RL environments for computer-use AI agents. Why? Current agents show 10x variance across minor UI changes. Here's how we're fixing it.

Cua

189,823 görüntüleme • 10 ay önce

8/ Computer Use Are A Sadness - 🎖️Engagement Award Exploiting computer use agents using LLM security vulnerabilities for fun and profit Hijack agents that try to access your computer
5:50

8/ Computer Use Are A Sadness - 🎖️Engagement Award Exploiting computer use agents using LLM security vulnerabilities for fun and profit Hijack agents that try to access your computer

Alex Reibman 🖇️

25,051 görüntüleme • 1 yıl önce

Build agents that can actually do real-world tasks! Agent Reinforcement Trainer (ART) is a framework to train multi-step LLM agents for real-world tasks using GRPO. Just a few lines of code. No manual rewards needed. vLLM + Unsloth combined 🚀 100% open-source.
18:15

Build agents that can actually do real-world tasks! Agent Reinforcement Trainer (ART) is a framework to train multi-step LLM agents for real-world tasks using GRPO. Just a few lines of code. No manual rewards needed. vLLM + Unsloth combined 🚀 100% open-source.

Akshay 🚀

38,297 görüntüleme • 9 ay önce

DeepSeek-R1 Computer Use Agents are WILD! 🤯
13:04

DeepSeek-R1 Computer Use Agents are WILD! 🤯

Julian Goldie SEO

369,775 görüntüleme • 1 yıl önce

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.
0:25

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

MIKE

14,387 görüntüleme • 1 ay önce

NEW FREE Computer Use AI Agents are INSANE 🤯
14:34

NEW FREE Computer Use AI Agents are INSANE 🤯

Julian Goldie SEO

49,308 görüntüleme • 1 yıl önce

Deepseek Computer Use AI Agents are INSANE (FREE!) 🤯
12:12

Deepseek Computer Use AI Agents are INSANE (FREE!) 🤯

Julian Goldie SEO

91,135 görüntüleme • 1 yıl önce