正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

OSWorld2.0 Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

AK

512,236 subscribers

29,144 次观看 • 1 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

LongHorizon-Harness Advancing Long-Horizon Agents for Real-World Tasks paper:

LongHorizon-Harness Advancing Long-Horizon Agents for Real-World Tasks paper:

AK

23,888 次观看 • 8 天前

🤔Can we assess agents across various apps & OS w.o. crafting new envs? OSWorld🖥️: A unified, real computer env for multimodal agents to evaluate open-ended computer tasks with arbitrary apps and interfaces on Ubuntu, Windows, & macOS. + annotated 369 real-world computer tasks 👇

🤔Can we assess agents across various apps & OS w.o. crafting new envs? OSWorld🖥️: A unified, real computer env for multimodal agents to evaluate open-ended computer tasks with arbitrary apps and interfaces on Ubuntu, Windows, & macOS. + annotated 369 real-world computer tasks 👇

Tianbao Xie

66,610 次观看 • 2 年前

Two years ago, we built OSWorld 1.0 — the benchmark that became the standard for computer-use agents. Agents now score 83.5% on it. Problem solved? Not even close. 🚀Today we introduce OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks. What's new: 🎯 108 real-world workflows, each ~1.6 hours ⏱️ for a skilled human ⚙️ ~318 tool calls/task vs. ~30 in OSWorld 1.0 🌍 Grounded in authentic artifacts & stateful user profiles ⚡ Captures real phenomena: dynamic environments, streaming interaction, cross-source reasoning, implicit-state inference & more 📊 Best results: Claude Opus 4.8 reaches the highest accuracy at 20.6%, while GPT-5.5 is far more token-efficient but plateaus near 13%. No one is close to solving real computer use. 🏠 Homepage: 📄 Paper: 💻 Code: 🤗 Dataset: 🧵 [1/8]

Two years ago, we built OSWorld 1.0 — the benchmark that became the standard for computer-use agents. Agents now score 83.5% on it. Problem solved? Not even close. 🚀Today we introduce OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks. What's new: 🎯 108 real-world workflows, each ~1.6 hours ⏱️ for a skilled human ⚙️ ~318 tool calls/task vs. ~30 in OSWorld 1.0 🌍 Grounded in authentic artifacts & stateful user profiles ⚡ Captures real phenomena: dynamic environments, streaming interaction, cross-source reasoning, implicit-state inference & more 📊 Best results: Claude Opus 4.8 reaches the highest accuracy at 20.6%, while GPT-5.5 is far more token-efficient but plateaus near 13%. No one is close to solving real computer use. 🏠 Homepage: 📄 Paper: 💻 Code: 🤗 Dataset: 🧵 [1/8]

XLANG NLP Lab

165,016 次观看 • 1 个月前

Computer use agents are slow and brittle. The fix isn’t just stronger models, but also deploying them as multi-agent systems. MACU is a general Multi-Agent Computer Use framework that consistently lifts success rates by 3.4-25.5% and is up to 1.5x faster on long-horizon tasks.🧵

Computer use agents are slow and brittle. The fix isn’t just stronger models, but also deploying them as multi-agent systems. MACU is a general Multi-Agent Computer Use framework that consistently lifts success rates by 3.4-25.5% and is up to 1.5x faster on long-horizon tasks.🧵

Jing Yu Koh

29,642 次观看 • 2 个月前

Need help in the real world? RoboVQA can guide robots and humans through long-horizon tasks on a phone via Google Meet. We release a dataset of 800k (video, question/answer) with robots & humans doing various long-horizon tasks. Data: Google DeepMind

Need help in the real world? RoboVQA can guide robots and humans through long-horizon tasks on a phone via Google Meet. We release a dataset of 800k (video, question/answer) with robots & humans doing various long-horizon tasks. Data: Google DeepMind

Pierre Sermanet

64,369 次观看 • 2 年前

How good is Opus 4.7 at long running computer use tasks?

How good is Opus 4.7 at long running computer use tasks?

AshutoshShrivastava

10,535 次观看 • 3 个月前

World Model meets robot policy! Robbyant's LingBot-VA: unifies video world modeling and robotic policy learning. - A single model generates both future video and the actions to make it real. - Long-term memory enables long-horizon tasks. - Claims significant outperformance over π₀.₅ in real-world tasks. - It's open-source

World Model meets robot policy! Robbyant's LingBot-VA: unifies video world modeling and robotic policy learning. - A single model generates both future video and the actions to make it real. - Long-term memory enables long-horizon tasks. - Claims significant outperformance over π₀.₅ in real-world tasks. - It's open-source

The Humanoid Hub

17,721 次观看 • 6 个月前

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.

Russ Salakhutdinov

22,518 次观看 • 3 个月前

Humanoid Robots can Think, Move, and Adapt in Real Time. LimX COSA (Cognitive OS of Agents), the first physical-world-native agentic OS, unifies cognitive reasoning and whole-body control, enabling robots to complete long-horizon, multi-step tasks.

Humanoid Robots can Think, Move, and Adapt in Real Time. LimX COSA (Cognitive OS of Agents), the first physical-world-native agentic OS, unifies cognitive reasoning and whole-body control, enabling robots to complete long-horizon, multi-step tasks.

LimX Dynamics

1,631,725 次观看 • 6 个月前

Introducing Lux, the most powerful and fastest Computer Use model. Lux outperforms Google Gemini CUA, OpenAI Operator and Anthropic Claude on benchmark with 300 real-world tasks. Try our developer-friendly SDK to build powerful, real-world applications. 🧵

Introducing Lux, the most powerful and fastest Computer Use model. Lux outperforms Google Gemini CUA, OpenAI Operator and Anthropic Claude on benchmark with 300 real-world tasks. Try our developer-friendly SDK to build powerful, real-world applications. 🧵

OpenAGI Labs

398,704 次观看 • 8 个月前

Today we're announcing cua-bench: a framework for benchmarking, training data, and RL environments for computer-use AI agents. Why? Current agents show 10x variance across minor UI changes. Here's how we're fixing it.

Today we're announcing cua-bench: a framework for benchmarking, training data, and RL environments for computer-use AI agents. Why? Current agents show 10x variance across minor UI changes. Here's how we're fixing it.

Cua

189,823 次观看 • 7 个月前

8/ Computer Use Are A Sadness - 🎖️Engagement Award Exploiting computer use agents using LLM security vulnerabilities for fun and profit Hijack agents that try to access your computer

8/ Computer Use Are A Sadness - 🎖️Engagement Award Exploiting computer use agents using LLM security vulnerabilities for fun and profit Hijack agents that try to access your computer

Alex Reibman 🖇️

25,051 次观看 • 1 年前

DeepSeek-R1 Computer Use Agents are WILD! 🤯

DeepSeek-R1 Computer Use Agents are WILD! 🤯

Julian Goldie SEO

369,297 次观看 • 1 年前

Build agents that can actually do real-world tasks! Agent Reinforcement Trainer (ART) is a framework to train multi-step LLM agents for real-world tasks using GRPO. Just a few lines of code. No manual rewards needed. vLLM + Unsloth combined 🚀 100% open-source.

Build agents that can actually do real-world tasks! Agent Reinforcement Trainer (ART) is a framework to train multi-step LLM agents for real-world tasks using GRPO. Just a few lines of code. No manual rewards needed. vLLM + Unsloth combined 🚀 100% open-source.

Akshay 🚀

38,297 次观看 • 6 个月前

Deepseek Computer Use AI Agents are INSANE (FREE!) 🤯

Deepseek Computer Use AI Agents are INSANE (FREE!) 🤯

Julian Goldie SEO

91,135 次观看 • 1 年前

NEW FREE Computer Use AI Agents are INSANE 🤯

NEW FREE Computer Use AI Agents are INSANE 🤯

Julian Goldie SEO

49,308 次观看 • 1 年前

Today we're launching Subconscious: a new platform for building agents with long-horizon reasoning and tool use, backed by MIT research. One API call. Tool use. Context beyond existing limits. If you're building agents, let's talk.

Today we're launching Subconscious: a new platform for building agents with long-horizon reasoning and tool use, backed by MIT research. One API call. Tool use. Context beyond existing limits. If you're building agents, let's talk.

Jack O'Brien

12,097 次观看 • 1 年前

Open Computer Agent - LLMs completing tasks using a VM. It's playground to test how well current LLM agents use a computer to solve everyday tasks. And this is just the start - very soon models will be 10x faster and 10x better at it! ❤️ built with e2b x qwen2.5-vl x smolagent

Open Computer Agent - LLMs completing tasks using a VM. It's playground to test how well current LLM agents use a computer to solve everyday tasks. And this is just the start - very soon models will be 10x faster and 10x better at it! ❤️ built with e2b x qwen2.5-vl x smolagent

Leandro von Werra

17,126 次观看 • 1 年前

🚀🚀🚀 Ever wondered what it takes for robots to handle real-world household tasks? long-horizon execution, deformable object dexterity, and unseen object generalization — meet GR-3, ByteDance Seed’s new Vision-Language-Action (VLA) model! GR-3 is a generalizable Vision-Language-Action (VLA) model with strong capabilities in complex long-horizon tasks. It understands unseen abstract concepts, manipulates deformable objects robustly, and adapts to novel settings with minimal human data. ✨ Generalization: Generalizes well to unseen objects, environments, and even instructions with abstract concepts. ✨ Long-Horizon Manipulation: Completes long-horizon tasks with strong instruction-following capabilities. ✨ Deformable Object Manipulation: Manipulate deformable objects robustly. Project Page: Arxiv: #ByteDance #ByteDanceSeed #GR3 #VLA #Robotics #FoundationModels

🚀🚀🚀 Ever wondered what it takes for robots to handle real-world household tasks? long-horizon execution, deformable object dexterity, and unseen object generalization — meet GR-3, ByteDance Seed’s new Vision-Language-Action (VLA) model! GR-3 is a generalizable Vision-Language-Action (VLA) model with strong capabilities in complex long-horizon tasks. It understands unseen abstract concepts, manipulates deformable objects robustly, and adapts to novel settings with minimal human data. ✨ Generalization: Generalizes well to unseen objects, environments, and even instructions with abstract concepts. ✨ Long-Horizon Manipulation: Completes long-horizon tasks with strong instruction-following capabilities. ✨ Deformable Object Manipulation: Manipulate deformable objects robustly. Project Page: Arxiv: #ByteDance #ByteDanceSeed #GR3 #VLA #Robotics #FoundationModels

Xiao Ma

46,323 次观看 • 1 年前