Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

How capable are web agents at solving knowledge work tasks? 🤔 Are LLMs up to the challenge? 🤖 Introducing WorkArena: a benchmark where agents meet the world 𝘸𝘪𝘭𝘥 web of enterprise software 🌐🖥️ Paper: Website: 🧵 1/7

Alexandre Lacoste

1,136 subscribers

24,550 Aufrufe • vor 2 Jahren •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

18 Kommentare

Profilbild von Alexandre Lacoste

Alexandre Lacostevor 2 Jahren

WorkArena is built on the ServiceNow platform, which is used by most of the Fortune 500 companies. It contains 29 tasks, which encompass the most common ways workers interact with the software. For example, searching for information in a company knowledge base ⬇️ 2/7

Profilbild von Alexandre Lacoste

Alexandre Lacostevor 2 Jahren

Developing enterprise web-based agents is no small feat. Challenges include: 🌐 Navigating the 'World 𝘞𝘪𝘭𝘥 Web' of Enterprise UI with huge and non-standard HTML files (100k+ tokens) 💾 Memory management (for POMPDs) 📈 Planning 📚 Domain-specific knowledge 3/7

Profilbild von Alexandre Lacoste

Alexandre Lacostevor 2 Jahren

To help agents navigate this space we propose a new web environment with: Multimodal observations (AXtree, HTML, screenshots) ☑High-level actions (click, fill, etc.) ☑Chat interface for user interaction ☑Robustness to the world wild web (e.g., iFrames, shadow DOMs) 4/7

Profilbild von Alexandre Lacoste

Alexandre Lacostevor 2 Jahren

WorkArena poses a real challenge! Even with GPT4 and our top-performing configs, we're seeing only a 55% success rate. A SOTA OSS code LLM hasn’t had a single successful episode yet. ✨We're calling on the open-source community to help bridge this significant gap. 5/7

Profilbild von Alexandre Lacoste

Alexandre Lacostevor 2 Jahren

All our experiments are conducted using BrowserGym, our new conversational gym environment for web agents! 🌐 Unified testing ground for MiniWob, WebArena, WorkArena 📣 We welcome new benchmark contributions! 🚀 Try our demo on the open web: 6/7

Profilbild von Alexandre Lacoste

Alexandre Lacostevor 2 Jahren

Special thanks to @ServiceNowRSRCH for the support!🙏 Shoutout to all authors: @alexandredrouin, @maxime_gasse, @MassCaccia, @ILaradji, @ManuelDelVerme, Tom Marty, @LeoBoisvert, @Megh1211, @qcappart , @dvazquezcv, @NicolasChapados. 👏 Also, thanks to @Mila_Quebec! 🎉

Profilbild von Alexandre Lacoste

Alexandre Lacostevor 2 Jahren

@alexandredrouin @maxime_gasse @MassCaccia @ILaradji @ManuelDelVerme @LeoBoisvert @Megh1211 @qcappart @dvazquezcv @NicolasChapados @Mila_Quebec Thanks to @ThirdCore42 (Tom Marty).

Profilbild von Xing Han Lu

Xing Han Luvor 2 Jahren

Amazing (and colossal) work! but I wanna stop for a moment and say how cool the intro video is - especially with the zoom out!

Profilbild von Rajiv Shah

Rajiv Shahvor 2 Jahren

Great work. It's nice to have data points on realistic tasks. This really helps level set expectations and know where we need to improve.

Profilbild von PG

PGvor 2 Jahren

That is so cool! Félicitations!

Profilbild von Prem Kumar Aparanji 🤖👶🏼🐘

Prem Kumar Aparanji 🤖👶🏼🐘vor 2 Jahren

Wow! Impressive work! And very true, inside the enterprise it is indeed a very wild web. BTW, would it not be better to just call the appropriate API of servicenow and accomplish these tasks? Is there any research on those lines too?

Profilbild von Frank Xu

Frank Xuvor 2 Jahren

Looking real nice! Great environment platform especially

Profilbild von AlexConway.eth ⌬⃤⟁

AlexConway.eth ⌬⃤⟁vor 2 Jahren

This is amazing but i can't wrap my head around how 2000 samples is a "large-scale" dataset?! 😅 Can you give an approximation on manhours required to produce this?

Profilbild von Alexandre Lacoste

Alexandre Lacostevor 2 Jahren

Lol. Large is always contextual :) @xhluca what were the resources involved in building weblinx?

Profilbild von Xing Han Lu

Xing Han Luvor 2 Jahren

@0x413x Yeah demonstrations usually take 15 to record and 15 for other things (upload, write summary, prepare, etc.) for 2 people. So ~60min per demonstration is around 2000h of work, without counting failed attempts and time to validate the demos afterwards.

Profilbild von AlexConway.eth ⌬⃤⟁

AlexConway.eth ⌬⃤⟁vor 2 Jahren

@alex_lacoste_ Thanks Xing! 🙏 where can i learn more about this?

Profilbild von Xing Han Lu

Xing Han Luvor 2 Jahren

@alex_lacoste_ The paper contains all the detailed information: The appendix is pretty extensive but you can use the ToC to navigate (right after references).

Profilbild von AlexConway.eth ⌬⃤⟁

AlexConway.eth ⌬⃤⟁vor 2 Jahren

@alex_lacoste_ Epic, thanks!

Ähnliche Videos

Meet Cabinet: Paper Clip + KB. for quite some time I've been thinking how LLMs are missing the knowledge base - where I can dump CSVs, PDFs, and most important - inline web app. running on Claude Code with agents with heartbeats and jobs

Meet Cabinet: Paper Clip + KB. for quite some time I've been thinking how LLMs are missing the knowledge base - where I can dump CSVs, PDFs, and most important - inline web app. running on Claude Code with agents with heartbeats and jobs

Hila Shmuel

524,927 Aufrufe • vor 7 Monaten

AI web agents like Operator and Anthropic’s Computer Use can operate a browser, but the LLMs inside are brittle, and you can’t trust what’s on the web. In this 🧵, I’ll show how adversaries can fool Anthropic’s web agent into sending phishing emails or revealing credit card info.

AI web agents like Operator and Anthropic’s Computer Use can operate a browser, but the LLMs inside are brittle, and you can’t trust what’s on the web. In this 🧵, I’ll show how adversaries can fool Anthropic’s web agent into sending phishing emails or revealing credit card info.

Micah Goldblum

42,969 Aufrufe • vor 1 Jahr

AI agents are getting real jobs now?! Introducing a managed marketplace where companies post roles, and agents get to work. Powered by in collaboration with Amazon Web Services, Anthropic, Wix, and Mesh. There's an agent out there who is the perfect fit for your team! Come and find it 🤖

AI agents are getting real jobs now?! Introducing a managed marketplace where companies post roles, and agents get to work. Powered by in collaboration with Amazon Web Services, Anthropic, Wix, and Mesh. There's an agent out there who is the perfect fit for your team! Come and find it 🤖

monday.com

35,696 Aufrufe • vor 7 Monaten

Introducing Notion Agents. You assign the tasks, your Agents do the work. Available today. Giddy up! 🤠

Introducing Notion Agents. You assign the tasks, your Agents do the work. Available today. Giddy up! 🤠

Notion

963,384 Aufrufe • vor 1 Jahr

🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. 🤔What's this all about? Why this benchmark? > Back in Nov 2023, when we released MMMU ( a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin ( which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.📸👀 > Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent ( showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages. 🌉 Bridging the Gap with VisualWebBench > To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites 🌐 across 12 domains 🏷️ and 87 sub-domains 🔍, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level 📊, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities 🧠💡. 😮 Surprising Findings > 🎉 Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro. > 🧠 Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs. > 🖼️ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent. > 🧱 Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements. 💡Fun Fact > Claude Sonnet is better than Opus on our benchmark :) 🎓 Conclusion > VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web. Kudos to the student leads Junpeng Liu Yifan Song and the team Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li! 👏 Check out more details in the Junpeng's thread👇

🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. 🤔What's this all about? Why this benchmark? > Back in Nov 2023, when we released MMMU ( a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin ( which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.📸👀 > Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent ( showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages. 🌉 Bridging the Gap with VisualWebBench > To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites 🌐 across 12 domains 🏷️ and 87 sub-domains 🔍, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level 📊, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities 🧠💡. 😮 Surprising Findings > 🎉 Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro. > 🧠 Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs. > 🖼️ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent. > 🧱 Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements. 💡Fun Fact > Claude Sonnet is better than Opus on our benchmark :) 🎓 Conclusion > VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web. Kudos to the student leads Junpeng Liu Yifan Song and the team Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li! 👏 Check out more details in the Junpeng's thread👇

Xiang Yue

56,697 Aufrufe • vor 2 Jahren

Build your next website with Dart and Jaspr 🌐 Craig catches up with Argel Bejarano 💙, Senior Staff Engineer at Aplazo to discuss how Jaspr compares to Flutter Web and how easily AI coding agents integrate into your web workflow.

Build your next website with Dart and Jaspr 🌐 Craig catches up with Argel Bejarano 💙, Senior Staff Engineer at Aplazo to discuss how Jaspr compares to Flutter Web and how easily AI coding agents integrate into your web workflow.

Flutter

12,160 Aufrufe • vor 1 Monat

we are building at the Y Combinator web agents hackathon this weekend. introducing clawtrace: on-call security for openclaw web agents. openclaw is powerful. but it’s hard to fully trust what your agents are actually doing. that’s why we built clawtrace. clawtrace lets you: - know when your agents go rogue - detect anomalies in behavior - track sessions, tool calls, thinking states here’s how it works:

we are building at the Y Combinator web agents hackathon this weekend. introducing clawtrace: on-call security for openclaw web agents. openclaw is powerful. but it’s hard to fully trust what your agents are actually doing. that’s why we built clawtrace. clawtrace lets you: - know when your agents go rogue - detect anomalies in behavior - track sessions, tool calls, thinking states here’s how it works:

Divya Saini

22,323 Aufrufe • vor 8 Monaten

we are building at the Y Combinator web agents hackathon this weekend. introducing clawtrace: on-call security for openclaw web agents. openclaw is powerful. but it’s hard to fully trust what your agents are actually doing. that’s why we built clawtrace. clawtrace lets you: - know when your agents go rogue - detect anomalies in behavior - track sessions, tool calls, thinking states here’s how it works:

we are building at the Y Combinator web agents hackathon this weekend. introducing clawtrace: on-call security for openclaw web agents. openclaw is powerful. but it’s hard to fully trust what your agents are actually doing. that’s why we built clawtrace. clawtrace lets you: - know when your agents go rogue - detect anomalies in behavior - track sessions, tool calls, thinking states here’s how it works:

Sri Laasya Nutheti

19,126 Aufrufe • vor 8 Monaten

NEW: DevRev raises a $100M Series A to build enterprise AI apps & agents - Valued at $1.1B 🦄 The core of their platform is an enterprise knowledge graph that powers AI apps & agents for customer service, product, software engineering, etc. Here's how devrev works:

NEW: DevRev raises a $100M Series A to build enterprise AI apps & agents - Valued at $1.1B 🦄 The core of their platform is an enterprise knowledge graph that powers AI apps & agents for customer service, product, software engineering, etc. Here's how devrev works:

Chief AI Officer

11,247 Aufrufe • vor 2 Jahren

Introducing: Browser Use agents support WebMCP 💯 > UI is made for humans > Direct HTTP calls are made for agents > WebMCP makes interaction 100% deterministic No more flaky UI. Let's build the new web for web agents.

Introducing: Browser Use agents support WebMCP 💯 > UI is made for humans > Direct HTTP calls are made for agents > WebMCP makes interaction 100% deterministic No more flaky UI. Let's build the new web for web agents.

Gregor Zunic

57,514 Aufrufe • vor 1 Monat

Everything you need to know about AI agents right now! 🤖🌐 Last week, I flew to NYC to interview @mateo_ventures and uncover the future of Decentralized AI, AI agents, and Open-source LLMs. Let’s dive in. 🧵

Everything you need to know about AI agents right now! 🤖🌐 Last week, I flew to NYC to interview @mateo_ventures and uncover the future of Decentralized AI, AI agents, and Open-source LLMs. Let’s dive in. 🧵

Francesco Andreoli ᵍᵐ

77,132 Aufrufe • vor 1 Jahr

AI agents are creating a new enterprise challenge: economics. At Dell Technologies Technologies World 2026, Ihab Tarazi of Dell and NVIDIA's Jason Hardy explained why the next phase of enterprise AI isn't about access to models, it's about controlling the cost of compute, storage, and the explosion of data agents create. The next wave of AI success will be defined by how organizations manage the economics behind agents at scale. $NVDA $DELL

AI agents are creating a new enterprise challenge: economics. At Dell Technologies Technologies World 2026, Ihab Tarazi of Dell and NVIDIA's Jason Hardy explained why the next phase of enterprise AI isn't about access to models, it's about controlling the cost of compute, storage, and the explosion of data agents create. The next wave of AI success will be defined by how organizations manage the economics behind agents at scale. $NVDA $DELL

Six Five Media

26,975 Aufrufe • vor 4 Monaten

Introducing 🦀 CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents 🦀 CRAB provides an end-to-end and easy-to-use framework to build multimodal agents, operate environments, and create benchmarks to evaluate them, featuring three key components: - 🔀 Cross-environment support - agents can operate tasks in 📱 Android and 💻 Ubuntu. - 🕸️ Graph evaluator - provides a fine-grain evaluation metric for agents. - 🤖 Task generation - composes subtasks to automatically generate tasks. By connecting all devices to agents, 🦀CRAB unlocks greater capabilities for human-like tasks than ever before. Use 🦀 CRAB to benchmark your multimodal agents! - 👨‍💻 Check out the repository: - 📝 Read the paper: - 🌐 Find out more via the project page: - 🐫 Join our community:

Introducing 🦀 CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents 🦀 CRAB provides an end-to-end and easy-to-use framework to build multimodal agents, operate environments, and create benchmarks to evaluate them, featuring three key components: - 🔀 Cross-environment support - agents can operate tasks in 📱 Android and 💻 Ubuntu. - 🕸️ Graph evaluator - provides a fine-grain evaluation metric for agents. - 🤖 Task generation - composes subtasks to automatically generate tasks. By connecting all devices to agents, 🦀CRAB unlocks greater capabilities for human-like tasks than ever before. Use 🦀 CRAB to benchmark your multimodal agents! - 👨‍💻 Check out the repository: - 📝 Read the paper: - 🌐 Find out more via the project page: - 🐫 Join our community:

CAMEL-AI.org

72,807 Aufrufe • vor 2 Jahren

🚀 Introducing Boson V3 - Decentralized AI Commerce Post Web commerce for humans and AI TL;DR Crypto x AI Agents -> Parabolic adoption = The end of the Web and Platforms Crypto needs AI to bridge complex UX AI agents need Crypto for trusted execution of transactions Discover how Boson V3 is the base layer for the Trillion-Dollar Post Web Commerce Era! 👇🧵

🚀 Introducing Boson V3 - Decentralized AI Commerce Post Web commerce for humans and AI TL;DR Crypto x AI Agents -> Parabolic adoption = The end of the Web and Platforms Crypto needs AI to bridge complex UX AI agents need Crypto for trusted execution of transactions Discover how Boson V3 is the base layer for the Trillion-Dollar Post Web Commerce Era! 👇🧵

Boson

2,294,157 Aufrufe • vor 1 Jahr

LLMs are capable of high-level planning, but they require pre-trained skills! Our #ICLR2024 paper instead uses LLM guidance to train RL agents from scratch to solve 25+ long-horizon robotics tasks across four benchmarks w/ >85% success rates Paper & code:

LLMs are capable of high-level planning, but they require pre-trained skills! Our #ICLR2024 paper instead uses LLM guidance to train RL agents from scratch to solve 25+ long-horizon robotics tasks across four benchmarks w/ >85% success rates Paper & code:

Murtaza Dalal

36,321 Aufrufe • vor 2 Jahren

The real challenge isn’t building agents. It’s discovering them. Trusting them. Coordinating them. Just like the early web had millions of websites, agentic AI will have millions of autonomous agents. Now, agents need infrastructure to find, verify, and work with other agents. Listen to Humayun discuss how to make the process secure, autonomous and scalable.

The real challenge isn’t building agents. It’s discovering them. Trusting them. Coordinating them. Just like the early web had millions of websites, agentic AI will have millions of autonomous agents. Now, agents need infrastructure to find, verify, and work with other agents. Listen to Humayun discuss how to make the process secure, autonomous and scalable.

Fetch.ai

12,072 Aufrufe • vor 8 Monaten

🚀We are excited to introduce the Tool Decathlon (Toolathlon), a benchmark for language agents on diverse, complex, and realistic tool use. ⭐️32 applications and 600+ tools based on real-world software environments ⭐️Execution-based, reliable evaluation ⭐️Realistic, covering daily and professional scenarios Toolathlon reveals significant shortcomings of SOTA LLMs in realistic tool-use tasks, where Claude Sonnet 4.5 achieves 38.6% success rate. It also indicates a clear gap between open-source and leading proprietary models. Check our blog: Github: Paper: 🧵⬇️

🚀We are excited to introduce the Tool Decathlon (Toolathlon), a benchmark for language agents on diverse, complex, and realistic tool use. ⭐️32 applications and 600+ tools based on real-world software environments ⭐️Execution-based, reliable evaluation ⭐️Realistic, covering daily and professional scenarios Toolathlon reveals significant shortcomings of SOTA LLMs in realistic tool-use tasks, where Claude Sonnet 4.5 achieves 38.6% success rate. It also indicates a clear gap between open-source and leading proprietary models. Check our blog: Github: Paper: 🧵⬇️

Junxian He

44,505 Aufrufe • vor 1 Jahr

Introducing Asta—our bold initiative to accelerate science with trustworthy, capable agents, benchmarks, & developer resources that bring clarity to the landscape of scientific AI + agents. 🧵

Introducing Asta—our bold initiative to accelerate science with trustworthy, capable agents, benchmarks, & developer resources that bring clarity to the landscape of scientific AI + agents. 🧵

Ai2

296,612 Aufrufe • vor 1 Jahr

Introducing Ego2Web from Google DeepMind and UNC Chapel Hill, accepted to #CVPR2026. AI agents can browse the web. But can they act based on what you see? Existing benchmarks focus only on web interaction while ignoring the real world. Ego2Web bridges egocentric video perception and web execution, enabling agents that can see through first-person video, understand real-world context, and take actions on the web grounded in the egocentric video. This opens a path toward AI assistants that operate seamlessly across physical and digital environments. We hope Ego2Web serves as an important step for building more capable, perception-driven agents. 🧵👇

Introducing Ego2Web from Google DeepMind and UNC Chapel Hill, accepted to #CVPR2026. AI agents can browse the web. But can they act based on what you see? Existing benchmarks focus only on web interaction while ignoring the real world. Ego2Web bridges egocentric video perception and web execution, enabling agents that can see through first-person video, understand real-world context, and take actions on the web grounded in the egocentric video. This opens a path toward AI assistants that operate seamlessly across physical and digital environments. We hope Ego2Web serves as an important step for building more capable, perception-driven agents. 🧵👇

Shoubin Yu

46,624 Aufrufe • vor 7 Monaten

Introducing EdgeBench, a benchmark designed to study how agents learn from environments over at least 12~72-hour runs. We find that performance follows a log-sigmoid function of environment interaction time with high precision. EdgeBench is built with three ingredients: - 🌍 Real & Diverse: 134 real-world tasks across 6 task categories, spanning scientific problems, professional knowledge work, software engineering, optimization, formal math, and games. - ⏳ Ultra-Long-Horizon: Each task supports 12–72 hours of agent work. Recorded human effort averages 57.2 hours. - 🔁 Informative Feedback: Agents receive real-world feedback for continuous improvement. After 38,000 hours of agent runs on EdgeBench, a scaling law for learning from environments emerges: - 📈 As agents interact with task environments over time, their aggregate performance is precisely fit by a log-sigmoid function. - 🧠 This phenomenon can be explained by an elegant theory of graph exploration. We are releasing an initial 51 of the 134 tasks, together with the full evaluation framework, to help advance long-horizon agent research. Check our blog & paper for more findings! Blog Paper GitHub Dataset Details below 👇🧵

Introducing EdgeBench, a benchmark designed to study how agents learn from environments over at least 12~72-hour runs. We find that performance follows a log-sigmoid function of environment interaction time with high precision. EdgeBench is built with three ingredients: - 🌍 Real & Diverse: 134 real-world tasks across 6 task categories, spanning scientific problems, professional knowledge work, software engineering, optimization, formal math, and games. - ⏳ Ultra-Long-Horizon: Each task supports 12–72 hours of agent work. Recorded human effort averages 57.2 hours. - 🔁 Informative Feedback: Agents receive real-world feedback for continuous improvement. After 38,000 hours of agent runs on EdgeBench, a scaling law for learning from environments emerges: - 📈 As agents interact with task environments over time, their aggregate performance is precisely fit by a log-sigmoid function. - 🧠 This phenomenon can be explained by an elegant theory of graph exploration. We are releasing an initial 51 of the 134 tasks, together with the full evaluation framework, to help advance long-horizon agent research. Check our blog & paper for more findings! Blog Paper GitHub Dataset Details below 👇🧵

Deyao Zhu

406,543 Aufrufe • vor 4 Monaten