Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

How capable are web agents at solving knowledge work tasks? 🤔 Are LLMs up to the challenge? 🤖 Introducing WorkArena: a benchmark where agents meet the world 𝘸𝘪𝘭𝘥 web of enterprise software 🌐🖥️ Paper: Website: 🧵 1/7

Alexandre Lacoste

1,136 subscribers

24,550 просмотров • 2 лет назад •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

Комментарии: 18

Фото профиля Alexandre Lacoste

Alexandre Lacoste2 лет назад

WorkArena is built on the ServiceNow platform, which is used by most of the Fortune 500 companies. It contains 29 tasks, which encompass the most common ways workers interact with the software. For example, searching for information in a company knowledge base ⬇️ 2/7

Фото профиля Alexandre Lacoste

Alexandre Lacoste2 лет назад

Developing enterprise web-based agents is no small feat. Challenges include: 🌐 Navigating the 'World 𝘞𝘪𝘭𝘥 Web' of Enterprise UI with huge and non-standard HTML files (100k+ tokens) 💾 Memory management (for POMPDs) 📈 Planning 📚 Domain-specific knowledge 3/7

Фото профиля Alexandre Lacoste

Alexandre Lacoste2 лет назад

To help agents navigate this space we propose a new web environment with: Multimodal observations (AXtree, HTML, screenshots) ☑High-level actions (click, fill, etc.) ☑Chat interface for user interaction ☑Robustness to the world wild web (e.g., iFrames, shadow DOMs) 4/7

Фото профиля Alexandre Lacoste

Alexandre Lacoste2 лет назад

WorkArena poses a real challenge! Even with GPT4 and our top-performing configs, we're seeing only a 55% success rate. A SOTA OSS code LLM hasn’t had a single successful episode yet. ✨We're calling on the open-source community to help bridge this significant gap. 5/7

Фото профиля Alexandre Lacoste

Alexandre Lacoste2 лет назад

All our experiments are conducted using BrowserGym, our new conversational gym environment for web agents! 🌐 Unified testing ground for MiniWob, WebArena, WorkArena 📣 We welcome new benchmark contributions! 🚀 Try our demo on the open web: 6/7

Фото профиля Alexandre Lacoste

Alexandre Lacoste2 лет назад

Special thanks to @ServiceNowRSRCH for the support!🙏 Shoutout to all authors: @alexandredrouin, @maxime_gasse, @MassCaccia, @ILaradji, @ManuelDelVerme, Tom Marty, @LeoBoisvert, @Megh1211, @qcappart , @dvazquezcv, @NicolasChapados. 👏 Also, thanks to @Mila_Quebec! 🎉

Фото профиля Alexandre Lacoste

Alexandre Lacoste2 лет назад

@alexandredrouin @maxime_gasse @MassCaccia @ILaradji @ManuelDelVerme @LeoBoisvert @Megh1211 @qcappart @dvazquezcv @NicolasChapados @Mila_Quebec Thanks to @ThirdCore42 (Tom Marty).

Фото профиля Xing Han Lu

Xing Han Lu2 лет назад

Amazing (and colossal) work! but I wanna stop for a moment and say how cool the intro video is - especially with the zoom out!

Фото профиля Rajiv Shah

Rajiv Shah2 лет назад

Great work. It's nice to have data points on realistic tasks. This really helps level set expectations and know where we need to improve.

Фото профиля PG

PG2 лет назад

That is so cool! Félicitations!

Фото профиля Prem Kumar Aparanji 🤖👶🏼🐘

Prem Kumar Aparanji 🤖👶🏼🐘2 лет назад

Wow! Impressive work! And very true, inside the enterprise it is indeed a very wild web. BTW, would it not be better to just call the appropriate API of servicenow and accomplish these tasks? Is there any research on those lines too?

Фото профиля Frank Xu

Frank Xu2 лет назад

Looking real nice! Great environment platform especially

Фото профиля AlexConway.eth ⌬⃤⟁

AlexConway.eth ⌬⃤⟁2 лет назад

This is amazing but i can't wrap my head around how 2000 samples is a "large-scale" dataset?! 😅 Can you give an approximation on manhours required to produce this?

Фото профиля Alexandre Lacoste

Alexandre Lacoste2 лет назад

Lol. Large is always contextual :) @xhluca what were the resources involved in building weblinx?

Фото профиля Xing Han Lu

Xing Han Lu2 лет назад

@0x413x Yeah demonstrations usually take 15 to record and 15 for other things (upload, write summary, prepare, etc.) for 2 people. So ~60min per demonstration is around 2000h of work, without counting failed attempts and time to validate the demos afterwards.

Фото профиля AlexConway.eth ⌬⃤⟁

AlexConway.eth ⌬⃤⟁2 лет назад

@alex_lacoste_ Thanks Xing! 🙏 where can i learn more about this?

Фото профиля Xing Han Lu

Xing Han Lu2 лет назад

@alex_lacoste_ The paper contains all the detailed information: The appendix is pretty extensive but you can use the ToC to navigate (right after references).

Фото профиля AlexConway.eth ⌬⃤⟁

AlexConway.eth ⌬⃤⟁2 лет назад

@alex_lacoste_ Epic, thanks!

Похожие видео

Meet Embodied Web Agents that bridge physical-digital realms. Imagine embodied agents that can search for online recipes, shop for ingredients and cook for you. Embodied web agents search internet information for implementing real-world embodied tasks. All data, codes and web environments are available at Paper link:

Meet Embodied Web Agents that bridge physical-digital realms. Imagine embodied agents that can search for online recipes, shop for ingredients and cook for you. Embodied web agents search internet information for implementing real-world embodied tasks. All data, codes and web environments are available at Paper link:

Yining Hong

55,062 просмотров • 1 год назад

Meet Cabinet: Paper Clip + KB. for quite some time I've been thinking how LLMs are missing the knowledge base - where I can dump CSVs, PDFs, and most important - inline web app. running on Claude Code with agents with heartbeats and jobs

Meet Cabinet: Paper Clip + KB. for quite some time I've been thinking how LLMs are missing the knowledge base - where I can dump CSVs, PDFs, and most important - inline web app. running on Claude Code with agents with heartbeats and jobs

Hila Shmuel

524,927 просмотров • 6 месяцев назад

AI web agents like Operator and Anthropic’s Computer Use can operate a browser, but the LLMs inside are brittle, and you can’t trust what’s on the web. In this 🧵, I’ll show how adversaries can fool Anthropic’s web agent into sending phishing emails or revealing credit card info.

AI web agents like Operator and Anthropic’s Computer Use can operate a browser, but the LLMs inside are brittle, and you can’t trust what’s on the web. In this 🧵, I’ll show how adversaries can fool Anthropic’s web agent into sending phishing emails or revealing credit card info.

Micah Goldblum

42,969 просмотров • 1 год назад

AI agents are getting real jobs now?! Introducing a managed marketplace where companies post roles, and agents get to work. Powered by in collaboration with Amazon Web Services, Anthropic, Wix, and Mesh. There's an agent out there who is the perfect fit for your team! Come and find it 🤖

AI agents are getting real jobs now?! Introducing a managed marketplace where companies post roles, and agents get to work. Powered by in collaboration with Amazon Web Services, Anthropic, Wix, and Mesh. There's an agent out there who is the perfect fit for your team! Come and find it 🤖

monday.com

35,696 просмотров • 7 месяцев назад

🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. 🤔What's this all about? Why this benchmark? > Back in Nov 2023, when we released MMMU ( a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin ( which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.📸👀 > Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent ( showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages. 🌉 Bridging the Gap with VisualWebBench > To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites 🌐 across 12 domains 🏷️ and 87 sub-domains 🔍, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level 📊, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities 🧠💡. 😮 Surprising Findings > 🎉 Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro. > 🧠 Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs. > 🖼️ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent. > 🧱 Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements. 💡Fun Fact > Claude Sonnet is better than Opus on our benchmark :) 🎓 Conclusion > VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web. Kudos to the student leads Junpeng Liu Yifan Song and the team Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li! 👏 Check out more details in the Junpeng's thread👇

🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. 🤔What's this all about? Why this benchmark? > Back in Nov 2023, when we released MMMU ( a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin ( which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.📸👀 > Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent ( showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages. 🌉 Bridging the Gap with VisualWebBench > To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites 🌐 across 12 domains 🏷️ and 87 sub-domains 🔍, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level 📊, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities 🧠💡. 😮 Surprising Findings > 🎉 Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro. > 🧠 Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs. > 🖼️ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent. > 🧱 Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements. 💡Fun Fact > Claude Sonnet is better than Opus on our benchmark :) 🎓 Conclusion > VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web. Kudos to the student leads Junpeng Liu Yifan Song and the team Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li! 👏 Check out more details in the Junpeng's thread👇

Xiang Yue

56,697 просмотров • 2 лет назад

we are building at the Y Combinator web agents hackathon this weekend. introducing clawtrace: on-call security for openclaw web agents. openclaw is powerful. but it’s hard to fully trust what your agents are actually doing. that’s why we built clawtrace. clawtrace lets you: - know when your agents go rogue - detect anomalies in behavior - track sessions, tool calls, thinking states here’s how it works:

we are building at the Y Combinator web agents hackathon this weekend. introducing clawtrace: on-call security for openclaw web agents. openclaw is powerful. but it’s hard to fully trust what your agents are actually doing. that’s why we built clawtrace. clawtrace lets you: - know when your agents go rogue - detect anomalies in behavior - track sessions, tool calls, thinking states here’s how it works:

Divya Saini

22,323 просмотров • 7 месяцев назад

we are building at the Y Combinator web agents hackathon this weekend. introducing clawtrace: on-call security for openclaw web agents. openclaw is powerful. but it’s hard to fully trust what your agents are actually doing. that’s why we built clawtrace. clawtrace lets you: - know when your agents go rogue - detect anomalies in behavior - track sessions, tool calls, thinking states here’s how it works:

we are building at the Y Combinator web agents hackathon this weekend. introducing clawtrace: on-call security for openclaw web agents. openclaw is powerful. but it’s hard to fully trust what your agents are actually doing. that’s why we built clawtrace. clawtrace lets you: - know when your agents go rogue - detect anomalies in behavior - track sessions, tool calls, thinking states here’s how it works:

Sri Laasya Nutheti

19,122 просмотров • 7 месяцев назад

NEW: DevRev raises a $100M Series A to build enterprise AI apps & agents - Valued at $1.1B 🦄 The core of their platform is an enterprise knowledge graph that powers AI apps & agents for customer service, product, software engineering, etc. Here's how devrev works:

NEW: DevRev raises a $100M Series A to build enterprise AI apps & agents - Valued at $1.1B 🦄 The core of their platform is an enterprise knowledge graph that powers AI apps & agents for customer service, product, software engineering, etc. Here's how devrev works:

Chief AI Officer

11,247 просмотров • 2 лет назад

🚀 Introducing Boson V3 - Decentralized AI Commerce Post Web commerce for humans and AI TL;DR Crypto x AI Agents -> Parabolic adoption = The end of the Web and Platforms Crypto needs AI to bridge complex UX AI agents need Crypto for trusted execution of transactions Discover how Boson V3 is the base layer for the Trillion-Dollar Post Web Commerce Era! 👇🧵

🚀 Introducing Boson V3 - Decentralized AI Commerce Post Web commerce for humans and AI TL;DR Crypto x AI Agents -> Parabolic adoption = The end of the Web and Platforms Crypto needs AI to bridge complex UX AI agents need Crypto for trusted execution of transactions Discover how Boson V3 is the base layer for the Trillion-Dollar Post Web Commerce Era! 👇🧵

Boson

2,294,157 просмотров • 1 год назад

AI agents are creating a new enterprise challenge: economics. At Dell Technologies Technologies World 2026, Ihab Tarazi of Dell and NVIDIA's Jason Hardy explained why the next phase of enterprise AI isn't about access to models, it's about controlling the cost of compute, storage, and the explosion of data agents create. The next wave of AI success will be defined by how organizations manage the economics behind agents at scale. $NVDA $DELL

AI agents are creating a new enterprise challenge: economics. At Dell Technologies Technologies World 2026, Ihab Tarazi of Dell and NVIDIA's Jason Hardy explained why the next phase of enterprise AI isn't about access to models, it's about controlling the cost of compute, storage, and the explosion of data agents create. The next wave of AI success will be defined by how organizations manage the economics behind agents at scale. $NVDA $DELL

Six Five Media

26,975 просмотров • 4 месяцев назад

Introducing 🦀 CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents 🦀 CRAB provides an end-to-end and easy-to-use framework to build multimodal agents, operate environments, and create benchmarks to evaluate them, featuring three key components: - 🔀 Cross-environment support - agents can operate tasks in 📱 Android and 💻 Ubuntu. - 🕸️ Graph evaluator - provides a fine-grain evaluation metric for agents. - 🤖 Task generation - composes subtasks to automatically generate tasks. By connecting all devices to agents, 🦀CRAB unlocks greater capabilities for human-like tasks than ever before. Use 🦀 CRAB to benchmark your multimodal agents! - 👨‍💻 Check out the repository: - 📝 Read the paper: - 🌐 Find out more via the project page: - 🐫 Join our community:

Introducing 🦀 CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents 🦀 CRAB provides an end-to-end and easy-to-use framework to build multimodal agents, operate environments, and create benchmarks to evaluate them, featuring three key components: - 🔀 Cross-environment support - agents can operate tasks in 📱 Android and 💻 Ubuntu. - 🕸️ Graph evaluator - provides a fine-grain evaluation metric for agents. - 🤖 Task generation - composes subtasks to automatically generate tasks. By connecting all devices to agents, 🦀CRAB unlocks greater capabilities for human-like tasks than ever before. Use 🦀 CRAB to benchmark your multimodal agents! - 👨‍💻 Check out the repository: - 📝 Read the paper: - 🌐 Find out more via the project page: - 🐫 Join our community:

CAMEL-AI.org

72,807 просмотров • 2 лет назад

The real challenge isn’t building agents. It’s discovering them. Trusting them. Coordinating them. Just like the early web had millions of websites, agentic AI will have millions of autonomous agents. Now, agents need infrastructure to find, verify, and work with other agents. Listen to Humayun discuss how to make the process secure, autonomous and scalable.

The real challenge isn’t building agents. It’s discovering them. Trusting them. Coordinating them. Just like the early web had millions of websites, agentic AI will have millions of autonomous agents. Now, agents need infrastructure to find, verify, and work with other agents. Listen to Humayun discuss how to make the process secure, autonomous and scalable.

Fetch.ai

12,072 просмотров • 8 месяцев назад

🚀We are excited to introduce the Tool Decathlon (Toolathlon), a benchmark for language agents on diverse, complex, and realistic tool use. ⭐️32 applications and 600+ tools based on real-world software environments ⭐️Execution-based, reliable evaluation ⭐️Realistic, covering daily and professional scenarios Toolathlon reveals significant shortcomings of SOTA LLMs in realistic tool-use tasks, where Claude Sonnet 4.5 achieves 38.6% success rate. It also indicates a clear gap between open-source and leading proprietary models. Check our blog: Github: Paper: 🧵⬇️

🚀We are excited to introduce the Tool Decathlon (Toolathlon), a benchmark for language agents on diverse, complex, and realistic tool use. ⭐️32 applications and 600+ tools based on real-world software environments ⭐️Execution-based, reliable evaluation ⭐️Realistic, covering daily and professional scenarios Toolathlon reveals significant shortcomings of SOTA LLMs in realistic tool-use tasks, where Claude Sonnet 4.5 achieves 38.6% success rate. It also indicates a clear gap between open-source and leading proprietary models. Check our blog: Github: Paper: 🧵⬇️

Junxian He

44,505 просмотров • 11 месяцев назад

Introducing Ego2Web from Google DeepMind and UNC Chapel Hill, accepted to #CVPR2026. AI agents can browse the web. But can they act based on what you see? Existing benchmarks focus only on web interaction while ignoring the real world. Ego2Web bridges egocentric video perception and web execution, enabling agents that can see through first-person video, understand real-world context, and take actions on the web grounded in the egocentric video. This opens a path toward AI assistants that operate seamlessly across physical and digital environments. We hope Ego2Web serves as an important step for building more capable, perception-driven agents. 🧵👇

Introducing Ego2Web from Google DeepMind and UNC Chapel Hill, accepted to #CVPR2026. AI agents can browse the web. But can they act based on what you see? Existing benchmarks focus only on web interaction while ignoring the real world. Ego2Web bridges egocentric video perception and web execution, enabling agents that can see through first-person video, understand real-world context, and take actions on the web grounded in the egocentric video. This opens a path toward AI assistants that operate seamlessly across physical and digital environments. We hope Ego2Web serves as an important step for building more capable, perception-driven agents. 🧵👇

Shoubin Yu @ COLM26

46,273 просмотров • 7 месяцев назад

Introducing EdgeBench, a benchmark designed to study how agents learn from environments over at least 12~72-hour runs. We find that performance follows a log-sigmoid function of environment interaction time with high precision. EdgeBench is built with three ingredients: - 🌍 Real & Diverse: 134 real-world tasks across 6 task categories, spanning scientific problems, professional knowledge work, software engineering, optimization, formal math, and games. - ⏳ Ultra-Long-Horizon: Each task supports 12–72 hours of agent work. Recorded human effort averages 57.2 hours. - 🔁 Informative Feedback: Agents receive real-world feedback for continuous improvement. After 38,000 hours of agent runs on EdgeBench, a scaling law for learning from environments emerges: - 📈 As agents interact with task environments over time, their aggregate performance is precisely fit by a log-sigmoid function. - 🧠 This phenomenon can be explained by an elegant theory of graph exploration. We are releasing an initial 51 of the 134 tasks, together with the full evaluation framework, to help advance long-horizon agent research. Check our blog & paper for more findings! Blog Paper GitHub Dataset Details below 👇🧵

Introducing EdgeBench, a benchmark designed to study how agents learn from environments over at least 12~72-hour runs. We find that performance follows a log-sigmoid function of environment interaction time with high precision. EdgeBench is built with three ingredients: - 🌍 Real & Diverse: 134 real-world tasks across 6 task categories, spanning scientific problems, professional knowledge work, software engineering, optimization, formal math, and games. - ⏳ Ultra-Long-Horizon: Each task supports 12–72 hours of agent work. Recorded human effort averages 57.2 hours. - 🔁 Informative Feedback: Agents receive real-world feedback for continuous improvement. After 38,000 hours of agent runs on EdgeBench, a scaling law for learning from environments emerges: - 📈 As agents interact with task environments over time, their aggregate performance is precisely fit by a log-sigmoid function. - 🧠 This phenomenon can be explained by an elegant theory of graph exploration. We are releasing an initial 51 of the 134 tasks, together with the full evaluation framework, to help advance long-horizon agent research. Check our blog & paper for more findings! Blog Paper GitHub Dataset Details below 👇🧵

Deyao Zhu

364,430 просмотров • 3 месяцев назад

.Aaron Levie says AI agents may take years to diffuse from engineering into the rest of knowledge work. "The models are really good at code, and the work is verifiable." "You have 5 or 10 things that make agents work in an enterprise context for engineering." "The users are less technical, the data is much more fragmented, the systems are much more legacy." "That's why it's going to be a number of years for this diffusion to roll from what we're seeing in Silicon Valley, into the rest of knowledge work."

.Aaron Levie says AI agents may take years to diffuse from engineering into the rest of knowledge work. "The models are really good at code, and the work is verifiable." "You have 5 or 10 things that make agents work in an enterprise context for engineering." "The users are less technical, the data is much more fragmented, the systems are much more legacy." "That's why it's going to be a number of years for this diffusion to roll from what we're seeing in Silicon Valley, into the rest of knowledge work."

MTS

13,046 просмотров • 6 месяцев назад

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.

Russ Salakhutdinov

22,518 просмотров • 5 месяцев назад

Sat down with Aaron Levie, CEO of Box, to discuss where AI agents are going, how agentic workflows are transforming software, why agents will require much more memory, and how token budgets for enterprises are evolving. Thank you Aaron Levie for taking the time! 00:00 - Intro 01:30 - Where are we in AI 06:25 - The Role of Agents in Software 11:40 - The Role of Memory for Agents 14:45 - TokenMaxxing 24:54 - Why Are There AI Doomers? 31:50 - Aaron's Take on Nvidia

Sat down with Aaron Levie, CEO of Box, to discuss where AI agents are going, how agentic workflows are transforming software, why agents will require much more memory, and how token budgets for enterprises are evolving. Thank you Aaron Levie for taking the time! 00:00 - Intro 01:30 - Where are we in AI 06:25 - The Role of Agents in Software 11:40 - The Role of Memory for Agents 14:45 - TokenMaxxing 24:54 - Why Are There AI Doomers? 31:50 - Aaron's Take on Nvidia

amit

96,221 просмотров • 3 месяцев назад

Meet Database Agents. A team of li’l librarians in your database — keeping it up to date automatically using context from the page, your workspace, and the web. No more “we’ll update the database later.” Starting to roll out on Biz + Enterprise.

Meet Database Agents. A team of li’l librarians in your database — keeping it up to date automatically using context from the page, your workspace, and the web. No more “we’ll update the database later.” Starting to roll out on Biz + Enterprise.

Notion

299,865 просмотров • 7 месяцев назад

I found an AI browser that is actually awesome FlowithOS is a web browser COMPLETELY powered by AI You give it tasks, the agents work for you in the background It's like an army of AI agents working for you 24/7 Here's a guide on how it works and how you can get started today

I found an AI browser that is actually awesome FlowithOS is a web browser COMPLETELY powered by AI You give it tasks, the agents work for you in the background It's like an army of AI agents working for you 24/7 Here's a guide on how it works and how you can get started today

Alex Finn

25,490 просмотров • 10 месяцев назад