Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. 🤔What's this all about? Why this benchmark? > Back in Nov 2023, when we released MMMU ( a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of...

xiangyue96's profile picture

Xiang Yue

6,579 subscribers

56,697 görüntüleme • 2 yıl önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

6 Yorum

Li Bo profil fotoğrafı
Li Bo2 yıl önce

Congrats! May I know if it's possible to PR this dataset into lmms-eval?

Xiang Yue@ICLR2025🇸🇬 profil fotoğrafı
Xiang Yue@ICLR2025🇸🇬2 yıl önce

Sure! Happy to do that!

Travala.com 🏨 ✈️ profil fotoğrafı
Travala.com 🏨 ✈️5 yıl önce

This one's for the Crypto Whales ⬇️ NEW exclusive & luxurious experiences have been added to Apply today & #TravelUnlikeAnyOther with your favorite cryptocurrency.

Homer Quan profil fotoğrafı
Homer Quan2 yıl önce

Nice work, this will be the foundational block of emerging “deep search” business model where Google is trying to pivot its search business into)

Janak Agrawal profil fotoğrafı
Janak Agrawal2 yıl önce

This benchmark is amazing!! Does this benchmark reference Gemini 1 Pro or Gemini 1.5 Pro?

Xiang Yue@ICLR2025🇸🇬 profil fotoğrafı
Xiang Yue@ICLR2025🇸🇬2 yıl önce

Thanks! We used Gemini 1 Pro.

Benzer Videolar

Working on multimodal instruction tuning and finding it hard to scale? Building Web/GUI agents but data is too narrow? Introducing 🚀MultiUI: 7.3M multimodal instructions from 1M webpage UIs, offering diverse data to boost text-rich visual understanding. Key takeaways: 🌟WebUI-trained models show major gains in visual web understanding and agent tasks. 💻 🌟Models also generalize well to non-UI tasks like DocVQA/OCR. 📄 How it works: We generate multimodal instructions with a text LLM using structured text from webpage accessibility trees. We then pair them with UI screenshots, to train multimodal models. Homepage: Paper: Dataset: Model: Congrats to the student lead Junpeng Liu and the team Tianyue Ou Yifan Song Yuxiao Qu Chenyan Xiong Wenhu Chen Graham Neubig ! More details are in the following threads ⬇️
1:19

Working on multimodal instruction tuning and finding it hard to scale? Building Web/GUI agents but data is too narrow? Introducing 🚀MultiUI: 7.3M multimodal instructions from 1M webpage UIs, offering diverse data to boost text-rich visual understanding. Key takeaways: 🌟WebUI-trained models show major gains in visual web understanding and agent tasks. 💻 🌟Models also generalize well to non-UI tasks like DocVQA/OCR. 📄 How it works: We generate multimodal instructions with a text LLM using structured text from webpage accessibility trees. We then pair them with UI screenshots, to train multimodal models. Homepage: Paper: Dataset: Model: Congrats to the student lead Junpeng Liu and the team Tianyue Ou Yifan Song Yuxiao Qu Chenyan Xiong Wenhu Chen Graham Neubig ! More details are in the following threads ⬇️

Xiang Yue

57,739 görüntüleme • 2 yıl önce

INTRODUCING Notte Building the agentic internet with the strongest web browser for LLM agents. We transform ANY webpage into structured text, enabling better web understanding and navigation. Plug any LLM to to build your own AI agent
0:39

INTRODUCING Notte Building the agentic internet with the strongest web browser for LLM agents. We transform ANY webpage into structured text, enabling better web understanding and navigation. Plug any LLM to to build your own AI agent

Notte

225,232 görüntüleme • 1 yıl önce

How capable are web agents at solving knowledge work tasks? 🤔 Are LLMs up to the challenge? 🤖 Introducing WorkArena: a benchmark where agents meet the world 𝘸𝘪𝘭𝘥 web of enterprise software 🌐🖥️ Paper: Website: 🧵 1/7
0:25

How capable are web agents at solving knowledge work tasks? 🤔 Are LLMs up to the challenge? 🤖 Introducing WorkArena: a benchmark where agents meet the world 𝘸𝘪𝘭𝘥 web of enterprise software 🌐🖥️ Paper: Website: 🧵 1/7

Alexandre Lacoste

24,550 görüntüleme • 2 yıl önce

Watch Gemini 3.7 Flash handle autonomous website optimization in Google Antigravity. Leveraging MCP servers, multimodal understanding, and long-horizon execution, the agent sets out to achieve a 100 across all Lighthouse metrics and modernize the site to the latest web standards.
2:17

Watch Gemini 3.7 Flash handle autonomous website optimization in Google Antigravity. Leveraging MCP servers, multimodal understanding, and long-horizon execution, the agent sets out to achieve a 100 across all Lighthouse metrics and modernize the site to the latest web standards.

Google Antigravity

100,350 görüntüleme • 2 ay önce

Introducing 🦀 CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents 🦀 CRAB provides an end-to-end and easy-to-use framework to build multimodal agents, operate environments, and create benchmarks to evaluate them, featuring three key components: - 🔀 Cross-environment support - agents can operate tasks in 📱 Android and 💻 Ubuntu. - 🕸️ Graph evaluator - provides a fine-grain evaluation metric for agents. - 🤖 Task generation - composes subtasks to automatically generate tasks. By connecting all devices to agents, 🦀CRAB unlocks greater capabilities for human-like tasks than ever before. Use 🦀 CRAB to benchmark your multimodal agents! - 👨‍💻 Check out the repository: - 📝 Read the paper: - 🌐 Find out more via the project page: - 🐫 Join our community:
1:05

Introducing 🦀 CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents 🦀 CRAB provides an end-to-end and easy-to-use framework to build multimodal agents, operate environments, and create benchmarks to evaluate them, featuring three key components: - 🔀 Cross-environment support - agents can operate tasks in 📱 Android and 💻 Ubuntu. - 🕸️ Graph evaluator - provides a fine-grain evaluation metric for agents. - 🤖 Task generation - composes subtasks to automatically generate tasks. By connecting all devices to agents, 🦀CRAB unlocks greater capabilities for human-like tasks than ever before. Use 🦀 CRAB to benchmark your multimodal agents! - 👨‍💻 Check out the repository: - 📝 Read the paper: - 🌐 Find out more via the project page: - 🐫 Join our community:

CAMEL-AI.org

72,807 görüntüleme • 2 yıl önce

The web was never meant to be flattened into text. Yet most web RAG systems start by parsing HTML --- a complex and lossy process. 🔥 Introducing PixelRAG: the first RAG system that retrieves and reads 30M+ web pages as pixels. Instead of extracting text, PixelRAG retrieves screenshots and lets a VLM read them directly. PixelRAG not only preserves visual information, but also outperforms text-based RAG on text-only QA benchmarks by +18.1%. Why? (1) HTML-to-text conversion often discards layout, structure, tables, and other useful signals. (2) We continued pretraining a VLM on web page screenshots and turned it into a surprisingly strong visual retriever. (3) Recent VLMs are remarkably good at understanding web pages, often with better accuracy and token efficiency than text-only pipelines. Takeaway: HTML parsing may be one of the biggest self-inflicted bottlenecks in web RAG. Demo below 👇 Code: Paper: Playground:
0:55

The web was never meant to be flattened into text. Yet most web RAG systems start by parsing HTML --- a complex and lossy process. 🔥 Introducing PixelRAG: the first RAG system that retrieves and reads 30M+ web pages as pixels. Instead of extracting text, PixelRAG retrieves screenshots and lets a VLM read them directly. PixelRAG not only preserves visual information, but also outperforms text-based RAG on text-only QA benchmarks by +18.1%. Why? (1) HTML-to-text conversion often discards layout, structure, tables, and other useful signals. (2) We continued pretraining a VLM on web page screenshots and turned it into a surprisingly strong visual retriever. (3) Recent VLMs are remarkably good at understanding web pages, often with better accuracy and token efficiency than text-only pipelines. Takeaway: HTML parsing may be one of the biggest self-inflicted bottlenecks in web RAG. Demo below 👇 Code: Paper: Playground:

Yichuan Wang

92,433 görüntüleme • 4 ay önce

New Short Course: Building AI Browser Agents! Learn how to build AI agents that interact and take actions on websites in this course, created in partnership with and taught by and @namangarg0, Co-founders of AGI Inc. AI browser agents can log into websites, fill out forms, click through web pages, or even place orders online for you. They use both visual information, like screenshots, and structural data, like the HTML or Document Object Model (DOM) of a web page, to reason and take action. With the complexity of webpages and multiple possible actions at each step, it can be challenging for an AI browser agent to complete an assigned task. Because these agents run long action sequences, a single error—like clicking the wrong button or misreading a field—can lead to unexpected outcomes or errors that compound over time. In this course, you'll understand how autonomous web agents work, their current limitations, and how AgentQ enables them to improve through self-correction. In detail, you'll: - Learn what web agents are, how they automate tasks online, their architecture, key components, limitations, and an overview of their decision-making strategies. - Build a web agent that can scrape website and return course recommendations in a structured output format. - Build an autonomous web agent that can execute multiple tasks, such as finding and summarizing webpages, filling out a form, and signing up for a newsletter. - Explore AgentQ, a framework that enables agents to self-correct by combining Monte Carlo Tree Search (MCTS), a self-critique mechanism for continuous improvement, and Direct Preference Optimization (DPO). - Deep dive into MCTS, learn how it finds an effective path, illustrated by an example of Gridworld animation, and use AgentQ to complete web tasks. - Understand AI agents' current state and future directions—including key factors shaping their evolution, such as hardware, algorithm innovation, and data availability. By the end of this course, you will have hands-on experience building browser agents and a deeper understanding of how to make them more robust and reliable. Please sign up here:
2:06

New Short Course: Building AI Browser Agents! Learn how to build AI agents that interact and take actions on websites in this course, created in partnership with and taught by and @namangarg0, Co-founders of AGI Inc. AI browser agents can log into websites, fill out forms, click through web pages, or even place orders online for you. They use both visual information, like screenshots, and structural data, like the HTML or Document Object Model (DOM) of a web page, to reason and take action. With the complexity of webpages and multiple possible actions at each step, it can be challenging for an AI browser agent to complete an assigned task. Because these agents run long action sequences, a single error—like clicking the wrong button or misreading a field—can lead to unexpected outcomes or errors that compound over time. In this course, you'll understand how autonomous web agents work, their current limitations, and how AgentQ enables them to improve through self-correction. In detail, you'll: - Learn what web agents are, how they automate tasks online, their architecture, key components, limitations, and an overview of their decision-making strategies. - Build a web agent that can scrape website and return course recommendations in a structured output format. - Build an autonomous web agent that can execute multiple tasks, such as finding and summarizing webpages, filling out a form, and signing up for a newsletter. - Explore AgentQ, a framework that enables agents to self-correct by combining Monte Carlo Tree Search (MCTS), a self-critique mechanism for continuous improvement, and Direct Preference Optimization (DPO). - Deep dive into MCTS, learn how it finds an effective path, illustrated by an example of Gridworld animation, and use AgentQ to complete web tasks. - Understand AI agents' current state and future directions—including key factors shaping their evolution, such as hardware, algorithm innovation, and data availability. By the end of this course, you will have hands-on experience building browser agents and a deeper understanding of how to make them more robust and reliable. Please sign up here:

Andrew Ng

186,182 görüntüleme • 1 yıl önce

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.
1:12

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.

Russ Salakhutdinov

22,518 görüntüleme • 6 ay önce

The wait is over!🚀 We are now live with Pera Web Wallet! 🌍 The ultimate open-source web wallet for managing all your #Algorand accounts and connecting to #dApps on desktop, built for security and ease of use. 🔗 Learn more:
0:40

The wait is over!🚀 We are now live with Pera Web Wallet! 🌍 The ultimate open-source web wallet for managing all your #Algorand accounts and connecting to #dApps on desktop, built for security and ease of use. 🔗 Learn more:

pera.algo

97,180 görüntüleme • 3 yıl önce

Achieve SOTA on multimodal, complex tasks with minimal supervision! Introducing In-Context Abstraction Learning (ICAL): a method that extracts insights from visual demos and human feedback to continually teach multimodal LLM agents 🚀🤖🧠 ICAL learns from sub-optimal demonstrations and human feedback by transforming noisy trajectories into optimized, annotated examples that improve multimodal decision-making in new domains. ICAL is the first to show state-of-the-art performance across three challenging domains: we obtain a 34% relative improvement on TEACh household robot instruction following from dialogue, a 59% relative improvement on Visual Web Arena for autonomous visual web agents, and beat few-shot GPT-4V and supervised baselines on Ego4D action forecasting.
0:30

Achieve SOTA on multimodal, complex tasks with minimal supervision! Introducing In-Context Abstraction Learning (ICAL): a method that extracts insights from visual demos and human feedback to continually teach multimodal LLM agents 🚀🤖🧠 ICAL learns from sub-optimal demonstrations and human feedback by transforming noisy trajectories into optimized, annotated examples that improve multimodal decision-making in new domains. ICAL is the first to show state-of-the-art performance across three challenging domains: we obtain a 34% relative improvement on TEACh household robot instruction following from dialogue, a 59% relative improvement on Visual Web Arena for autonomous visual web agents, and beat few-shot GPT-4V and supervised baselines on Ego4D action forecasting.

Gabriel Sarch

19,340 görüntüleme • 2 yıl önce

Introducing Tidewave Web for Rails and Phoenix: a coding agent that runs in the browser alongside your web application, with full page and code context. Tidewave deeply integrates with your stack, from the database to the UI, making AI development more seamless and efficient.
1:12

Introducing Tidewave Web for Rails and Phoenix: a coding agent that runs in the browser alongside your web application, with full page and code context. Tidewave deeply integrates with your stack, from the database to the UI, making AI development more seamless and efficient.

José Valim

101,322 görüntüleme • 1 yıl önce

Papers With Code SOTA is back 🚀 We used Gemini 3 to process millions of charts and tables across arXiv + the web to surface state-of-the-art AI research in reasoning, computer-use, OCR, and more Track the best-performing methods and see which benchmarks are getting adopted
0:36

Papers With Code SOTA is back 🚀 We used Gemini 3 to process millions of charts and tables across arXiv + the web to surface state-of-the-art AI research in reasoning, computer-use, OCR, and more Track the best-performing methods and see which benchmarks are getting adopted

alphaXiv

17,856 görüntüleme • 11 ay önce

LLaVA-3D A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D-awareness for 3D scene understanding has been hindered by the lack of large-scale 3D vision-language datasets and powerful 3D encoders. In this paper, we introduce a simple yet effective framework called LLaVA-3D. Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D efficiently adapts LLaVA for 3D scene understanding without compromising 2D understanding capabilities. To achieve this, we employ a simple yet effective representation, 3D Patch, which connects 2D CLIP patch features with their corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs and employing joint 2D and 3D vision-language instruction tuning, we establish a unified architecture for both 2D image understanding and 3D scene understanding. Experimental results show that LLaVA-3D converges 3.5x faster than existing 3D LMMs when trained on 3D vision-language datasets. Moreover, LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks but also maintains comparable 2D image understanding and vision-language conversation capabilities with LLaVA.
0:31

LLaVA-3D A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D-awareness for 3D scene understanding has been hindered by the lack of large-scale 3D vision-language datasets and powerful 3D encoders. In this paper, we introduce a simple yet effective framework called LLaVA-3D. Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D efficiently adapts LLaVA for 3D scene understanding without compromising 2D understanding capabilities. To achieve this, we employ a simple yet effective representation, 3D Patch, which connects 2D CLIP patch features with their corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs and employing joint 2D and 3D vision-language instruction tuning, we establish a unified architecture for both 2D image understanding and 3D scene understanding. Experimental results show that LLaVA-3D converges 3.5x faster than existing 3D LMMs when trained on 3D vision-language datasets. Moreover, LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks but also maintains comparable 2D image understanding and vision-language conversation capabilities with LLaVA.

AK

41,747 görüntüleme • 2 yıl önce

LLM agents have demonstrated promise in their ability to automate computer tasks, but face challenges with multi-step reasoning and planning. Towards addressing this, we propose an inference-time tree search algorithm for LLM agents to explicitly perform exploration and multi-step planning in interactive web environments. It is the first tree search algorithm for LLM agents that shows effectiveness on realistic and complex web environments: on the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%.
0:20

LLM agents have demonstrated promise in their ability to automate computer tasks, but face challenges with multi-step reasoning and planning. Towards addressing this, we propose an inference-time tree search algorithm for LLM agents to explicitly perform exploration and multi-step planning in interactive web environments. It is the first tree search algorithm for LLM agents that shows effectiveness on realistic and complex web environments: on the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%.

Jing Yu Koh

124,894 görüntüleme • 2 yıl önce

AI agents are becoming increasingly capable of taking actions beyond the chat window and interacting with tools in the real world, such as web browsers. We explored this capability by having an AI agent present one of our interactive web-based demos. In this video, it's presenting our election interference demo. This agent is a Cursor coding agent and is equipped with tools to read and interact with the browser (using Playwright) and perform text-to-speech (with ElevenLabs), in addition to the standard Cursor tools.
3:23

AI agents are becoming increasingly capable of taking actions beyond the chat window and interacting with tools in the real world, such as web browsers. We explored this capability by having an AI agent present one of our interactive web-based demos. In this video, it's presenting our election interference demo. This agent is a Cursor coding agent and is equipped with tools to read and interact with the browser (using Playwright) and perform text-to-speech (with ElevenLabs), in addition to the standard Cursor tools.

CivAI

18,451 görüntüleme • 11 ay önce

Today we are announcing Keenable AI, an AI-native index of the best human knowledge we have, starting with the open web. AI seems to know just about everything until you ask it about something you know deeply. The answer isn’t wrong, but it's just very average. We started Keenable to fix this problem: every model and every agent should be able to query, reason over, and continuously learn from the living web. Backed by a $26M Seed from Accel and Conviction. Built by the team that took on Google at Yandex Search, with researchers and engineers from Amazon AGI, X, and Perplexity. The Web Search API and Web Query Language for AI are live now. Together they make the web queryable at AI scale. Both are free until end of September. Seven days of benchmarks, stories, and launches ahead. Let your agents search.
1:08

Today we are announcing Keenable AI, an AI-native index of the best human knowledge we have, starting with the open web. AI seems to know just about everything until you ask it about something you know deeply. The answer isn’t wrong, but it's just very average. We started Keenable to fix this problem: every model and every agent should be able to query, reason over, and continuously learn from the living web. Backed by a $26M Seed from Accel and Conviction. Built by the team that took on Google at Yandex Search, with researchers and engineers from Amazon AGI, X, and Perplexity. The Web Search API and Web Query Language for AI are live now. Together they make the web queryable at AI scale. Both are free until end of September. Seven days of benchmarks, stories, and launches ahead. Let your agents search.

Andrey Styskin

554,978 görüntüleme • 2 ay önce

Agents may need to read documents, interpret charts, and search the web to complete a task. Nemotron 3.5 Super VL brings text and visual understanding into one model, reducing the need to pass context between separate text and vision models.
3:01

Agents may need to read documents, interpret charts, and search the web to complete a task. Nemotron 3.5 Super VL brings text and visual understanding into one model, reducing the need to pass context between separate text and vision models.

NVIDIA AI

26,250 görüntüleme • 19 gün önce

Sierra co-founder Bret Taylor says the era of web apps is over: "We made these databases, these systems of record in the 80s and 90s. And then as the web browser came out, we said, 'We're going to give you some forms and fields in the web browser to manipulate these systems of record.'" "Now we think, in the future, most people are just going to be talking to AI agents, and they're just going to be performing those actions on your behalf."
2:11

Sierra co-founder Bret Taylor says the era of web apps is over: "We made these databases, these systems of record in the 80s and 90s. And then as the web browser came out, we said, 'We're going to give you some forms and fields in the web browser to manipulate these systems of record.'" "Now we think, in the future, most people are just going to be talking to AI agents, and they're just going to be performing those actions on your behalf."

TBPN

94,265 görüntüleme • 7 ay önce

😈 Today, we introduce WebGym, the largest-to-date open-source RL environment for web agent training that contains 300k tasks and a rollout framework optimized specifically for web environments' rollout speed. We reveal the effects of essential scaling directions we observe with WebGym. 1/n
0:26

😈 Today, we introduce WebGym, the largest-to-date open-source RL environment for web agent training that contains 300k tasks and a rollout framework optimized specifically for web environments' rollout speed. We reveal the effects of essential scaling directions we observe with WebGym. 1/n

Jack Bai

45,194 görüntüleme • 9 ay önce