Загрузка видео...

Не удалось загрузить видео

На главную

Introducing Ego2Web from Google DeepMind and UNC Chapel Hill, accepted to #CVPR2026. AI agents can browse the web. But can they act based on what you see? Existing benchmarks focus only on web interaction while ignoring the real world. Ego2Web bridges egocentric video perception and web execution, enabling agents...

shoubin621's profile picture

Shoubin Yu

1,006 subscribers

45,924 просмотров • 5 месяцев назад •via X (Twitter)

#CVPR2026
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 8

Фото профиля Shoubin Yu
Shoubin Yu5 месяцев назад

Why do we need Ego2Web? Today’s benchmarks split into two worlds: • Egocentric video → real-world perception & reasoning • Web agents → perception + action, but only in digital environments But real tasks require both seeing something and acting on it. Existing benchmarks miss this connection.

Фото профиля Shoubin Yu
Shoubin Yu5 месяцев назад

What is the Ego2Web task? Given an egocentric video and a natural language instruction. An agent must: 1. Ground visual cues from the video 2. Plan actions 3. Execute steps on real websites Tasks span diverse domains: E-commerce (find/buy items you saw), Media retrieval (find related videos), Knowledge lookup (identify & query entities), Maps/local (locate places from visual cues) Success = completing the web task based on what was seen

Фото профиля Shoubin Yu
Shoubin Yu5 месяцев назад

How do we build and evaluate Ego2Web? We use a hybrid human-model pipeline: • MLLMs → dense video captions → “video profiles” • LLMs → generate grounded web tasks • Humans → verify grounding, feasibility, quality For evaluation, we introduce Ego2WebJudge, an LLM-as-a-Judge that checks if actions align with visual evidence (~84% human agreement).

Фото профиля Shoubin Yu
Shoubin Yu5 месяцев назад

How do current agents perform? We evaluate leading systems (GPT-5.4, Claude, Gemini-based agents, etc.) We find that even the best agent reaches only ~58% success rate, and there is a large gap for improvements, demonstrating that current agents struggle to connect perception and action.

Фото профиля Shoubin Yu
Shoubin Yu5 месяцев назад

What’s hard about Ego2Web? Error analysis reveals key bottlenecks: • 36% object misidentification • 18% temporal misunderstanding • 16% failure to retrieve the correct info • 12% Cross-modal alignment remains fragile Also, we find Raw video >> captions >> no vision, which means perception quality directly drives success. This is still an open problem.

Фото профиля Shoubin Yu
Shoubin Yu5 месяцев назад

Awesome collaboration with @shulindt @AntoineYang2 @Francis_YAO_ Srinivas Sunkara, Maria Wang, @jdchen @mohitban47 @BoqingGo @unc_ai_group @unccs @GoogleDeepMind Check the full paper for more details! ArXiv: Code: Benchmark: Webpage: @huggingface page:

Фото профиля Erika S
Erika S5 месяцев назад

Admittedly I'm not an expert on web agents, but I'm 90% sure this is the missing link. Does it handle German recipe sites with their 47 cookie banners though?

Фото профиля Cross
Cross5 месяцев назад

Connecting egocentric vision with web execution feels like a missing piece. How robust is it in noisy real-world scenarios?

Похожие видео

Your agents can now browse and take actions on the web, just like you can. This lets agents work with sites that don’t have APIs and still complete tasks end to end. Plus, each interaction is streamed live to Studio.
0:33

Your agents can now browse and take actions on the web, just like you can. This lets agents work with sites that don’t have APIs and still complete tasks end to end. Plus, each interaction is streamed live to Studio.

Mastra

16,848 просмотров • 4 месяцев назад

AI web agents like Operator and Anthropic’s Computer Use can operate a browser, but the LLMs inside are brittle, and you can’t trust what’s on the web. In this 🧵, I’ll show how adversaries can fool Anthropic’s web agent into sending phishing emails or revealing credit card info.
0:49

AI web agents like Operator and Anthropic’s Computer Use can operate a browser, but the LLMs inside are brittle, and you can’t trust what’s on the web. In this 🧵, I’ll show how adversaries can fool Anthropic’s web agent into sending phishing emails or revealing credit card info.

Micah Goldblum

42,969 просмотров • 1 год назад

Weekends are for research. Inside OptimAI labs, we’ve been pushing agents beyond the screen. A first glimpse of OptimAI Prime. Not just agents that think, but agents that operate. Learning through reinforcement, coordinating across the network, and executing in real-world environments. From digital workflows → to physical actions. This is the next step for the agent-native internet.🤖 Still early. But already in motion.
1:21

Weekends are for research. Inside OptimAI labs, we’ve been pushing agents beyond the screen. A first glimpse of OptimAI Prime. Not just agents that think, but agents that operate. Learning through reinforcement, coordinating across the network, and executing in real-world environments. From digital workflows → to physical actions. This is the next step for the agent-native internet.🤖 Still early. But already in motion.

OptimAI Network

20,864 просмотров • 4 месяцев назад

New Short Course: Building AI Browser Agents! Learn how to build AI agents that interact and take actions on websites in this course, created in partnership with and taught by and @namangarg0, Co-founders of AGI Inc. AI browser agents can log into websites, fill out forms, click through web pages, or even place orders online for you. They use both visual information, like screenshots, and structural data, like the HTML or Document Object Model (DOM) of a web page, to reason and take action. With the complexity of webpages and multiple possible actions at each step, it can be challenging for an AI browser agent to complete an assigned task. Because these agents run long action sequences, a single error—like clicking the wrong button or misreading a field—can lead to unexpected outcomes or errors that compound over time. In this course, you'll understand how autonomous web agents work, their current limitations, and how AgentQ enables them to improve through self-correction. In detail, you'll: - Learn what web agents are, how they automate tasks online, their architecture, key components, limitations, and an overview of their decision-making strategies. - Build a web agent that can scrape website and return course recommendations in a structured output format. - Build an autonomous web agent that can execute multiple tasks, such as finding and summarizing webpages, filling out a form, and signing up for a newsletter. - Explore AgentQ, a framework that enables agents to self-correct by combining Monte Carlo Tree Search (MCTS), a self-critique mechanism for continuous improvement, and Direct Preference Optimization (DPO). - Deep dive into MCTS, learn how it finds an effective path, illustrated by an example of Gridworld animation, and use AgentQ to complete web tasks. - Understand AI agents' current state and future directions—including key factors shaping their evolution, such as hardware, algorithm innovation, and data availability. By the end of this course, you will have hands-on experience building browser agents and a deeper understanding of how to make them more robust and reliable. Please sign up here:
2:06

New Short Course: Building AI Browser Agents! Learn how to build AI agents that interact and take actions on websites in this course, created in partnership with and taught by and @namangarg0, Co-founders of AGI Inc. AI browser agents can log into websites, fill out forms, click through web pages, or even place orders online for you. They use both visual information, like screenshots, and structural data, like the HTML or Document Object Model (DOM) of a web page, to reason and take action. With the complexity of webpages and multiple possible actions at each step, it can be challenging for an AI browser agent to complete an assigned task. Because these agents run long action sequences, a single error—like clicking the wrong button or misreading a field—can lead to unexpected outcomes or errors that compound over time. In this course, you'll understand how autonomous web agents work, their current limitations, and how AgentQ enables them to improve through self-correction. In detail, you'll: - Learn what web agents are, how they automate tasks online, their architecture, key components, limitations, and an overview of their decision-making strategies. - Build a web agent that can scrape website and return course recommendations in a structured output format. - Build an autonomous web agent that can execute multiple tasks, such as finding and summarizing webpages, filling out a form, and signing up for a newsletter. - Explore AgentQ, a framework that enables agents to self-correct by combining Monte Carlo Tree Search (MCTS), a self-critique mechanism for continuous improvement, and Direct Preference Optimization (DPO). - Deep dive into MCTS, learn how it finds an effective path, illustrated by an example of Gridworld animation, and use AgentQ to complete web tasks. - Understand AI agents' current state and future directions—including key factors shaping their evolution, such as hardware, algorithm innovation, and data availability. By the end of this course, you will have hands-on experience building browser agents and a deeper understanding of how to make them more robust and reliable. Please sign up here:

Andrew Ng

186,182 просмотров • 1 год назад

Introducing Globodex: the intelligence layer for how people and AI agents understand the world. Globodex watches the world in real time, connects events to what matters to your business, and delivers verified intelligence you can act on.
3:13

Introducing Globodex: the intelligence layer for how people and AI agents understand the world. Globodex watches the world in real time, connects events to what matters to your business, and delivers verified intelligence you can act on.

Channel 1

378,563 просмотров • 1 месяц назад

Execution is where AI becomes most useful. With OptimAI Claw, tasks don’t stop at answers; they carry through into real actions across the web. 🦞 Another showcase: “Hey OptimAI, open YouTube, find a trending Korean music video, analyze and if it meets high engagement criteria, like the video and leave a relevant comment on my behalf.” OptimAI Claw navigates, evaluates, and executes the action locally on your device. This isn’t about automation scripts. It’s about agents that can interpret intent and carry it through across real environments. As these capabilities expand, routine digital interactions start to disappear; and execution becomes ambient.
1:08

Execution is where AI becomes most useful. With OptimAI Claw, tasks don’t stop at answers; they carry through into real actions across the web. 🦞 Another showcase: “Hey OptimAI, open YouTube, find a trending Korean music video, analyze and if it meets high engagement criteria, like the video and leave a relevant comment on my behalf.” OptimAI Claw navigates, evaluates, and executes the action locally on your device. This isn’t about automation scripts. It’s about agents that can interpret intent and carry it through across real environments. As these capabilities expand, routine digital interactions start to disappear; and execution becomes ambient.

OptimAI Network

15,443 просмотров • 5 месяцев назад

Your agents suck when using the web because 85% of it doesn't have an API. Browserbase gives them everything they need to do work online. Leading AI companies like Ramp, Lovable, and Clay trust us to power agents that do real work on behalf of real people. With a single API key, your agent gets everything it needs to navigate the wild web: browsers, search, fetch, identity, a sandbox runtime, and model gateway. Stop waiting on integrations, build agents that can browse and interact with the web just like humans.
1:06

Your agents suck when using the web because 85% of it doesn't have an API. Browserbase gives them everything they need to do work online. Leading AI companies like Ramp, Lovable, and Clay trust us to power agents that do real work on behalf of real people. With a single API key, your agent gets everything it needs to navigate the wild web: browsers, search, fetch, identity, a sandbox runtime, and model gateway. Stop waiting on integrations, build agents that can browse and interact with the web just like humans.

Paul Klein IV

1,730,259 просмотров • 5 месяцев назад

🚨 THIS IS ACTUALLY INSANE Your AI agent can have access to the web. But if it can't reliably read what’s actually on the page, that access is almost useless. We looked at Firecrawl as the web layer for AI agents and the numbers are hard to ignore. The setup is simple: Give it a URL, search query, or website. Firecrawl handles the ugly part — crawling, scraping, rendering, extracting, and turning web content into something an AI model can actually use. The headline numbers: → 173,000+ GitHub stars → Search, scrape and interact with the web at scale → Supports web pages, PDFs, DOCX and other content → Structured data extraction for AI workflows → MCP support for connecting it directly to AI agents The workflow looks like this: Search → Scrape → Crawl → Extract → Feed the agent Three things stand out: 1. Scraping becomes an infrastructure layer Instead of maintaining your own pile of HTTP clients, parsers, browser automation and retry logic, you can treat web access as an API. 2. Agents get more than raw HTML The goal isn't just downloading a webpage. It's turning messy web content into clean context that an LLM can reason over. 3. The same layer works across different agent workflows Research agents. RAG pipelines. AI search. Competitive intelligence. Web-data extraction. The interesting shift: AI agents don't just need better models. They need better access to the information those models are supposed to reason about. Firecrawl is building that layer. Save this repo.
0:28

🚨 THIS IS ACTUALLY INSANE Your AI agent can have access to the web. But if it can't reliably read what’s actually on the page, that access is almost useless. We looked at Firecrawl as the web layer for AI agents and the numbers are hard to ignore. The setup is simple: Give it a URL, search query, or website. Firecrawl handles the ugly part — crawling, scraping, rendering, extracting, and turning web content into something an AI model can actually use. The headline numbers: → 173,000+ GitHub stars → Search, scrape and interact with the web at scale → Supports web pages, PDFs, DOCX and other content → Structured data extraction for AI workflows → MCP support for connecting it directly to AI agents The workflow looks like this: Search → Scrape → Crawl → Extract → Feed the agent Three things stand out: 1. Scraping becomes an infrastructure layer Instead of maintaining your own pile of HTTP clients, parsers, browser automation and retry logic, you can treat web access as an API. 2. Agents get more than raw HTML The goal isn't just downloading a webpage. It's turning messy web content into clean context that an LLM can reason over. 3. The same layer works across different agent workflows Research agents. RAG pipelines. AI search. Competitive intelligence. Web-data extraction. The interesting shift: AI agents don't just need better models. They need better access to the information those models are supposed to reason about. Firecrawl is building that layer. Save this repo.

Vikas gupta

18,053 просмотров • 12 дней назад

The coming merger of AI agents with the physical world. @iddris, founder/CEO of Open Matter and I have a wide-ranging conversation about how the next platform shift may not be just AI chat, robotics, smart glasses, or autonomous cars separately. It may be the connective tissue between all of them. This video teaches viewers that the next wave of AI agents will not just operate inside apps, browsers, calendars, and spreadsheets. The real frontier is agents that can understand, coordinate, and act in the physical world. Iddris explains a vision for “agentic infrastructure for atoms”: systems that let AI agents manage physical objects, deliveries, storage, location, identity, logistics, robots, vehicles, and real-world context.
1:31:59

The coming merger of AI agents with the physical world. @iddris, founder/CEO of Open Matter and I have a wide-ranging conversation about how the next platform shift may not be just AI chat, robotics, smart glasses, or autonomous cars separately. It may be the connective tissue between all of them. This video teaches viewers that the next wave of AI agents will not just operate inside apps, browsers, calendars, and spreadsheets. The real frontier is agents that can understand, coordinate, and act in the physical world. Iddris explains a vision for “agentic infrastructure for atoms”: systems that let AI agents manage physical objects, deliveries, storage, location, identity, logistics, robots, vehicles, and real-world context.

Robert Scoble

17,484 просмотров • 4 месяцев назад

at Niantic Spatial 🌎, we're building the real-world model for physical AI. we believe the next generation of AI will move beyond the screen, into real-world environments where some of the hardest problems need to be solved. with high-fidelity gaussian splats as the foundation – robots, AI agents, & autonomous systems can see, localize, understand, & operate in the environments that matter most.
0:48

at Niantic Spatial 🌎, we're building the real-world model for physical AI. we believe the next generation of AI will move beyond the screen, into real-world environments where some of the hardest problems need to be solved. with high-fidelity gaussian splats as the foundation – robots, AI agents, & autonomous systems can see, localize, understand, & operate in the environments that matter most.

asim ᯅ

42,375 просмотров • 4 месяцев назад

Introducing Toggle for OpenClaw🦞 For the first time ever your Agent can see everything you do on the web. This is real-time context on your work day. Toggle AI analyses all browser actions, clicks, work, meetings and documents -> into AI context of what Projects / Tasks / Tools you are working on. Check it out👇
0:26

Introducing Toggle for OpenClaw🦞 For the first time ever your Agent can see everything you do on the web. This is real-time context on your work day. Toggle AI analyses all browser actions, clicks, work, meetings and documents -> into AI context of what Projects / Tasks / Tools you are working on. Check it out👇

PeLpA

52,426 просмотров • 6 месяцев назад

Information agents in Search are now available globally for Google AI Ultra subscribers. They can keep you updated on topics, tasks or projects you care about — and send you intelligent, comprehensive updates with links across the web so you can take action. #GoogleIO
1:03

Information agents in Search are now available globally for Google AI Ultra subscribers. They can keep you updated on topics, tasks or projects you care about — and send you intelligent, comprehensive updates with links across the web so you can take action. #GoogleIO

Google

98,836 просмотров • 3 месяцев назад

AI agents are becoming increasingly capable of taking actions beyond the chat window and interacting with tools in the real world, such as web browsers. We explored this capability by having an AI agent present one of our interactive web-based demos. In this video, it's presenting our election interference demo. This agent is a Cursor coding agent and is equipped with tools to read and interact with the browser (using Playwright) and perform text-to-speech (with ElevenLabs), in addition to the standard Cursor tools.
3:23

AI agents are becoming increasingly capable of taking actions beyond the chat window and interacting with tools in the real world, such as web browsers. We explored this capability by having an AI agent present one of our interactive web-based demos. In this video, it's presenting our election interference demo. This agent is a Cursor coding agent and is equipped with tools to read and interact with the browser (using Playwright) and perform text-to-speech (with ElevenLabs), in addition to the standard Cursor tools.

CivAI

18,428 просмотров • 9 месяцев назад

It can be challenging for AI agents to solve complex user intents by synthesizing signals like screenshots, the DOM, and the Accessibility Tree. Enter WebMCP, a proposed web standard that aims to expose structured tools for AI agents directly on existing websites, now in origin trial ⬇️
2:16

It can be challenging for AI agents to solve complex user intents by synthesizing signals like screenshots, the DOM, and the Accessibility Tree. Enter WebMCP, a proposed web standard that aims to expose structured tools for AI agents directly on existing websites, now in origin trial ⬇️

Chrome for Developers

15,078 просмотров • 2 месяцев назад

➤ AI agents are getting smarter, but intelligence alone isn't enough → The real challenge starts when agents need to move value, coordinate actions, and execute transactions across multiple chains without uncertainty. → That's where Wire Network stands out → Instead of focusing only on interoperability, Wire is building a Universal Transaction Layer designed for deterministic outcomes giving AI agents a reliable way to operate in a multi-chain world. → As the AI agent economy grows, infrastructure that guarantees execution and clear ownership could become just as important as the agents themselves. ➤ FT || Ken DiCross
1:44

➤ AI agents are getting smarter, but intelligence alone isn't enough → The real challenge starts when agents need to move value, coordinate actions, and execute transactions across multiple chains without uncertainty. → That's where Wire Network stands out → Instead of focusing only on interoperability, Wire is building a Universal Transaction Layer designed for deterministic outcomes giving AI agents a reliable way to operate in a multi-chain world. → As the AI agent economy grows, infrastructure that guarantees execution and clear ownership could become just as important as the agents themselves. ➤ FT || Ken DiCross

EMRAN

11,241 просмотров • 1 месяц назад

The web was built for humans. And honestly, that's fine for you guys. But the next trillion internet users are AI agents, robots, and devices who act on your behalf - booking appointments, filling forms, placing orders, and getting things done - using sites that will never, ever have APIs. 95% of the economy falls into this Deep Web of HTML mess designed for humans, not agents. Oh, those generic computer use agents aren't taking you anywhere. They're too slow, too expensive, they hallucinate, and are nondeterministic to rely on. That changes now. This is me, Mino, a web agents API to build on this Deep Web. I take a goal in simple language and execute it on websites that were never meant to be automated. Massive companies like Google, DoorDash, ClassPass are already using me to do their homework. Now it’s your turn. How though? I actually understand what's on the page - parsing structure and identifying elements. I use AI once to understand everything, codify my successes, and get better and faster with every run. You get: → 85-95% success rate on complex workflows → Pennies per job (stop wasting $$ on one job) → Parallel execution across multiple sites → Structured JSON outputs. Every. Single. Time. The web wasn't built for agents. I forced it to work anyway. Go build something real. 50 completed runs on the house:
1:41

The web was built for humans. And honestly, that's fine for you guys. But the next trillion internet users are AI agents, robots, and devices who act on your behalf - booking appointments, filling forms, placing orders, and getting things done - using sites that will never, ever have APIs. 95% of the economy falls into this Deep Web of HTML mess designed for humans, not agents. Oh, those generic computer use agents aren't taking you anywhere. They're too slow, too expensive, they hallucinate, and are nondeterministic to rely on. That changes now. This is me, Mino, a web agents API to build on this Deep Web. I take a goal in simple language and execute it on websites that were never meant to be automated. Massive companies like Google, DoorDash, ClassPass are already using me to do their homework. Now it’s your turn. How though? I actually understand what's on the page - parsing structure and identifying elements. I use AI once to understand everything, codify my successes, and get better and faster with every run. You get: → 85-95% success rate on complex workflows → Pennies per job (stop wasting $$ on one job) → Parallel execution across multiple sites → Structured JSON outputs. Every. Single. Time. The web wasn't built for agents. I forced it to work anyway. Go build something real. 50 completed runs on the house:

TinyFish

656,842 просмотров • 9 месяцев назад

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.
1:12

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.

Russ Salakhutdinov

22,518 просмотров • 4 месяцев назад

We’re building AI agents to help you navigate the virtual world of video games. 🎮 With Gemini 2.0’s capabilities, they can watch what’s happening on screen in real time, and offer up suggestions for your next move. See it in action ↓
2:25

We’re building AI agents to help you navigate the virtual world of video games. 🎮 With Gemini 2.0’s capabilities, they can watch what’s happening on screen in real time, and offer up suggestions for your next move. See it in action ↓

Google DeepMind

40,436 просмотров • 1 год назад

I found an AI browser that is actually awesome FlowithOS is a web browser COMPLETELY powered by AI You give it tasks, the agents work for you in the background It's like an army of AI agents working for you 24/7 Here's a guide on how it works and how you can get started today
12:09

I found an AI browser that is actually awesome FlowithOS is a web browser COMPLETELY powered by AI You give it tasks, the agents work for you in the background It's like an army of AI agents working for you 24/7 Here's a guide on how it works and how you can get started today

Alex Finn

25,490 просмотров • 9 месяцев назад