Loading video...

Video Failed to Load

Go Home

When evaluated against the WebVoyager benchmark, which tests agent performance on end-to-end real world web tasks, Project Mariner achieved a state-of-the-art result of 83.5% working as a single agent setup.

GoogleDeepMind's profile picture

Google DeepMind

1,125,017 subscribers

23,958 views • 1 year ago •via X (Twitter)

News & PoliticsScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.
0:18

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.

AK

19,684 views • 1 year ago

Finally, we're so excited to bring Project Mariner capabilities to AI Mode in Search, Agent Mode in Google Gemini, and as a standalone web app. This will allow the products you know and love to intelligently manage complex multi-step tasks on your behalf.
0:15

Finally, we're so excited to bring Project Mariner capabilities to AI Mode in Search, Agent Mode in Google Gemini, and as a standalone web app. This will allow the products you know and love to intelligently manage complex multi-step tasks on your behalf.

Google AI

29,189 views • 1 year ago

I set a Ralph loop using agent-browser + Parallel Web Systems to do a ton of market research. Perfect way to have an agent working while I do other things. Decided to setup my iMac for a lot of the looping tasks because it makes it easier to keep the repo clones and work product separated.
0:11

I set a Ralph loop using agent-browser + Parallel Web Systems to do a ton of market research. Perfect way to have an agent working while I do other things. Decided to setup my iMac for a lot of the looping tasks because it makes it easier to keep the repo clones and work product separated.

Ryan Carson

34,837 views • 4 months ago

my new vibe code setup: 1 orchestrator agent which controls 85 sub-agents working in parallel each sub-agent spawns from my stream of consciousness and tests from the main orchestrator Here's how it works:
0:16

my new vibe code setup: 1 orchestrator agent which controls 85 sub-agents working in parallel each sub-agent spawns from my stream of consciousness and tests from the main orchestrator Here's how it works:

Greg Kamradt

316,172 views • 11 months ago

The end result of Project 2025
0:14

The end result of Project 2025

The Daily Sneed™

32,800,259 views • 1 year ago

We're excited to power LemonSlice's new video agent product. Turn your real-time voice agent into a video agent using a single photo. The underlying model—which autoscales on Modal 😉—streams infinite-length talking video at 20fps using a single GPU.
0:17

We're excited to power LemonSlice's new video agent product. Turn your real-time voice agent into a video agent using a single photo. The underlying model—which autoscales on Modal 😉—streams infinite-length talking video at 20fps using a single GPU.

Modal

17,225 views • 5 months ago

It's true, most AI coding tools make you do the grunt testing and debugging work. That's why we invested in researching and building a state-of-the-art app testing tool. Replit Agent 3 will spin up a browser and test your app end-to-end.
0:13

It's true, most AI coding tools make you do the grunt testing and debugging work. That's why we invested in researching and building a state-of-the-art app testing tool. Replit Agent 3 will spin up a browser and test your app end-to-end.

Amjad Masad

109,857 views • 6 months ago

A very short look at our original animation project, The End Of The World. We're still working on it, and it's a ton of fun. Hopefully I'll be able to provide a release date soon.
0:14

A very short look at our original animation project, The End Of The World. We're still working on it, and it's a ton of fun. Hopefully I'll be able to provide a release date soon.

Kannas Ricebowl

108,216 views • 5 months ago

JISOO reacting to the performance of the group Allday Project at Cartier's year-end party!
0:18

JISOO reacting to the performance of the group Allday Project at Cartier's year-end party!

َ

10,680 views • 6 months ago

Sonnet-3.7 is available in Cursor! We've been very impressed by its coding ability, especially on real-world agentic tasks. It appears to be the new state of the art.
0:17

Sonnet-3.7 is available in Cursor! We've been very impressed by its coding ability, especially on real-world agentic tasks. It appears to be the new state of the art.

Cursor

1,114,871 views • 1 year ago

SOMEONE OPEN SOURCED A SMALL MODEL TRAINED SPECIFICALLY AS A PERSONAL AGENT ROUTER DECIDES WHAT RUNS LOCAL VS CLOUD AUTOMATICALLY ROUTES TASKS TO THE RIGHT MODEL BASED ON COMPLEXITY SMARTER AGENT ORCHESTRATION WITHOUT THE OVERHEAD
0:15

SOMEONE OPEN SOURCED A SMALL MODEL TRAINED SPECIFICALLY AS A PERSONAL AGENT ROUTER DECIDES WHAT RUNS LOCAL VS CLOUD AUTOMATICALLY ROUTES TASKS TO THE RIGHT MODEL BASED ON COMPLEXITY SMARTER AGENT ORCHESTRATION WITHOUT THE OVERHEAD

0xMarioNawfal

157,414 views • 2 months ago

As a naive citizen of Pakistan, I may have grievances against you, but as a student of realpolitik and State craftsmanship, I call you a successful Statesman!!! Your State was your ultimate end, and you successfully achieved that end: Politics is normalised; economy, stabilised, defence, reinforced and diplomacy, bolstered. Ladies and Gentlemen! Although clad in uniform, he is the Statesman from within.
0:18

As a naive citizen of Pakistan, I may have grievances against you, but as a student of realpolitik and State craftsmanship, I call you a successful Statesman!!! Your State was your ultimate end, and you successfully achieved that end: Politics is normalised; economy, stabilised, defence, reinforced and diplomacy, bolstered. Ladies and Gentlemen! Although clad in uniform, he is the Statesman from within.

Suleman Shah Rashdi

169,379 views • 7 months ago

MAMAMOO Hwasa forgot which finger to bite at the end of a Gogobebe performance. She is so adorable^^
0:11

MAMAMOO Hwasa forgot which finger to bite at the end of a Gogobebe performance. She is so adorable^^

DragonKazoo77

17,722 views • 8 months ago

2 Cursor Composer agents working together on the same project. one builds the project, the other reviews once the first agent is done and writes a report. cursor rules file is in comment to achieve this. first agent is instructed to build the project second agent is instructed to write a monitoring script to determine when the project is done then to review the project for errors and write a report This prompt can be improved. This is meant as a baseline If you are interested in learning how to use Cursor more in depth, I have a 28 chapter 1000x Cursor Course building full apps from scratch with 19 hours of content. link is in my bio if you are interested
0:19

2 Cursor Composer agents working together on the same project. one builds the project, the other reviews once the first agent is done and writes a report. cursor rules file is in comment to achieve this. first agent is instructed to build the project second agent is instructed to write a monitoring script to determine when the project is done then to review the project for errors and write a report This prompt can be improved. This is meant as a baseline If you are interested in learning how to use Cursor more in depth, I have a 28 chapter 1000x Cursor Course building full apps from scratch with 19 hours of content. link is in my bio if you are interested

echo.hive

149,034 views • 1 year ago

this might be the end of Sketch & Agent after what just happened 💔
0:19

this might be the end of Sketch & Agent after what just happened 💔

yoxic

14,790,627 views • 1 year ago

GPU Web Acceleration Test (This is not a benchmark, it reflects real-life web GPU performance)
0:17

GPU Web Acceleration Test (This is not a benchmark, it reflects real-life web GPU performance)

Pixel Gamer 4k

66,214 views • 1 year ago

when she single-handley brings state-of-the-art hair physics to 2K
0:11

when she single-handley brings state-of-the-art hair physics to 2K

latrell

530,562 views • 9 months ago

MILEY CYRUS NEW SINGLE TONIGHT END OF THE WORLD
0:14

MILEY CYRUS NEW SINGLE TONIGHT END OF THE WORLD

Miley Topic

21,283 views • 1 year ago

The Gemini 2.0 era is here. And we’re excited for you to start building with it. A quick rewind of what we just released ⏪ Gemini 2.0 Flash ⚡ comes with low latency and better performance. 🔵 You can now access an experimental version in G3mini on the web, while Gemini Advanced users can try Deep Research, a new AI research assistant. 🔵 Developers can begin building through the Gemini API in Google AI Studio and Vertex AI 2.0 is also enabling new research prototypes of AI agents, including: 🔵 Project Astra, which explores future capabilities of a universal AI assistant 🔵 Project Mariner, which shows what’s possible for human-agent interaction, starting with your browser 🔵 Jules, an experimental AI-powered coding agent Finally, we’re exploring how 2.0 can be used in agents across domains — from navigating the virtual world of video games to applying its spatial reasoning capabilities to robotics. 🤖
0:14

The Gemini 2.0 era is here. And we’re excited for you to start building with it. A quick rewind of what we just released ⏪ Gemini 2.0 Flash ⚡ comes with low latency and better performance. 🔵 You can now access an experimental version in G3mini on the web, while Gemini Advanced users can try Deep Research, a new AI research assistant. 🔵 Developers can begin building through the Gemini API in Google AI Studio and Vertex AI 2.0 is also enabling new research prototypes of AI agents, including: 🔵 Project Astra, which explores future capabilities of a universal AI assistant 🔵 Project Mariner, which shows what’s possible for human-agent interaction, starting with your browser 🔵 Jules, an experimental AI-powered coding agent Finally, we’re exploring how 2.0 can be used in agents across domains — from navigating the virtual world of video games to applying its spatial reasoning capabilities to robotics. 🤖

Google DeepMind

231,798 views • 1 year ago