Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

🧵🚀 New WebAgent Benchmark Alert! 🚀 There is hope for human workers! We released WorkArena++, a new challenging benchmark for WebAgents. Our best agent achieves 0% accuracy on this benchmark, while human evaluators still obtain 94%! 🔗

alex_lacoste_'s profile picture

Alexandre Lacoste

1,139 subscribers

11,271 görüntüleme • 1 yıl önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

new collab from Paradigm and OpenAI: evmbench is a benchmark and agent harness for exploiting smart contract bugs a few months ago, the best models found <20% of critical, fund-draining @Code4rena bugs in our benchmark. today they find > 70%
0:15

new collab from Paradigm and OpenAI: evmbench is a benchmark and agent harness for exploiting smart contract bugs a few months ago, the best models found <20% of critical, fund-draining @Code4rena bugs in our benchmark. today they find > 70%

Alpin Yukseloglu

187,689 görüntüleme • 4 ay önce

D4RL is a great benchmark, but is saturated. Introducing OGBench, a new benchmark for offline goal-conditioned RL and offline RL! Tasks include HumanoidMaze, Puzzle, Drawing, and more 🙂 Project page: GitHub: 🧵↓
0:16

D4RL is a great benchmark, but is saturated. Introducing OGBench, a new benchmark for offline goal-conditioned RL and offline RL! Tasks include HumanoidMaze, Puzzle, Drawing, and more 🙂 Project page: GitHub: 🧵↓

Seohong Park

36,410 görüntüleme • 1 yıl önce

We are finally releasing a new benchmark!! #RSS2023 Main highlights are: * For real-world * Reproducible (3D printing) * 9 furniture assembly long-horizon tasks * 200+ hrs of demonstrations This apparently is my 3rd furniture-related benchmark.🥹
0:14

We are finally releasing a new benchmark!! #RSS2023 Main highlights are: * For real-world * Reproducible (3D printing) * 9 furniture assembly long-horizon tasks * 200+ hrs of demonstrations This apparently is my 3rd furniture-related benchmark.🥹

Joseph Lim

26,911 görüntüleme • 3 yıl önce

The Chinese Zeekr 9X SUV sets a new benchmark for luxury
0:13

The Chinese Zeekr 9X SUV sets a new benchmark for luxury

Massimo

31,818 görüntüleme • 2 ay önce

Lowering the benchmark 🚀 Marc Márquez #HungarianGP 🇭🇺
0:18

Lowering the benchmark 🚀 Marc Márquez #HungarianGP 🇭🇺

MotoGP™🏁

33,141 görüntüleme • 10 ay önce

Revolut is the #1 finance app in Europe — once again (a thread 🧵) Thanks to your trust, we set a new benchmark in 2024.
0:15

Revolut is the #1 finance app in Europe — once again (a thread 🧵) Thanks to your trust, we set a new benchmark in 2024.

Revolut

810,156 görüntüleme • 1 yıl önce

Lamborghini Revuelto set a new benchmark for performance. Now yours to build for real with LEGO
0:15

Lamborghini Revuelto set a new benchmark for performance. Now yours to build for real with LEGO

Lamborghini

36,358 görüntüleme • 10 ay önce

agent-browser + Lightpanda + batch + multi-session is a dangerous combo Completed this benchmark task in 5.85s
0:12

agent-browser + Lightpanda + batch + multi-session is a dangerous combo Completed this benchmark task in 5.85s

Chris Tate

86,869 görüntüleme • 2 ay önce

New benchmark set by Valentin Perrone! 🚨 #CatalanGP 🏁
0:13

New benchmark set by Valentin Perrone! 🚨 #CatalanGP 🏁

MotoGP™🏁

20,928 görüntüleme • 9 ay önce

$BENCH & Benchmark Pass will launch within 72 hours 🚀 This thread tells you everything you need to know.
0:12

$BENCH & Benchmark Pass will launch within 72 hours 🚀 This thread tells you everything you need to know.

Benchmark @ERC-7660

37,734 görüntüleme • 2 yıl önce

Black Myth Wukong not only exceeds our expectations, it also sets a new benchmark for modern action RPGs. Black Myth #BlackMythWukong Read our full review here:
0:11

Black Myth Wukong not only exceeds our expectations, it also sets a new benchmark for modern action RPGs. Black Myth #BlackMythWukong Read our full review here:

Sportskeeda Gaming

131,330 görüntüleme • 1 yıl önce

New open world benchmark coming soon! 100+ hours of puzzle gameplay Previews available on
0:18

New open world benchmark coming soon! 100+ hours of puzzle gameplay Previews available on

💺

23,610 görüntüleme • 2 ay önce

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.
0:18

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.

AK

19,684 görüntüleme • 1 yıl önce

📢 NEW ADOPTION ALERT 📢 Excited to announce Paima Studios is now part of the Vanar gaming ecosystem! Their cutting-edge gaming engine is the catalyst for developers moving from Web2 to Web3, setting a new benchmark in the gaming industry.
0:12

📢 NEW ADOPTION ALERT 📢 Excited to announce Paima Studios is now part of the Vanar gaming ecosystem! Their cutting-edge gaming engine is the catalyst for developers moving from Web2 to Web3, setting a new benchmark in the gaming industry.

Vanar

66,913 görüntüleme • 2 yıl önce

Introduce EAGLE, a new method for fast LLM decoding based on compression: - 3x🚀than vanilla - 2x🚀 than Lookahead (on its benchmark) - 1.6x🚀 than Medusa (on its benchmark) - provably maintains text distribution - trainable (in 1~2 days) and testable on RTX 3090s Playground: Blog: Code: ⚒️First Principle: Compression! Yi Ma We find that the sequence of second-top-layer features is compressible, making the prediction of subsequent feature vectors from previous ones easy by a small model. 🙏Acknowledge: This project is greatly inspired by the Medusa team (Tianle Cai @yli3521 Zhengyang Geng Hongwu Peng Tri Dao), the Lookahead team (Hao Zhang LMSYS Org), and others. Joint work with Yuhui Li and Chao Zhang
0:15

Introduce EAGLE, a new method for fast LLM decoding based on compression: - 3x🚀than vanilla - 2x🚀 than Lookahead (on its benchmark) - 1.6x🚀 than Medusa (on its benchmark) - provably maintains text distribution - trainable (in 1~2 days) and testable on RTX 3090s Playground: Blog: Code: ⚒️First Principle: Compression! Yi Ma We find that the sequence of second-top-layer features is compressible, making the prediction of subsequent feature vectors from previous ones easy by a small model. 🙏Acknowledge: This project is greatly inspired by the Medusa team (Tianle Cai @yli3521 Zhengyang Geng Hongwu Peng Tri Dao), the Lookahead team (Hao Zhang LMSYS Org), and others. Joint work with Yuhui Li and Chao Zhang

Hongyang Zhang

118,810 görüntüleme • 2 yıl önce

Can MLLMs actually track what's happening in a video? Introducing VSTAT 🎯, our new benchmark for visual state tracking. The tasks are simple: count cups, read typed words, count page flips. Humans solve them easily. MLLMs don't. 🧵 [1/11]
0:15

Can MLLMs actually track what's happening in a video? Introducing VSTAT 🎯, our new benchmark for visual state tracking. The tasks are simple: count cups, read typed words, count page flips. Humans solve them easily. MLLMs don't. 🧵 [1/11]

Sihyun Yu

160,611 görüntüleme • 17 gün önce

Today we’re announcing a new open-source SOTA for AI drone autonomy SOTApilot hit 95.34% success on Swarm — our public, 1,000-seed benchmark for general UAV navigation across 6 procedural environments GitHub in the first reply 👇 400+ submissions 25 top-model improvements 2 months Our challenge for the next 3 months: make Swarm the benchmark the world uses to measure AI drone autonomy SOTApilot begins now 🐝
0:15

Today we’re announcing a new open-source SOTA for AI drone autonomy SOTApilot hit 95.34% success on Swarm — our public, 1,000-seed benchmark for general UAV navigation across 6 procedural environments GitHub in the first reply 👇 400+ submissions 25 top-model improvements 2 months Our challenge for the next 3 months: make Swarm the benchmark the world uses to measure AI drone autonomy SOTApilot begins now 🐝

Swarm

24,362 görüntüleme • 6 gün önce

#Excellent initiative for #mega #warehouse to bring #Synergy and #efficiency for group airlines . Thanks to all who had worked #tirelessly to make this #possible. Setting new #benchmark
0:10

#Excellent initiative for #mega #warehouse to bring #Synergy and #efficiency for group airlines . Thanks to all who had worked #tirelessly to make this #possible. Setting new #benchmark

Sisira Kanta Dash

21,253 görüntüleme • 2 yıl önce

🚀 ASAP is now FULLY open-source! 🚀 ✅ Humanoid RL motion tracking & delta actions ✅ Motion retargeting to any humanoid ✅ ASAP Benchmark motions + pretrained policies ✅ Sim2Sim & Sim2Real ready — run ASAP in sim or on your G1 robot! 🔗 🎥👇
0:12

🚀 ASAP is now FULLY open-source! 🚀 ✅ Humanoid RL motion tracking & delta actions ✅ Motion retargeting to any humanoid ✅ ASAP Benchmark motions + pretrained policies ✅ Sim2Sim & Sim2Real ready — run ASAP in sim or on your G1 robot! 🔗 🎥👇

Tairan He

39,412 görüntüleme • 10 ay önce

New benchmark set by Quartararo! Into the 1:36s he goes 📈 #SpanishGP 🇪🇸
0:17

New benchmark set by Quartararo! Into the 1:36s he goes 📈 #SpanishGP 🇪🇸

MotoGP™🏁

52,102 görüntüleme • 1 yıl önce