🧵🚀 New WebAgent Benchmark Alert! 🚀 There is hope... for human workers! We released WorkArena++, a new challenging benchmark for WebAgents. Our best agent achieves 0% accuracy on this benchmark, while human evaluators still obtain 94%! 🔗show more

new collab from Paradigm and OpenAI: evmbench is a... benchmark and agent harness for exploiting smart contract bugs a few months ago, the best models found <20% of critical, fund-draining @Code4rena bugs in our benchmark. today they find > 70%show more

D4RL is a great benchmark, but is saturated. Introducing... OGBench, a new benchmark for offline goal-conditioned RL and offline RL! Tasks include HumanoidMaze, Puzzle, Drawing, and more 🙂 Project page: GitHub: 🧵↓show more

New highly advanced benchmark for AI models.

We are finally releasing a new benchmark!! #RSS2023 Main... highlights are: * For real-world * Reproducible (3D printing) * 9 furniture assembly long-horizon tasks * 200+ hrs of demonstrations This apparently is my 3rd furniture-related benchmark.🥹show more

The Chinese Zeekr 9X SUV sets a new benchmark... for luxuryshow more

Revolut is the #1 finance app in Europe —... once again (a thread 🧵) Thanks to your trust, we set a new benchmark in 2024.show more

ok Grok 4.5 is wild It just built me... this rocket tracking app with live data and a 3D globe 🚀 I might need a new benchmark after this 😭show more

Lamborghini Revuelto set a new benchmark for performance. Now... yours to build for real with LEGOshow more

I genuinely feel #King is going to bring something... we haven’t seen before and set a completely new benchmark for Indian cinema 🔥show more

Bez sets a new benchmark ⏱️🔥 #DutchGP 🇳🇱

🚀 Prisma Next performs close to raw `pg`. It... reaches ~90% of raw `pg` peak throughput in our benchmark, while shipping a 148.5 KB gzipped bundle.show more

Join us for the unveiling of the new benchmark... supercar. 25 April 23.58 BSTshow more

agent-browser + Lightpanda + batch + multi-session is a... dangerous combo Completed this benchmark task in 5.85sshow more

Seedance 2.0 in native 4K just set a new... benchmark for AI filmmaking. 7 examples and prompts tested on Magnific 👇show more

some genius invented a kebab benchmark for llms

$BENCH & Benchmark Pass will launch within 72 hours... 🚀 This thread tells you everything you need to know.show more

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents... at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.show more

New open world benchmark coming soon! 100+ hours of... puzzle gameplay Previews available onshow more