🧵🚀 New WebAgent Benchmark Alert! 🚀 There is hope... for human workers! We released WorkArena++, a new challenging benchmark for WebAgents. Our best agent achieves 0% accuracy on this benchmark, while human evaluators still obtain 94%! 🔗show more

new collab from Paradigm and OpenAI: evmbench is a... benchmark and agent harness for exploiting smart contract bugs a few months ago, the best models found <20% of critical, fund-draining @Code4rena bugs in our benchmark. today they find > 70%show more

D4RL is a great benchmark, but is saturated. Introducing... OGBench, a new benchmark for offline goal-conditioned RL and offline RL! Tasks include HumanoidMaze, Puzzle, Drawing, and more 🙂 Project page: GitHub: 🧵↓show more

We are finally releasing a new benchmark!! #RSS2023 Main... highlights are: * For real-world * Reproducible (3D printing) * 9 furniture assembly long-horizon tasks * 200+ hrs of demonstrations This apparently is my 3rd furniture-related benchmark.🥹show more

The Chinese Zeekr 9X SUV sets a new benchmark... for luxuryshow more

Lowering the benchmark 🚀 Marc Márquez #HungarianGP 🇭🇺

Revolut is the #1 finance app in Europe —... once again (a thread 🧵) Thanks to your trust, we set a new benchmark in 2024.show more

Lamborghini Revuelto set a new benchmark for performance. Now... yours to build for real with LEGOshow more

Bez sets a new benchmark ⏱️🔥 #DutchGP 🇳🇱

🚀 Prisma Next performs close to raw `pg`. It... reaches ~90% of raw `pg` peak throughput in our benchmark, while shipping a 148.5 KB gzipped bundle.show more

agent-browser + Lightpanda + batch + multi-session is a... dangerous combo Completed this benchmark task in 5.85sshow more

New benchmark set by Valentin Perrone! 🚨 #CatalanGP 🏁

Seedance 2.0 in native 4K just set a new... benchmark for AI filmmaking. 7 examples and prompts tested on Magnific 👇show more

$BENCH & Benchmark Pass will launch within 72 hours... 🚀 This thread tells you everything you need to know.show more

Black Myth Wukong not only exceeds our expectations, it... also sets a new benchmark for modern action RPGs. Black Myth #BlackMythWukong Read our full review here:show more

some genius invented a kebab benchmark for llms

New open world benchmark coming soon! 100+ hours of... puzzle gameplay Previews available onshow more

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents... at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.show more

📢 NEW ADOPTION ALERT 📢 Excited to announce Paima... Studios is now part of the Vanar gaming ecosystem! Their cutting-edge gaming engine is the catalyst for developers moving from Web2 to Web3, setting a new benchmark in the gaming industry.show more

Introduce EAGLE, a new method for fast LLM decoding... based on compression: - 3x🚀than vanilla - 2x🚀 than Lookahead (on its benchmark) - 1.6x🚀 than Medusa (on its benchmark) - provably maintains text distribution - trainable (in 1~2 days) and testable on RTX 3090s Playground: Blog: Code: ⚒️First Principle: Compression! Yi Ma We find that the sequence of second-top-layer features is compressible, making the prediction of subsequent feature vectors from previous ones easy by a small model. 🙏Acknowledge: This project is greatly inspired by the Medusa team (Tianle Cai @yli3521 Zhengyang Geng Hongwu Peng Tri Dao), the Lookahead team (Hao Zhang LMSYS Org), and others. Joint work with Yuhui Li and Chao Zhangshow more

#Excellent initiative for #mega #warehouse to bring #Synergy and... #efficiency for group airlines . Thanks to all who had worked #tirelessly to make this #possible. Setting new #benchmarkshow more