Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Agent Arena measures complex, real-world tasks. Data shows that these long-horizon sessions amplify differences in frontier models. Long-horizon tasks in Agent Arena span hundreds of turns, from porting a codebase to Rust and auditing an investment strategy document to writing a math textbook, and more. Comparing the top model...

arena's profile picture

Arena.ai

239,690 subscribers

29,334 Aufrufe • vor 5 Tagen •via X (Twitter)

Wissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

In the last 14 days, nearly 30,000 tasks ran for over an hour on Agent Arena. Within that hour, the average model made 332 tool calls. In these long-horizon tasks, the right model choice can cut the cost by up to 66%. In Agent Arena, an hour of work costs a median $10.51 per task with Gemini 4 Carbon, $21.13 with Claude Opus 5.5 by Anthropic and $31.08 with GPT-6 Astra by OpenAI. That makes Gemini 4 Carbon by Google DeepMind 50% cheaper than Opus and 66% cheaper than Astra for an hour-long task. For long-running jobs, weighing cost against capability becomes critical: Claude Opus 5.5 still scores higher, but costs twice as much per hour of work.
0:13

In the last 14 days, nearly 30,000 tasks ran for over an hour on Agent Arena. Within that hour, the average model made 332 tool calls. In these long-horizon tasks, the right model choice can cut the cost by up to 66%. In Agent Arena, an hour of work costs a median $10.51 per task with Gemini 4 Carbon, $21.13 with Claude Opus 5.5 by Anthropic and $31.08 with GPT-6 Astra by OpenAI. That makes Gemini 4 Carbon by Google DeepMind 50% cheaper than Opus and 66% cheaper than Astra for an hour-long task. For long-running jobs, weighing cost against capability becomes critical: Claude Opus 5.5 still scores higher, but costs twice as much per hour of work.

Arena.ai

12,783 Aufrufe • vor 5 Tagen

Kimi K3.1 Pro vs. Gemini 4 Carbon (Arena Checkpoint) vs. Gemini 4 Argon vs. GPT-6 Astra >Kimi K3.1 is giving frontier models some real competition and cooked Astra >By the way, I am not sponsored by Kimi.
0:15

Kimi K3.1 Pro vs. Gemini 4 Carbon (Arena Checkpoint) vs. Gemini 4 Argon vs. GPT-6 Astra >Kimi K3.1 is giving frontier models some real competition and cooked Astra >By the way, I am not sponsored by Kimi.

Pranav Reddy

25,166 Aufrufe • vor 16 Tagen

🚨 HUGE: Moonshot AI unveils Kimi K3.1 Pro, a 2.8-TRILLION-parameter model claiming to beat Claude Opus 5.5 and GPT-6 Astra on long-horizon coding tasks. The model features a 1 MILLION-token context window, with 104 billion parameters active per token. Moonshot reports leading scores on SWE Marathon and agent benchmarks. Its ambitions go beyond chat with building game projects in Unreal Engine, Unity and Godot, creating parametric 3D models, and improving financial-modeling reliability. Moonshot promises OPEN WEIGHTS by November 4. Independent testing will determine whether those benchmark claims translate into reliable, working projects.
0:19

🚨 HUGE: Moonshot AI unveils Kimi K3.1 Pro, a 2.8-TRILLION-parameter model claiming to beat Claude Opus 5.5 and GPT-6 Astra on long-horizon coding tasks. The model features a 1 MILLION-token context window, with 104 billion parameters active per token. Moonshot reports leading scores on SWE Marathon and agent benchmarks. Its ambitions go beyond chat with building game projects in Unreal Engine, Unity and Godot, creating parametric 3D models, and improving financial-modeling reliability. Moonshot promises OPEN WEIGHTS by November 4. Independent testing will determine whether those benchmark claims translate into reliable, working projects.

Coin Bureau

36,692 Aufrufe • vor 16 Tagen

Nearly a year of Code Arena: WebDev progress compressed into 15 seconds. Each line follows the highest-scoring model from top labs over time, showing the pace of improvement across the ecosystem. In just the last year, the model in the leading spot increased score by +340 pts and the number of frontier labs competing for the top spot expanded from 6 to 10. Anthropic has dominated throughout the year. Although standout releases have jumped to the top spot, most notably the Chinese open-source model Kimi K3 from Kimi.ai in July. Today, GPT-6 Astra by OpenAI leads with 1796 pts, followed by Claude Fable 5.1 at 1764 pts. The next closest lab is 103 pts away, Qwen with 1685 pts. Code Arena: WebDev ranks models through head-to-head user preference on real front-end web development tasks. These votes drive the leaderboard that is tracking the frontier.
0:15

Nearly a year of Code Arena: WebDev progress compressed into 15 seconds. Each line follows the highest-scoring model from top labs over time, showing the pace of improvement across the ecosystem. In just the last year, the model in the leading spot increased score by +340 pts and the number of frontier labs competing for the top spot expanded from 6 to 10. Anthropic has dominated throughout the year. Although standout releases have jumped to the top spot, most notably the Chinese open-source model Kimi K3 from Kimi.ai in July. Today, GPT-6 Astra by OpenAI leads with 1796 pts, followed by Claude Fable 5.1 at 1764 pts. The next closest lab is 103 pts away, Qwen with 1685 pts. Code Arena: WebDev ranks models through head-to-head user preference on real front-end web development tasks. These votes drive the leaderboard that is tracking the frontier.

Arena.ai

93,946 Aufrufe • vor 1 Monat

tested 3 models against opus 5.5 gemini 4 carbon (updated checkpoint gemini 4 argon Kimi k3.1 (kibitz) Qwen 4 (kalmia) all tested in arena, my bet is on gemini which of them came close to opus 5.5 ?
0:12

tested 3 models against opus 5.5 gemini 4 carbon (updated checkpoint gemini 4 argon Kimi k3.1 (kibitz) Qwen 4 (kalmia) all tested in arena, my bet is on gemini which of them came close to opus 5.5 ?

Haleemah

80,948 Aufrufe • vor 18 Tagen

we compared Gemini 3.1 Pro, Opus 4.7, and GPT 5.5 to Kimi K2.6, Xiaomi Mimo v2.5, and Qwen 3.6 Max in average, closed source are faster. GPT 5.5 and Opus 4.7 the fastest. Kimi K2.6 comes after. It keeps up thanks to its native INT4 quantization. MiMo took the longest, but the refinement and aesthetic only ranks after Gemini 3.1. as if a mini Gemini. Its slow cuz it's trained for long-horizon agentic work. Claude Opus 4.7 is blooming in a very different style, probably because Anthropic trains for taste, not just accuracy.
0:16

we compared Gemini 3.1 Pro, Opus 4.7, and GPT 5.5 to Kimi K2.6, Xiaomi Mimo v2.5, and Qwen 3.6 Max in average, closed source are faster. GPT 5.5 and Opus 4.7 the fastest. Kimi K2.6 comes after. It keeps up thanks to its native INT4 quantization. MiMo took the longest, but the refinement and aesthetic only ranks after Gemini 3.1. as if a mini Gemini. Its slow cuz it's trained for long-horizon agentic work. Claude Opus 4.7 is blooming in a very different style, probably because Anthropic trains for taste, not just accuracy.

GMI Cloud

39,221 Aufrufe • vor 6 Monaten

Introducing Claude Opus 5.5, the first model in our new Claude 5.5 family. It performs at the level of Claude Fable 5.1 for most tasks, and costs 40% less to run than Opus 5.
0:20

Introducing Claude Opus 5.5, the first model in our new Claude 5.5 family. It performs at the level of Claude Fable 5.1 for most tasks, and costs 40% less to run than Opus 5.

Claude

29,652,563 Aufrufe • vor 1 Monat

Kimi K3.1 is basically the open-weight version of Opus 5.5 / GPT-6 Astra, but way cheaper I asked them to build a whale breaching scene. Kimi: $0.75 Opus 5.5: $3.01 Astra: $2.00 Gemini 4 Argon: $2.30 Comparable quality at a fraction of the cost.
0:20

Kimi K3.1 is basically the open-weight version of Opus 5.5 / GPT-6 Astra, but way cheaper I asked them to build a whale breaching scene. Kimi: $0.75 Opus 5.5: $3.01 Astra: $2.00 Gemini 4 Argon: $2.30 Comparable quality at a fraction of the cost.

Bhavy☄️

94,917 Aufrufe • vor 17 Tagen

Same robot, same task, two models with very different writing styles ➤ On the left, Claude Opus 5.5 dabs out "hello world" dot by dot: 3,270 arm moves over 34 minutes ➤ On the right, GPT-6 Astra writes in continuous strokes: 514 moves in 13 minutes From AA-Robotics v0.1, seen from the model’s view See more examples here:
0:11

Same robot, same task, two models with very different writing styles ➤ On the left, Claude Opus 5.5 dabs out "hello world" dot by dot: 3,270 arm moves over 34 minutes ➤ On the right, GPT-6 Astra writes in continuous strokes: 514 moves in 13 minutes From AA-Robotics v0.1, seen from the model’s view See more examples here:

Artificial Analysis

62,902 Aufrufe • vor 12 Tagen

The biggest gap between AI labs isn't whether their agents get the job done. It's whether users actually enjoy working with them. We tracked the best agent from Anthropic, OpenAI, Google DeepMind, Kimi.ai, and DeepSeek across 4 key Agent Arena signals used between late June and the start of October to see how the industry is shifting. Three things stood out: 1. Users like some agents far more than others. On Praise vs Complaint, OpenAI and Anthropic sit at +35 points over the leaderboard average. Moonshot, xAI and DeepSeek land between +5 and +13. No other signal splits the labs this widely. 2. Agents are getting much better at delivery. On Confirmed Success, the average lab's best model went from -4 to +14 points in three months. Anthropic leads with Claude Opus 5.5 (Max) at +20. 3. Labs are converging on error recovery and steering. Since June, the spread between the six labs has more than halved on Bash Recovery (34 → 16 points) and nearly halved on Steerability (19 → 11). In short, the labs are catching up on avoiding failures. Leaving the user happy is where the top three are pulling away.
0:16

The biggest gap between AI labs isn't whether their agents get the job done. It's whether users actually enjoy working with them. We tracked the best agent from Anthropic, OpenAI, Google DeepMind, Kimi.ai, and DeepSeek across 4 key Agent Arena signals used between late June and the start of October to see how the industry is shifting. Three things stood out: 1. Users like some agents far more than others. On Praise vs Complaint, OpenAI and Anthropic sit at +35 points over the leaderboard average. Moonshot, xAI and DeepSeek land between +5 and +13. No other signal splits the labs this widely. 2. Agents are getting much better at delivery. On Confirmed Success, the average lab's best model went from -4 to +14 points in three months. Anthropic leads with Claude Opus 5.5 (Max) at +20. 3. Labs are converging on error recovery and steering. Since June, the spread between the six labs has more than halved on Bash Recovery (34 → 16 points) and nearly halved on Steerability (19 → 11). In short, the labs are catching up on avoiding failures. Leaving the user happy is where the top three are pulling away.

Arena.ai

28,096 Aufrufe • vor 24 Tagen

I’m releasing OpenTTD-Bench, a benchmark for measuring frontier LLM agents on real-time, long-horizon tasks, based on a complex business simulation game OpenTTD, where models run competing transportation companies and must make the most amount of profit.
0:15

I’m releasing OpenTTD-Bench, a benchmark for measuring frontier LLM agents on real-time, long-horizon tasks, based on a complex business simulation game OpenTTD, where models run competing transportation companies and must make the most amount of profit.

Sophia Xu

35,408 Aufrufe • vor 11 Tagen

Grok 4.5 barely had time to enjoy first place. Grok 4.5 Medium just took the top spot on LaurenBench with 56.9%, beating Claude Sonnet 5, Claude Opus 5, GLM 5.2 and GPT 5.6 on real world agent tasks. And Elon says Grok 4.6 arrives next week. Looks like Grok 4.5 won’t be in the spotlight for long. xAI Grok / Writer: Annette, Grok Imagine Designer: Janné
0:15

Grok 4.5 barely had time to enjoy first place. Grok 4.5 Medium just took the top spot on LaurenBench with 56.9%, beating Claude Sonnet 5, Claude Opus 5, GLM 5.2 and GPT 5.6 on real world agent tasks. And Elon says Grok 4.6 arrives next week. Looks like Grok 4.5 won’t be in the spotlight for long. xAI Grok / Writer: Annette, Grok Imagine Designer: Janné

Mario Nawfal

64,010 Aufrufe • vor 3 Monaten

My honest take on Kimi K3.1 Pro 🥶 I asked Kimi to turn an art concept into a complete 3D environment at around 10:25. At the same time, I gave Claude Opus 5.5 the exact same task. About an hour later, Kimi produced what you see below. A complete 3D scene running right in the browser, with reflective icy seas, drifting floes, rugged snow-covered fjords, and warm windows glowing against the cold. Claude finished around the same time, but a little earlier than Kimi. So, clearly, Kimi has improved in terms of execution speed. It feels faster than its previous models. Now, let's talk about cost. The task consumed 62% of my 5-hour session on the $100 Claude plan, whereas it used just 0.53% of my Kimi Vivace plan's session. I followed up with another task to make some refinements, and that barely cost anything either with Kimi. Just 0.35% of my session credits. You can do the math and see how much cheaper it is. Finally, talking about the results, I think Kimi K3.1 Pro is really solid. When it comes to UI, it reminds me of GPT-6 Astra, while its overall capabilities feel closer to Claude Opus 5.5. Although I preferred Claude's result overall in terms of 3D modeling, as it followed the original requirements in greater detail, Kimi K3.1 Pro came pretty close. In fact, Kimi presented the overall scene better than Claude. Overall, I'd highly recommend giving Kimi a try. It really feels like it sits somewhere between the frontier models. True to its benchmarks, i would say. P.S. This is not a paid post. Kimi hasn't paid me anything, nor have they ever paid me in the past 😅 Live:
0:19

My honest take on Kimi K3.1 Pro 🥶 I asked Kimi to turn an art concept into a complete 3D environment at around 10:25. At the same time, I gave Claude Opus 5.5 the exact same task. About an hour later, Kimi produced what you see below. A complete 3D scene running right in the browser, with reflective icy seas, drifting floes, rugged snow-covered fjords, and warm windows glowing against the cold. Claude finished around the same time, but a little earlier than Kimi. So, clearly, Kimi has improved in terms of execution speed. It feels faster than its previous models. Now, let's talk about cost. The task consumed 62% of my 5-hour session on the $100 Claude plan, whereas it used just 0.53% of my Kimi Vivace plan's session. I followed up with another task to make some refinements, and that barely cost anything either with Kimi. Just 0.35% of my session credits. You can do the math and see how much cheaper it is. Finally, talking about the results, I think Kimi K3.1 Pro is really solid. When it comes to UI, it reminds me of GPT-6 Astra, while its overall capabilities feel closer to Claude Opus 5.5. Although I preferred Claude's result overall in terms of 3D modeling, as it followed the original requirements in greater detail, Kimi K3.1 Pro came pretty close. In fact, Kimi presented the overall scene better than Claude. Overall, I'd highly recommend giving Kimi a try. It really feels like it sits somewhere between the frontier models. True to its benchmarks, i would say. P.S. This is not a paid post. Kimi hasn't paid me anything, nor have they ever paid me in the past 😅 Live:

The Bugged Dev

26,084 Aufrufe • vor 16 Tagen

holo CD effect by Muse Wonder, GPT-6 Astra, Claude Opus 5.5, and Claude Fable 5.1 muse wonder seems wonderful this time
0:16

holo CD effect by Muse Wonder, GPT-6 Astra, Claude Opus 5.5, and Claude Fable 5.1 muse wonder seems wonderful this time

Ann Nguyen

417,466 Aufrufe • vor 27 Tagen

The redesigned Arena Leaderboard Overview is live! The frontier is moving quickly, so we built a central hub to help you keep up. The new Arena Leaderboard experience brings all our live signals into one place for researchers, builders, and our global community of users who power our model evaluations. Here’s what’s new: - New Release Models: a real-time feed of the latest models added to Arena - Performance by Category: surfacing live sessions, and the top models by categories like Agents, Images, and Coding. - Model Capabilities: first impressions of new models, straight from the Arena team - Arena News: the latest research, features and updates from around the Arena universe For example, now you can see scores across arenas for models like GPT-6 Sol and Claude Opus 5.5, alongside signals like their category standings and capabilities, their latest news and more. Explore scores and now much more at our redesigned leaderboard. Find the link below.
0:17

The redesigned Arena Leaderboard Overview is live! The frontier is moving quickly, so we built a central hub to help you keep up. The new Arena Leaderboard experience brings all our live signals into one place for researchers, builders, and our global community of users who power our model evaluations. Here’s what’s new: - New Release Models: a real-time feed of the latest models added to Arena - Performance by Category: surfacing live sessions, and the top models by categories like Agents, Images, and Coding. - Model Capabilities: first impressions of new models, straight from the Arena team - Arena News: the latest research, features and updates from around the Arena universe For example, now you can see scores across arenas for models like GPT-6 Sol and Claude Opus 5.5, alongside signals like their category standings and capabilities, their latest news and more. Explore scores and now much more at our redesigned leaderboard. Find the link below.

Arena.ai

24,761 Aufrufe • vor 1 Monat

Today, we're open sourcing the first computer-use dataset for design. We captured 200+ hours and 3400+ trajectories of real design work in Figma, recording step by step how designers execute complex, long-horizon tasks. FigmaTrace dataset, paper, and blog below :) Patronus AI
0:19

Today, we're open sourcing the first computer-use dataset for design. We captured 200+ hours and 3400+ trajectories of real design work in Figma, recording step by step how designers execute complex, long-horizon tasks. FigmaTrace dataset, paper, and blog below :) Patronus AI

Anand Kannappan

137,887 Aufrufe • vor 2 Monaten

K3.1 Pro works directly in the tools used by the 3D modeling and game industries - Blender, Unreal Engine, Godot, and CAD. On BenchCAD, which tests 3D reconstruction from multi-view renderings through CAD code generation, it scores 92.7% with tools, approaching Claude Opus 5.5 (96.2%) and GPT-6 Astra (95.9%). K3.1 Pro can deliver a fairly complete project, from asset construction to scene assembly, animation, and rendering.
0:19

K3.1 Pro works directly in the tools used by the 3D modeling and game industries - Blender, Unreal Engine, Godot, and CAD. On BenchCAD, which tests 3D reconstruction from multi-view renderings through CAD code generation, it scores 92.7% with tools, approaching Claude Opus 5.5 (96.2%) and GPT-6 Astra (95.9%). K3.1 Pro can deliver a fairly complete project, from asset construction to scene assembly, animation, and rendering.

Kimi.ai

159,280 Aufrufe • vor 17 Tagen

If you want the task actually done, Grok 4.5 is making a very strong case. #1 on Long-Horizon Terminal-Bench. It completed 13 out of 46 long-horizon terminal tasks. The previous top models were sitting around 7. These aren't cute little prompts either. Multi-step workflows. Software engineering. Scientific computing. Millions of tokens. Long sessions. Grok 4.5 nearly doubled the mark. Turns out “get it done” is a very useful xAI feature. Grok xAI / Writer: Annette, Designer: Janné
0:16

If you want the task actually done, Grok 4.5 is making a very strong case. #1 on Long-Horizon Terminal-Bench. It completed 13 out of 46 long-horizon terminal tasks. The previous top models were sitting around 7. These aren't cute little prompts either. Multi-step workflows. Software engineering. Scientific computing. Millions of tokens. Long sessions. Grok 4.5 nearly doubled the mark. Turns out “get it done” is a very useful xAI feature. Grok xAI / Writer: Annette, Designer: Janné

Mario Nawfal

42,372 Aufrufe • vor 3 Monaten

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.
0:18

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.

AK

19,684 Aufrufe • vor 2 Jahren

Meta just dropped Muse Wonder 1, its new frontier AI model. They're positioning it against Claude Opus 5.5 and GPT-6 Astra, with major improvements in coding and agentic capabilities. Naturally, I had to throw my Japanese voxel world benchmark at it. Here's what it built.
0:10

Meta just dropped Muse Wonder 1, its new frontier AI model. They're positioning it against Claude Opus 5.5 and GPT-6 Astra, with major improvements in coding and agentic capabilities. Naturally, I had to throw my Japanese voxel world benchmark at it. Here's what it built.

Tak 🦞

18,275 Aufrufe • vor 28 Tagen