Loading video...

Video Failed to Load

Go Home

How good are frontier models at writing Polytopia bots? Introducing Polytopia Bench. 12 models each get 40 minutes to write a Rust bot from scratch.

kzdagoof's profile picture

Kevin Z

9,382 subscribers

33,271 views • 9 days ago •via X (Twitter)

Science & TechnologyGaming
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

How it feels using the latest AI tools to summarize earnings transcripts and create excel models from scratch in minutes (your fund is down 26% YTD tho)
0:13

How it feels using the latest AI tools to summarize earnings transcripts and create excel models from scratch in minutes (your fund is down 26% YTD tho)

litquidity

102,237 views • 1 year ago

8 coding agents (written in rust) who are each writing rust for a simulator while I scroll Grokipedia.
0:13

8 coding agents (written in rust) who are each writing rust for a simulator while I scroll Grokipedia.

Elliot Arledge

26,976 views • 1 year ago

Agent Arena measures complex, real-world tasks. Data shows that these long-horizon sessions amplify differences in frontier models. Long-horizon tasks in Agent Arena span hundreds of turns, from porting a codebase to Rust and auditing an investment strategy document to writing a math textbook, and more. Comparing the top model from each of the 5 leading labs on Agent Arena reveals a 55-minute gap in how long they spend on long-horizon tasks. The models are: - Claude Opus 5.5 by Anthropic - Gemini 4 Carbon by Google DeepMind - GPT-6 Astra by OpenAI - Muse Wonder Full by AI at Meta - Kimi K3.1 Pro by Kimi.ai In this group, Claude Opus 5.5, #1 in Agent Arena, takes the longest at every task length, peaking at 1 hour 51 minutes. In contrast, Kimi K3.1 Pro spends the least time at every length, with 56 minutes on long-horizon tasks.
0:14

Agent Arena measures complex, real-world tasks. Data shows that these long-horizon sessions amplify differences in frontier models. Long-horizon tasks in Agent Arena span hundreds of turns, from porting a codebase to Rust and auditing an investment strategy document to writing a math textbook, and more. Comparing the top model from each of the 5 leading labs on Agent Arena reveals a 55-minute gap in how long they spend on long-horizon tasks. The models are: - Claude Opus 5.5 by Anthropic - Gemini 4 Carbon by Google DeepMind - GPT-6 Astra by OpenAI - Muse Wonder Full by AI at Meta - Kimi K3.1 Pro by Kimi.ai In this group, Claude Opus 5.5, #1 in Agent Arena, takes the longest at every task length, peaking at 1 hour 51 minutes. In contrast, Kimi K3.1 Pro spends the least time at every length, with 56 minutes on long-horizon tasks.

Arena.ai

29,334 views • 7 days ago

Grok 4.1 Fast outperforms all major frontier models on τ²-Bench Telecom for agentic tool use Beating the newest models from Google (Gemini 3 Pro), Anthropic (Claude Opus 4.5), and OpenAI (GPT 5.2 xhigh)
0:15

Grok 4.1 Fast outperforms all major frontier models on τ²-Bench Telecom for agentic tool use Beating the newest models from Google (Gemini 3 Pro), Anthropic (Claude Opus 4.5), and OpenAI (GPT 5.2 xhigh)

X Freeze

41,019 views • 10 months ago

From frogs to the frontier, Scott Wu covered a lot at Runtime: → Agents are moving from tab completion to “virtual employees” that own outcomes → Frontier models may only need ~2% of workloads; routing handles the rest → 91% of Cognition’s internal Devin sessions start without a human
0:17

From frogs to the frontier, Scott Wu covered a lot at Runtime: → Agents are moving from tab completion to “virtual employees” that own outcomes → Frontier models may only need ~2% of workloads; routing handles the rest → 91% of Cognition’s internal Devin sessions start without a human

Modal

47,471 views • 1 month ago

We're open sourcing a 9B model that extracts structured data from documents at near-frontier performance. - 90.2% on our bench, vs Gemini 3.5 Flash at 91.3% - Leads extraction models like NuExtract3 (81.5%) - 9.5s p50 timings - Pass JSON schema
0:16

We're open sourcing a 9B model that extracts structured data from documents at near-frontier performance. - 90.2% on our bench, vs Gemini 3.5 Flash at 91.3% - Leads extraction models like NuExtract3 (81.5%) - 9.5s p50 timings - Pass JSON schema

Vik Paruchuri

229,345 views • 4 months ago

Intelligence should not have a gatekeeper. Today, 40+ frontier and open models stand behind a single endpoint. Gata Inference is live.
0:14

Intelligence should not have a gatekeeper. Today, 40+ frontier and open models stand behind a single endpoint. Gata Inference is live.

Gata

18,190 views • 3 months ago

Introducing ReactBench A benchmark for coding agents on real React work Models write bad React code - useEffect, slow performance, memory leaks
0:10

Introducing ReactBench A benchmark for coding agents on real React work Models write bad React code - useEffect, slow performance, memory leaks

Aiden Bai

345,103 views • 3 months ago

introducing Krea for Grok bot. generate images and video with the best models in the world, right from Grok. try it now 👇
0:17

introducing Krea for Grok bot. generate images and video with the best models in the world, right from Grok. try it now 👇

Krea

242,668 views • 19 days ago

Today we're introducing Verda Inference, in preview, with day-0 access to Kimi K3.1 Pro, in partnership with Kimi.ai 🧵 Frontier open-weight models through one API, engineered for every request, from a single prompt to long-horizon agents.
0:10

Today we're introducing Verda Inference, in preview, with day-0 access to Kimi K3.1 Pro, in partnership with Kimi.ai 🧵 Frontier open-weight models through one API, engineered for every request, from a single prompt to long-horizon agents.

Verda

444,375 views • 7 days ago

Introducing SVG Arena! SVGs are XML-based text files that encode geometry, and they're surprisingly tough for state-of-the-art LLMs to get right Here's how bad models (still) are at drawing a pelican on a bicycle - creds to Simon Willison Leaderboard coming soon! In the meantime, here are some recent generations
0:13

Introducing SVG Arena! SVGs are XML-based text files that encode geometry, and they're surprisingly tough for state-of-the-art LLMs to get right Here's how bad models (still) are at drawing a pelican on a bicycle - creds to Simon Willison Leaderboard coming soon! In the meantime, here are some recent generations

Design Arena

29,136 views • 10 months ago

Language models understand natural proteins. But can they generalize beyond, to design completely new proteins from scratch? New preprint: A 🧵
0:13

Language models understand natural proteins. But can they generalize beyond, to design completely new proteins from scratch? New preprint: A 🧵

Tom Sercu

275,285 views • 3 years ago

Even when you get a D rank he looks cool. Sonic team master how to make this crusty models look good good for them lmao
0:10

Even when you get a D rank he looks cool. Sonic team master how to make this crusty models look good good for them lmao

Ace super💥

283,632 views • 2 years ago

I did it. I built "Cursor for Writing". Introducing Lucid ✨ - Write at the speed of thought - Pull in context from anywhere - Chat + Deep Research
3:45

I did it. I built "Cursor for Writing". Introducing Lucid ✨ - Write at the speed of thought - Pull in context from anywhere - Chat + Deep Research

Ray

443,911 views • 1 year ago

Today we're excited to announce a major collaboration with Takeda to deploy our frontier models across their entire organization. Takeda scientists now get access to BoltzMol-1, BoltzProt-1, and fine-tuned models for biomolecular modeling and design.
0:13

Today we're excited to announce a major collaboration with Takeda to deploy our frontier models across their entire organization. Takeda scientists now get access to BoltzMol-1, BoltzProt-1, and fine-tuned models for biomolecular modeling and design.

Boltz

33,078 views • 4 months ago

A major question in multimodal modeling is how to leverage strong pre-trained models, such as industry-scale LLMs and VLMs, during training to avoid starting from scratch. This is important as the demand for rich and domain-specific multimodal models continues to increase, while training them from scratch is obviously impractical due to limited data and compute. The so-called “any-to-any” multimodal models can model a large diverse dictionary of modalities. That’s good. Their downside is that their architectures are often not decoder-only, which has limited their performance in practice and prevents them from leveraging strong pre-trained decoder-only models as priors. We are releasing MODUS (ICML Conference '26) a decoder-only any-to-any multimodal model to address some of these questions. A single transformer decoder predicts any modality from any others with no modality-specific heads, losses, or task pipelines. We show efficient adaptation of established models, e.g., BAGEL, to rich any-to-any multimodal modeling. We are releasing 14B to 77B parameter models. All materials are open-source. The download links and demos here 🧵
0:18

A major question in multimodal modeling is how to leverage strong pre-trained models, such as industry-scale LLMs and VLMs, during training to avoid starting from scratch. This is important as the demand for rich and domain-specific multimodal models continues to increase, while training them from scratch is obviously impractical due to limited data and compute. The so-called “any-to-any” multimodal models can model a large diverse dictionary of modalities. That’s good. Their downside is that their architectures are often not decoder-only, which has limited their performance in practice and prevents them from leveraging strong pre-trained decoder-only models as priors. We are releasing MODUS (ICML Conference '26) a decoder-only any-to-any multimodal model to address some of these questions. A single transformer decoder predicts any modality from any others with no modality-specific heads, losses, or task pipelines. We show efficient adaptation of established models, e.g., BAGEL, to rich any-to-any multimodal modeling. We are releasing 14B to 77B parameter models. All materials are open-source. The download links and demos here 🧵

Amir Zamir

24,521 views • 4 months ago

Qwen 3.7-max beats Opus 4.7 and GPT-5.5 We tested three frontier models on a real agentic task: write a Tetris bot that plays the game and trains itself. Each model could read its own code, run benchmarks, and rewrite itself across 10 iterations. Then we compared the final bots head to head. Qwen 3.7-Max: training cost $1.32, bot improvement +56% Claude Opus 4.7: training cost $12.15, bot improvement +28% GPT-5.5: training cost $2.85, bot improvement +7% Qwen won on every dimension - biggest jump, 9× cheaper than Claude, 2× cheaper than GPT. Long agentic loops is where Qwen Max actually delivers.
0:10

Qwen 3.7-max beats Opus 4.7 and GPT-5.5 We tested three frontier models on a real agentic task: write a Tetris bot that plays the game and trains itself. Each model could read its own code, run benchmarks, and rewrite itself across 10 iterations. Then we compared the final bots head to head. Qwen 3.7-Max: training cost $1.32, bot improvement +56% Claude Opus 4.7: training cost $12.15, bot improvement +28% GPT-5.5: training cost $2.85, bot improvement +7% Qwen won on every dimension - biggest jump, 9× cheaper than Claude, 2× cheaper than GPT. Long agentic loops is where Qwen Max actually delivers.

atomic.chat

871,419 views • 5 months ago