Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Overview of our recent launch of Coding Agent benchmarks on Artificial Analysis and our first Youtube Video! We walk through the performance, cost, token usage and speed differences across different coding agents. This includes looking at Opus 4.7 in Claude Code's leading performance and Composer 2.5's strong positioning on... show more

Artificial Analysis

106,825 subscribers

10,623 views • 1 month ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Our new Gemini 3.5 Flash is our strongest agentic and coding model yet, delivering frontier-level performance at 4x the speed of comparable frontier models at less than half the cost. We had @TulseeDoshi break down what you need to know about it.

Our new Gemini 3.5 Flash is our strongest agentic and coding model yet, delivering frontier-level performance at 4x the speed of comparable frontier models at less than half the cost. We had @TulseeDoshi break down what you need to know about it.

Google

77,414 views • 1 month ago

Announcing Opcode Everyone's favorite Claude Code GUI Claudia is now Opcode... with a complete revamp. And we just hit 15K+ stars on GitHub! The support from the community has been immense! ❤️ What's new: - Feature parity with all new Claude Code features in the CLI. - A complete UI revamp, faster and cleaner. What's coming: - Opcode's own coding agent that exclusively uses open models while achieving SOTA performance on coding benchmarks. We're currently piloting this at companies requiring fully local on-prem coding agents. If you're one of them, reach out to us and we'll set this up for you! Try Opcode at opcode [dot] sh Like/retweet and reply to this tweet with "join" to get an invite to the Discord server and beta test our coding agent for free.

Announcing Opcode Everyone's favorite Claude Code GUI Claudia is now Opcode... with a complete revamp. And we just hit 15K+ stars on GitHub! The support from the community has been immense! ❤️ What's new: - Feature parity with all new Claude Code features in the CLI. - A complete UI revamp, faster and cleaner. What's coming: - Opcode's own coding agent that exclusively uses open models while achieving SOTA performance on coding benchmarks. We're currently piloting this at companies requiring fully local on-prem coding agents. If you're one of them, reach out to us and we'll set this up for you! Try Opcode at opcode [dot] sh Like/retweet and reply to this tweet with "join" to get an invite to the Discord server and beta test our coding agent for free.

winfunc

38,639 views • 10 months ago

$Everyone's sleeping on MiniMax. Again. They just shipped M3. The first open-weights model to combine frontier coding, 1M context, and native multimodality in one drop. I plugged it into Claude Code this morning. Pasted a design from Dribbble. Watched M3 write production-ready React code in one session. At the agency, I just replaced Opus 4.8 with M3 for 80% of our coding tasks. The output is the same and we are running everything at a fraction of the cost. Open infrastructure is the future.$

Everyone's sleeping on MiniMax. Again. They just shipped M3. The first open-weights model to combine frontier coding, 1M context, and native multimodality in one drop. I plugged it into Claude Code this morning. Pasted a design from Dribbble. Watched M3 write production-ready React code in one session. At the agency, I just replaced Opus 4.8 with M3 for 80% of our coding tasks. The output is the same and we are running everything at a fraction of the cost. Open infrastructure is the future.

Prajwal Tomar

12,904 views • 1 month ago

Following up on our Intelligence Index v4.1 release yesterday, in the video below, Daniel from our team shares a short overview of what's changed: 1. Three upgraded evaluations: Terminal-Bench 2.1, τ³-Bench Banking and GDPval-AA v2 2. Cost, time, and tokens per task: Understand the cost, time, and tokens of tasks across our Index and for individual evals, and how these trade off against Intelligence 3. Cached input token reporting: We now report the amount of cached tokens a particular model uses and how this influences cost

Following up on our Intelligence Index v4.1 release yesterday, in the video below, Daniel from our team shares a short overview of what's changed: 1. Three upgraded evaluations: Terminal-Bench 2.1, τ³-Bench Banking and GDPval-AA v2 2. Cost, time, and tokens per task: Understand the cost, time, and tokens of tasks across our Index and for individual evals, and how these trade off against Intelligence 3. Cached input token reporting: We now report the amount of cached tokens a particular model uses and how this influences cost

Artificial Analysis

12,491 views • 28 days ago

Embedder is the world's first hardware-aware coding agent. By understanding and interacting directly with your hardware, it achieves state of the art performance in an embedded systems (C++/Rust) context. Our latest update (v0.3.0) features a stunning new terminal UI, and our fastest, most capable firmware agent yet: Embedder

Embedder is the world's first hardware-aware coding agent. By understanding and interacting directly with your hardware, it achieves state of the art performance in an embedded systems (C++/Rust) context. Our latest update (v0.3.0) features a stunning new terminal UI, and our fastest, most capable firmware agent yet: Embedder

Ethan Gibbs

435,204 views • 5 months ago

🔗 Announcing LangChain OSS Skills LangChain has the most popular frameworks for building AI agents — and now your coding agent can be an expert in it. We're excited to release the first iteration of LangChain OSS Skills, giving your agent expertise in our open source frameworks. The skills include guidance on how to use langchain, langgraph, and deepagents to effectively build agents. ➡️ Install our OSS skills for your coding agent here: ➡️ Read more:

🔗 Announcing LangChain OSS Skills LangChain has the most popular frameworks for building AI agents — and now your coding agent can be an expert in it. We're excited to release the first iteration of LangChain OSS Skills, giving your agent expertise in our open source frameworks. The skills include guidance on how to use langchain, langgraph, and deepagents to effectively build agents. ➡️ Install our OSS skills for your coding agent here: ➡️ Read more:

LangChain OSS

65,840 views • 4 months ago

Gemini 3.5 Flash is a step forward for Google on speed and agentic capabilities but comes at a trade-off of being higher cost than prior models We have measured up to ~280 output tokens/sec, placing it on the speed/intelligence Pareto frontier and well ahead of Gemini 3 Flash. It also shows a major uplift on agentic tasks, reaching ~1650 ELO on GDPVal-AA. The trade-off: cost is up ~5x versus Gemini 3 Flash, driven by higher token prices (3x higher than Gemini 3 Flash) and higher token usage. In this video, Declan Jackson, Member of Technical Staff at Artificial Analysis, breaks it down.

Gemini 3.5 Flash is a step forward for Google on speed and agentic capabilities but comes at a trade-off of being higher cost than prior models We have measured up to ~280 output tokens/sec, placing it on the speed/intelligence Pareto frontier and well ahead of Gemini 3 Flash. It also shows a major uplift on agentic tasks, reaching ~1650 ELO on GDPVal-AA. The trade-off: cost is up ~5x versus Gemini 3 Flash, driven by higher token prices (3x higher than Gemini 3 Flash) and higher token usage. In this video, Declan Jackson, Member of Technical Staff at Artificial Analysis, breaks it down.

Artificial Analysis

16,055 views • 1 month ago

🚀New Amazon Q Developer agent for software development is available to customers: This agent is based on a new agent architecture that has exciting results coming from the SWE-bench scores (on the full and verified benchmarks) representing AI models’ ability to resolve real-world coding problems. Interesting aspect of Q Agent is that with these newest updates, Q drove nearly 50% more successful coding tasks completed. What makes Q Dev Agent remarkable? The agent architecture is not just about using the best LLMs (which we do), but also giving the agent the ability to constantly explore multiple paths to find the best way to resolve a particular problem (and back tracking when it has reached dead end like a developer would do). Needless to say, we are just getting started on the developer agent and we are constantly pushing to advance our AI capabilities while maintaining quality, security, privacy, and reliability to keep Amazon Q Developer an innovative and trusted option available to our customers using agents for software development. We highlighted the results of our first SWE-bench submission of Amazon Q Developer back in June blog post; with these updates, our new agent resolves 51% more coding tasks than its previous iteration on the SWE-bench verified dataset, and 43% more on the full dataset. That’s the difference a few months make, and I can’t wait to share what our teams will deliver at re:Invent this December. Here's a quick demo showcasing our new Agent in action:

🚀New Amazon Q Developer agent for software development is available to customers: This agent is based on a new agent architecture that has exciting results coming from the SWE-bench scores (on the full and verified benchmarks) representing AI models’ ability to resolve real-world coding problems. Interesting aspect of Q Agent is that with these newest updates, Q drove nearly 50% more successful coding tasks completed. What makes Q Dev Agent remarkable? The agent architecture is not just about using the best LLMs (which we do), but also giving the agent the ability to constantly explore multiple paths to find the best way to resolve a particular problem (and back tracking when it has reached dead end like a developer would do). Needless to say, we are just getting started on the developer agent and we are constantly pushing to advance our AI capabilities while maintaining quality, security, privacy, and reliability to keep Amazon Q Developer an innovative and trusted option available to our customers using agents for software development. We highlighted the results of our first SWE-bench submission of Amazon Q Developer back in June blog post; with these updates, our new agent resolves 51% more coding tasks than its previous iteration on the SWE-bench verified dataset, and 43% more on the full dataset. That’s the difference a few months make, and I can’t wait to share what our teams will deliver at re:Invent this December. Here's a quick demo showcasing our new Agent in action:

Swami Sivasubramanian

28,946 views • 1 year ago

AI agents are becoming increasingly capable of taking actions beyond the chat window and interacting with tools in the real world, such as web browsers. We explored this capability by having an AI agent present one of our interactive web-based demos. In this video, it's presenting our election interference demo. This agent is a Cursor coding agent and is equipped with tools to read and interact with the browser (using Playwright) and perform text-to-speech (with ElevenLabs), in addition to the standard Cursor tools.

AI agents are becoming increasingly capable of taking actions beyond the chat window and interacting with tools in the real world, such as web browsers. We explored this capability by having an AI agent present one of our interactive web-based demos. In this video, it's presenting our election interference demo. This agent is a Cursor coding agent and is equipped with tools to read and interact with the browser (using Playwright) and perform text-to-speech (with ElevenLabs), in addition to the standard Cursor tools.

CivAI

18,428 views • 7 months ago

What if language models didn't have to generate one token at a time? Our CEO Stefano Ermon joined @TBPN to break down how Mercury 2's diffusion LLM hits 1,000+ tok/s on standard NVIDIA GPUs — and why speed changes the product for coding, voice agents, and search.

What if language models didn't have to generate one token at a time? Our CEO Stefano Ermon joined @TBPN to break down how Mercury 2's diffusion LLM hits 1,000+ tok/s on standard NVIDIA GPUs — and why speed changes the product for coding, voice agents, and search.

Inception

27,108 views • 4 months ago

We are excited to announce a powerful step for the future of FOMO! Taking a page out of Virtuals book on BASE, FOMO will be releasing the ability for future projects to be paired in $FOMO in the coming weeks. This is the biggest release we have ever announced. Launch your AI Agent Token + $FOMO trading pair Every individual agent token is paired with the $FOMO token in its liquidity pool. When launching an agent on you will need $FOMO tokens, which are used to create the liquidity pool. This process creates deflationary pressure for FOMO and the entire agent ecosystem. When creating your agent and token, you will have the option to pair your launch with FOMO or SOL, as our goal is not to alienate any project, but rather invite the best communities, CTO’s and builders to launch with us. If you decide to pair your project with FOMO you in turn get full marketing and dev support, once your project graduates the bonding curve and reaches Raydium. Further, as an added incentive, as our revenue grows we will be using part of the funds to support projects that have paired in FOMO. And Devs who launch tokens paired in FOMO will earn fees from their AI Agent token launch. Building the most robust agents using our framework will catapult us as one of the most prominent standards of the Solana ecosystem. Not only have we developed our own core infrastructure, but we also pull from some of the best repo’s and developer talent in all of AI, not just blockchain. Our team is comprised of 9 world class artificial intelligence engineers, PHDs in mathematics and engineering from the top companies on the cutting edge of AI. The future of AI Agents will be on Solana and we will help lead the way.

We are excited to announce a powerful step for the future of FOMO! Taking a page out of Virtuals book on BASE, FOMO will be releasing the ability for future projects to be paired in $FOMO in the coming weeks. This is the biggest release we have ever announced. Launch your AI Agent Token + $FOMO trading pair Every individual agent token is paired with the $FOMO token in its liquidity pool. When launching an agent on you will need $FOMO tokens, which are used to create the liquidity pool. This process creates deflationary pressure for FOMO and the entire agent ecosystem. When creating your agent and token, you will have the option to pair your launch with FOMO or SOL, as our goal is not to alienate any project, but rather invite the best communities, CTO’s and builders to launch with us. If you decide to pair your project with FOMO you in turn get full marketing and dev support, once your project graduates the bonding curve and reaches Raydium. Further, as an added incentive, as our revenue grows we will be using part of the funds to support projects that have paired in FOMO. And Devs who launch tokens paired in FOMO will earn fees from their AI Agent token launch. Building the most robust agents using our framework will catapult us as one of the most prominent standards of the Solana ecosystem. Not only have we developed our own core infrastructure, but we also pull from some of the best repo’s and developer talent in all of AI, not just blockchain. Our team is comprised of 9 world class artificial intelligence engineers, PHDs in mathematics and engineering from the top companies on the cutting edge of AI. The future of AI Agents will be on Solana and we will help lead the way.

FOMO

129,867 views • 1 year ago

AI that senses coding frustration, is this the future of learning? In our latest GitHub Podcast episode we talk to Angie Jones about Goose, Block’s open source AI agent and reference implementation of the Model Context Protocol (MCP).

AI that senses coding frustration, is this the future of learning? In our latest GitHub Podcast episode we talk to Angie Jones about Goose, Block’s open source AI agent and reference implementation of the Model Context Protocol (MCP).

GitHub

16,414 views • 7 months ago

Introducing ALE-Bench, ALE-Agent! Towards Automating Long-Horizon Algorithm Engineering for Hard Optimization Problems Blog: Paper: ALE-Bench is a coding benchmark primarily focused on hard optimization (NP-hard) problems. We developed this benchmark with AtCoder Inc., a leading coding contest platform company. What makes ALE-Bench unique is its focus on hard optimization problems that demand long-horizon and creative reasoning. It’s open-ended, in the sense that true optima are out of reach (NP-hard) and scores can continuously improve. We believe this benchmark has the potential to become one of the key benchmarks for reasoning and coding in the next generation. ALE-Agent is our end-to-end agent that we specifically designed for this challenging domain. In fact, our ALE-Agent has already built an impressive track record in the wild! In May 2025, our agent participated in a live AtCoder Heuristic Competition (AHC), alongside 1,000 other participants in real-time. AHC is considered to be one of the most challenging coding competitions in this domain. Our ALE-Agent achieved an impressive ranking of 21st out of 1,000 human participants in the competition (top 2%), marking a turning point for AI discovery of solutions to hard optimization problems with a wide spectrum of important real world applications such as logistics, routing, packing, factory production planning, power-grid balancing. We look forward to applying this technology to real industrial optimization opportunities. Building on the insights from this study, Sakana AI will continue to tackle the challenge of developing AI with even greater algorithm engineering capabilities. ALE-Bench Dataset: ALE-Bench Code: This research was conducted in collaboration with AtCoder Inc. (AtCoder). We are deeply grateful for their outstanding expertise and contributions in optimization and algorithms, which were invaluable in providing data, analyzing results, and enabling our AI agent’s participation in their contests.

Introducing ALE-Bench, ALE-Agent! Towards Automating Long-Horizon Algorithm Engineering for Hard Optimization Problems Blog: Paper: ALE-Bench is a coding benchmark primarily focused on hard optimization (NP-hard) problems. We developed this benchmark with AtCoder Inc., a leading coding contest platform company. What makes ALE-Bench unique is its focus on hard optimization problems that demand long-horizon and creative reasoning. It’s open-ended, in the sense that true optima are out of reach (NP-hard) and scores can continuously improve. We believe this benchmark has the potential to become one of the key benchmarks for reasoning and coding in the next generation. ALE-Agent is our end-to-end agent that we specifically designed for this challenging domain. In fact, our ALE-Agent has already built an impressive track record in the wild! In May 2025, our agent participated in a live AtCoder Heuristic Competition (AHC), alongside 1,000 other participants in real-time. AHC is considered to be one of the most challenging coding competitions in this domain. Our ALE-Agent achieved an impressive ranking of 21st out of 1,000 human participants in the competition (top 2%), marking a turning point for AI discovery of solutions to hard optimization problems with a wide spectrum of important real world applications such as logistics, routing, packing, factory production planning, power-grid balancing. We look forward to applying this technology to real industrial optimization opportunities. Building on the insights from this study, Sakana AI will continue to tackle the challenge of developing AI with even greater algorithm engineering capabilities. ALE-Bench Dataset: ALE-Bench Code: This research was conducted in collaboration with AtCoder Inc. (AtCoder). We are deeply grateful for their outstanding expertise and contributions in optimization and algorithms, which were invaluable in providing data, analyzing results, and enabling our AI agent’s participation in their contests.

Sakana AI

237,195 views • 1 year ago

We are proud to announce the launch of our platform and the THREE initial agents: the Data Aggregation Agent, Sentiment Analysis Agent and Risk Manager Agent. This demo clip is a culmination of months of hard work and dedication to our vision. The platform have now entered a private alpha test which will be followed by a public beta. Tomorrow we will be doing a live streamed demo of the platform as it exists right now and will also be opening our waitlist to the general public. Thank you for your patience, belief and support. $cainam

We are proud to announce the launch of our platform and the THREE initial agents: the Data Aggregation Agent, Sentiment Analysis Agent and Risk Manager Agent. This demo clip is a culmination of months of hard work and dedication to our vision. The platform have now entered a private alpha test which will be followed by a public beta. Tomorrow we will be doing a live streamed demo of the platform as it exists right now and will also be opening our waitlist to the general public. Thank you for your patience, belief and support. $cainam

Cainam Ventures

12,101 views • 1 year ago

Inviting early beta testers for our upcoming SWE agent, Asteria. We're solving AI code comprehension, where the context window and the size of your codebase are no longer limitations, and reinventing the UX of how coding agents are currently implemented. We initially built this to address false positives and enhance accuracy for our SAST agent, which autonomously identifies security vulnerabilities, traces how a payload would flow through code, and ultimately demonstrates an actual exploit. The performance improved so significantly that we decided to test it on realistic engineering tasks on large codebases and witnessed SOTA results. Reply "access" if you'd like to be an early beta tester, a lot of quirks to fix.

Inviting early beta testers for our upcoming SWE agent, Asteria. We're solving AI code comprehension, where the context window and the size of your codebase are no longer limitations, and reinventing the UX of how coding agents are currently implemented. We initially built this to address false positives and enhance accuracy for our SAST agent, which autonomously identifies security vulnerabilities, traces how a payload would flow through code, and ultimately demonstrates an actual exploit. The performance improved so significantly that we decided to test it on realistic engineering tasks on large codebases and witnessed SOTA results. Reply "access" if you'd like to be an early beta tester, a lot of quirks to fix.

mufeed vh

12,408 views • 1 year ago

Llama-4 Maverick Is Now Available On ChatLLM First impressions - extremely economical - searches the web and does a decent job of analysis and writing - doesn't seem very strong at coding. Check it out on our playground Seems like a solid alternative to Flash 2.0.

Llama-4 Maverick Is Now Available On ChatLLM First impressions - extremely economical - searches the web and does a decent job of analysis and writing - doesn't seem very strong at coding. Check it out on our playground Seems like a solid alternative to Flash 2.0.

Bindu Reddy

33,577 views • 1 year ago

RouteLLM - Route To The Best LLM Based On Your Prompt Our RouteLLM got an upgrade this week.., it got smarter at picking the right LLM - o1 for complex queries - gpt4o for quick answers - sonnet for coding - deepseek for simple code - gemini for long context - llama and mini for simple question It optimizes for performance, speed and cost!

RouteLLM - Route To The Best LLM Based On Your Prompt Our RouteLLM got an upgrade this week.., it got smarter at picking the right LLM - o1 for complex queries - gpt4o for quick answers - sonnet for coding - deepseek for simple code - gemini for long context - llama and mini for simple question It optimizes for performance, speed and cost!

Bindu Reddy

17,860 views • 1 year ago

IN NEWS: Crusoe acquires Atero We spoke with Chase Lochmiller (CEO of Crusoe) and Alon Yariv (CEO of Atero) about what the deal unlocks. “Our investment is about accelerating performance, enhancing GPU utilization, and driving reliability. By optimizing storage and usage, we can drive down the cost per token, allowing customers to maximize value.” He added: “Our goal is to invest in the platform and drive massive performance gains to enable our customers to create more value using Crusoe Cloud.”

IN NEWS: Crusoe acquires Atero We spoke with Chase Lochmiller (CEO of Crusoe) and Alon Yariv (CEO of Atero) about what the deal unlocks. “Our investment is about accelerating performance, enhancing GPU utilization, and driving reliability. By optimizing storage and usage, we can drive down the cost per token, allowing customers to maximize value.” He added: “Our goal is to invest in the platform and drive massive performance gains to enable our customers to create more value using Crusoe Cloud.”

TBPN

12,721 views • 10 months ago

Why should we be prioritising social and emotional development? Hear what some of our experts friends from across the early childhood sector have to say. ▶️Watch the full film on our YouTube channel: #ShapingUs

Why should we be prioritising social and emotional development? Hear what some of our experts friends from across the early childhood sector have to say. ▶️Watch the full film on our YouTube channel: #ShapingUs

The Royal Foundation Centre for Early Childhood

55,935 views • 1 year ago