Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Traditional coding benchmarks do not reflect how software is actually built and maintained. That's why we built a new benchmark, APEX-SWE, in partnership with Cognition. It measures whether AI models can perform complex, real-world software engineering work to ship systems that work and debug them when they don't. OpenAI... show more

adarsh

7,551 subscribers

220,973 views • 7 months ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

33 Comments

adarsh7 months ago

APEX-SWE Leaderboard | Pass@1 🥇@OpenAI GPT-5.3 Codex (High) at 41.5% 🥈@AnthropicAI Opus 4.6 (High) at 40.5% 🥉@AnthropicAI Opus 4.5 (High) at 38.7% Every frontier model fails on nearly 60% of real production tasks. Check out the full APEX-SWE leaderboard:

adarsh7 months ago

We have an open-source dev set on @huggingface: And, our eval harness on @GitHub:

adarsh7 months ago

You can read more about APEX-SWE here:

Logan Watchorn7 months ago

@cognition Let the hill climbing begin!

Vidit Agrawal7 months ago

@cognition APEX-SWE is a much better proxy for real engineering work than toy coding benchmarks—testing whether models can actually build, ship, and debug working systems. Great to see GPT 5.3 Codex (High) leading at 41.5% Pass@1.

khushi suri7 months ago

@cognition :o our quants

Robert Wachen7 months ago

@cognition Very exciting

musho7 months ago

@cognition 🚀

Dhruv Ahuja7 months ago

@cognition super insightful stuff

Grace Li7 months ago

@cognition congrats!

Sahil Panhotra | Indie Builder7 months ago

@cognition i am surprised to know that OpenAi is winning in this when i thought it gonna be claude opus 4.6

Kevin Cherian7 months ago

@suryamidha @cognition Go @AbhiKottamasu!

Jase7 months ago

@cognition any chance of adding xhigh for 5.4 and 5.3 codex?

Om Londhe7 months ago

@cognition Amazing @OpenAI @AnthropicAI 🚀

Nikhil Sharma7 months ago

@cognition The companies most motivated to prove AI can replace developers built the benchmark measuring it. Mercor does AI hiring. Cognition made Devin. These are not neutral observers. The incentive structure is baked into the methodology.

Ege Erdil7 months ago

@cognition opus 4.5 > opus 4.6? lol

Armin Hamrah7 months ago

@cognition Congrats Adarsh & team!

Leonard Tang7 months ago

@rhythmrg @cognition oh this is fire

Ravi Riley7 months ago

@cognition ultimate indians + asians collab

aDaD5bC6 months ago

@cognition ????

Kaodi Dike7 months ago

@cognition 🚀🚀

Kai August7 months ago

@cognition 🔥

Tom's KI Ecke7 months ago

@cognition Definitely makes sense, appreciate your take!

Omkar Savant7 months ago

@cognition congrats! Seems great

Austin Jordan7 months ago

@cognition Great share

LapMatt6 months ago

@cognition Very insightful! APEX-SWE sets a new standard for evaluating AI in software engineering. It's crucial that our benchmarks reflect real-world challenges, as almost 60% of models struggle with production tasks. Exciting times ahead! 🚀

Kiran Bharambe7 months ago

@cognition Real-world complexity is where most models struggle.

T_Khanh2026.eth 🐬TermMax7 months ago

@cognition Finally measuring what matters

Bethany Hyde7 months ago

@cognition Great one

Daily Wisdom7 months ago

@cognition Numbers aside, the direction here matters more.

Max Rimpel7 months ago

@cognition 🔥🔥🔥

Mr. Luis7 months ago

@cognition All

Maanas Baraya7 months ago

@cognition 🔥🔥🔥

Related Videos

Wave 9 is here: a frontier model built for software engineering. Introducing our new family of models: SWE-1, SWE-1-lite, and SWE-1-mini. Based on internal evals, it has performance nearing that of frontier models from the foundation labs. Available now, only in Windsurf!

Wave 9 is here: a frontier model built for software engineering. Introducing our new family of models: SWE-1, SWE-1-lite, and SWE-1-mini. Based on internal evals, it has performance nearing that of frontier models from the foundation labs. Available now, only in Windsurf!

Devin Developers

743,638 views • 1 year ago

It's month close. AI models can pass the CPA exam, but do you trust them to close the books? Introducing APEX-Accounting, built in collaboration with Ramp and Ramp Labs, to test whether frontier AI can do real accounting work. It’s a new benchmark comprising 160 tasks across 10 simulated companies, authored and graded by accounting professionals with experience at firms like Deloitte, PwC, EY, and KPMG.

It's month close. AI models can pass the CPA exam, but do you trust them to close the books? Introducing APEX-Accounting, built in collaboration with Ramp and Ramp Labs, to test whether frontier AI can do real accounting work. It’s a new benchmark comprising 160 tasks across 10 simulated companies, authored and graded by accounting professionals with experience at firms like Deloitte, PwC, EY, and KPMG.

Mercor

142,289 views • 2 months ago

Vibe coding is causing the death of SaaS... Because we want better software: more features, more workflows, more EVERYTHING We're not happy with existing software, that's why we vibe code something ourselves. But what if... existing software came built-in with an AI vibe coding layer? If it did: that software can sell more, beat AI-first companies, and reduce churn. We've already proven this works - that's how we added $1,000,000 in sales and prevented $120,000 in churn for our Series B customers. Gigacatalyst embeds a vibe coding layer inside your software. See how it works: If you could improve any daily software that you use, what would you change?

Vibe coding is causing the death of SaaS... Because we want better software: more features, more workflows, more EVERYTHING We're not happy with existing software, that's why we vibe code something ourselves. But what if... existing software came built-in with an AI vibe coding layer? If it did: that software can sell more, beat AI-first companies, and reduce churn. We've already proven this works - that's how we added $1,000,000 in sales and prevented $120,000 in churn for our Series B customers. Gigacatalyst embeds a vibe coding layer inside your software. See how it works: If you could improve any daily software that you use, what would you change?

Namanyay

19,446 views • 4 months ago

When the way software is built changes, we help companies adopt it. For over a decade, that meant bringing React Native to production across mobile, web, desktop, and TV. Now we are expanding that same mission to AI-native engineering: helping teams ship software with agents, safely and at scale. This is our new manifesto.

When the way software is built changes, we help companies adopt it. For over a decade, that meant bringing React Native to production across mobile, web, desktop, and TV. Now we are expanding that same mission to AI-native engineering: helping teams ship software with agents, safely and at scale. This is our new manifesto.

Callstack Engineers

24,865 views • 4 months ago

Are AI agents ready to be your virtual coworker? Can they write your emails, build Excel models, and create slide decks? Introducing APEX-Agents, a frontier benchmark that tests how well AI agents complete real, long-horizon professional services deliverables in Google Workspace. Mercor APEX-Agents Pass@1 leaderboard: 🥇 Google Gemini 3 Flash (High): 24.0% 🥈 OpenAI GPT-5.2 (High): 23.0% 🥉 Paul Jankura Claude Opus 4.5 (High): 18.4%

Are AI agents ready to be your virtual coworker? Can they write your emails, build Excel models, and create slide decks? Introducing APEX-Agents, a frontier benchmark that tests how well AI agents complete real, long-horizon professional services deliverables in Google Workspace. Mercor APEX-Agents Pass@1 leaderboard: 🥇 Google Gemini 3 Flash (High): 24.0% 🥈 OpenAI GPT-5.2 (High): 23.0% 🥉 Paul Jankura Claude Opus 4.5 (High): 18.4%

Brendan (can/do)

145,326 views • 9 months ago

Building a star on Earth is a big engineering challenge. To tackle it, we’ve partnered with two industry powerhouses to create a high-fidelity digital twin of our #SPARC fusion machine. Here’s how this partnership is coming to life: ⚡️ Siemens: Data and Design — #Siemens software like Teamcenter and Designcenter NX let us manage SPARC’s complex design, housing data for our digital twin and handling traditional computer-aided design work. ⚡️ NVIDIA: AI Accelerator — NVIDIA’s Omniverse library provides a visually intuitive, 3D mirror of real-world SPARC that could improve planning and operations. And #NVIDIA software tools can help us build AI models of physics simulations that run enormously faster than traditional software. ⚡️ CFS: The Fusion Frontier — Integrating real-world and simulation data will let us benefit from AI-boosted physics software and compare its results with real-world data from how SPARC actually operates. Dive deeper into the details here and learn more about how our power trio is accelerating commercial fusion: #FusionEnergy #DigitalTwin #AI

Building a star on Earth is a big engineering challenge. To tackle it, we’ve partnered with two industry powerhouses to create a high-fidelity digital twin of our #SPARC fusion machine. Here’s how this partnership is coming to life: ⚡️ Siemens: Data and Design — #Siemens software like Teamcenter and Designcenter NX let us manage SPARC’s complex design, housing data for our digital twin and handling traditional computer-aided design work. ⚡️ NVIDIA: AI Accelerator — NVIDIA’s Omniverse library provides a visually intuitive, 3D mirror of real-world SPARC that could improve planning and operations. And #NVIDIA software tools can help us build AI models of physics simulations that run enormously faster than traditional software. ⚡️ CFS: The Fusion Frontier — Integrating real-world and simulation data will let us benefit from AI-boosted physics software and compare its results with real-world data from how SPARC actually operates. Dive deeper into the details here and learn more about how our power trio is accelerating commercial fusion: #FusionEnergy #DigitalTwin #AI

Commonwealth Fusion Systems

13,875 views • 9 months ago

Jev and the System One Model: RLCD, intelligence/$, reliable AI, & the end of chat-first AI TypeSafe AI CEO Diogo Almeida explains why AI can solve extraordinarily hard problems yet still fail to automate basic work, why Jev is built for reliable decisions inside software instead of chat, why TypeSafe rejects public benchmarks and refusals at the API layer, why data and the right task matter more than brute-force compute, how System One Models could reshape coding agents and software, and why even with $1 billion he wouldn’t pre-train a model from scratch.

Jev and the System One Model: RLCD, intelligence/$, reliable AI, & the end of chat-first AI TypeSafe AI CEO Diogo Almeida explains why AI can solve extraordinarily hard problems yet still fail to automate basic work, why Jev is built for reliable decisions inside software instead of chat, why TypeSafe rejects public benchmarks and refusals at the API layer, why data and the right task matter more than brute-force compute, how System One Models could reshape coding agents and software, and why even with $1 billion he wouldn’t pre-train a model from scratch.

Latent.Space

415,091 views • 1 month ago

Most AI coding tools focus on helping one developer work faster. Reason is taking a different approach. The idea of a multiplayer coding agent is interesting because software development rarely happens in isolation. Reason lets AI agents and humans work together, split tasks, debug issues and verify solutions instead of relying on a single agent to do everything. 🔹 Parallel collaboration 🔹 Shared context across sessions 🔹 From code changes to tested solutions AI coding is moving beyond autocomplete. The next step is building systems that can actually work together.

Most AI coding tools focus on helping one developer work faster. Reason is taking a different approach. The idea of a multiplayer coding agent is interesting because software development rarely happens in isolation. Reason lets AI agents and humans work together, split tasks, debug issues and verify solutions instead of relying on a single agent to do everything. 🔹 Parallel collaboration 🔹 Shared context across sessions 🔹 From code changes to tested solutions AI coding is moving beyond autocomplete. The next step is building systems that can actually work together.

MΞK

10,892 views • 7 days ago

Today we're excited to introduce Devin, the first AI software engineer. Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork. Devin is an autonomous agent that solves engineering tasks through the use of its own shell, code editor, and web browser. When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted. Check out what Devin can do in the thread below.

Today we're excited to introduce Devin, the first AI software engineer. Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork. Devin is an autonomous agent that solves engineering tasks through the use of its own shell, code editor, and web browser. When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted. Check out what Devin can do in the thread below.

Cognition

31,484,525 views • 2 years ago

Can AI actually automate jobs? Scale AI and are launching the Remote Labor Index (RLI), the first benchmark and public leaderboard that test how well AI agents can complete real, paid freelance work in domains like software engineering, design, architecture, data analysis, and more. Early results show the limits of today’s models. The top AI agent completed just 2.5% of real freelance jobs better than humans. AI is powerful, but not yet reliable enough to replace skilled labor. RLI gives us a transparent way to track progress over time and bring clarity to the future of work.

Can AI actually automate jobs? Scale AI and are launching the Remote Labor Index (RLI), the first benchmark and public leaderboard that test how well AI agents can complete real, paid freelance work in domains like software engineering, design, architecture, data analysis, and more. Early results show the limits of today’s models. The top AI agent completed just 2.5% of real freelance jobs better than humans. AI is powerful, but not yet reliable enough to replace skilled labor. RLI gives us a transparent way to track progress over time and bring clarity to the future of work.

Bing Liu

424,662 views • 11 months ago

The new Factory software AI agents just combined OpenAI’s Codex, Devin, and Cursor into one product. They are fully autonomous agents called Droids that perform deep code research, technical docs, and do autonomous end-to-end coding. Here’s how it works (with real examples)👇:

The new Factory software AI agents just combined OpenAI’s Codex, Devin, and Cursor into one product. They are fully autonomous agents called Droids that perform deep code research, technical docs, and do autonomous end-to-end coding. Here’s how it works (with real examples)👇:

Alvaro Cintas

94,884 views • 1 year ago

TypeSafe AI's Diogo Almeida's elevator pitch for Jev: "Where the f**k is all the automation?" "AI is so unbelievably smart. Not that I hate on chatbots or coding agents, I love them myself, but it's so useless at all other stuff. It's tragic." "TypeSafe is making AI for software. We want to make AI powerful, not just for humans in the loop, but to actually build real software. Jev, to us, is our first model in this whole space to make [automation] way, way better." "I like Claude Code and Codex. I love the description from Garry Tan and them: It's just-in-time software. It makes software on the fly, and you can program software in natural language, but it has the same expressive power as software." "What I want instead is smart software. Instead of automating software engineering, I want to expand what software itself can do, such that things that should be automatable can then be automatable." "I want to express things like intent. I want to expand the vocabulary of what we can do." "Programming is hyper-specifying valuable things and then infinitely replicating them. It's so freaking cool." Diogo Almeida

TypeSafe AI's Diogo Almeida's elevator pitch for Jev: "Where the f**k is all the automation?" "AI is so unbelievably smart. Not that I hate on chatbots or coding agents, I love them myself, but it's so useless at all other stuff. It's tragic." "TypeSafe is making AI for software. We want to make AI powerful, not just for humans in the loop, but to actually build real software. Jev, to us, is our first model in this whole space to make [automation] way, way better." "I like Claude Code and Codex. I love the description from Garry Tan and them: It's just-in-time software. It makes software on the fly, and you can program software in natural language, but it has the same expressive power as software." "What I want instead is smart software. Instead of automating software engineering, I want to expand what software itself can do, such that things that should be automatable can then be automatable." "I want to express things like intent. I want to expand the vocabulary of what we can do." "Programming is hyper-specifying valuable things and then infinitely replicating them. It's so freaking cool." Diogo Almeida

a16z

39,453 views • 24 days ago

“AI agents will outperform humans at almost all jobs by 2026–2027.” - The forecast is everywhere. So we built the exam to test that claim, on real labor-market aligned work. On the hardest tier, top agents pass 2.6%. Meet Agents' Last Exam (ALE), a rolling benchmark measuring whether agents can actually do real jobs. 🧵👇

“AI agents will outperform humans at almost all jobs by 2026–2027.” - The forecast is everywhere. So we built the exam to test that claim, on real labor-market aligned work. On the hardest tier, top agents pass 2.6%. Meet Agents' Last Exam (ALE), a rolling benchmark measuring whether agents can actually do real jobs. 🧵👇

Yiyou Sun

98,576 views • 4 months ago

OpenAI Greg Brockman: GPT-5.5 is not an endpoint — it's a beginning It is an early step toward stronger models arriving in the coming months, with larger gains across many capabilities The focus is not just on better benchmarks, but usefulness in the real world, for real users and real-world work

OpenAI Greg Brockman: GPT-5.5 is not an endpoint — it's a beginning It is an early step toward stronger models arriving in the coming months, with larger gains across many capabilities The focus is not just on better benchmarks, but usefulness in the real world, for real users and real-world work

Haider.

40,201 views • 6 months ago

Most of the market is priced for a world that's coming to an end. Traditional software, legacy platforms, much of the SaaS layer of the S&P 500… all built on the assumption that humans will remain the primary customers. AI agents don't buy software the way we do. They don't need dashboards, subscriptions, or logins. And once the buyer changes, so does everything built around them. The investable opportunity ahead is the agentic economy… what gets built for agents as customers. That's where the next wave of capital ends up, and most people aren't positioned for it.

Most of the market is priced for a world that's coming to an end. Traditional software, legacy platforms, much of the SaaS layer of the S&P 500… all built on the assumption that humans will remain the primary customers. AI agents don't buy software the way we do. They don't need dashboards, subscriptions, or logins. And once the buyer changes, so does everything built around them. The investable opportunity ahead is the agentic economy… what gets built for agents as customers. That's where the next wave of capital ends up, and most people aren't positioned for it.

Raoul Pal

215,413 views • 3 months ago

Here’s the problem: software interfaces are abstract, hardware is instinctive. We don’t learn objects, we recognize them. Buttons, wheels, friction, position they map to how we exist in the real world. Menus, layers, hidden states they fight against it. This is why this feels different. It’s not an app pretending to be a recorder. It’s a recorder that happens to exist in software. Built in wabi

Here’s the problem: software interfaces are abstract, hardware is instinctive. We don’t learn objects, we recognize them. Buttons, wheels, friction, position they map to how we exist in the real world. Menus, layers, hidden states they fight against it. This is why this feels different. It’s not an app pretending to be a recorder. It’s a recorder that happens to exist in software. Built in wabi

Tykra

30,343 views • 7 months ago

Mind-1: Physical AI at Human Speed, Built for Real Work Completing a task is only the beginning. Doing it at a pace that works in the real world is the next step—and that is what Mind-1 is built to achieve.

Mind-1: Physical AI at Human Speed, Built for Real Work Completing a task is only the beginning. Doing it at a pace that works in the real world is the next step—and that is what Mind-1 is built to achieve.

MindOn

141,343 views • 23 days ago

Handshake CEO Garrett Lord explains why AI agents exploded in software engineering but not the rest of white-collar work: "In software engineering, the average engineer is two, three, four times more productive. Software engineering agents are able to operate for multiple days at a time, automating hundreds of hours of work." "CEOs are not seeing that level of automation and ROI in other functions beyond software engineering. Software engineering is accelerated in such a massive way because everything was written down in GitHub. And it's extremely clear with what the right answer is." "When you have enough verifiable rewards and you scale up enough compute against verifiable rewards, you get what happened in software engineering, which is agents able to operate and do the job of humans." Sahil Bhaiwala Handshake

Handshake CEO Garrett Lord explains why AI agents exploded in software engineering but not the rest of white-collar work: "In software engineering, the average engineer is two, three, four times more productive. Software engineering agents are able to operate for multiple days at a time, automating hundreds of hours of work." "CEOs are not seeing that level of automation and ROI in other functions beyond software engineering. Software engineering is accelerated in such a massive way because everything was written down in GitHub. And it's extremely clear with what the right answer is." "When you have enough verifiable rewards and you scale up enough compute against verifiable rewards, you get what happened in software engineering, which is agents able to operate and do the job of humans." Sahil Bhaiwala Handshake

MTS

14,513 views • 1 month ago

Introducing Autobot 🤖 Software agencies still overcharge clients by the hour. AI has fundamentally changed how software gets built, but clients aren't seeing those savings. So we built Autobot, the first autonomous Replit ⠕ agency that reduce the cost of agency-built software by 90%. Describe what you want to build, answer a few questions, and about an hour later your first working version is ready, for free. Try it, share it and if you love it, you can keep it. If not, you don't pay anything. That's how custom software should be bought.

Introducing Autobot 🤖 Software agencies still overcharge clients by the hour. AI has fundamentally changed how software gets built, but clients aren't seeing those savings. So we built Autobot, the first autonomous Replit ⠕ agency that reduce the cost of agency-built software by 90%. Describe what you want to build, answer a few questions, and about an hour later your first working version is ready, for free. Try it, share it and if you love it, you can keep it. If not, you don't pay anything. That's how custom software should be bought.

viktorTJM

91,220 views • 3 months ago