正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Traditional coding benchmarks do not reflect how software is actually built and maintained. That's why we built a new benchmark, APEX-SWE, in partnership with Cognition. It measures whether AI models can perform complex, real-world software engineering work to ship systems that work and debug them when they don't. OpenAI... show more

adarsh

7,551 subscribers

220,973 次观看 • 7 个月前 •via X (Twitter)

教育科学技术

Anya Rossi• Live Now

Private livecam show

33 条评论

adarsh 的头像

adarsh7 个月前

APEX-SWE Leaderboard | Pass@1 🥇@OpenAI GPT-5.3 Codex (High) at 41.5% 🥈@AnthropicAI Opus 4.6 (High) at 40.5% 🥉@AnthropicAI Opus 4.5 (High) at 38.7% Every frontier model fails on nearly 60% of real production tasks. Check out the full APEX-SWE leaderboard:

adarsh 的头像

adarsh7 个月前

We have an open-source dev set on @huggingface: And, our eval harness on @GitHub:

adarsh 的头像

adarsh7 个月前

You can read more about APEX-SWE here:

Logan Watchorn 的头像

Logan Watchorn7 个月前

@cognition Let the hill climbing begin!

Vidit Agrawal 的头像

Vidit Agrawal7 个月前

@cognition APEX-SWE is a much better proxy for real engineering work than toy coding benchmarks—testing whether models can actually build, ship, and debug working systems. Great to see GPT 5.3 Codex (High) leading at 41.5% Pass@1.

khushi suri 的头像

khushi suri7 个月前

@cognition :o our quants

Robert Wachen 的头像

Robert Wachen7 个月前

@cognition Very exciting

musho 的头像

musho7 个月前

@cognition 🚀

Dhruv Ahuja 的头像

Dhruv Ahuja7 个月前

@cognition super insightful stuff

Grace Li 的头像

Grace Li7 个月前

@cognition congrats!

Sahil Panhotra | Indie Builder 的头像

Sahil Panhotra | Indie Builder7 个月前

@cognition i am surprised to know that OpenAi is winning in this when i thought it gonna be claude opus 4.6

Kevin Cherian 的头像

Kevin Cherian7 个月前

@suryamidha @cognition Go @AbhiKottamasu!

Jase 的头像

Jase7 个月前

@cognition any chance of adding xhigh for 5.4 and 5.3 codex?

Om Londhe 的头像

Om Londhe7 个月前

@cognition Amazing @OpenAI @AnthropicAI 🚀

Nikhil Sharma 的头像

Nikhil Sharma7 个月前

@cognition The companies most motivated to prove AI can replace developers built the benchmark measuring it. Mercor does AI hiring. Cognition made Devin. These are not neutral observers. The incentive structure is baked into the methodology.

Ege Erdil 的头像

Ege Erdil7 个月前

@cognition opus 4.5 > opus 4.6? lol

Armin Hamrah 的头像

Armin Hamrah7 个月前

@cognition Congrats Adarsh & team!

Leonard Tang 的头像

Leonard Tang7 个月前

@rhythmrg @cognition oh this is fire

Ravi Riley 的头像

Ravi Riley7 个月前

@cognition ultimate indians + asians collab

aDaD5bC 的头像

aDaD5bC6 个月前

@cognition ????

Kaodi Dike 的头像

Kaodi Dike7 个月前

@cognition 🚀🚀

Kai August 的头像

Kai August7 个月前

@cognition 🔥

Tom's KI Ecke 的头像

Tom's KI Ecke7 个月前

@cognition Definitely makes sense, appreciate your take!

Omkar Savant 的头像

Omkar Savant7 个月前

@cognition congrats! Seems great

Austin Jordan 的头像

Austin Jordan7 个月前

@cognition Great share

LapMatt 的头像

LapMatt6 个月前

@cognition Very insightful! APEX-SWE sets a new standard for evaluating AI in software engineering. It's crucial that our benchmarks reflect real-world challenges, as almost 60% of models struggle with production tasks. Exciting times ahead! 🚀

Kiran Bharambe 的头像

Kiran Bharambe7 个月前

@cognition Real-world complexity is where most models struggle.

T_Khanh2026.eth 🐬TermMax 的头像

T_Khanh2026.eth 🐬TermMax7 个月前

@cognition Finally measuring what matters

Bethany Hyde 的头像

Bethany Hyde7 个月前

@cognition Great one

Daily Wisdom 的头像

Daily Wisdom7 个月前

@cognition Numbers aside, the direction here matters more.

Max Rimpel 的头像

Max Rimpel7 个月前

@cognition 🔥🔥🔥

Mr. Luis 的头像

Mr. Luis7 个月前

@cognition All

Maanas Baraya 的头像

Maanas Baraya7 个月前

@cognition 🔥🔥🔥

相关视频

Wave 9 is here: a frontier model built for software engineering. Introducing our new family of models: SWE-1, SWE-1-lite, and SWE-1-mini. Based on internal evals, it has performance nearing that of frontier models from the foundation labs. Available now, only in Windsurf!

Wave 9 is here: a frontier model built for software engineering. Introducing our new family of models: SWE-1, SWE-1-lite, and SWE-1-mini. Based on internal evals, it has performance nearing that of frontier models from the foundation labs. Available now, only in Windsurf!

Devin Developers

743,638 次观看 • 1 年前

It's month close. AI models can pass the CPA exam, but do you trust them to close the books? Introducing APEX-Accounting, built in collaboration with Ramp and Ramp Labs, to test whether frontier AI can do real accounting work. It’s a new benchmark comprising 160 tasks across 10 simulated companies, authored and graded by accounting professionals with experience at firms like Deloitte, PwC, EY, and KPMG.

It's month close. AI models can pass the CPA exam, but do you trust them to close the books? Introducing APEX-Accounting, built in collaboration with Ramp and Ramp Labs, to test whether frontier AI can do real accounting work. It’s a new benchmark comprising 160 tasks across 10 simulated companies, authored and graded by accounting professionals with experience at firms like Deloitte, PwC, EY, and KPMG.

Mercor

142,289 次观看 • 2 个月前

Vibe coding is causing the death of SaaS... Because we want better software: more features, more workflows, more EVERYTHING We're not happy with existing software, that's why we vibe code something ourselves. But what if... existing software came built-in with an AI vibe coding layer? If it did: that software can sell more, beat AI-first companies, and reduce churn. We've already proven this works - that's how we added $1,000,000 in sales and prevented $120,000 in churn for our Series B customers. Gigacatalyst embeds a vibe coding layer inside your software. See how it works: If you could improve any daily software that you use, what would you change?

Vibe coding is causing the death of SaaS... Because we want better software: more features, more workflows, more EVERYTHING We're not happy with existing software, that's why we vibe code something ourselves. But what if... existing software came built-in with an AI vibe coding layer? If it did: that software can sell more, beat AI-first companies, and reduce churn. We've already proven this works - that's how we added $1,000,000 in sales and prevented $120,000 in churn for our Series B customers. Gigacatalyst embeds a vibe coding layer inside your software. See how it works: If you could improve any daily software that you use, what would you change?

Namanyay

19,446 次观看 • 4 个月前

When the way software is built changes, we help companies adopt it. For over a decade, that meant bringing React Native to production across mobile, web, desktop, and TV. Now we are expanding that same mission to AI-native engineering: helping teams ship software with agents, safely and at scale. This is our new manifesto.

When the way software is built changes, we help companies adopt it. For over a decade, that meant bringing React Native to production across mobile, web, desktop, and TV. Now we are expanding that same mission to AI-native engineering: helping teams ship software with agents, safely and at scale. This is our new manifesto.

Callstack Engineers

24,865 次观看 • 4 个月前

Are AI agents ready to be your virtual coworker? Can they write your emails, build Excel models, and create slide decks? Introducing APEX-Agents, a frontier benchmark that tests how well AI agents complete real, long-horizon professional services deliverables in Google Workspace. Mercor APEX-Agents Pass@1 leaderboard: 🥇 Google Gemini 3 Flash (High): 24.0% 🥈 OpenAI GPT-5.2 (High): 23.0% 🥉 Paul Jankura Claude Opus 4.5 (High): 18.4%

Are AI agents ready to be your virtual coworker? Can they write your emails, build Excel models, and create slide decks? Introducing APEX-Agents, a frontier benchmark that tests how well AI agents complete real, long-horizon professional services deliverables in Google Workspace. Mercor APEX-Agents Pass@1 leaderboard: 🥇 Google Gemini 3 Flash (High): 24.0% 🥈 OpenAI GPT-5.2 (High): 23.0% 🥉 Paul Jankura Claude Opus 4.5 (High): 18.4%

Brendan (can/do)

145,326 次观看 • 9 个月前

Building a star on Earth is a big engineering challenge. To tackle it, we’ve partnered with two industry powerhouses to create a high-fidelity digital twin of our #SPARC fusion machine. Here’s how this partnership is coming to life: ⚡️ Siemens: Data and Design — #Siemens software like Teamcenter and Designcenter NX let us manage SPARC’s complex design, housing data for our digital twin and handling traditional computer-aided design work. ⚡️ NVIDIA: AI Accelerator — NVIDIA’s Omniverse library provides a visually intuitive, 3D mirror of real-world SPARC that could improve planning and operations. And #NVIDIA software tools can help us build AI models of physics simulations that run enormously faster than traditional software. ⚡️ CFS: The Fusion Frontier — Integrating real-world and simulation data will let us benefit from AI-boosted physics software and compare its results with real-world data from how SPARC actually operates. Dive deeper into the details here and learn more about how our power trio is accelerating commercial fusion: #FusionEnergy #DigitalTwin #AI

Building a star on Earth is a big engineering challenge. To tackle it, we’ve partnered with two industry powerhouses to create a high-fidelity digital twin of our #SPARC fusion machine. Here’s how this partnership is coming to life: ⚡️ Siemens: Data and Design — #Siemens software like Teamcenter and Designcenter NX let us manage SPARC’s complex design, housing data for our digital twin and handling traditional computer-aided design work. ⚡️ NVIDIA: AI Accelerator — NVIDIA’s Omniverse library provides a visually intuitive, 3D mirror of real-world SPARC that could improve planning and operations. And #NVIDIA software tools can help us build AI models of physics simulations that run enormously faster than traditional software. ⚡️ CFS: The Fusion Frontier — Integrating real-world and simulation data will let us benefit from AI-boosted physics software and compare its results with real-world data from how SPARC actually operates. Dive deeper into the details here and learn more about how our power trio is accelerating commercial fusion: #FusionEnergy #DigitalTwin #AI

Commonwealth Fusion Systems

13,875 次观看 • 9 个月前

Jev and the System One Model: RLCD, intelligence/$, reliable AI, & the end of chat-first AI TypeSafe AI CEO Diogo Almeida explains why AI can solve extraordinarily hard problems yet still fail to automate basic work, why Jev is built for reliable decisions inside software instead of chat, why TypeSafe rejects public benchmarks and refusals at the API layer, why data and the right task matter more than brute-force compute, how System One Models could reshape coding agents and software, and why even with $1 billion he wouldn’t pre-train a model from scratch.

Jev and the System One Model: RLCD, intelligence/$, reliable AI, & the end of chat-first AI TypeSafe AI CEO Diogo Almeida explains why AI can solve extraordinarily hard problems yet still fail to automate basic work, why Jev is built for reliable decisions inside software instead of chat, why TypeSafe rejects public benchmarks and refusals at the API layer, why data and the right task matter more than brute-force compute, how System One Models could reshape coding agents and software, and why even with $1 billion he wouldn’t pre-train a model from scratch.

Latent.Space

415,091 次观看 • 1 个月前

Most AI coding tools focus on helping one developer work faster. Reason is taking a different approach. The idea of a multiplayer coding agent is interesting because software development rarely happens in isolation. Reason lets AI agents and humans work together, split tasks, debug issues and verify solutions instead of relying on a single agent to do everything. 🔹 Parallel collaboration 🔹 Shared context across sessions 🔹 From code changes to tested solutions AI coding is moving beyond autocomplete. The next step is building systems that can actually work together.

Most AI coding tools focus on helping one developer work faster. Reason is taking a different approach. The idea of a multiplayer coding agent is interesting because software development rarely happens in isolation. Reason lets AI agents and humans work together, split tasks, debug issues and verify solutions instead of relying on a single agent to do everything. 🔹 Parallel collaboration 🔹 Shared context across sessions 🔹 From code changes to tested solutions AI coding is moving beyond autocomplete. The next step is building systems that can actually work together.

MΞK

10,892 次观看 • 7 天前

Today we're excited to introduce Devin, the first AI software engineer. Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork. Devin is an autonomous agent that solves engineering tasks through the use of its own shell, code editor, and web browser. When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted. Check out what Devin can do in the thread below.

Today we're excited to introduce Devin, the first AI software engineer. Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork. Devin is an autonomous agent that solves engineering tasks through the use of its own shell, code editor, and web browser. When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted. Check out what Devin can do in the thread below.

Cognition

31,484,525 次观看 • 2 年前

Can AI actually automate jobs? Scale AI and are launching the Remote Labor Index (RLI), the first benchmark and public leaderboard that test how well AI agents can complete real, paid freelance work in domains like software engineering, design, architecture, data analysis, and more. Early results show the limits of today’s models. The top AI agent completed just 2.5% of real freelance jobs better than humans. AI is powerful, but not yet reliable enough to replace skilled labor. RLI gives us a transparent way to track progress over time and bring clarity to the future of work.

Can AI actually automate jobs? Scale AI and are launching the Remote Labor Index (RLI), the first benchmark and public leaderboard that test how well AI agents can complete real, paid freelance work in domains like software engineering, design, architecture, data analysis, and more. Early results show the limits of today’s models. The top AI agent completed just 2.5% of real freelance jobs better than humans. AI is powerful, but not yet reliable enough to replace skilled labor. RLI gives us a transparent way to track progress over time and bring clarity to the future of work.

Bing Liu

424,662 次观看 • 11 个月前

The new Factory software AI agents just combined OpenAI’s Codex, Devin, and Cursor into one product. They are fully autonomous agents called Droids that perform deep code research, technical docs, and do autonomous end-to-end coding. Here’s how it works (with real examples)👇:

The new Factory software AI agents just combined OpenAI’s Codex, Devin, and Cursor into one product. They are fully autonomous agents called Droids that perform deep code research, technical docs, and do autonomous end-to-end coding. Here’s how it works (with real examples)👇:

Alvaro Cintas

94,884 次观看 • 1 年前

TypeSafe AI's Diogo Almeida's elevator pitch for Jev: "Where the f**k is all the automation?" "AI is so unbelievably smart. Not that I hate on chatbots or coding agents, I love them myself, but it's so useless at all other stuff. It's tragic." "TypeSafe is making AI for software. We want to make AI powerful, not just for humans in the loop, but to actually build real software. Jev, to us, is our first model in this whole space to make [automation] way, way better." "I like Claude Code and Codex. I love the description from Garry Tan and them: It's just-in-time software. It makes software on the fly, and you can program software in natural language, but it has the same expressive power as software." "What I want instead is smart software. Instead of automating software engineering, I want to expand what software itself can do, such that things that should be automatable can then be automatable." "I want to express things like intent. I want to expand the vocabulary of what we can do." "Programming is hyper-specifying valuable things and then infinitely replicating them. It's so freaking cool." Diogo Almeida

TypeSafe AI's Diogo Almeida's elevator pitch for Jev: "Where the f**k is all the automation?" "AI is so unbelievably smart. Not that I hate on chatbots or coding agents, I love them myself, but it's so useless at all other stuff. It's tragic." "TypeSafe is making AI for software. We want to make AI powerful, not just for humans in the loop, but to actually build real software. Jev, to us, is our first model in this whole space to make [automation] way, way better." "I like Claude Code and Codex. I love the description from Garry Tan and them: It's just-in-time software. It makes software on the fly, and you can program software in natural language, but it has the same expressive power as software." "What I want instead is smart software. Instead of automating software engineering, I want to expand what software itself can do, such that things that should be automatable can then be automatable." "I want to express things like intent. I want to expand the vocabulary of what we can do." "Programming is hyper-specifying valuable things and then infinitely replicating them. It's so freaking cool." Diogo Almeida

a16z

39,453 次观看 • 24 天前

“AI agents will outperform humans at almost all jobs by 2026–2027.” - The forecast is everywhere. So we built the exam to test that claim, on real labor-market aligned work. On the hardest tier, top agents pass 2.6%. Meet Agents' Last Exam (ALE), a rolling benchmark measuring whether agents can actually do real jobs. 🧵👇

“AI agents will outperform humans at almost all jobs by 2026–2027.” - The forecast is everywhere. So we built the exam to test that claim, on real labor-market aligned work. On the hardest tier, top agents pass 2.6%. Meet Agents' Last Exam (ALE), a rolling benchmark measuring whether agents can actually do real jobs. 🧵👇

Yiyou Sun

98,576 次观看 • 4 个月前

OpenAI Greg Brockman: GPT-5.5 is not an endpoint — it's a beginning It is an early step toward stronger models arriving in the coming months, with larger gains across many capabilities The focus is not just on better benchmarks, but usefulness in the real world, for real users and real-world work

OpenAI Greg Brockman: GPT-5.5 is not an endpoint — it's a beginning It is an early step toward stronger models arriving in the coming months, with larger gains across many capabilities The focus is not just on better benchmarks, but usefulness in the real world, for real users and real-world work

Haider.

40,201 次观看 • 6 个月前

Most of the market is priced for a world that's coming to an end. Traditional software, legacy platforms, much of the SaaS layer of the S&P 500… all built on the assumption that humans will remain the primary customers. AI agents don't buy software the way we do. They don't need dashboards, subscriptions, or logins. And once the buyer changes, so does everything built around them. The investable opportunity ahead is the agentic economy… what gets built for agents as customers. That's where the next wave of capital ends up, and most people aren't positioned for it.

Most of the market is priced for a world that's coming to an end. Traditional software, legacy platforms, much of the SaaS layer of the S&P 500… all built on the assumption that humans will remain the primary customers. AI agents don't buy software the way we do. They don't need dashboards, subscriptions, or logins. And once the buyer changes, so does everything built around them. The investable opportunity ahead is the agentic economy… what gets built for agents as customers. That's where the next wave of capital ends up, and most people aren't positioned for it.

Raoul Pal

215,413 次观看 • 3 个月前

Here’s the problem: software interfaces are abstract, hardware is instinctive. We don’t learn objects, we recognize them. Buttons, wheels, friction, position they map to how we exist in the real world. Menus, layers, hidden states they fight against it. This is why this feels different. It’s not an app pretending to be a recorder. It’s a recorder that happens to exist in software. Built in wabi

Here’s the problem: software interfaces are abstract, hardware is instinctive. We don’t learn objects, we recognize them. Buttons, wheels, friction, position they map to how we exist in the real world. Menus, layers, hidden states they fight against it. This is why this feels different. It’s not an app pretending to be a recorder. It’s a recorder that happens to exist in software. Built in wabi

Tykra

30,343 次观看 • 7 个月前

Mind-1: Physical AI at Human Speed, Built for Real Work Completing a task is only the beginning. Doing it at a pace that works in the real world is the next step—and that is what Mind-1 is built to achieve.

Mind-1: Physical AI at Human Speed, Built for Real Work Completing a task is only the beginning. Doing it at a pace that works in the real world is the next step—and that is what Mind-1 is built to achieve.

MindOn

141,343 次观看 • 23 天前

Handshake CEO Garrett Lord explains why AI agents exploded in software engineering but not the rest of white-collar work: "In software engineering, the average engineer is two, three, four times more productive. Software engineering agents are able to operate for multiple days at a time, automating hundreds of hours of work." "CEOs are not seeing that level of automation and ROI in other functions beyond software engineering. Software engineering is accelerated in such a massive way because everything was written down in GitHub. And it's extremely clear with what the right answer is." "When you have enough verifiable rewards and you scale up enough compute against verifiable rewards, you get what happened in software engineering, which is agents able to operate and do the job of humans." Sahil Bhaiwala Handshake

Handshake CEO Garrett Lord explains why AI agents exploded in software engineering but not the rest of white-collar work: "In software engineering, the average engineer is two, three, four times more productive. Software engineering agents are able to operate for multiple days at a time, automating hundreds of hours of work." "CEOs are not seeing that level of automation and ROI in other functions beyond software engineering. Software engineering is accelerated in such a massive way because everything was written down in GitHub. And it's extremely clear with what the right answer is." "When you have enough verifiable rewards and you scale up enough compute against verifiable rewards, you get what happened in software engineering, which is agents able to operate and do the job of humans." Sahil Bhaiwala Handshake

MTS

14,513 次观看 • 1 个月前

Introducing Autobot 🤖 Software agencies still overcharge clients by the hour. AI has fundamentally changed how software gets built, but clients aren't seeing those savings. So we built Autobot, the first autonomous Replit ⠕ agency that reduce the cost of agency-built software by 90%. Describe what you want to build, answer a few questions, and about an hour later your first working version is ready, for free. Try it, share it and if you love it, you can keep it. If not, you don't pay anything. That's how custom software should be bought.

Introducing Autobot 🤖 Software agencies still overcharge clients by the hour. AI has fundamentally changed how software gets built, but clients aren't seeing those savings. So we built Autobot, the first autonomous Replit ⠕ agency that reduce the cost of agency-built software by 90%. Describe what you want to build, answer a few questions, and about an hour later your first working version is ready, for free. Try it, share it and if you love it, you can keep it. If not, you don't pay anything. That's how custom software should be bought.

viktorTJM

91,220 次观看 • 3 个月前