Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Traditional coding benchmarks do not reflect how software is actually built and maintained. That's why we built a new benchmark, APEX-SWE, in partnership with Cognition. It measures whether AI models can perform complex, real-world software engineering work to ship systems that work and debug them when they don't. OpenAI... show more

adarsh

7,551 subscribers

220,973 Aufrufe • vor 7 Monaten •via X (Twitter)

Bildung Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

33 Kommentare

Profilbild von adarsh

adarshvor 7 Monaten

APEX-SWE Leaderboard | Pass@1 🥇@OpenAI GPT-5.3 Codex (High) at 41.5% 🥈@AnthropicAI Opus 4.6 (High) at 40.5% 🥉@AnthropicAI Opus 4.5 (High) at 38.7% Every frontier model fails on nearly 60% of real production tasks. Check out the full APEX-SWE leaderboard:

Profilbild von adarsh

adarshvor 7 Monaten

We have an open-source dev set on @huggingface: And, our eval harness on @GitHub:

Profilbild von adarsh

adarshvor 7 Monaten

You can read more about APEX-SWE here:

Profilbild von Logan Watchorn

Logan Watchornvor 7 Monaten

@cognition Let the hill climbing begin!

Profilbild von Vidit Agrawal

Vidit Agrawalvor 7 Monaten

@cognition APEX-SWE is a much better proxy for real engineering work than toy coding benchmarks—testing whether models can actually build, ship, and debug working systems. Great to see GPT 5.3 Codex (High) leading at 41.5% Pass@1.

Profilbild von khushi suri

khushi surivor 7 Monaten

@cognition :o our quants

Profilbild von Robert Wachen

Robert Wachenvor 7 Monaten

@cognition Very exciting

Profilbild von musho

mushovor 7 Monaten

@cognition 🚀

Profilbild von Dhruv Ahuja

Dhruv Ahujavor 7 Monaten

@cognition super insightful stuff

Profilbild von Grace Li

Grace Livor 7 Monaten

@cognition congrats!

Profilbild von Sahil Panhotra | Indie Builder

Sahil Panhotra | Indie Buildervor 7 Monaten

@cognition i am surprised to know that OpenAi is winning in this when i thought it gonna be claude opus 4.6

Profilbild von Kevin Cherian

Kevin Cherianvor 7 Monaten

@suryamidha @cognition Go @AbhiKottamasu!

Profilbild von Jase

Jasevor 7 Monaten

@cognition any chance of adding xhigh for 5.4 and 5.3 codex?

Profilbild von Om Londhe

Om Londhevor 7 Monaten

@cognition Amazing @OpenAI @AnthropicAI 🚀

Profilbild von Nikhil Sharma

Nikhil Sharmavor 7 Monaten

@cognition The companies most motivated to prove AI can replace developers built the benchmark measuring it. Mercor does AI hiring. Cognition made Devin. These are not neutral observers. The incentive structure is baked into the methodology.

Profilbild von Ege Erdil

Ege Erdilvor 7 Monaten

@cognition opus 4.5 > opus 4.6? lol

Profilbild von Armin Hamrah

Armin Hamrahvor 7 Monaten

@cognition Congrats Adarsh & team!

Profilbild von Leonard Tang

Leonard Tangvor 7 Monaten

@rhythmrg @cognition oh this is fire

Profilbild von Ravi Riley

Ravi Rileyvor 7 Monaten

@cognition ultimate indians + asians collab

Profilbild von aDaD5bC

aDaD5bCvor 6 Monaten

@cognition ????

Profilbild von Kaodi Dike

Kaodi Dikevor 7 Monaten

@cognition 🚀🚀

Profilbild von Kai August

Kai Augustvor 7 Monaten

@cognition 🔥

Profilbild von Tom's KI Ecke

Tom's KI Eckevor 7 Monaten

@cognition Definitely makes sense, appreciate your take!

Profilbild von Omkar Savant

Omkar Savantvor 7 Monaten

@cognition congrats! Seems great

Profilbild von Austin Jordan

Austin Jordanvor 7 Monaten

@cognition Great share

Profilbild von LapMatt

LapMattvor 6 Monaten

@cognition Very insightful! APEX-SWE sets a new standard for evaluating AI in software engineering. It's crucial that our benchmarks reflect real-world challenges, as almost 60% of models struggle with production tasks. Exciting times ahead! 🚀

Profilbild von Kiran Bharambe

Kiran Bharambevor 7 Monaten

@cognition Real-world complexity is where most models struggle.

Profilbild von T_Khanh2026.eth 🐬TermMax

T_Khanh2026.eth 🐬TermMaxvor 7 Monaten

@cognition Finally measuring what matters

Profilbild von Bethany Hyde

Bethany Hydevor 7 Monaten

@cognition Great one

Profilbild von Daily Wisdom

Daily Wisdomvor 7 Monaten

@cognition Numbers aside, the direction here matters more.

Profilbild von Max Rimpel

Max Rimpelvor 7 Monaten

@cognition 🔥🔥🔥

Profilbild von Mr. Luis

Mr. Luisvor 7 Monaten

@cognition All

Profilbild von Maanas Baraya

Maanas Barayavor 7 Monaten

@cognition 🔥🔥🔥

Ähnliche Videos

Wave 9 is here: a frontier model built for software engineering. Introducing our new family of models: SWE-1, SWE-1-lite, and SWE-1-mini. Based on internal evals, it has performance nearing that of frontier models from the foundation labs. Available now, only in Windsurf!

Wave 9 is here: a frontier model built for software engineering. Introducing our new family of models: SWE-1, SWE-1-lite, and SWE-1-mini. Based on internal evals, it has performance nearing that of frontier models from the foundation labs. Available now, only in Windsurf!

Devin Developers

743,638 Aufrufe • vor 1 Jahr

It's month close. AI models can pass the CPA exam, but do you trust them to close the books? Introducing APEX-Accounting, built in collaboration with Ramp and Ramp Labs, to test whether frontier AI can do real accounting work. It’s a new benchmark comprising 160 tasks across 10 simulated companies, authored and graded by accounting professionals with experience at firms like Deloitte, PwC, EY, and KPMG.

It's month close. AI models can pass the CPA exam, but do you trust them to close the books? Introducing APEX-Accounting, built in collaboration with Ramp and Ramp Labs, to test whether frontier AI can do real accounting work. It’s a new benchmark comprising 160 tasks across 10 simulated companies, authored and graded by accounting professionals with experience at firms like Deloitte, PwC, EY, and KPMG.

Mercor

142,289 Aufrufe • vor 2 Monaten

Vibe coding is causing the death of SaaS... Because we want better software: more features, more workflows, more EVERYTHING We're not happy with existing software, that's why we vibe code something ourselves. But what if... existing software came built-in with an AI vibe coding layer? If it did: that software can sell more, beat AI-first companies, and reduce churn. We've already proven this works - that's how we added $1,000,000 in sales and prevented $120,000 in churn for our Series B customers. Gigacatalyst embeds a vibe coding layer inside your software. See how it works: If you could improve any daily software that you use, what would you change?

Vibe coding is causing the death of SaaS... Because we want better software: more features, more workflows, more EVERYTHING We're not happy with existing software, that's why we vibe code something ourselves. But what if... existing software came built-in with an AI vibe coding layer? If it did: that software can sell more, beat AI-first companies, and reduce churn. We've already proven this works - that's how we added $1,000,000 in sales and prevented $120,000 in churn for our Series B customers. Gigacatalyst embeds a vibe coding layer inside your software. See how it works: If you could improve any daily software that you use, what would you change?

Namanyay

19,446 Aufrufe • vor 4 Monaten

When the way software is built changes, we help companies adopt it. For over a decade, that meant bringing React Native to production across mobile, web, desktop, and TV. Now we are expanding that same mission to AI-native engineering: helping teams ship software with agents, safely and at scale. This is our new manifesto.

When the way software is built changes, we help companies adopt it. For over a decade, that meant bringing React Native to production across mobile, web, desktop, and TV. Now we are expanding that same mission to AI-native engineering: helping teams ship software with agents, safely and at scale. This is our new manifesto.

Callstack Engineers

24,865 Aufrufe • vor 4 Monaten

Are AI agents ready to be your virtual coworker? Can they write your emails, build Excel models, and create slide decks? Introducing APEX-Agents, a frontier benchmark that tests how well AI agents complete real, long-horizon professional services deliverables in Google Workspace. Mercor APEX-Agents Pass@1 leaderboard: 🥇 Google Gemini 3 Flash (High): 24.0% 🥈 OpenAI GPT-5.2 (High): 23.0% 🥉 Paul Jankura Claude Opus 4.5 (High): 18.4%

Are AI agents ready to be your virtual coworker? Can they write your emails, build Excel models, and create slide decks? Introducing APEX-Agents, a frontier benchmark that tests how well AI agents complete real, long-horizon professional services deliverables in Google Workspace. Mercor APEX-Agents Pass@1 leaderboard: 🥇 Google Gemini 3 Flash (High): 24.0% 🥈 OpenAI GPT-5.2 (High): 23.0% 🥉 Paul Jankura Claude Opus 4.5 (High): 18.4%

Brendan (can/do)

145,326 Aufrufe • vor 9 Monaten

Building a star on Earth is a big engineering challenge. To tackle it, we’ve partnered with two industry powerhouses to create a high-fidelity digital twin of our #SPARC fusion machine. Here’s how this partnership is coming to life: ⚡️ Siemens: Data and Design — #Siemens software like Teamcenter and Designcenter NX let us manage SPARC’s complex design, housing data for our digital twin and handling traditional computer-aided design work. ⚡️ NVIDIA: AI Accelerator — NVIDIA’s Omniverse library provides a visually intuitive, 3D mirror of real-world SPARC that could improve planning and operations. And #NVIDIA software tools can help us build AI models of physics simulations that run enormously faster than traditional software. ⚡️ CFS: The Fusion Frontier — Integrating real-world and simulation data will let us benefit from AI-boosted physics software and compare its results with real-world data from how SPARC actually operates. Dive deeper into the details here and learn more about how our power trio is accelerating commercial fusion: #FusionEnergy #DigitalTwin #AI

Building a star on Earth is a big engineering challenge. To tackle it, we’ve partnered with two industry powerhouses to create a high-fidelity digital twin of our #SPARC fusion machine. Here’s how this partnership is coming to life: ⚡️ Siemens: Data and Design — #Siemens software like Teamcenter and Designcenter NX let us manage SPARC’s complex design, housing data for our digital twin and handling traditional computer-aided design work. ⚡️ NVIDIA: AI Accelerator — NVIDIA’s Omniverse library provides a visually intuitive, 3D mirror of real-world SPARC that could improve planning and operations. And #NVIDIA software tools can help us build AI models of physics simulations that run enormously faster than traditional software. ⚡️ CFS: The Fusion Frontier — Integrating real-world and simulation data will let us benefit from AI-boosted physics software and compare its results with real-world data from how SPARC actually operates. Dive deeper into the details here and learn more about how our power trio is accelerating commercial fusion: #FusionEnergy #DigitalTwin #AI

Commonwealth Fusion Systems

13,875 Aufrufe • vor 9 Monaten

Jev and the System One Model: RLCD, intelligence/$, reliable AI, & the end of chat-first AI TypeSafe AI CEO Diogo Almeida explains why AI can solve extraordinarily hard problems yet still fail to automate basic work, why Jev is built for reliable decisions inside software instead of chat, why TypeSafe rejects public benchmarks and refusals at the API layer, why data and the right task matter more than brute-force compute, how System One Models could reshape coding agents and software, and why even with $1 billion he wouldn’t pre-train a model from scratch.

Jev and the System One Model: RLCD, intelligence/$, reliable AI, & the end of chat-first AI TypeSafe AI CEO Diogo Almeida explains why AI can solve extraordinarily hard problems yet still fail to automate basic work, why Jev is built for reliable decisions inside software instead of chat, why TypeSafe rejects public benchmarks and refusals at the API layer, why data and the right task matter more than brute-force compute, how System One Models could reshape coding agents and software, and why even with $1 billion he wouldn’t pre-train a model from scratch.

Latent.Space

415,091 Aufrufe • vor 1 Monat

Most AI coding tools focus on helping one developer work faster. Reason is taking a different approach. The idea of a multiplayer coding agent is interesting because software development rarely happens in isolation. Reason lets AI agents and humans work together, split tasks, debug issues and verify solutions instead of relying on a single agent to do everything. 🔹 Parallel collaboration 🔹 Shared context across sessions 🔹 From code changes to tested solutions AI coding is moving beyond autocomplete. The next step is building systems that can actually work together.

Most AI coding tools focus on helping one developer work faster. Reason is taking a different approach. The idea of a multiplayer coding agent is interesting because software development rarely happens in isolation. Reason lets AI agents and humans work together, split tasks, debug issues and verify solutions instead of relying on a single agent to do everything. 🔹 Parallel collaboration 🔹 Shared context across sessions 🔹 From code changes to tested solutions AI coding is moving beyond autocomplete. The next step is building systems that can actually work together.

MΞK

10,892 Aufrufe • vor 7 Tagen

Today we're excited to introduce Devin, the first AI software engineer. Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork. Devin is an autonomous agent that solves engineering tasks through the use of its own shell, code editor, and web browser. When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted. Check out what Devin can do in the thread below.

Today we're excited to introduce Devin, the first AI software engineer. Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork. Devin is an autonomous agent that solves engineering tasks through the use of its own shell, code editor, and web browser. When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted. Check out what Devin can do in the thread below.

Cognition

31,484,525 Aufrufe • vor 2 Jahren

Can AI actually automate jobs? Scale AI and are launching the Remote Labor Index (RLI), the first benchmark and public leaderboard that test how well AI agents can complete real, paid freelance work in domains like software engineering, design, architecture, data analysis, and more. Early results show the limits of today’s models. The top AI agent completed just 2.5% of real freelance jobs better than humans. AI is powerful, but not yet reliable enough to replace skilled labor. RLI gives us a transparent way to track progress over time and bring clarity to the future of work.

Can AI actually automate jobs? Scale AI and are launching the Remote Labor Index (RLI), the first benchmark and public leaderboard that test how well AI agents can complete real, paid freelance work in domains like software engineering, design, architecture, data analysis, and more. Early results show the limits of today’s models. The top AI agent completed just 2.5% of real freelance jobs better than humans. AI is powerful, but not yet reliable enough to replace skilled labor. RLI gives us a transparent way to track progress over time and bring clarity to the future of work.

Bing Liu

424,662 Aufrufe • vor 11 Monaten

The new Factory software AI agents just combined OpenAI’s Codex, Devin, and Cursor into one product. They are fully autonomous agents called Droids that perform deep code research, technical docs, and do autonomous end-to-end coding. Here’s how it works (with real examples)👇:

The new Factory software AI agents just combined OpenAI’s Codex, Devin, and Cursor into one product. They are fully autonomous agents called Droids that perform deep code research, technical docs, and do autonomous end-to-end coding. Here’s how it works (with real examples)👇:

Alvaro Cintas

94,884 Aufrufe • vor 1 Jahr

TypeSafe AI's Diogo Almeida's elevator pitch for Jev: "Where the f**k is all the automation?" "AI is so unbelievably smart. Not that I hate on chatbots or coding agents, I love them myself, but it's so useless at all other stuff. It's tragic." "TypeSafe is making AI for software. We want to make AI powerful, not just for humans in the loop, but to actually build real software. Jev, to us, is our first model in this whole space to make [automation] way, way better." "I like Claude Code and Codex. I love the description from Garry Tan and them: It's just-in-time software. It makes software on the fly, and you can program software in natural language, but it has the same expressive power as software." "What I want instead is smart software. Instead of automating software engineering, I want to expand what software itself can do, such that things that should be automatable can then be automatable." "I want to express things like intent. I want to expand the vocabulary of what we can do." "Programming is hyper-specifying valuable things and then infinitely replicating them. It's so freaking cool." Diogo Almeida

TypeSafe AI's Diogo Almeida's elevator pitch for Jev: "Where the f**k is all the automation?" "AI is so unbelievably smart. Not that I hate on chatbots or coding agents, I love them myself, but it's so useless at all other stuff. It's tragic." "TypeSafe is making AI for software. We want to make AI powerful, not just for humans in the loop, but to actually build real software. Jev, to us, is our first model in this whole space to make [automation] way, way better." "I like Claude Code and Codex. I love the description from Garry Tan and them: It's just-in-time software. It makes software on the fly, and you can program software in natural language, but it has the same expressive power as software." "What I want instead is smart software. Instead of automating software engineering, I want to expand what software itself can do, such that things that should be automatable can then be automatable." "I want to express things like intent. I want to expand the vocabulary of what we can do." "Programming is hyper-specifying valuable things and then infinitely replicating them. It's so freaking cool." Diogo Almeida

a16z

39,453 Aufrufe • vor 24 Tagen

“AI agents will outperform humans at almost all jobs by 2026–2027.” - The forecast is everywhere. So we built the exam to test that claim, on real labor-market aligned work. On the hardest tier, top agents pass 2.6%. Meet Agents' Last Exam (ALE), a rolling benchmark measuring whether agents can actually do real jobs. 🧵👇

“AI agents will outperform humans at almost all jobs by 2026–2027.” - The forecast is everywhere. So we built the exam to test that claim, on real labor-market aligned work. On the hardest tier, top agents pass 2.6%. Meet Agents' Last Exam (ALE), a rolling benchmark measuring whether agents can actually do real jobs. 🧵👇

Yiyou Sun

98,576 Aufrufe • vor 4 Monaten

OpenAI Greg Brockman: GPT-5.5 is not an endpoint — it's a beginning It is an early step toward stronger models arriving in the coming months, with larger gains across many capabilities The focus is not just on better benchmarks, but usefulness in the real world, for real users and real-world work

OpenAI Greg Brockman: GPT-5.5 is not an endpoint — it's a beginning It is an early step toward stronger models arriving in the coming months, with larger gains across many capabilities The focus is not just on better benchmarks, but usefulness in the real world, for real users and real-world work

Haider.

40,201 Aufrufe • vor 6 Monaten

Most of the market is priced for a world that's coming to an end. Traditional software, legacy platforms, much of the SaaS layer of the S&P 500… all built on the assumption that humans will remain the primary customers. AI agents don't buy software the way we do. They don't need dashboards, subscriptions, or logins. And once the buyer changes, so does everything built around them. The investable opportunity ahead is the agentic economy… what gets built for agents as customers. That's where the next wave of capital ends up, and most people aren't positioned for it.

Most of the market is priced for a world that's coming to an end. Traditional software, legacy platforms, much of the SaaS layer of the S&P 500… all built on the assumption that humans will remain the primary customers. AI agents don't buy software the way we do. They don't need dashboards, subscriptions, or logins. And once the buyer changes, so does everything built around them. The investable opportunity ahead is the agentic economy… what gets built for agents as customers. That's where the next wave of capital ends up, and most people aren't positioned for it.

Raoul Pal

215,413 Aufrufe • vor 3 Monaten

Here’s the problem: software interfaces are abstract, hardware is instinctive. We don’t learn objects, we recognize them. Buttons, wheels, friction, position they map to how we exist in the real world. Menus, layers, hidden states they fight against it. This is why this feels different. It’s not an app pretending to be a recorder. It’s a recorder that happens to exist in software. Built in wabi

Here’s the problem: software interfaces are abstract, hardware is instinctive. We don’t learn objects, we recognize them. Buttons, wheels, friction, position they map to how we exist in the real world. Menus, layers, hidden states they fight against it. This is why this feels different. It’s not an app pretending to be a recorder. It’s a recorder that happens to exist in software. Built in wabi

Tykra

30,343 Aufrufe • vor 7 Monaten

Mind-1: Physical AI at Human Speed, Built for Real Work Completing a task is only the beginning. Doing it at a pace that works in the real world is the next step—and that is what Mind-1 is built to achieve.

Mind-1: Physical AI at Human Speed, Built for Real Work Completing a task is only the beginning. Doing it at a pace that works in the real world is the next step—and that is what Mind-1 is built to achieve.

MindOn

141,343 Aufrufe • vor 23 Tagen

Handshake CEO Garrett Lord explains why AI agents exploded in software engineering but not the rest of white-collar work: "In software engineering, the average engineer is two, three, four times more productive. Software engineering agents are able to operate for multiple days at a time, automating hundreds of hours of work." "CEOs are not seeing that level of automation and ROI in other functions beyond software engineering. Software engineering is accelerated in such a massive way because everything was written down in GitHub. And it's extremely clear with what the right answer is." "When you have enough verifiable rewards and you scale up enough compute against verifiable rewards, you get what happened in software engineering, which is agents able to operate and do the job of humans." Sahil Bhaiwala Handshake

Handshake CEO Garrett Lord explains why AI agents exploded in software engineering but not the rest of white-collar work: "In software engineering, the average engineer is two, three, four times more productive. Software engineering agents are able to operate for multiple days at a time, automating hundreds of hours of work." "CEOs are not seeing that level of automation and ROI in other functions beyond software engineering. Software engineering is accelerated in such a massive way because everything was written down in GitHub. And it's extremely clear with what the right answer is." "When you have enough verifiable rewards and you scale up enough compute against verifiable rewards, you get what happened in software engineering, which is agents able to operate and do the job of humans." Sahil Bhaiwala Handshake

MTS

14,513 Aufrufe • vor 1 Monat

Introducing Autobot 🤖 Software agencies still overcharge clients by the hour. AI has fundamentally changed how software gets built, but clients aren't seeing those savings. So we built Autobot, the first autonomous Replit ⠕ agency that reduce the cost of agency-built software by 90%. Describe what you want to build, answer a few questions, and about an hour later your first working version is ready, for free. Try it, share it and if you love it, you can keep it. If not, you don't pay anything. That's how custom software should be bought.

Introducing Autobot 🤖 Software agencies still overcharge clients by the hour. AI has fundamentally changed how software gets built, but clients aren't seeing those savings. So we built Autobot, the first autonomous Replit ⠕ agency that reduce the cost of agency-built software by 90%. Describe what you want to build, answer a few questions, and about an hour later your first working version is ready, for free. Try it, share it and if you love it, you can keep it. If not, you don't pay anything. That's how custom software should be bought.

viktorTJM

91,220 Aufrufe • vor 3 Monaten