Loading video...

Video Failed to Load

Go Home

What are the largest software engineering tasks AI can perform? To answer this, we built MirrorCode, our long-horizon SWE benchmark that lets AI code autonomously for days at a time. The best models complete some tasks we estimate would take human engineers several weeks.

EpochAIResearch's profile picture

Epoch AI

47,193 subscribers

40,224 views • 18 hours ago •via X (Twitter)

EducationScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

New AI models can now perform many of the tasks that once defined entry-level jobs. In another year, imagine where we are at.
0:42

New AI models can now perform many of the tasks that once defined entry-level jobs. In another year, imagine where we are at.

Financial Dystopia

22,716 views • 3 months ago

Geoffrey Hinton says that in 2026, AI will gain the capability to replace many jobs Every 7 months, AI doubles the length of tasks it can handle. It went from 1-minute coding to hour-long projects Soon, it'll manage month-long software engineering tasks "then few people will be needed for SWE projects"
0:40

Geoffrey Hinton says that in 2026, AI will gain the capability to replace many jobs Every 7 months, AI doubles the length of tasks it can handle. It went from 1-minute coding to hour-long projects Soon, it'll manage month-long software engineering tasks "then few people will be needed for SWE projects"

Haider.

66,008 views • 5 months ago

Scott Wu says Karpathy's idea of jagged intelligence explains why humans and AI work best together. AI is superhuman at some tasks, shockingly weak at others. “As long as that tail exists of what humans can do, that's what we'll own.” We still decide what to build. In the end, software engineering is just telling the computer what to do.
1:09

Scott Wu says Karpathy's idea of jagged intelligence explains why humans and AI work best together. AI is superhuman at some tasks, shockingly weak at others. “As long as that tail exists of what humans can do, that's what we'll own.” We still decide what to build. In the end, software engineering is just telling the computer what to do.

vitrupo

38,496 views • 1 year ago

Traditional coding benchmarks do not reflect how software is actually built and maintained. That's why we built a new benchmark, APEX-SWE, in partnership with Cognition. It measures whether AI models can perform complex, real-world software engineering work to ship systems that work and debug them when they don't. OpenAI GPT 5.3 Codex (High) tops the leaderboard at 41.5% on Pass@1.
2:30

Traditional coding benchmarks do not reflect how software is actually built and maintained. That's why we built a new benchmark, APEX-SWE, in partnership with Cognition. It measures whether AI models can perform complex, real-world software engineering work to ship systems that work and debug them when they don't. OpenAI GPT 5.3 Codex (High) tops the leaderboard at 41.5% on Pass@1.

adarsh

208,673 views • 3 months ago

Windsurf proudly launches SWE-1, their first AI models designed with a comprehensive approach beyond just code. They enhance the entire software engineering process. Here's what this means for the exciting future of software engineering:
1:40

Windsurf proudly launches SWE-1, their first AI models designed with a comprehensive approach beyond just code. They enhance the entire software engineering process. Here's what this means for the exciting future of software engineering:

Fakhr

34,460 views • 1 year ago

In the future AI Engineers won't replace us. Instead, we will orchestrate them! And that's what we built this weekend at the Mistral AI Cerebral Valley hackathon! · Spin up devs or reviewers · Define tasks for them on a board · Check their code on Github and more coming ⚡️
0:42

In the future AI Engineers won't replace us. Instead, we will orchestrate them! And that's what we built this weekend at the Mistral AI Cerebral Valley hackathon! · Spin up devs or reviewers · Define tasks for them on a board · Check their code on Github and more coming ⚡️

Axel Peytavin

18,100 views • 2 years ago

Sam Altman: "We now have AI systems smart enough that our best scientists are saying this changes the game for science." "These models are discovering new knowledge on their own. This is the first time a tool humans have built is autonomously contributing to the sum total of human knowledge."
0:34

Sam Altman: "We now have AI systems smart enough that our best scientists are saying this changes the game for science." "These models are discovering new knowledge on their own. This is the first time a tool humans have built is autonomously contributing to the sum total of human knowledge."

vitrupo

26,992 views • 1 month ago

For the first time in human history, we are teaching a Foundation Model to master the diverse tasks of medicinal chemists, biologists, and computational scientists all in one place. In our latest collaboration with Liquid AI, we are moving away from fragmented, specialized tools toward a single, super-intelligent model. What surprised me most? This model isn't just performing at reasonable levels—it has started outperforming specialist models across physics-based tasks, imaging, and longitudinal data. Why this changes everything: -Synergy over Specialization: Fine-tuning on specific tasks has unlocked unexpected capabilities in synergetic areas, opening a new frontier in multimodal AI research. -Zero-Shot Potential: We are building a model that can perform out-of-scope tasks, moving us closer to an "AI deity" for drug discovery. -Quality First: The goal isn't just to bypass regulations to save time; it’s about using these synergies to develop better, more effective drugs. We are no longer just looking at linear regression or simple text; we are looking at the future of how humanity fights disease. #LiquidAI #InsilicoMedicine #GenerativeAI #DrugDiscovery #DeepTech #BiotechInnovation
1:53

For the first time in human history, we are teaching a Foundation Model to master the diverse tasks of medicinal chemists, biologists, and computational scientists all in one place. In our latest collaboration with Liquid AI, we are moving away from fragmented, specialized tools toward a single, super-intelligent model. What surprised me most? This model isn't just performing at reasonable levels—it has started outperforming specialist models across physics-based tasks, imaging, and longitudinal data. Why this changes everything: -Synergy over Specialization: Fine-tuning on specific tasks has unlocked unexpected capabilities in synergetic areas, opening a new frontier in multimodal AI research. -Zero-Shot Potential: We are building a model that can perform out-of-scope tasks, moving us closer to an "AI deity" for drug discovery. -Quality First: The goal isn't just to bypass regulations to save time; it’s about using these synergies to develop better, more effective drugs. We are no longer just looking at linear regression or simple text; we are looking at the future of how humanity fights disease. #LiquidAI #InsilicoMedicine #GenerativeAI #DrugDiscovery #DeepTech #BiotechInnovation

Alex Zhavoronkov, PhD (aka Aleksandrs Zavoronkovs)

10,544 views • 2 months ago

"current AI models are still embarrassingly bad at many tasks" Sam Altman explains why current AI models aren't considered AGI: 1) they don't continuously learn/improve 2) can't discover new science autonomously 3) cannot perform basic knowledge work, such as completing job tasks using the internet and local files
0:49

"current AI models are still embarrassingly bad at many tasks" Sam Altman explains why current AI models aren't considered AGI: 1) they don't continuously learn/improve 2) can't discover new science autonomously 3) cannot perform basic knowledge work, such as completing job tasks using the internet and local files

Haider.

42,689 views • 1 year ago

Prior to release, we shared a version of Cohere North Mini Code with AI engineers and answered some questions. Here's a quick illustrated walkthrough of the model's architecture and training process. Small models fill an important niche. They: 1. run on more widely available hardware 2. handle tasks within a certain range of complexity 3. take on sub-tasks that larger models delegate
4:53

Prior to release, we shared a version of Cohere North Mini Code with AI engineers and answered some questions. Here's a quick illustrated walkthrough of the model's architecture and training process. Small models fill an important niche. They: 1. run on more widely available hardware 2. handle tasks within a certain range of complexity 3. take on sub-tasks that larger models delegate

Jay Alammar

87,948 views • 10 days ago

Inviting early beta testers for our upcoming SWE agent, Asteria. We're solving AI code comprehension, where the context window and the size of your codebase are no longer limitations, and reinventing the UX of how coding agents are currently implemented. We initially built this to address false positives and enhance accuracy for our SAST agent, which autonomously identifies security vulnerabilities, traces how a payload would flow through code, and ultimately demonstrates an actual exploit. The performance improved so significantly that we decided to test it on realistic engineering tasks on large codebases and witnessed SOTA results. Reply "access" if you'd like to be an early beta tester, a lot of quirks to fix.
0:45

Inviting early beta testers for our upcoming SWE agent, Asteria. We're solving AI code comprehension, where the context window and the size of your codebase are no longer limitations, and reinventing the UX of how coding agents are currently implemented. We initially built this to address false positives and enhance accuracy for our SAST agent, which autonomously identifies security vulnerabilities, traces how a payload would flow through code, and ultimately demonstrates an actual exploit. The performance improved so significantly that we decided to test it on realistic engineering tasks on large codebases and witnessed SOTA results. Reply "access" if you'd like to be an early beta tester, a lot of quirks to fix.

mufeed vh

12,400 views • 1 year ago

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.
1:12

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.

Russ Salakhutdinov

22,518 views • 1 month ago

Ray Kurzweil says AI models will exceed human verbal ability in a year or two and the next step is large event models that enable robots to perform physical tasks
1:38

Ray Kurzweil says AI models will exceed human verbal ability in a year or two and the next step is large event models that enable robots to perform physical tasks

Tsarathustra

71,501 views • 2 years ago

I'm excited to share that we've built the world's most capable AI software engineer, achieving 30.08% on SWE-Bench – ahead of Amazon and Cognition. This model is so much more than a benchmark score: it was trained from the start to think and behave like a human SWE.
3:58

I'm excited to share that we've built the world's most capable AI software engineer, achieving 30.08% on SWE-Bench – ahead of Amazon and Cognition. This model is so much more than a benchmark score: it was trained from the start to think and behave like a human SWE.

Alistair

820,015 views • 1 year ago

We now run evals to help you know what AI models perform best to write Nuxt code.
0:21

We now run evals to help you know what AI models perform best to write Nuxt code.

Nuxt

27,970 views • 4 months ago

Introducing MARS, the first Personal AI Robot. In technology, getting to the future we want starts with building it together. This is why we designed Mars. Inspired by early PCs, Mars is a powerful, complete, extendable robot. It runs BASIC, an open embodied AI agent we created for you to program with code, demonstrations, and prompts. This allows Mars to perform complex long-horizon tasks involving spatial memory, reasoning, manipulation and navigation. Pre-orders open now ⬇️ Demos & details in the thread
1:10

Introducing MARS, the first Personal AI Robot. In technology, getting to the future we want starts with building it together. This is why we designed Mars. Inspired by early PCs, Mars is a powerful, complete, extendable robot. It runs BASIC, an open embodied AI agent we created for you to program with code, demonstrations, and prompts. This allows Mars to perform complex long-horizon tasks involving spatial memory, reasoning, manipulation and navigation. Pre-orders open now ⬇️ Demos & details in the thread

Axel Peytavin

313,416 views • 9 months ago

Additionally, looking towards the future, we’re releasing PARTNR: a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration. Built on Habitat 3.0, it’s the largest benchmark of its kind to study and evaluate human-robot collaboration in household activities By providing a standardized benchmark and dataset we hope to enable new research on robots that can not only operate in isolation, but in collaboration with people. Details and code ➡️
1:01

Additionally, looking towards the future, we’re releasing PARTNR: a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration. Built on Habitat 3.0, it’s the largest benchmark of its kind to study and evaluate human-robot collaboration in household activities By providing a standardized benchmark and dataset we hope to enable new research on robots that can not only operate in isolation, but in collaboration with people. Details and code ➡️

AI at Meta

20,244 views • 1 year ago

Announcing Agent Composer: AI that works when it actually IS rocket science. Technical teams can now automate the routine (but complex) tasks that used to take hours every week—root cause analysis, production planning, test code generation— and reduce them down to minutes, so you can get your real work done. Here's how we solved the AI context problem for hard engineering tasks (like rocket science 🚀) 🧵
2:19

Announcing Agent Composer: AI that works when it actually IS rocket science. Technical teams can now automate the routine (but complex) tasks that used to take hours every week—root cause analysis, production planning, test code generation— and reduce them down to minutes, so you can get your real work done. Here's how we solved the AI context problem for hard engineering tasks (like rocket science 🚀) 🧵

Contextual AI

1,552,286 views • 5 months ago