Загрузка видео...

Не удалось загрузить видео

На главную

What are the largest software engineering tasks AI can perform? To answer this, we built MirrorCode, our long-horizon SWE benchmark that lets AI code autonomously for days at a time. The best models complete some tasks we estimate would take human engineers several weeks.

EpochAIResearch's profile picture

Epoch AI

47,193 subscribers

40,224 просмотров • 20 часов назад •via X (Twitter)

ОбразованиеНаука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

New AI models can now perform many of the tasks that once defined entry-level jobs. In another year, imagine where we are at.
0:42

New AI models can now perform many of the tasks that once defined entry-level jobs. In another year, imagine where we are at.

Financial Dystopia

22,716 просмотров • 3 месяцев назад

Geoffrey Hinton says that in 2026, AI will gain the capability to replace many jobs Every 7 months, AI doubles the length of tasks it can handle. It went from 1-minute coding to hour-long projects Soon, it'll manage month-long software engineering tasks "then few people will be needed for SWE projects"
0:40

Geoffrey Hinton says that in 2026, AI will gain the capability to replace many jobs Every 7 months, AI doubles the length of tasks it can handle. It went from 1-minute coding to hour-long projects Soon, it'll manage month-long software engineering tasks "then few people will be needed for SWE projects"

Haider.

66,008 просмотров • 5 месяцев назад

Scott Wu says Karpathy's idea of jagged intelligence explains why humans and AI work best together. AI is superhuman at some tasks, shockingly weak at others. “As long as that tail exists of what humans can do, that's what we'll own.” We still decide what to build. In the end, software engineering is just telling the computer what to do.
1:09

Scott Wu says Karpathy's idea of jagged intelligence explains why humans and AI work best together. AI is superhuman at some tasks, shockingly weak at others. “As long as that tail exists of what humans can do, that's what we'll own.” We still decide what to build. In the end, software engineering is just telling the computer what to do.

vitrupo

38,496 просмотров • 1 год назад

Traditional coding benchmarks do not reflect how software is actually built and maintained. That's why we built a new benchmark, APEX-SWE, in partnership with Cognition. It measures whether AI models can perform complex, real-world software engineering work to ship systems that work and debug them when they don't. OpenAI GPT 5.3 Codex (High) tops the leaderboard at 41.5% on Pass@1.
2:30

Traditional coding benchmarks do not reflect how software is actually built and maintained. That's why we built a new benchmark, APEX-SWE, in partnership with Cognition. It measures whether AI models can perform complex, real-world software engineering work to ship systems that work and debug them when they don't. OpenAI GPT 5.3 Codex (High) tops the leaderboard at 41.5% on Pass@1.

adarsh

208,673 просмотров • 3 месяцев назад

Windsurf proudly launches SWE-1, their first AI models designed with a comprehensive approach beyond just code. They enhance the entire software engineering process. Here's what this means for the exciting future of software engineering:
1:40

Windsurf proudly launches SWE-1, their first AI models designed with a comprehensive approach beyond just code. They enhance the entire software engineering process. Here's what this means for the exciting future of software engineering:

Fakhr

34,460 просмотров • 1 год назад

In the future AI Engineers won't replace us. Instead, we will orchestrate them! And that's what we built this weekend at the Mistral AI Cerebral Valley hackathon! · Spin up devs or reviewers · Define tasks for them on a board · Check their code on Github and more coming ⚡️
0:42

In the future AI Engineers won't replace us. Instead, we will orchestrate them! And that's what we built this weekend at the Mistral AI Cerebral Valley hackathon! · Spin up devs or reviewers · Define tasks for them on a board · Check their code on Github and more coming ⚡️

Axel Peytavin

18,100 просмотров • 2 лет назад

Sam Altman: "We now have AI systems smart enough that our best scientists are saying this changes the game for science." "These models are discovering new knowledge on their own. This is the first time a tool humans have built is autonomously contributing to the sum total of human knowledge."
0:34

Sam Altman: "We now have AI systems smart enough that our best scientists are saying this changes the game for science." "These models are discovering new knowledge on their own. This is the first time a tool humans have built is autonomously contributing to the sum total of human knowledge."

vitrupo

26,992 просмотров • 1 месяц назад

For the first time in human history, we are teaching a Foundation Model to master the diverse tasks of medicinal chemists, biologists, and computational scientists all in one place. In our latest collaboration with Liquid AI, we are moving away from fragmented, specialized tools toward a single, super-intelligent model. What surprised me most? This model isn't just performing at reasonable levels—it has started outperforming specialist models across physics-based tasks, imaging, and longitudinal data. Why this changes everything: -Synergy over Specialization: Fine-tuning on specific tasks has unlocked unexpected capabilities in synergetic areas, opening a new frontier in multimodal AI research. -Zero-Shot Potential: We are building a model that can perform out-of-scope tasks, moving us closer to an "AI deity" for drug discovery. -Quality First: The goal isn't just to bypass regulations to save time; it’s about using these synergies to develop better, more effective drugs. We are no longer just looking at linear regression or simple text; we are looking at the future of how humanity fights disease. #LiquidAI #InsilicoMedicine #GenerativeAI #DrugDiscovery #DeepTech #BiotechInnovation
1:53

For the first time in human history, we are teaching a Foundation Model to master the diverse tasks of medicinal chemists, biologists, and computational scientists all in one place. In our latest collaboration with Liquid AI, we are moving away from fragmented, specialized tools toward a single, super-intelligent model. What surprised me most? This model isn't just performing at reasonable levels—it has started outperforming specialist models across physics-based tasks, imaging, and longitudinal data. Why this changes everything: -Synergy over Specialization: Fine-tuning on specific tasks has unlocked unexpected capabilities in synergetic areas, opening a new frontier in multimodal AI research. -Zero-Shot Potential: We are building a model that can perform out-of-scope tasks, moving us closer to an "AI deity" for drug discovery. -Quality First: The goal isn't just to bypass regulations to save time; it’s about using these synergies to develop better, more effective drugs. We are no longer just looking at linear regression or simple text; we are looking at the future of how humanity fights disease. #LiquidAI #InsilicoMedicine #GenerativeAI #DrugDiscovery #DeepTech #BiotechInnovation

Alex Zhavoronkov, PhD (aka Aleksandrs Zavoronkovs)

10,544 просмотров • 2 месяцев назад

"current AI models are still embarrassingly bad at many tasks" Sam Altman explains why current AI models aren't considered AGI: 1) they don't continuously learn/improve 2) can't discover new science autonomously 3) cannot perform basic knowledge work, such as completing job tasks using the internet and local files
0:49

"current AI models are still embarrassingly bad at many tasks" Sam Altman explains why current AI models aren't considered AGI: 1) they don't continuously learn/improve 2) can't discover new science autonomously 3) cannot perform basic knowledge work, such as completing job tasks using the internet and local files

Haider.

42,689 просмотров • 1 год назад

Prior to release, we shared a version of Cohere North Mini Code with AI engineers and answered some questions. Here's a quick illustrated walkthrough of the model's architecture and training process. Small models fill an important niche. They: 1. run on more widely available hardware 2. handle tasks within a certain range of complexity 3. take on sub-tasks that larger models delegate
4:53

Prior to release, we shared a version of Cohere North Mini Code with AI engineers and answered some questions. Here's a quick illustrated walkthrough of the model's architecture and training process. Small models fill an important niche. They: 1. run on more widely available hardware 2. handle tasks within a certain range of complexity 3. take on sub-tasks that larger models delegate

Jay Alammar

87,948 просмотров • 10 дней назад

Inviting early beta testers for our upcoming SWE agent, Asteria. We're solving AI code comprehension, where the context window and the size of your codebase are no longer limitations, and reinventing the UX of how coding agents are currently implemented. We initially built this to address false positives and enhance accuracy for our SAST agent, which autonomously identifies security vulnerabilities, traces how a payload would flow through code, and ultimately demonstrates an actual exploit. The performance improved so significantly that we decided to test it on realistic engineering tasks on large codebases and witnessed SOTA results. Reply "access" if you'd like to be an early beta tester, a lot of quirks to fix.
0:45

Inviting early beta testers for our upcoming SWE agent, Asteria. We're solving AI code comprehension, where the context window and the size of your codebase are no longer limitations, and reinventing the UX of how coding agents are currently implemented. We initially built this to address false positives and enhance accuracy for our SAST agent, which autonomously identifies security vulnerabilities, traces how a payload would flow through code, and ultimately demonstrates an actual exploit. The performance improved so significantly that we decided to test it on realistic engineering tasks on large codebases and witnessed SOTA results. Reply "access" if you'd like to be an early beta tester, a lot of quirks to fix.

mufeed vh

12,400 просмотров • 1 год назад

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.
1:12

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.

Russ Salakhutdinov

22,518 просмотров • 1 месяц назад

Ray Kurzweil says AI models will exceed human verbal ability in a year or two and the next step is large event models that enable robots to perform physical tasks
1:38

Ray Kurzweil says AI models will exceed human verbal ability in a year or two and the next step is large event models that enable robots to perform physical tasks

Tsarathustra

71,501 просмотров • 2 лет назад

I'm excited to share that we've built the world's most capable AI software engineer, achieving 30.08% on SWE-Bench – ahead of Amazon and Cognition. This model is so much more than a benchmark score: it was trained from the start to think and behave like a human SWE.
3:58

I'm excited to share that we've built the world's most capable AI software engineer, achieving 30.08% on SWE-Bench – ahead of Amazon and Cognition. This model is so much more than a benchmark score: it was trained from the start to think and behave like a human SWE.

Alistair

820,015 просмотров • 1 год назад

We now run evals to help you know what AI models perform best to write Nuxt code.
0:21

We now run evals to help you know what AI models perform best to write Nuxt code.

Nuxt

28,001 просмотров • 4 месяцев назад

Introducing MARS, the first Personal AI Robot. In technology, getting to the future we want starts with building it together. This is why we designed Mars. Inspired by early PCs, Mars is a powerful, complete, extendable robot. It runs BASIC, an open embodied AI agent we created for you to program with code, demonstrations, and prompts. This allows Mars to perform complex long-horizon tasks involving spatial memory, reasoning, manipulation and navigation. Pre-orders open now ⬇️ Demos & details in the thread
1:10

Introducing MARS, the first Personal AI Robot. In technology, getting to the future we want starts with building it together. This is why we designed Mars. Inspired by early PCs, Mars is a powerful, complete, extendable robot. It runs BASIC, an open embodied AI agent we created for you to program with code, demonstrations, and prompts. This allows Mars to perform complex long-horizon tasks involving spatial memory, reasoning, manipulation and navigation. Pre-orders open now ⬇️ Demos & details in the thread

Axel Peytavin

313,416 просмотров • 9 месяцев назад

Additionally, looking towards the future, we’re releasing PARTNR: a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration. Built on Habitat 3.0, it’s the largest benchmark of its kind to study and evaluate human-robot collaboration in household activities By providing a standardized benchmark and dataset we hope to enable new research on robots that can not only operate in isolation, but in collaboration with people. Details and code ➡️
1:01

Additionally, looking towards the future, we’re releasing PARTNR: a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration. Built on Habitat 3.0, it’s the largest benchmark of its kind to study and evaluate human-robot collaboration in household activities By providing a standardized benchmark and dataset we hope to enable new research on robots that can not only operate in isolation, but in collaboration with people. Details and code ➡️

AI at Meta

20,244 просмотров • 1 год назад

Announcing Agent Composer: AI that works when it actually IS rocket science. Technical teams can now automate the routine (but complex) tasks that used to take hours every week—root cause analysis, production planning, test code generation— and reduce them down to minutes, so you can get your real work done. Here's how we solved the AI context problem for hard engineering tasks (like rocket science 🚀) 🧵
2:19

Announcing Agent Composer: AI that works when it actually IS rocket science. Technical teams can now automate the routine (but complex) tasks that used to take hours every week—root cause analysis, production planning, test code generation— and reduce them down to minutes, so you can get your real work done. Here's how we solved the AI context problem for hard engineering tasks (like rocket science 🚀) 🧵

Contextual AI

1,552,286 просмотров • 5 месяцев назад