Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Today we’re releasing Ramp SWE-Bench: a private, production-grounded coding benchmark created from real engineering problems we've faced at Ramp.

Ramp Labs

14,212 subscribers

188,015 views • 2 months ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Why did a spend management platform start its own AI research lab? Karim Atiyeh describes Ramp Labs as a “collection of crazy ambitious people that don’t like to be told that something is impossible”. It started about a year ago and has since worked on projects like building its own production-grounded coding benchmark “Ramp SWE-Bench”, putting Claude Code in RollerCoaster Tycoon, and a mechanistic interpretability playground. The Redpoint team spent the day with Ramp Labs to answer questions like: ▪️ How did Ramp Labs get started? ▪️ How do they decide what research to pursue? ▪️ The long-term vision for Ramp Labs

Why did a spend management platform start its own AI research lab? Karim Atiyeh describes Ramp Labs as a “collection of crazy ambitious people that don’t like to be told that something is impossible”. It started about a year ago and has since worked on projects like building its own production-grounded coding benchmark “Ramp SWE-Bench”, putting Claude Code in RollerCoaster Tycoon, and a mechanistic interpretability playground. The Redpoint team spent the day with Ramp Labs to answer questions like: ▪️ How did Ramp Labs get started? ▪️ How do they decide what research to pursue? ▪️ The long-term vision for Ramp Labs

Redpoint

100,292 views • 8 days ago

Verdent achieved top performance on SWE-bench Verified today! We ran the benchmark using the same configuration our users work with in production, and no modifications to the standard setup. We're focused on helping engineers solve real problems with reliable code.

Verdent achieved top performance on SWE-bench Verified today! We ran the benchmark using the same configuration our users work with in production, and no modifications to the standard setup. We're focused on helping engineers solve real problems with reliable code.

Verdent

2,856,539 views • 9 months ago

I'm excited to share that we've built the world's most capable AI software engineer, achieving 30.08% on SWE-Bench – ahead of Amazon and Cognition. This model is so much more than a benchmark score: it was trained from the start to think and behave like a human SWE.

I'm excited to share that we've built the world's most capable AI software engineer, achieving 30.08% on SWE-Bench – ahead of Amazon and Cognition. This model is so much more than a benchmark score: it was trained from the start to think and behave like a human SWE.

Alistair

820,273 views • 2 years ago

Today we're excited to introduce Devin, the first AI software engineer. Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork. Devin is an autonomous agent that solves engineering tasks through the use of its own shell, code editor, and web browser. When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted. Check out what Devin can do in the thread below.

Today we're excited to introduce Devin, the first AI software engineer. Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork. Devin is an autonomous agent that solves engineering tasks through the use of its own shell, code editor, and web browser. When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted. Check out what Devin can do in the thread below.

Cognition

31,457,315 views • 2 years ago

Introducing Warp Code: a partner you can trust, from prompt to production 🤠 Warp Code features: 👉 Top coding agent: #1 on Terminal-Bench, #3 on SWE-bench Verified 👉 Code review panel 👉 Lightweight code editor 👉 Slash commands, and agent profiles

Introducing Warp Code: a partner you can trust, from prompt to production 🤠 Warp Code features: 👉 Top coding agent: #1 on Terminal-Bench, #3 on SWE-bench Verified 👉 Code review panel 👉 Lightweight code editor 👉 Slash commands, and agent profiles

Warp

112,863 views • 11 months ago

At Ramp, we've seen AI token spend skyrocket 13x among our customers since last January. We ran experiments where coding agents managed their own token budgets. They ignored them completely, so we employed a separate controller model to approve spend on their behalf.

At Ramp, we've seen AI token spend skyrocket 13x among our customers since last January. We ran experiments where coding agents managed their own token budgets. They ignored them completely, so we employed a separate controller model to approve spend on their behalf.

Ramp Labs

37,115 views • 3 months ago

The ramp had a different kind of glow today. Former Naxal women from Bastar walked the ramp at a #NationalHandloomDay event, bringing their stories, confidence and the beauty of handloom to the spotlight. Their journey reflects the spirit of a #NewIndia, where transformation opens doors to new possibilities. #NariShakti #Bastar

The ramp had a different kind of glow today. Former Naxal women from Bastar walked the ramp at a #NationalHandloomDay event, bringing their stories, confidence and the beauty of handloom to the spotlight. Their journey reflects the spirit of a #NewIndia, where transformation opens doors to new possibilities. #NariShakti #Bastar

MyGovIndia

20,271 views • 2 days ago

Existing IR/RAG benchmarks are unrealistic: they’re often derived from easily retrievable topics, rather than grounded in solving real user problems. 🧵Introducing 𝐅𝐫𝐞𝐬𝐡𝐒𝐭𝐚𝐜𝐤, a challenging RAG benchmark on niche, recent topics. Work done during intern Databricks 🧱

Existing IR/RAG benchmarks are unrealistic: they’re often derived from easily retrievable topics, rather than grounded in solving real user problems. 🧵Introducing 𝐅𝐫𝐞𝐬𝐡𝐒𝐭𝐚𝐜𝐤, a challenging RAG benchmark on niche, recent topics. Work done during intern Databricks 🧱

Nandan Thakur

40,063 views • 1 year ago

Introducing ALE-Bench, ALE-Agent! Towards Automating Long-Horizon Algorithm Engineering for Hard Optimization Problems Blog: Paper: ALE-Bench is a coding benchmark primarily focused on hard optimization (NP-hard) problems. We developed this benchmark with AtCoder Inc., a leading coding contest platform company. What makes ALE-Bench unique is its focus on hard optimization problems that demand long-horizon and creative reasoning. It’s open-ended, in the sense that true optima are out of reach (NP-hard) and scores can continuously improve. We believe this benchmark has the potential to become one of the key benchmarks for reasoning and coding in the next generation. ALE-Agent is our end-to-end agent that we specifically designed for this challenging domain. In fact, our ALE-Agent has already built an impressive track record in the wild! In May 2025, our agent participated in a live AtCoder Heuristic Competition (AHC), alongside 1,000 other participants in real-time. AHC is considered to be one of the most challenging coding competitions in this domain. Our ALE-Agent achieved an impressive ranking of 21st out of 1,000 human participants in the competition (top 2%), marking a turning point for AI discovery of solutions to hard optimization problems with a wide spectrum of important real world applications such as logistics, routing, packing, factory production planning, power-grid balancing. We look forward to applying this technology to real industrial optimization opportunities. Building on the insights from this study, Sakana AI will continue to tackle the challenge of developing AI with even greater algorithm engineering capabilities. ALE-Bench Dataset: ALE-Bench Code: This research was conducted in collaboration with AtCoder Inc. (AtCoder). We are deeply grateful for their outstanding expertise and contributions in optimization and algorithms, which were invaluable in providing data, analyzing results, and enabling our AI agent’s participation in their contests.

Introducing ALE-Bench, ALE-Agent! Towards Automating Long-Horizon Algorithm Engineering for Hard Optimization Problems Blog: Paper: ALE-Bench is a coding benchmark primarily focused on hard optimization (NP-hard) problems. We developed this benchmark with AtCoder Inc., a leading coding contest platform company. What makes ALE-Bench unique is its focus on hard optimization problems that demand long-horizon and creative reasoning. It’s open-ended, in the sense that true optima are out of reach (NP-hard) and scores can continuously improve. We believe this benchmark has the potential to become one of the key benchmarks for reasoning and coding in the next generation. ALE-Agent is our end-to-end agent that we specifically designed for this challenging domain. In fact, our ALE-Agent has already built an impressive track record in the wild! In May 2025, our agent participated in a live AtCoder Heuristic Competition (AHC), alongside 1,000 other participants in real-time. AHC is considered to be one of the most challenging coding competitions in this domain. Our ALE-Agent achieved an impressive ranking of 21st out of 1,000 human participants in the competition (top 2%), marking a turning point for AI discovery of solutions to hard optimization problems with a wide spectrum of important real world applications such as logistics, routing, packing, factory production planning, power-grid balancing. We look forward to applying this technology to real industrial optimization opportunities. Building on the insights from this study, Sakana AI will continue to tackle the challenge of developing AI with even greater algorithm engineering capabilities. ALE-Bench Dataset: ALE-Bench Code: This research was conducted in collaboration with AtCoder Inc. (AtCoder). We are deeply grateful for their outstanding expertise and contributions in optimization and algorithms, which were invaluable in providing data, analyzing results, and enabling our AI agent’s participation in their contests.

Sakana AI

237,195 views • 1 year ago

Traditional coding benchmarks do not reflect how software is actually built and maintained. That's why we built a new benchmark, APEX-SWE, in partnership with Cognition. It measures whether AI models can perform complex, real-world software engineering work to ship systems that work and debug them when they don't. OpenAI GPT 5.3 Codex (High) tops the leaderboard at 41.5% on Pass@1.

Traditional coding benchmarks do not reflect how software is actually built and maintained. That's why we built a new benchmark, APEX-SWE, in partnership with Cognition. It measures whether AI models can perform complex, real-world software engineering work to ship systems that work and debug them when they don't. OpenAI GPT 5.3 Codex (High) tops the leaderboard at 41.5% on Pass@1.

adarsh

215,281 views • 4 months ago

It's month close. AI models can pass the CPA exam, but do you trust them to close the books? Introducing APEX-Accounting, built in collaboration with Ramp and Ramp Labs, to test whether frontier AI can do real accounting work. It’s a new benchmark comprising 160 tasks across 10 simulated companies, authored and graded by accounting professionals with experience at firms like Deloitte, PwC, EY, and KPMG.

It's month close. AI models can pass the CPA exam, but do you trust them to close the books? Introducing APEX-Accounting, built in collaboration with Ramp and Ramp Labs, to test whether frontier AI can do real accounting work. It’s a new benchmark comprising 160 tasks across 10 simulated companies, authored and graded by accounting professionals with experience at firms like Deloitte, PwC, EY, and KPMG.

Mercor

41,967 views • 15 days ago

New Benchtalks with John Yang: on ProgramBench (0% frontier models at launch) and the lineage/future of coding benchmarks, from SWE-bench/InterCode to now 01:29 ProgramBench launch and reception 03:41 Why artifact-level evaluation, not code-level 06:03 Why models love Python 08:29 ProgramBench as a research tool 12:45 From SWE-bench & InterCode to ProgramBench 17:47 How to grade a coding model 21:53 The position paper & humans in the loop 25:01 Managing quality with agents-in-the-loop 28:40 Internet access and benchmark integrity 35:26 Where models may surpass human abilities 38:56 When a model hits 80% on ProgramBench 43:55 Benchmarks worth paying attention to 46:24 What benchmark do you wish existed 49:32 Will benchmarks still look like benchmarks in 5 years 52:02 How to contribute to ProgramBench

New Benchtalks with John Yang: on ProgramBench (0% frontier models at launch) and the lineage/future of coding benchmarks, from SWE-bench/InterCode to now 01:29 ProgramBench launch and reception 03:41 Why artifact-level evaluation, not code-level 06:03 Why models love Python 08:29 ProgramBench as a research tool 12:45 From SWE-bench & InterCode to ProgramBench 17:47 How to grade a coding model 21:53 The position paper & humans in the loop 25:01 Managing quality with agents-in-the-loop 28:40 Internet access and benchmark integrity 35:26 Where models may surpass human abilities 38:56 When a model hits 80% on ProgramBench 43:55 Benchmarks worth paying attention to 46:24 What benchmark do you wish existed 49:32 Will benchmarks still look like benchmarks in 5 years 52:02 How to contribute to ProgramBench

vincent sunn chen

26,810 views • 2 months ago

Alibaba just released a coding model that hits 82 percent on SWE-Bench Verified. That is the highest score ever published for an open-source model. The weights are free. The license is Apache 2.0. You can run it today. The model is Qwen 4 Coder 32B. Here is what 82 percent on SWE-Bench Verified actually means. SWE-Bench Verified tests whether an AI can autonomously resolve real bugs pulled from real production GitHub repositories. Not synthetic exercises. Real open-source projects that real teams depend on. A model gets a bug report, reads the code, writes a fix, and either passes the test suite or it does not. At 82 percent, Qwen 4 Coder 32B resolves 82 out of every 100 real production bugs it is given. Without a human guiding it. On code it has never seen before. For comparison: Qwen 4 Coder 32B: 82 percent SWE-Bench Verified. Open source. Apache 2.0. Claude Fable 5: 80.3 percent SWE-Bench Pro. $10 input / $50 output per million tokens. Currently suspended. GPT-5.6 Sol: Competitive on Terminal-Bench. $5 input / $30 output per million tokens. An open-weight model that you can download and run for free just beat both of them on the benchmark designed to measure real software engineering capability. Here is the architecture. Qwen 4 Coder 32B is a 32 billion parameter dense model. Not a Mixture-of-Experts. Every parameter is active on every request. This matters for inference: a dense 32B model runs on 22 gigabytes of VRAM, which fits on a single high-end consumer GPU or a MacBook Pro with 64GB of unified memory. The smaller variant, Qwen 4 Coder 4B, runs at approximately 135 tokens per second on an M5 Max and fits inside 8 gigabytes of RAM. For a model with usable coding capability, that is a new bar for what fits in a single laptop. The training methodology continued Alibaba's approach of reinforcement learning on verifiable coding tasks. The model gets rewarded when its code passes tests. It gets penalized when it fails. Over millions of training steps, the model learns to write code that actually runs rather than code that looks plausible. License: Apache 2.0. Full commercial use. No attribution requirement. No revenue threshold. No monthly active user ceiling. Weights: Hugging Face, available today. Runs on: vLLM, Ollama, SGLang, and any standard GGUF-compatible inference engine. Qwen 4 32B also runs at approximately 135 tokens per second on an M5 Max chip, setting a new bar for what a sub-8GB model can do on Apple Silicon. The open-source coding model just beat the best closed-source model in the world on the benchmark designed to test whether AI can actually do software engineering. The weights are free. The subscription is optional. Source: Autom8Labs AI Insight July 2026, State of Open Source LLMs June 2026, Kunal Ganglani blog June 2026.

Alibaba just released a coding model that hits 82 percent on SWE-Bench Verified. That is the highest score ever published for an open-source model. The weights are free. The license is Apache 2.0. You can run it today. The model is Qwen 4 Coder 32B. Here is what 82 percent on SWE-Bench Verified actually means. SWE-Bench Verified tests whether an AI can autonomously resolve real bugs pulled from real production GitHub repositories. Not synthetic exercises. Real open-source projects that real teams depend on. A model gets a bug report, reads the code, writes a fix, and either passes the test suite or it does not. At 82 percent, Qwen 4 Coder 32B resolves 82 out of every 100 real production bugs it is given. Without a human guiding it. On code it has never seen before. For comparison: Qwen 4 Coder 32B: 82 percent SWE-Bench Verified. Open source. Apache 2.0. Claude Fable 5: 80.3 percent SWE-Bench Pro. $10 input / $50 output per million tokens. Currently suspended. GPT-5.6 Sol: Competitive on Terminal-Bench. $5 input / $30 output per million tokens. An open-weight model that you can download and run for free just beat both of them on the benchmark designed to measure real software engineering capability. Here is the architecture. Qwen 4 Coder 32B is a 32 billion parameter dense model. Not a Mixture-of-Experts. Every parameter is active on every request. This matters for inference: a dense 32B model runs on 22 gigabytes of VRAM, which fits on a single high-end consumer GPU or a MacBook Pro with 64GB of unified memory. The smaller variant, Qwen 4 Coder 4B, runs at approximately 135 tokens per second on an M5 Max and fits inside 8 gigabytes of RAM. For a model with usable coding capability, that is a new bar for what fits in a single laptop. The training methodology continued Alibaba's approach of reinforcement learning on verifiable coding tasks. The model gets rewarded when its code passes tests. It gets penalized when it fails. Over millions of training steps, the model learns to write code that actually runs rather than code that looks plausible. License: Apache 2.0. Full commercial use. No attribution requirement. No revenue threshold. No monthly active user ceiling. Weights: Hugging Face, available today. Runs on: vLLM, Ollama, SGLang, and any standard GGUF-compatible inference engine. Qwen 4 32B also runs at approximately 135 tokens per second on an M5 Max chip, setting a new bar for what a sub-8GB model can do on Apple Silicon. The open-source coding model just beat the best closed-source model in the world on the benchmark designed to test whether AI can actually do software engineering. The weights are free. The subscription is optional. Source: Autom8Labs AI Insight July 2026, State of Open Source LLMs June 2026, Kunal Ganglani blog June 2026.

Harman

41,278 views • 1 month ago

Looking forward to meeting with #NATO Defence Ministers. Our top priority is to boost defence spending & ramp up defence production. We are at a pivotal moment for our security. Make no mistake, NATO is strong today & we will become even stronger. #DefMin

Looking forward to meeting with #NATO Defence Ministers. Our top priority is to boost defence spending & ramp up defence production. We are at a pivotal moment for our security. Make no mistake, NATO is strong today & we will become even stronger. #DefMin

Mark Rutte

47,276 views • 1 year ago

Over 1 billion PDFs are created every day, but your agents still can’t read them reliably. Today we’re releasing Parse 2.0, the most accurate document parsing API in the world. Extend already processes millions of pages daily for leading AI teams like Brex, Mercury, Opendoor, Flatiron Health, and hundreds of others. Now, its even better. Parse 2.0 is SOTA quality on RealDoc-Bench, our open source benchmark that measures agent success rate on real world docs that agents actually encounter in production. We trained Parse 2.0 on 1M+ pages of the hardest documents seen in production. Here’s how it stacks up: - #1 in healthcare, real estate, logistics, and financial services - 95.7% agent Q&A accuracy on 581 docs (next best: 92%) - 0.847 F1 on layout (next best: 0.759) Give it a try today and build production-ready document agents with Extend.

Over 1 billion PDFs are created every day, but your agents still can’t read them reliably. Today we’re releasing Parse 2.0, the most accurate document parsing API in the world. Extend already processes millions of pages daily for leading AI teams like Brex, Mercury, Opendoor, Flatiron Health, and hundreds of others. Now, its even better. Parse 2.0 is SOTA quality on RealDoc-Bench, our open source benchmark that measures agent success rate on real world docs that agents actually encounter in production. We trained Parse 2.0 on 1M+ pages of the hardest documents seen in production. Here’s how it stacks up: - #1 in healthcare, real estate, logistics, and financial services - 95.7% agent Q&A accuracy on 581 docs (next best: 92%) - 0.847 F1 on layout (next best: 0.759) Give it a try today and build production-ready document agents with Extend.

Kushal Byatnal

586,891 views • 2 months ago

Today we announced first U3O8 production from our 85%-owned Kayelekera #Uranium Mine in Malawi. This major milestone is a key final step in Kayelekera’s successful processing plant commissioning, with production planned to ramp up to 2.4M lbs of U3O8 per annum in Q1 of 2026. Hear Lotus' $LOT Managing Director Greg Bittar discuss in the investor webinar held earlier today – watch it here:

Today we announced first U3O8 production from our 85%-owned Kayelekera #Uranium Mine in Malawi. This major milestone is a key final step in Kayelekera’s successful processing plant commissioning, with production planned to ramp up to 2.4M lbs of U3O8 per annum in Q1 of 2026. Hear Lotus' $LOT Managing Director Greg Bittar discuss in the investor webinar held earlier today – watch it here:

Lotus Resources

13,848 views • 11 months ago

MiniMax M2.1 is officially live🚀 Built for real-world coding and AI-native organizations — from vibe builds to serious workflows. A SOTA 10B-activated OSS coding & agent model, scoring 72.5% on SWE-multilingual and 88.6% on our newly open-sourced VIBE-bench, exceeding leading closed-source models like Gemini 3 Pro and Claude 4.5 Sonnet. The most powerful OSS model for the agentic era is here.

MiniMax M2.1 is officially live🚀 Built for real-world coding and AI-native organizations — from vibe builds to serious workflows. A SOTA 10B-activated OSS coding & agent model, scoring 72.5% on SWE-multilingual and 88.6% on our newly open-sourced VIBE-bench, exceeding leading closed-source models like Gemini 3 Pro and Claude 4.5 Sonnet. The most powerful OSS model for the agentic era is here.

MiniMax (official)

1,074,195 views • 7 months ago

Obamacare destroyed families like mine. Our premiums jumped from $800 to over $2,400 a month!! I’ve lived the failure Democrats created. Democrats set this ACA tax credit cliff, and on January 1 millions of Americans are about to hit a financial brick wall. We need an off-ramp from Obamacare and an on-ramp to REAL healthcare freedom! That could be price transparency, competition, strong HSAs, and giving industries the power to negotiate lower costs. This entire crisis is a Democrat-made mess. It’s time Republicans step up and actually fix it!!

Obamacare destroyed families like mine. Our premiums jumped from $800 to over $2,400 a month!! I’ve lived the failure Democrats created. Democrats set this ACA tax credit cliff, and on January 1 millions of Americans are about to hit a financial brick wall. We need an off-ramp from Obamacare and an on-ramp to REAL healthcare freedom! That could be price transparency, competition, strong HSAs, and giving industries the power to negotiate lower costs. This entire crisis is a Democrat-made mess. It’s time Republicans step up and actually fix it!!

Former Congresswoman Marjorie Taylor Greene🇺🇸

460,762 views • 9 months ago

Fresh off the heels of being granted a temporary restraining order in the legal dispute with Angstrom Automotive, it looks like Cybertruck production has made a bit of a rebound today at the Giga Texas outbound lot! Many (perhaps 100 or more) were parked all over the lot today, some intermixed with Model Y’s and a few Cybercabs, others in groups by themselves, more autonomously driving to the EOL facility for processing and many at the Direct Delivery center. It’s great to see this & I hope signals a new ramp of Cybertruck production!

Fresh off the heels of being granted a temporary restraining order in the legal dispute with Angstrom Automotive, it looks like Cybertruck production has made a bit of a rebound today at the Giga Texas outbound lot! Many (perhaps 100 or more) were parked all over the lot today, some intermixed with Model Y’s and a few Cybercabs, others in groups by themselves, more autonomously driving to the EOL facility for processing and many at the Direct Delivery center. It’s great to see this & I hope signals a new ramp of Cybertruck production!

Joe Tegtmeyer 🚀 🤠🛸😎

26,896 views • 10 days ago

Hoping your coding agents could understand you and adapt to your preferences? Meet TOM-SWE, our new framework for coding agents that don’t just write code, but model the user's mind persistently (ranging from general preferences to small details) arxiv: ❓Motivation: Most coding agents today can plan, edit, run, and test code. But they still fail at a key part of real-world development, understanding the user! Underspecified, shifting, or context-dependent instructions can easily break them. You must have those moments when coding agents were running for 10 minutes and ended up producing things largely misaligned. (1/)

Hoping your coding agents could understand you and adapt to your preferences? Meet TOM-SWE, our new framework for coding agents that don’t just write code, but model the user's mind persistently (ranging from general preferences to small details) arxiv: ❓Motivation: Most coding agents today can plan, edit, run, and test code. But they still fail at a key part of real-world development, understanding the user! Underspecified, shifting, or context-dependent instructions can easily break them. You must have those moments when coding agents were running for 10 minutes and ended up producing things largely misaligned. (1/)

Xuhui Zhou

40,191 views • 9 months ago