Video yükleniyor...
Video Yüklenemedi
Introducing RSI-Inference Bench from 🍎 Apple ML Research 🌐 💻 Can AI agents improve the systems that run AI? We spent $100K API tokens, 50K GPU hours to find out. We built environments and benchmark from real-world #vLLM and #SGLang tasks, covering the full lifecycle of model inference engineering:... show more
79,918 görüntüleme • 1 ay önce •via X (Twitter)
35 Yorum

this was a team effort with @junteng88716710 @richard_baihe @UnderGroundJeg @YizheZhangNLP. None of it would have been possible without their amazing work and persistent push, all credits to the team

interesting to see that our InferenceBench has ~same rankings! Claude models are consistently ahead of GPT models.

Great work. But why is it called RSI?

thanks @woosuk_k! we believe inference is the heart of every AI agent, so we built envs + evals that, e.g. let agents served by @vllm_project improve @vllm_project itself. that's a perfect example of a self-improving loop in our view of RSI

@ivanfioravanti imagine this for m7 ❤️

Apple ML Research spent $100K in API tokens and 50K GPU hours asking whether AI agents can improve the systems that run AI. best agent so far: 26.7% pass@1 on real vLLM and SGLang tasks. picture a mechanic tuning the engine of the car that's driving them to the shop.

Nice work! The per-model cost looks much better than the aggregated one :) Would love to see a write-up!

amazing!

thanks @bingyikang the 🐐!

26.7 percent pass@1 on real vLLM and SGLang work is a useful place to start. Is bug fix much easier than day-0 model support, or do the three splits land near the same rate?

The breakdown by task type would be revealing. Are agents better at fixing a localized bug than improving performance without breaking correctness elsewhere? The aggregate pass rate could hide very different levels of usefulness.

Interesting research! Bookmarked and gonna to deep dive on it soon!

Interesting to see AI agents tested on real inference engineering tasks. The 26.7% pass rate shows how much room there still is for improvement.

agents improving their own inference is a cool benchmark

i didn’t know you guys did cool experiments at apple!! this is super fun

how useful

are you guys sure that sonnet didn't cheat?

Nice work! Do you have data on for what % of tasks, Opus 5.5 refuses and downgrades to a lower model quality

Good detail in the scoring: a result that fails correctness counts as 1.00x, so an agent can't win on speed alone. 🧪 It is 37 tasks across vLLM and SGLang, each with 1 to 8 GPUs and up to 8 hours offline. Curious how often agents trade correctness for speed. 🤔

I’ll tweak it a bit for my own needs, looks amazing! Great work guys!!

50K GPU hours to ask whether agents can tune inference — a benchmark that measures the actual money question

烧十万美元 token、五万卡时来验证 AI 能不能优化跑 AI 的系统,这题本身就很有趣,等于花钱请 AI 帮自己省钱。用 vLLM 和 SGLang 真实负载做 benchmark,比玩具任务有说服力得多,蹲一个结果看看 agent 到底挖出了哪些系统级优化。

The continuous rewards are intriguing. Does a faster solution have to pass correctness checks across multiple batch sizes and context lengths, or only the task’s target workload?

why you are not using the Verified Organization instead @Apple

Best agent only nails 26.7% on its first try (pass@1 = success on the first attempt). They burned 50K GPU hours to prove that. Honestly, I'm glad inference engineering is still a human job.

100k in API tokens?? oh lordy

This benchmark fills a real gap in inference engineering, the test coverage looks really solid.

How do you prevent steering & shortcuts ?

@grok do they release the environments and the recipe to build them?

nice progress on the benchmark

Great job!

想看换一批负载还能涨多少,别只把跑分调上去了

Hey. Is RSI-Inference using any new vLLM/SGLang endpoints or just the existing standard ones? I’d like to add support for our inference engine LayerScale. Cheers, Victor

Apple not known for any good LLM Model. It continues Apple’s AI is the classic “we’ll take the easy wins” strategy, not a “we build the smartest models” one. They’re not leading on raw intelligence and never really tried to. Their new AFM 3 models (the ones powering the 2026 Siri AI overhaul) show solid jumps against their own previous versions — human preference rates went from the low 20s% to the mid-40s/60s% on text tasks. But Apple still doesn’t publish proper head-to-heads against GPT-6, Gemini 3, Claude Opus 5.5, etc. New CEO John Ternus has been pretty open about it: they don’t need to beat ChatGPT. They want personal context (everything already on your iPhone) and tight OS integration instead. The models themselves lean heavily on Google. AFM 3 was built in collaboration with Gemini tech under a multi-year deal. Apple is effectively renting a lot of the hard capability work while wrapping it in Private Cloud Compute and on-device silicon. That gives them real advantages in speed, battery life, and privacy on Apple Silicon devices — their on-device models are often faster than comparable small open models even if they’re not smarter — but it’s not independent frontier work. The RSI-Inference Bench you were looking at fits the pattern. It’s solid research from their ML team on whether agents can actually improve real inference stacks (vLLM/SGLang tasks). Best result so far is still only 26.7% pass@1 after serious spend. That’s useful signal that the hard systems problems remain hard, but it’s not evidence Apple is suddenly a top AI lab. They’re good at the polished, private, “it just works in your apps” layer that normal people notice. For anything that needs the actual smartest model, most people are still going elsewhere.

ds41f 1017? 🙏 👉👈



