Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Introducing RSI-Inference Bench from 🍎 Apple ML Research 🌐 💻 Can AI agents improve the systems that run AI? We spent $100K API tokens, 50K GPU hours to find out. We built environments and benchmark from real-world #vLLM and #SGLang tasks, covering the full lifecycle of model inference engineering:...

79,918 görüntüleme • 1 ay önce •via X (Twitter)

35 Yorum

Ruixiang Zhang profil fotoğrafı
Ruixiang Zhang1 ay önce

this was a team effort with @junteng88716710 @richard_baihe @UnderGroundJeg @YizheZhangNLP. None of it would have been possible without their amazing work and persistent push, all credits to the team

Maksym Andriushchenko profil fotoğrafı
Maksym Andriushchenko1 ay önce

interesting to see that our InferenceBench has ~same rankings! Claude models are consistently ahead of GPT models.

Woosuk Kwon profil fotoğrafı
Woosuk Kwon1 ay önce

Great work. But why is it called RSI?

Ruixiang Zhang profil fotoğrafı
Ruixiang Zhang1 ay önce

thanks @woosuk_k! we believe inference is the heart of every AI agent, so we built envs + evals that, e.g. let agents served by @vllm_project improve @vllm_project itself. that's a perfect example of a self-improving loop in our view of RSI

Wolfbane profil fotoğrafı
Wolfbane1 ay önce

@ivanfioravanti imagine this for m7 ❤️

David Batista profil fotoğrafı
David Batista1 ay önce

Apple ML Research spent $100K in API tokens and 50K GPU hours asking whether AI agents can improve the systems that run AI. best agent so far: 26.7% pass@1 on real vLLM and SGLang tasks. picture a mechanic tuning the engine of the car that's driving them to the shop.

Dimitrios Skarlatos profil fotoğrafı
Dimitrios Skarlatos1 ay önce

Nice work! The per-model cost looks much better than the aggregated one :) Would love to see a write-up!

Bingyi Kang profil fotoğrafı
Bingyi Kang1 ay önce

amazing!

Ruixiang Zhang profil fotoğrafı
Ruixiang Zhang1 ay önce

thanks @bingyikang the 🐐!

AI Quanting profil fotoğrafı
AI Quanting1 ay önce

26.7 percent pass@1 on real vLLM and SGLang work is a useful place to start. Is bug fix much easier than day-0 model support, or do the three splits land near the same rate?

Almost Tomorrow profil fotoğrafı
Almost Tomorrow1 ay önce

The breakdown by task type would be revealing. Are agents better at fixing a localized bug than improving performance without breaking correctness elsewhere? The aggregate pass rate could hide very different levels of usefulness.

Ivan Fioravanti profil fotoğrafı
Ivan Fioravanti1 ay önce

Interesting research! Bookmarked and gonna to deep dive on it soon!

Flix Muller profil fotoğrafı
Flix Muller1 ay önce

Interesting to see AI agents tested on real inference engineering tasks. The 26.7% pass rate shows how much room there still is for improvement.

BreezeOg profil fotoğrafı
BreezeOg1 ay önce

agents improving their own inference is a cool benchmark

Rach profil fotoğrafı
Rach1 ay önce

i didn’t know you guys did cool experiments at apple!! this is super fun

Andrey Kolesnikov profil fotoğrafı
Andrey Kolesnikov1 ay önce

how useful

noel profil fotoğrafı
noel1 ay önce

are you guys sure that sonnet didn't cheat?

Varun Kumar profil fotoğrafı
Varun Kumar1 ay önce

Nice work! Do you have data on for what % of tasks, Opus 5.5 refuses and downgrades to a lower model quality

Stanlin Dias profil fotoğrafı
Stanlin Dias29 gün önce

Good detail in the scoring: a result that fails correctness counts as 1.00x, so an agent can't win on speed alone. 🧪 It is 37 tasks across vLLM and SGLang, each with 1 to 8 GPUs and up to 8 hours offline. Curious how often agents trade correctness for speed. 🤔

Will {AI} profil fotoğrafı
Will {AI}1 ay önce

I’ll tweak it a bit for my own needs, looks amazing! Great work guys!!

CloudDog profil fotoğrafı
CloudDog29 gün önce

50K GPU hours to ask whether agents can tune inference — a benchmark that measures the actual money question

Wen profil fotoğrafı
Wen1 ay önce

烧十万美元 token、五万卡时来验证 AI 能不能优化跑 AI 的系统,这题本身就很有趣,等于花钱请 AI 帮自己省钱。用 vLLM 和 SGLang 真实负载做 benchmark,比玩具任务有说服力得多,蹲一个结果看看 agent 到底挖出了哪些系统级优化。

飞叔 profil fotoğrafı
飞叔1 ay önce

The continuous rewards are intriguing. Does a faster solution have to pass correctness checks across multiple batch sizes and context lengths, or only the task’s target workload?

J.D. Nicholls is building in public profil fotoğrafı
J.D. Nicholls is building in public1 ay önce

why you are not using the Verified Organization instead @Apple

zdj profil fotoğrafı
zdj29 gün önce

Best agent only nails 26.7% on its first try (pass@1 = success on the first attempt). They burned 50K GPU hours to prove that. Honestly, I'm glad inference engineering is still a human job.

Ratel profil fotoğrafı
Ratel1 ay önce

100k in API tokens?? oh lordy

Crio Songo profil fotoğrafı
Crio Songo1 ay önce

This benchmark fills a real gap in inference engineering, the test coverage looks really solid.

Francesco Massa profil fotoğrafı
Francesco Massa29 gün önce

How do you prevent steering & shortcuts ?

aldo pareja profil fotoğrafı
aldo pareja1 ay önce

@grok do they release the environments and the recipe to build them?

Marcel Butucea profil fotoğrafı
Marcel Butucea1 ay önce

nice progress on the benchmark

Boyuan Wang profil fotoğrafı
Boyuan Wang1 ay önce

Great job!

安叫兽|Bird🕊️ 🔶 BNB profil fotoğrafı
安叫兽|Bird🕊️ 🔶 BNB1 ay önce

想看换一批负载还能涨多少,别只把跑分调上去了

logotype profil fotoğrafı
logotype1 ay önce

Hey. Is RSI-Inference using any new vLLM/SGLang endpoints or just the existing standard ones? I’d like to add support for our inference engine LayerScale. Cheers, Victor

Chris Coyne profil fotoğrafı
Chris Coyne1 ay önce

Apple not known for any good LLM Model. It continues Apple’s AI is the classic “we’ll take the easy wins” strategy, not a “we build the smartest models” one. They’re not leading on raw intelligence and never really tried to. Their new AFM 3 models (the ones powering the 2026 Siri AI overhaul) show solid jumps against their own previous versions — human preference rates went from the low 20s% to the mid-40s/60s% on text tasks. But Apple still doesn’t publish proper head-to-heads against GPT-6, Gemini 3, Claude Opus 5.5, etc. New CEO John Ternus has been pretty open about it: they don’t need to beat ChatGPT. They want personal context (everything already on your iPhone) and tight OS integration instead. The models themselves lean heavily on Google. AFM 3 was built in collaboration with Gemini tech under a multi-year deal. Apple is effectively renting a lot of the hard capability work while wrapping it in Private Cloud Compute and on-device silicon. That gives them real advantages in speed, battery life, and privacy on Apple Silicon devices — their on-device models are often faster than comparable small open models even if they’re not smarter — but it’s not independent frontier work. The RSI-Inference Bench you were looking at fits the pattern. It’s solid research from their ML team on whether agents can actually improve real inference stacks (vLLM/SGLang tasks). Best result so far is still only 26.7% pass@1 after serious spend. That’s useful signal that the hard systems problems remain hard, but it’s not evidence Apple is suddenly a top AI lab. They’re good at the polished, private, “it just works in your apps” layer that normal people notice. For anything that needs the actual smartest model, most people are still going elsewhere.

Christopher Owen profil fotoğrafı
Christopher Owen1 ay önce

ds41f 1017? 🙏 👉👈

Benzer Videolar