New blog is out: vLLM x AgentX: Optimizing for Real-World Agentic Serving. Agent traffic stresses every layer of the serving stack at once. This post walks the full-stack work for optimizing vLLM on Agentic workloads, including the architecture, framework, and runtime optimizations, measured on AgentX, SemiAnalysis's public agentic benchmark. 🧵1/6

DeepSeek V4 Pro sustains 83K tokens per GPU-second at a strict p90 interactivity SLO of 50+ tok/s per user, and 130K at the top of the frontier. For context: serving the same workload on Opus 5 at the same cache-hit rate costs 106x more. This shows how much headroom open-weight models have when the stack is tuned for them. 2/6

What agentic traffic actually looks like, from real coding-agent traces: 🔷 43 turns per session, median 🔷 142K-token median input against a 444-token median output 🔷 96%+ prefix-cache hit rate 🔷 44% of sessions fork subagents Long prefixes, tiny outputs, constant reuse. 3/6

The work broadly spans across three planes in the stack. Here are some highlights: 🔶 KV cache management matters. A packed KV layout for DeepSeek V4 saves ~10% KV memory and cuts 92 tensors per block down to 1 🔶 Parallelism follows the model. Decode context parallelism gives Kimi K3 2.7x decode throughput at the same TPOT; prefill context parallelism runs DeepSeek V4's sparse MLA 2.65x faster than head sharding 🔶 Simple scheduling makes an impact. Simply capping prefill breaks head-of-line blocking and achieves +93% TPGS and ~2.3x better p90 interactivity 4/6

Lessons learned, the hard way: 🔷 Pipeline parallelism is great on cold, long prompts. On warm agent turns that add a few hundred tokens, the bubbles eat the gain. 🔷 Decode context parallelism won on Kimi K3 but only matched DEP on DeepSeek V4. Parallelism has to follow the model's attention stack. 🔷 Load balancing doesn’t always beat simple session-sticky routing: for workloads with short inter-turn delays, preserving a warm KV cache can matter more than balancing the queue. 5/6

Everything is on a live public dashboard: tokens per dollar or tokens per GPU against interactivity, across GB300 NVL72 and B300, with public configs to reproduce each point. Our blog covers each of the above sections in detail, from optimizations to performance, along with what did not work. Read more at the full blog post below: 🔗 6/6

Those gaps between turns are mostly tool latency, not think time - the agent waiting on a page load or a build. Even when a human is watching, they aren't reading at 50 tok/s. The deadline that bites on our side is time to finish the turn, not tokens per second per user.

Full-stack agentic serving opts with a dashboard you can actually open — long prefix, tiny decode is the traffic shape that kills naive stacks.

Agentic traffic really does stress the whole stack—especially when agents interleave long context with many small requests. Curious how you handled scheduling for those bursty patterns?

142K in, 444 out. That ratio is why agent traffic looks nothing like chat. Does the 96% cache hit hold through the whole session? One miss at that context length would spike the response time for that turn.

the trace shape matches our fleet: agent turns are prefill-bound, decode is nearly noise. our 8-bit glm-5.3-flash fleet prefills ~350 tok/s at 4k ctx, ~160 cold at 38k - that 96% prefix-hit rate is the whole ballgame. every miss re-pays the prefix at cold speed

The 106x is the figure people will quote, so its denominator matters: your side is GPU-seconds you incur, the API side is a list price with margin inside it. Those are not the same quantity. Is the 106x against Opus 5 list pricing, or against an estimated serving cost?