New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys LMSYS Org and RadixArk RadixArk, and taught by Richard Chen Richard Chen, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation.... This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!show more

@lmsysorg @radixark @richardczl Thank you Andrew, Richard, and people worked behind, for the great course!

@lmsysorg @radixark Wowowowowoow!

@lmsysorg @radixark @richardczl Exactly what LLM deployment needs, cutting redundant compute with smart caching like SGLang can slash costs and boost speed. Great to see real-world efficiency in focus. Looking forward to this course!

@lmsysorg @radixark @richardczl SGLang is so underrated for inference optimization. Will check this out, Andrew!

@lmsysorg @radixark @richardczl This is great! Wondering how to add new courses? Would love to take one for Harness building and self-evolving agents.

@lmsysorg @radixark @richardczl Great course on inference

@lmsysorg @radixark @richardczl Most people focus on training LLMs, but the real leverage is in efficient inference. KV cache + RadixAttention in SGLang could significantly change how production GenAI systems are optimized. Excited to explore this.

@lmsysorg @radixark @richardczl UNBE

@lmsysorg @radixark @richardczl Inference cost is the quiet drag on every AI product's unit economics — compute per token compounds faster than most founders expect. SGLang's radix caching is one of the highest-leverage optimizations available right now, especially for multi-turn and batched workloads.

@lmsysorg @radixark @richardczl The cost of running LLMs in production is a real bottleneck. So many founders are wrestling with this.

Efficient inference is exactly what we're wrestling with building PawMe — an open-source AI robot for pets. Running vision + behavior models on embedded hardware at the edge is painful when compute is scarce. KV caching strategies like this are a game-changer for embodied AI at home.

@lmsysorg @radixark @richardczl I just upgraded sglang to 0.5.10.post1 2 days ago for my local inference. Sglang is my favorite😛

@lmsysorg @radixark @richardczl 🔥

@lmsysorg @radixark @richardczl sglang has been quietly becoming the go-to for running inference at scale. the radix attention approach cuts costs significantly vs naive serving. glad someone's teaching this properly, most teams learn it the hard way after their first production bill.

@lmsysorg @radixark @richardczl Understanding SGLang's caching and multi-GPU parallelism I think it will help me a lot with my technical skills on LLM inference efficiency.

@lmsysorg @radixark @richardczl SGLang sounds promising for optimizing LLM costs! I've seen inference expenses balloon quickly. Efficient text *and* image gen? Color me intrigued. #LLMs #AI #Efficiency

@lmsysorg @radixark @richardczl Efficient inference starts with smart reuse.

@lmsysorg @radixark @richardczl inference efficiency is becoming the real competitive moat. as models get smarter the question shifts from capability to cost per useful output. SGLang doing prefix caching and continuous batching right is huge for anyone running production workloads

@lmsysorg @radixark @richardczl sglang's chunked prefill actually lets you squeeze way more throughput from the same hardware compared to naive batching

@lmsysorg @radixark @richardczl Andrew Ng course on efficiency feels like we've come full circle from "throw more compute at it" back to actually caring about performance.

@lmsysorg @radixark @richardczl This is a must-watch for anyone who loves getting their hands dirty with inference frameworks. Moving from theory to actually implementing RadixAttention is the best way to understand how to cut production costs and latency. Great move by the LMSys and RadixArk teams!

@lmsysorg @radixark @richardczl Andrew Ng的可贵之处是把复杂的AI技术讲得任何人都能听懂。但听懂和理解之间，隔着一段工程实践。

@lmsysorg @radixark @richardczl sglang handling the batching is the right move. inference cost is the bottleneck for most real apps, not the training

@lmsysorg @radixark @richardczl Nice! Will need to have our team check it out

@lmsysorg @radixark @richardczl Inference efficiency is the quiet bottleneck slowing AI deployment at scale. Most teams focus on model quality, but KV cache sharing and prefix reuse like SGLang does can cut costs dramatically - making previously impossible use cases suddenly viable. Solid course.

@lmsysorg @radixark @richardczl Andrew Ng的可贵之处是把复杂的AI技术讲得任何人都能听懂。但听懂和理解之间，隔着一段工程实践。

@lmsysorg @radixark @richardczl inference efficiency is the part that doesn't get enough attention. faster generation changes what you can do in an agentic loop. latency at p95 is a different constraint than throughput and i think most agent builders are optimizing for the wrong one

@lmsysorg @radixark @richardczl Solid approach—scale and save on inference.

@lmsysorg @radixark @richardczl Caching shared context like this is such a practical way to cut down on redundant compute. Always appreciate these hands-on courses.

@lmsysorg @radixark @richardczl Smart caching could redefine LLM cost at scale.

Inference costs are the thing nobody warns you about when you start building with LLMs. I use Gemini Flash for question generation in my app and even with a cheaper model, optimizing how you batch and cache requests makes a massive difference at scale. Curious if SGLang handles the structured output case well (JSON schemas etc). That's where I burn the most tokens.

@lmsysorg @radixark @richardczl Significant update. Interesting implications for the future of work.

@lmsysorg @radixark @richardczl running LLMs is a challenge. Cost management is key. What specific efficiencies will the course cover?