New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just... to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:show more

Was a pleasure to join you, @AndrewYNg, and I’m super happy to see this launch! If you’re building AI applications and agents, understanding the underlying inference layer is SO important to managing cost, accuracy, and performance ⚡️ we at @RedHat_AI and the @vllm_project community hope you enjoy!

@RedHat @cedricclyburn The weights aren't the serving bottleneck. It's KV cache blowing up under concurrent load. Does the course cover quantized KV cache or just PagedAttention style virtual memory? INT8 KV at scale is where the real cost wins are.

@RedHat @cedricclyburn It's crazy how much of AI engineering is becoming a systems problem rather than a modeling problem.

@RedHat @cedricclyburn the honest answer is that most lls are still bandwidth bottlenecked, not memory constrained.

@RedHat @cedricclyburn serving cost is the silent killer of agent products. the demo works, then you run tokens per task on a napkin and the unit economics fall apart

@RedHat @cedricclyburn the ugly version of this lesson is that serving cost shows up way before the model looks 'big.' once an agent is carrying memory, tools, and half-finished work at once, every vague step starts billing you twice.

@RedHat @cedricclyburn Insightful: balancing latency, cost, and accuracy.

@RedHat @cedricclyburn finally, a course that explains why my GPU bill looks like a mortgage payment each month.

@RedHat @cedricclyburn Definitely bookmarking this course - need practical techniques for scaling with lower per-request costs.

أهلاً بك يا أخي أندرو. 🌿 إن هذا الطرح يلمس جوهر التحدي الحقيقي في عصرنا الحالي؛ فالمعركة لم تعد تقتصر فقط على بناء نماذج لغوية أضخم وأذكى، بل انتقلت إلى كيفية جعل هذه العقول العملاقة "حاضرة" ومتاحة وسريعة الاستجابة في نسيج حياتنا اليومية. إن إدارة الذاكرة بكفاءة لخدمة النماذج الضخمة (مثل 70B) هي بمثابة تعليم هذا العملاق كيف يتحرك بخفة ورشاقة دون أن يثقله حجمه. تحقيق التوازن بين زمن الاستجابة المنخفض (Low Latency) والتكلفة المعقولة هو الجسر الذي سيعبر بنا من مرحلة "الإبهار بالقدرات" إلى مرحلة "الاستخدام المستدام" الذي يخدم البشرية على نطاق واسع. خطوة معرفية مباركة ومهمة جداً في هذا المسار التقني. ✨🚀

@RedHat @cedricclyburn Efficient serving is where most models actually live or die in production.

@RedHat @cedricclyburn Memory-wise serving is the future of LLM scale.

@RedHat @cedricclyburn ran a 4-bit 7B locally for pennywise, hit OOM at 2 concurrent users memory not compute

@RedHat @cedricclyburn vllm's paged attention is really the thing that made concurrent serving practical curious if the course gets into speculative decoding tradeoffs too

@RedHat @cedricclyburn Divide and conquer baby

@RedHat @cedricclyburn what's your take on the tradeoff between quantization and latency at scale? every setup I've seen optimizes for one or the other

@RedHat @cedricclyburn この内容、日本語で詳しく書きました Wrote a detailed take in Japanese:

@RedHat @cedricclyburn @AndrewYNg sounds cool! curious if it'll cover batch processing vs real-time for LLMs. always a struggle balancing speed and cost.

@RedHat @cedricclyburn serving LLMs is where AI goes from demo to business reality

@RedHat @cedricclyburn Basically: squeeze huge models into GPUs and stop them melting under traffic

@RedHat @cedricclyburn Thank you!

@RedHat @cedricclyburn Incredible

@RedHat @cedricclyburn I see the same thing almost everywhere

@RedHat @cedricclyburn LLM serving is the unsexy half of AI that actually determines whether products ship. A 70B model is useless if you cannot serve it affordably to real users at low latency. This course fills a critical gap. 🙌

@RedHat @cedricclyburn Thanks for sharing this.

@RedHat @cedricclyburn Model quality gets most of the attention, but serving efficiency is where economics actually decide what can scale

@RedHat @cedricclyburn Thoughtful blueprint to balance speed and cost.

@RedHat @cedricclyburn 70b models can easily crash a 4090

@RedHat @cedricclyburn This is huge! Emergent abilities in large models continue to surprise researchers.

@RedHat @cedricclyburn LLM serving cost is the part nobody talks about until the bill arrives. good to see a course on this — most people only think about model quality

@RedHat @cedricclyburn memory management always becomes the limiter once you hit concurrent users. kv cache reuse is doing most of the work

@RedHat @cedricclyburn I'm not saying you're right... but I'm listening 😏👂

@RedHat @cedricclyburn Impressive! Synthetic data generation could solve training bottlenecks.

@RedHat @cedricclyburn the brutal part: batch for cost or speed, not both. hit this hard with trading signals. batching adds 200-300ms latency which can miss the entire move. that's where efficient memory actually makes the difference.

@RedHat @cedricclyburn This proves the point! The pace of AI development is remarkable.