Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

I built an interactive simulator that shows why LLM inference capacity is hard. Batching, queues, KV cache, GPU decode, priority tiers - all coupled. Change one knob and the bottleneck moves.

Khushboo Verma

164,918 subscribers

25,217 Aufrufe • vor 4 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

23 Kommentare

Profilbild von Khushboo Verma

Khushboo Vermavor 4 Monaten

The core problem: GPUs hate serving one request at a time. Decode is memory-bandwidth-bound, every forward pass streams the whole model's weights from memory to produce... one token. Serve requests sequentially and your $40k GPU idles between tokens.

Profilbild von Khushboo Verma

Khushboo Vermavor 4 Monaten

The fix: batching. One forward pass = one next token for EVERY sequence in the batch. 16 requests, one pass, 16 tokens. The weight reads are shared, so step cost grows much slower than batch size. Throughput is almost free. Almost.

Profilbild von Khushboo Verma

Khushboo Vermavor 4 Monaten

But requests finish at different lengths. Static batching strands empty slots until the longest sequence finishes, wasted bandwidth every step. Continuous batching backfills each slot the moment it frees, between decode steps. Utilization goes from ~60% to ~95%.

Profilbild von Khushboo Verma

Khushboo Vermavor 4 Monaten

And priority tiers make it worse. Strict priority is great for enterprise traffic and starves free-tier users forever under load. Real schedulers bound starvation: weighted fair queueing, max-wait promotion, reserved slots.

Profilbild von Khushboo Verma

Khushboo Vermavor 4 Monaten

The part nobody tells you: the bottleneck doesn't stay put. Long prompts? Prefill steals decode time → inter-token latency spikes Long outputs? Decode-bound → queues explode Small KV cache? Memory blocks admissions while GPUs sit idle Few CPU workers? GPUs starve

Profilbild von Khushboo Verma

Khushboo Vermavor 4 Monaten

So I built a simulator where you can watch all of this happen. ⚙️ Tune request rate, token distributions, GPUs, batch size, wait deadline, tier mix, SLOs, see throughput, TTFT, p99 and a live "current bottleneck" banner respond instantly.

Profilbild von Khushboo Verma

Khushboo Vermavor 4 Monaten

7 one-click failure scenarios included: - decode saturation - prefill storm - CPU starvation - batching tradeoff - tier starvation - KV cache crunch Each one teaches a different capacity lesson in ~10 seconds. Try now! 👀

Profilbild von PsudoMike 🇨🇦

PsudoMike 🇨🇦vor 4 Monaten

KV cache fragmentation is what actually surprises engineers the first time. You've got headroom in theory, but contiguous memory requirements choke it. Spent two weeks tuning batch sizes before realizing the bottleneck was cache eviction, not GPU utilization.

Profilbild von Khushboo Verma

Khushboo Vermavor 4 Monaten

Yeah, complex systems

Profilbild von Harnoor Singh

Harnoor Singhvor 4 Monaten

That’s a great explanation!

Profilbild von Khushboo Verma

Khushboo Vermavor 4 Monaten

glad you liked it :)

Profilbild von Shivay Lamba

Shivay Lambavor 4 Monaten

great visuals!

Profilbild von Khushboo Verma

Khushboo Vermavor 4 Monaten

Thanks, so far, haven't had much luck in video generation using AI models, this one turned out nice

Profilbild von David Uche

David Uchevor 4 Monaten

Nice stuff

Profilbild von Khushboo Verma

Khushboo Vermavor 4 Monaten

Thanks 🙏

Profilbild von NoiselessTrading

NoiselessTradingvor 4 Monaten

Insanely well done,! 🙌 The interactive simulator + failure scenarios make LLM serving concepts finally click. Super valuable tool!”

Profilbild von Khushboo Verma

Khushboo Vermavor 4 Monaten

Thanks :)

Profilbild von Aniket Maurya

Aniket Mauryavor 4 Monaten

great explaination!

Profilbild von Khushboo Verma

Khushboo Vermavor 4 Monaten

Thank you :)

Profilbild von PsudoMike 🇨🇦

PsudoMike 🇨🇦vor 4 Monaten

Seen this in payments. Optimize auth latency and now settlement throughput is the ceiling. Fix settlement, reconciliation piles up. Every component looks fine in isolation. You only see the coupling when one part starts to scale and the others don't follow.

Profilbild von PsudoMike 🇨🇦

PsudoMike 🇨🇦vor 4 Monaten

The KV cache piece is what gets people. Adding more GPUs sounds like the fix but long sequences fill KV memory fast. You end up memory bound before you're ever compute bound. What latency targets are you modeling at different request rates?

Profilbild von Gaurav Nardia

Gaurav Nardiavor 4 Monaten

Khushboo, I DMed you something. Can you please check? I think it's in the request tab.

Profilbild von Ishwar | Infrastructure Systems

Ishwar | Infrastructure Systemsvor 4 Monaten

The moving bottleneck aspect is underrated. A lot of engineering intuition assumes "find bottleneck -> optimize bottleneck." In serving systems, fixing one bottleneck often just reveals the next one.

Ähnliche Videos

Batching strategies in LLM inference, clearly explained! (bookmark it) - Static - Dynamic - And continuous batching I wrote a detailed article explaining how each works and why serving LLMs is a different problem from traditional ML inference. The article is quoted below.

Batching strategies in LLM inference, clearly explained! (bookmark it) - Static - Dynamic - And continuous batching I wrote a detailed article explaining how each works and why serving LLMs is a different problem from traditional ML inference. The article is quoted below.

Akshay 🚀

30,234 Aufrufe • vor 1 Monat

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 Aufrufe • vor 1 Monat

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.

h100envy

61,470 Aufrufe • vor 3 Monaten

LLM inference speed with vs. without KV caching: (learn how and why it works below)

LLM inference speed with vs. without KV caching: (learn how and why it works below)

Daily Dose of Data Science

59,218 Aufrufe • vor 6 Monaten

LLM inference speed with vs. without KV caching: (learn how and why it works below)

LLM inference speed with vs. without KV caching: (learn how and why it works below)

Avi Chawla

396,012 Aufrufe • vor 7 Monaten

We’re thrilled to open-source TriAttention! 🚀 🦞 Deploy OpenClaw (32B LLM) on a single 24GB RTX 4090 locally 💻Full code open-source & vLLM-ready for one-click deployment ⚡️ 2.5× faster inference speed & 10.7× less KV cache memory usage TriAttention is a novel KV cache compression method built on rigorous trigonometric analysis in the Pre‑RoPE space for efficient LLM long reasoning. Github Repo: Paper Link: Homepage:

We’re thrilled to open-source TriAttention! 🚀 🦞 Deploy OpenClaw (32B LLM) on a single 24GB RTX 4090 locally 💻Full code open-source & vLLM-ready for one-click deployment ⚡️ 2.5× faster inference speed & 10.7× less KV cache memory usage TriAttention is a novel KV cache compression method built on rigorous trigonometric analysis in the Pre‑RoPE space for efficient LLM long reasoning. Github Repo: Paper Link: Homepage:

Yukang Chen

199,279 Aufrufe • vor 6 Monaten

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

Reese Chong

53,026 Aufrufe • vor 6 Monaten

The most important 40 seconds of the Dario pod…. Why do these models feel like magic all of a sudden (fintwit)!? “KV CACHE OPTIMIZATION”…which is still a major major bottleneck btw…

The most important 40 seconds of the Dario pod…. Why do these models feel like magic all of a sudden (fintwit)!? “KV CACHE OPTIMIZATION”…which is still a major major bottleneck btw…

Jared L Kubin

137,743 Aufrufe • vor 8 Monaten

Jim Cramer on Leopold Aschenbrenner’s $NBIS stake “When this man touches something, it turns to gold.” The next AI bottleneck is efficient inference at scale which is why Nebius Token Factory is built around optimization, orchestration and agentic deployment rather than basic GPU rental.

Jim Cramer on Leopold Aschenbrenner’s $NBIS stake “When this man touches something, it turns to gold.” The next AI bottleneck is efficient inference at scale which is why Nebius Token Factory is built around optimization, orchestration and agentic deployment rather than basic GPU rental.

Shay Boloor

289,153 Aufrufe • vor 4 Monaten

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

h100envy

32,605 Aufrufe • vor 3 Monaten

KV caching is the fundamental optimization underpinning autoregressive LLM inference. Transformer layers store keys and values from earlier tokens, then reuse them as new tokens are generated. This avoids recomputation, but at the cost of memory capacity and bandwidth.

KV caching is the fundamental optimization underpinning autoregressive LLM inference. Transformer layers store keys and values from earlier tokens, then reuse them as new tokens are generated. This avoids recomputation, but at the cost of memory capacity and bandwidth.

Alec Helbling

36,668 Aufrufe • vor 1 Monat

I made an interactive enzyme simulator. Change the temperature, add or remove inhibitors, and tune the probability that enzymes bind their substrates. Watch as the reaction rate moves up or down in real-time. Maybe this will be a helpful teaching tool.

I made an interactive enzyme simulator. Change the temperature, add or remove inhibitors, and tune the probability that enzymes bind their substrates. Watch as the reaction rate moves up or down in real-time. Maybe this will be a helpful teaching tool.

Niko McCarty.

60,605 Aufrufe • vor 1 Jahr

The reason I own $AAOI is because it builds the optical connectivity that lets AI chips move data across data centers. As AI scales toward 500K+ GPU factories, copper becomes the bottleneck & optics become plumbing that moves data fast enough for training & inference to work.

The reason I own $AAOI is because it builds the optical connectivity that lets AI chips move data across data centers. As AI scales toward 500K+ GPU factories, copper becomes the bottleneck & optics become plumbing that moves data fast enough for training & inference to work.

Shay Boloor

188,217 Aufrufe • vor 4 Monaten

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Avi Chawla

16,140 Aufrufe • vor 2 Monaten

Researchers made LLM inference 14x faster and 90% cheaper. The video below depicts the speed up in action. Providers discount cached input tokens by as much as 90% because a cache hit skips prefill compute entirely. For stable system prompts and tool definitions, hit rates of 60 to 85% are achievable, which makes it the highest-leverage inference optimization. But the cost saving only works when the cached text is an exact, byte-for-byte prefix of the new request. If you change one character anywhere before it, the entire cached region is missed. Three common request patterns produce full cache misses: - A query that needs documents A and B together can't reuse B's standalone cache, because those KV entries were computed without A in front of them. - The same three documents retrieved in a different order produce a full cache miss, even though nothing about the documents changed. - In multi-turn conversations, every new turn invalidates whatever was cached beyond the stable prefix. Alibaba's production data did a study on this and found that just 10% of cached KV blocks serve 77% of all cache hits. So most of what gets cached sits in storage and is never used a single time. And the root cause is that KV entries are position-dependent. Each token's KV encodes attention to everything before it, so a cached block is only valid in the exact context it was computed in. There's a second, less discussed problem as well. Cache management runs inside the inference engine's process. Moving KV tensors between GPU, CPU, and disk competes with inference for the same resources. This is why Google's TurboQuant compresses KV caches to 3 bits with no accuracy loss and still causes a 20%+ slowdown when it runs in-process. Fixing both problems means restructuring where caching lives. Cache management moves into its own process, the engine only exchanges block IDs over shared GPU memory, and heavy data movement runs across GPU, CPU, disk, and remote storage in parallel. Non-prefix reuse gets handled by selectively recomputing only the small set of tokens that attend across document boundaries. LMCache is the open-source project (10k+ stars) that implements this exact architecture, and it plugs into vLLM, SGLang, and TensorRT-LLM. The selective recomputation part is implemented in its CacheBlend technique, which makes cached docs in any order and combination, with 2-4x faster multi-document processing. On H200s running Qwen3-235B with 50 concurrent users, LMCache's multiprocess mode delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. GitHub repo: (don't forget to star 🌟) My co-founder wrote a full breakdown of KV cache management. It covers the disaggregated architecture behind the 14x speed up, how CacheBlend preserves generation quality while skipping recomputation, and how to turn every document in a knowledge base into a reusable cached asset. Read it below.

Researchers made LLM inference 14x faster and 90% cheaper. The video below depicts the speed up in action. Providers discount cached input tokens by as much as 90% because a cache hit skips prefill compute entirely. For stable system prompts and tool definitions, hit rates of 60 to 85% are achievable, which makes it the highest-leverage inference optimization. But the cost saving only works when the cached text is an exact, byte-for-byte prefix of the new request. If you change one character anywhere before it, the entire cached region is missed. Three common request patterns produce full cache misses: - A query that needs documents A and B together can't reuse B's standalone cache, because those KV entries were computed without A in front of them. - The same three documents retrieved in a different order produce a full cache miss, even though nothing about the documents changed. - In multi-turn conversations, every new turn invalidates whatever was cached beyond the stable prefix. Alibaba's production data did a study on this and found that just 10% of cached KV blocks serve 77% of all cache hits. So most of what gets cached sits in storage and is never used a single time. And the root cause is that KV entries are position-dependent. Each token's KV encodes attention to everything before it, so a cached block is only valid in the exact context it was computed in. There's a second, less discussed problem as well. Cache management runs inside the inference engine's process. Moving KV tensors between GPU, CPU, and disk competes with inference for the same resources. This is why Google's TurboQuant compresses KV caches to 3 bits with no accuracy loss and still causes a 20%+ slowdown when it runs in-process. Fixing both problems means restructuring where caching lives. Cache management moves into its own process, the engine only exchanges block IDs over shared GPU memory, and heavy data movement runs across GPU, CPU, disk, and remote storage in parallel. Non-prefix reuse gets handled by selectively recomputing only the small set of tokens that attend across document boundaries. LMCache is the open-source project (10k+ stars) that implements this exact architecture, and it plugs into vLLM, SGLang, and TensorRT-LLM. The selective recomputation part is implemented in its CacheBlend technique, which makes cached docs in any order and combination, with 2-4x faster multi-document processing. On H200s running Qwen3-235B with 50 concurrent users, LMCache's multiprocess mode delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. GitHub repo: (don't forget to star 🌟) My co-founder wrote a full breakdown of KV cache management. It covers the disaggregated architecture behind the 14x speed up, how CacheBlend preserves generation quality while skipping recomputation, and how to turn every document in a knowledge base into a reusable cached asset. Read it below.

Avi Chawla

30,717 Aufrufe • vor 3 Monaten

Introducing Universal Simulator (UniSim), an interactive simulator of the real world. Interactive website: Paper:

Introducing Universal Simulator (UniSim), an interactive simulator of the real world. Interactive website: Paper:

Sherry Yang

440,007 Aufrufe • vor 3 Jahren

This guy built a real motion racing simulator that actually moves, spins and drifts with the game.

This guy built a real motion racing simulator that actually moves, spins and drifts with the game.

Science girl

41,677 Aufrufe • vor 6 Monaten

SITUATION EXPLAINED: Cerebras raised $5.55 billion in their IPO and closing their first day of trading valued at $66 billion, making it the biggest US tech IPO since Snowflake in 2020. Cerebras makes Wafer-Scale Engine chips built for AI inference. We asked Sarah Fong the main difference between wafer-scale chips and traditional GPUs: - GPUs are great at parallel work (graphics, training) - AI inference is sequential, AKA one token at a time This causes the "memory wall" problem: - Every GPU core needs model weights, KV cache, and activations to do its math - On a GPU, that data lives in off-chip memory (HBM) - Cores constantly load and offload from off-chip memory, which is a huge bottleneck; hardware accounts for ~70% of inference latency Cerebras' chips: -Dinner-plate sized (vs. GPUs which are palm-sized) with tens of thousands of cores -Memory sits directly on top of the cores as distributed SRAM -Weights and KV cache can be accessed at on-chip speeds in the PB/s range, compared with off-chip speeds in the TB/s range achieved by GPUs with HBM.

SITUATION EXPLAINED: Cerebras raised $5.55 billion in their IPO and closing their first day of trading valued at $66 billion, making it the biggest US tech IPO since Snowflake in 2020. Cerebras makes Wafer-Scale Engine chips built for AI inference. We asked Sarah Fong the main difference between wafer-scale chips and traditional GPUs: - GPUs are great at parallel work (graphics, training) - AI inference is sequential, AKA one token at a time This causes the "memory wall" problem: - Every GPU core needs model weights, KV cache, and activations to do its math - On a GPU, that data lives in off-chip memory (HBM) - Cores constantly load and offload from off-chip memory, which is a huge bottleneck; hardware accounts for ~70% of inference latency Cerebras' chips: -Dinner-plate sized (vs. GPUs which are palm-sized) with tens of thousands of cores -Memory sits directly on top of the cores as distributed SRAM -Weights and KV cache can be accessed at on-chip speeds in the PB/s range, compared with off-chip speeds in the TB/s range achieved by GPUs with HBM.

MTS

44,057 Aufrufe • vor 5 Monaten

i spent 3 hours finding the sweet spot for hermes 4.3 36B on a single RTX 3090. saving you the trouble, anon. the model is 21.8GB at Q4_K_M. that leaves 2.2GB free on 24GB VRAM. not much room for KV cache. here's what actually happened: 4K: 35.3 tok/s 8K: 35.2 tok/s 16K: 34.8 tok/s 32K: 34.6 tok/s 64K: 6.4 tok/s 128K: 1.9 tok/s flat from 4K to 32K. then it falls off a cliff at 64K. the trick is quantized KV cache. without it you OOM at 16K. with quantized KV cache you get 32K at full speed. all 64 layers on GPU. at 64K something weird happens. ngl 99 (all layers on GPU) = 3.96 tok/s. the KV cache silently spills to CPU. drop to ngl 55 and speed jumps to 6.37. drop to ngl 48 and it gets worse again (3.46). there's an offload sweet spot where you free just enough VRAM for the cache without losing too much compute to PCIe transfers. 128K works at ngl 32 but you're at 1.95 tok/s. half the model on CPU. usable for batch work, not for interactive. the sweet spot command: llama-server -m hermes-4.3-36b-Q4_K_M.gguf -ngl 99 -c 32768 --cache-type-k q4_0 --cache-type-v q4_0 32K context. 34.6 tok/s. all on GPU. this is where dense 36B lives on 24GB. for comparison, qwen 3.5 (35B MoE, 3B active) holds 112 tok/s from 4K all the way to 262K on the same GPU. no speed drop. same total params, completely different architecture. hybrid linear attention means flat context scaling. dense pays for every token in the KV cache. code and quality comparison coming next. fast vs slow generation side by side in the videos below.

i spent 3 hours finding the sweet spot for hermes 4.3 36B on a single RTX 3090. saving you the trouble, anon. the model is 21.8GB at Q4_K_M. that leaves 2.2GB free on 24GB VRAM. not much room for KV cache. here's what actually happened: 4K: 35.3 tok/s 8K: 35.2 tok/s 16K: 34.8 tok/s 32K: 34.6 tok/s 64K: 6.4 tok/s 128K: 1.9 tok/s flat from 4K to 32K. then it falls off a cliff at 64K. the trick is quantized KV cache. without it you OOM at 16K. with quantized KV cache you get 32K at full speed. all 64 layers on GPU. at 64K something weird happens. ngl 99 (all layers on GPU) = 3.96 tok/s. the KV cache silently spills to CPU. drop to ngl 55 and speed jumps to 6.37. drop to ngl 48 and it gets worse again (3.46). there's an offload sweet spot where you free just enough VRAM for the cache without losing too much compute to PCIe transfers. 128K works at ngl 32 but you're at 1.95 tok/s. half the model on CPU. usable for batch work, not for interactive. the sweet spot command: llama-server -m hermes-4.3-36b-Q4_K_M.gguf -ngl 99 -c 32768 --cache-type-k q4_0 --cache-type-v q4_0 32K context. 34.6 tok/s. all on GPU. this is where dense 36B lives on 24GB. for comparison, qwen 3.5 (35B MoE, 3B active) holds 112 tok/s from 4K all the way to 262K on the same GPU. no speed drop. same total params, completely different architecture. hybrid linear attention means flat context scaling. dense pays for every token in the KV cache. code and quality comparison coming next. fast vs slow generation side by side in the videos below.

Sudo su

52,268 Aufrufe • vor 7 Monaten

$NBIS co-founder Roman Chernin explained why the next AI bottleneck is running inference efficiently at scale. That’s why Nebius is building Token Factory around model optimization, orchestration & agentic AI deployment instead of acting like a basic GPU rental business.

$NBIS co-founder Roman Chernin explained why the next AI bottleneck is running inference efficiently at scale. That’s why Nebius is building Token Factory around model optimization, orchestration & agentic AI deployment instead of acting like a basic GPU rental business.

Shay Boloor

130,014 Aufrufe • vor 5 Monaten