Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Need faster LLM inference without sacrificing accuracy? Speculative decoding can help. Choosing the right draft length and drafting method depends on your model, workload and hardware. We break down five practical guidelines for balancing throughput and latency.

NVIDIA AI

338,344 subscribers

35,445 Aufrufe • vor 1 Monat •via X (Twitter)

Wissenschaft & Technologie Bildung Comedy

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Everyone talks about our hardware @Cerebras. Few notice the software. Ryan Loney breaks down the hidden optimizations powering 20× faster LLM inference than GPUs, speculative decoding, token reuse, and why we’re just getting started. Watch the full story here

Everyone talks about our hardware @Cerebras. Few notice the software. Ryan Loney breaks down the hidden optimizations powering 20× faster LLM inference than GPUs, speculative decoding, token reuse, and why we’re just getting started. Watch the full story here

Cerebras

222,816 Aufrufe • vor 9 Monaten

🚀 Our DSpark for Qwen3.8-27B beats native MTP with the same 8 speculative tokens on 4×H200. Up to 52% faster single-stream decoding and 23% higher peak throughput. On 8-needle MRCR, it averages 4.30 accepted tokens beyond 1M context. Model: Demo below, see for yourself!

🚀 Our DSpark for Qwen3.8-27B beats native MTP with the same 8 speculative tokens on 4×H200. Up to 52% faster single-stream decoding and 23% higher peak throughput. On 8-needle MRCR, it averages 4.30 accepted tokens beyond 1M context. Model: Demo below, see for yourself!

Red Hat AI

14,041 Aufrufe • vor 1 Monat

While exploring diffusion language models (DLMs), we kept asking ourselves: given speculative decoding, a lossless acceleration technique, why do we still need DLMs? 🤔 Our latest work, ORL, tries to answer this question: AR optimizes intelligence 👆, speculative decoding optimizes speed 👉, and DLMs optimize the Pareto frontier of speed and intelligence 🚀. Through co-design across the algorithms, the training and inference engine, and the kernels, ORL pushes out a new Pareto frontier of intelligence and speed. Similar performance, much faster than speculative decoding. You say lossy spec? ORL-DLM is better too! 😎 🌐 Project + live demos:

While exploring diffusion language models (DLMs), we kept asking ourselves: given speculative decoding, a lossless acceleration technique, why do we still need DLMs? 🤔 Our latest work, ORL, tries to answer this question: AR optimizes intelligence 👆, speculative decoding optimizes speed 👉, and DLMs optimize the Pareto frontier of speed and intelligence 🚀. Through co-design across the algorithms, the training and inference engine, and the kernels, ORL pushes out a new Pareto frontier of intelligence and speed. Similar performance, much faster than speculative decoding. You say lossy spec? ORL-DLM is better too! 😎 🌐 Project + live demos:

Chenfeng_X

21,240 Aufrufe • vor 17 Tagen

Production-ready RAG systems need observability. From tracking latency and throughput to evaluating response quality with human feedback or LLM-as-a-judge, robust observability gives you visibility into both system performance and output quality, on both a component and system-wide level. This lesson from our Retrieval Augmented Generation course breaks down the core components of an effective eval system and how to balance cost, automation, and accuracy when choosing your metrics. 📚 Learn more in the full course:

Production-ready RAG systems need observability. From tracking latency and throughput to evaluating response quality with human feedback or LLM-as-a-judge, robust observability gives you visibility into both system performance and output quality, on both a component and system-wide level. This lesson from our Retrieval Augmented Generation course breaks down the core components of an effective eval system and how to balance cost, automation, and accuracy when choosing your metrics. 📚 Learn more in the full course:

DeepLearning.AI

18,793 Aufrufe • vor 9 Monaten

Already running an inference engine? So where does NVIDIA Dynamo fit in? In five minutes, we break down how Dynamo sits around engines like SGLang, vLLM and TensorRT-LLM to scale inference across GPUs and nodes. Full video in the comments 🔽

Already running an inference engine? So where does NVIDIA Dynamo fit in? In five minutes, we break down how Dynamo sits around engines like SGLang, vLLM and TensorRT-LLM to scale inference across GPUs and nodes. Full video in the comments 🔽

NVIDIA AI

30,908 Aufrufe • vor 2 Monaten

Inference providers hate this man... Your LLM deployment can deliver 30 to 50% better performance after initial deployment. If your current inference provider isn't learning your workload and tuning the stack continuously after deployment, you need another provider. Wafer's continual inference agents do that work. Wafer's continual inference agents profile your production workload to identify where scheduling, decoding, or GPU memory pressure limits performance. The agents benchmark changes to batching, serving engines, and GPU kernels against your workload. Each candidate must preserve correctness and meet your reliability targets before it ships. The agents keep profiling as traffic changes, identifying new bottlenecks and further opportunities to improve performance per dollar. Watch emilio andere explain how that work continues after deployment in his conversation with theCUBE.

wafer

42,840 Aufrufe • vor 1 Monat

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Avi Chawla

158,352 Aufrufe • vor 5 Monaten

Just launched #CES2026, the new open-source NVIDIA Nemotron Speech ASR model is here to solve latency drift and redundant compute. Its cache-aware streaming architecture eliminates the need for buffered inference, giving you stable, sub-100ms latency (24ms median T-T-F) and up to 3x more throughput on your GPU. 🤗 Read the technical blog with real-world results from Daily and Modal on Hugging Face:

Just launched #CES2026, the new open-source NVIDIA Nemotron Speech ASR model is here to solve latency drift and redundant compute. Its cache-aware streaming architecture eliminates the need for buffered inference, giving you stable, sub-100ms latency (24ms median T-T-F) and up to 3x more throughput on your GPU. 🤗 Read the technical blog with real-world results from Daily and Modal on Hugging Face:

NVIDIA AI Developer

138,518 Aufrufe • vor 9 Monaten

Introducing EVO Router - a smart router that continuously optimizes and hill-climbs on your AI inference workloads. It learns from your code, prompts, production traffic, use cases, and SLAs, then searches for the best setup for every workload. That can mean more than choosing just a single model or routing through a classifier. @EVO__HQ can optimize across model × provider combinations, fusion systems, cascades, routing policies, and prompts — whatever the workload allows, while staying within your quality, latency, and reliability constraints. As open-source models move closer to frontier-model performance, we believe this kind of workload-specific hill-climbing will become an important part of how AI systems are deployed and operated. Our early users have seen 30–60% lower inference costs across use cases ranging from complex multi-turn agents to asynchronous batch workloads. EVO Router has also reached the Pareto frontier on several benchmarks, outperforming existing setups on cost, quality, or both.

Introducing EVO Router - a smart router that continuously optimizes and hill-climbs on your AI inference workloads. It learns from your code, prompts, production traffic, use cases, and SLAs, then searches for the best setup for every workload. That can mean more than choosing just a single model or routing through a classifier. @EVO__HQ can optimize across model × provider combinations, fusion systems, cascades, routing policies, and prompts — whatever the workload allows, while staying within your quality, latency, and reliability constraints. As open-source models move closer to frontier-model performance, we believe this kind of workload-specific hill-climbing will become an important part of how AI systems are deployed and operated. Our early users have seen 30–60% lower inference costs across use cases ranging from complex multi-turn agents to asynchronous batch workloads. EVO Router has also reached the Pareto frontier on several benchmarks, outperforming existing setups on cost, quality, or both.

Alok Bishoyi

16,327 Aufrufe • vor 3 Monaten

The Inference Engineering Masterclass: 10x faster models, quantization, speculative decoding, Rubin, & self-optimizing AI Baseten Philip Kiely and ali explain what actually happens after a model is trained, why turning weights into a fast and reliable product creates an entirely new optimization problem, how quantization errors can cancel out to unlock more throughput, why inference teams are still finding 20–200% performance gains, how video generation runs into a quadratic attention wall, and how GLM-5.2 helped rewrite and optimize the GPU kernels serving GLM-5.2 itself.

The Inference Engineering Masterclass: 10x faster models, quantization, speculative decoding, Rubin, & self-optimizing AI Baseten Philip Kiely and ali explain what actually happens after a model is trained, why turning weights into a fast and reliable product creates an entirely new optimization problem, how quantization errors can cancel out to unlock more throughput, why inference teams are still finding 20–200% performance gains, how video generation runs into a quadratic attention wall, and how GLM-5.2 helped rewrite and optimize the GPU kernels serving GLM-5.2 itself.

Latent.Space

149,249 Aufrufe • vor 3 Monaten

Massive breakthrough in speculative decoding! It made a 9B LLM on my MacBook 4x faster, and 2.6x faster than regular speculative decoding. I ran Qwen3.5 9B on my M5 MacBook Pro with the same prompt, three ways: - No speculative decoding: done in 42s - Regular speculative decoding: done in 30s - DFlash + Weaver: done in 11s The video below shows all three side by side. Here's why it works: Speculative decoding lets a cheap drafter guess the next few tokens, and the main model checks them all in one pass. The fastest drafters, like DFlash, guess every position in parallel. But each guess is made on its own, so position 5 never sees what was picked at position 4. One early mismatch and everything after it gets thrown away. Weaver fixes this. It's a tiny adapter (one Transformer layer) on top of DFlash that: ↳ picks among DFlash's top candidates left to right, so each pick depends on the last ↳ grows a tree of likely continuations instead of a single chain ↳ lets the main model verify the whole tree in one pass On my Mac, every pass of the model produced 7.5 tokens instead of 1. On a B200 GPU, the paper reports 4x faster decoding across chat, math and code. Mirai Labs open-sourced the whole stack in their engine, uzu, built for Apple Silicon. GitHub repo: (don't forget to star it 🌟) Trying it takes two commands: 𝗯𝗿𝗲𝘄 𝗶𝗻𝘀𝘁𝗮𝗹𝗹 𝗺𝗶𝗿𝗮𝗶 𝗺𝗶𝗿𝗮𝗶 --𝗺𝗼𝗱𝗲𝗹 𝘁𝗿𝘆𝗺𝗶𝗿𝗮𝗶/𝗤𝘄𝗲𝗻𝟯.𝟱-𝟵𝗕-𝗠 -𝗺 "𝗪𝗿𝗶𝘁𝗲 𝗮 𝗽𝘆𝘁𝗵𝗼𝗻 𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻 𝘁𝗼 𝗺𝗲𝗿𝗴𝗲 𝘁𝘄𝗼 𝘀𝗼𝗿𝘁𝗲𝗱 𝗹𝗶𝘀𝘁𝘀" --- I wrote a full breakdown of why local inference is memory-bound and how uzu gets past it, with reproducible code. The article is quoted below. In the replies, there's an illustration of standard vs. speculative vs. uzu decoding.

Massive breakthrough in speculative decoding! It made a 9B LLM on my MacBook 4x faster, and 2.6x faster than regular speculative decoding. I ran Qwen3.5 9B on my M5 MacBook Pro with the same prompt, three ways: - No speculative decoding: done in 42s - Regular speculative decoding: done in 30s - DFlash + Weaver: done in 11s The video below shows all three side by side. Here's why it works: Speculative decoding lets a cheap drafter guess the next few tokens, and the main model checks them all in one pass. The fastest drafters, like DFlash, guess every position in parallel. But each guess is made on its own, so position 5 never sees what was picked at position 4. One early mismatch and everything after it gets thrown away. Weaver fixes this. It's a tiny adapter (one Transformer layer) on top of DFlash that: ↳ picks among DFlash's top candidates left to right, so each pick depends on the last ↳ grows a tree of likely continuations instead of a single chain ↳ lets the main model verify the whole tree in one pass On my Mac, every pass of the model produced 7.5 tokens instead of 1. On a B200 GPU, the paper reports 4x faster decoding across chat, math and code. Mirai Labs open-sourced the whole stack in their engine, uzu, built for Apple Silicon. GitHub repo: (don't forget to star it 🌟) Trying it takes two commands: 𝗯𝗿𝗲𝘄 𝗶𝗻𝘀𝘁𝗮𝗹𝗹 𝗺𝗶𝗿𝗮𝗶 𝗺𝗶𝗿𝗮𝗶 --𝗺𝗼𝗱𝗲𝗹 𝘁𝗿𝘆𝗺𝗶𝗿𝗮𝗶/𝗤𝘄𝗲𝗻𝟯.𝟱-𝟵𝗕-𝗠 -𝗺 "𝗪𝗿𝗶𝘁𝗲 𝗮 𝗽𝘆𝘁𝗵𝗼𝗻 𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻 𝘁𝗼 𝗺𝗲𝗿𝗴𝗲 𝘁𝘄𝗼 𝘀𝗼𝗿𝘁𝗲𝗱 𝗹𝗶𝘀𝘁𝘀" --- I wrote a full breakdown of why local inference is memory-bound and how uzu gets past it, with reproducible code. The article is quoted below. In the replies, there's an illustration of standard vs. speculative vs. uzu decoding.

Akshay 🚀

26,337 Aufrufe • vor 3 Tagen

.AMD Instinct™ MI350X GPU Droplets are live on DigitalOcean's Agentic Inference Cloud. 🆕 Lower latency. Higher throughput. ✅ Transparent pricing and production-ready scale. ✅ AI without the infrastructure headache. 💯

.AMD Instinct™ MI350X GPU Droplets are live on DigitalOcean's Agentic Inference Cloud. 🆕 Lower latency. Higher throughput. ✅ Transparent pricing and production-ready scale. ✅ AI without the infrastructure headache. 💯

DigitalOcean

25,539 Aufrufe • vor 8 Monaten

SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉 DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales. We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec. At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads. With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300. DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks DeepSeek for open-sourcing! Blog with full technical details and commands to run below 👇

SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉 DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales. We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec. At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads. With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300. DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks DeepSeek for open-sourcing! Blog with full technical details and commands to run below 👇

LMSYS Org

170,752 Aufrufe • vor 3 Monaten

We just added speculative decoding for Carbon, so you can now generate DNA sequences 2-3x faster with llama.cpp This enables anyone to process huge genomes locally! GGUF weights and quants:

We just added speculative decoding for Carbon, so you can now generate DNA sequences 2-3x faster with llama.cpp This enables anyone to process huge genomes locally! GGUF weights and quants:

Lewis Tunstall

16,007 Aufrufe • vor 5 Monaten

🎉 MiniMax M3.1 Flash from MiniMax (official) has Day-0 support in vLLM! Built for multi-step automation, real-codebase engineering and visual reasoning—with native NVFP4 KV, NVFP4 MoE weights and DSpark speculative decoding in the serving stack. What the model can do, and the vLLM engineering that makes it practical to serve. 👇 Model: 1/5

🎉 MiniMax M3.1 Flash from MiniMax (official) has Day-0 support in vLLM! Built for multi-step automation, real-codebase engineering and visual reasoning—with native NVFP4 KV, NVFP4 MoE weights and DSpark speculative decoding in the serving stack. What the model can do, and the vLLM engineering that makes it practical to serve. 👇 Model: 1/5

vLLM

11,710 Aufrufe • vor 10 Tagen

Why are there so many ways to connect to Postgres? Because each method has tradeoffs. The right choice depends on your architecture. Let's break down 5 connection methods including direct, PgBouncer, and Hyperdrive.

Why are there so many ways to connect to Postgres? Because each method has tradeoffs. The right choice depends on your architecture. Let's break down 5 connection methods including direct, PgBouncer, and Hyperdrive.

PlanetScale

16,681 Aufrufe • vor 10 Monaten

Crusoe Managed Inference is now available. Run model inference faster, easier, and more cost-effectively. Large-context AI solutions demand fast outputs, massive throughput, and resilient scaling. Crusoe Managed Inference delivers breakthrough speed & reduced input-token spend.

Crusoe Managed Inference is now available. Run model inference faster, easier, and more cost-effectively. Large-context AI solutions demand fast outputs, massive throughput, and resilient scaling. Crusoe Managed Inference delivers breakthrough speed & reduced input-token spend.

Crusoe

12,096,815 Aufrufe • vor 11 Monaten

The Tokamak Milestone reduces latency and increases throughput, delivering faster, smoother blockchain performance and bringing Web3 closer to Web2 standards 🚀

The Tokamak Milestone reduces latency and increases throughput, delivering faster, smoother blockchain performance and bringing Web3 closer to Web2 standards 🚀

DFINITY Foundation

19,211 Aufrufe • vor 1 Jahr

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding. To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8. We are open sourcing it today. 1/2

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding. To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8. We are open sourcing it today. 1/2

Inferact

245,826 Aufrufe • vor 1 Monat