Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Sequential Monte Carlo speculative decoding from Makora keeps multiple draft tokens alive in parallel instead of rewinding failed matches.

SemiAnalysis_'s profile picture

SemiAnalysis

117,028 subscribers

24,216 Aufrufe • vor 25 Tagen •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Speculative decoding speeds up generation from LLMs significantly by computing several potential tokens in parallel. Learn about this technique and how it has been utilized to achieve 2–3x speed-ups at inference:
0:32

Speculative decoding speeds up generation from LLMs significantly by computing several potential tokens in parallel. Learn about this technique and how it has been utilized to achieve 2–3x speed-ups at inference:

Google AI

38,566 Aufrufe • vor 1 Jahr

The example below is using prompt-based speculative decoding. Specifically, ngram hashing is utilized to suggest drafts of up to 64 tokens. The hasher keeps track of ngrams in the observed contexts, so mostly effective for coding tasks. Here is another demo:
0:47

The example below is using prompt-based speculative decoding. Specifically, ngram hashing is utilized to suggest drafts of up to 64 tokens. The hasher keeps track of ngrams in the observed contexts, so mostly effective for coding tasks. Here is another demo:

Georgi Gerganov

29,798 Aufrufe • vor 2 Monaten

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 · M5 Max · greedy exact match ▸ 85 tok/s, 3.3× at 1024 tokens (runtime) ▸ ~70 tok/s, 2.6× in the video (terminal I/O overhead) ▸ 80 tok/s, 3.1× at 2048 tokens (runtime) Currently working on: → Long context (speedup degrades past 4K tokens, KV cache growth) → Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.
0:39

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 · M5 Max · greedy exact match ▸ 85 tok/s, 3.3× at 1024 tokens (runtime) ▸ ~70 tok/s, 2.6× in the video (terminal I/O overhead) ▸ 80 tok/s, 3.1× at 2048 tokens (runtime) Currently working on: → Long context (speedup degrades past 4K tokens, KV cache growth) → Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.

bstn 👁️

36,942 Aufrufe • vor 2 Monaten

Unbelievable shot from Musetti in Monte-Carlo
0:29

Unbelievable shot from Musetti in Monte-Carlo

José Morgado

45,846 Aufrufe • vor 1 Jahr

What a shot from Altmaier in Monte-Carlo.
0:29

What a shot from Altmaier in Monte-Carlo.

José Morgado

32,290 Aufrufe • vor 1 Jahr

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?
0:26

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Avi Chawla

157,390 Aufrufe • vor 1 Monat

Is thumbnail A/B testing sequential or in parallel? Here’s the answer!
0:23

Is thumbnail A/B testing sequential or in parallel? Here’s the answer!

YouTube Creator Insider

27,158 Aufrufe • vor 1 Jahr

With love, from Monte Carlo 🇲🇨🇫🇷
1:05

With love, from Monte Carlo 🇲🇨🇫🇷

Thomas Casanova ✈️🏇🏑🏌️🏂

19,250 Aufrufe • vor 1 Jahr

Getting warm in Monte Carlo with last year’s finalists 💪 🔥 Carlos Alcaraz Rolex Monte-Carlo Masters
0:42

Getting warm in Monte Carlo with last year’s finalists 💪 🔥 Carlos Alcaraz Rolex Monte-Carlo Masters

Tennis TV

27,372 Aufrufe • vor 2 Monaten

Lorenzo Musetti. In the fading evening light of Monte-Carlo. Stunning.
0:30

Lorenzo Musetti. In the fading evening light of Monte-Carlo. Stunning.

Rolex Monte-Carlo Masters

32,852 Aufrufe • vor 1 Jahr

Learning from the best 🤩 Rolex Monte-Carlo Masters #RolexMonteCarloMasters
0:40

Learning from the best 🤩 Rolex Monte-Carlo Masters #RolexMonteCarloMasters

Tennis TV

13,103 Aufrufe • vor 2 Monaten

Having some fun with the new speculative generation feature in LM Studio: -MLX 4-bit Qwen 32B/0.5B draft runs a lot faster for coding tasks than the 32B model alone -Nice to visualizase the draft tokens generated:
0:25

Having some fun with the new speculative generation feature in LM Studio: -MLX 4-bit Qwen 32B/0.5B draft runs a lot faster for coding tasks than the 32B model alone -Nice to visualizase the draft tokens generated:

Awni Hannun

12,642 Aufrufe • vor 1 Jahr

Soak it in Lorenzo 🥹🇮🇹 Lorenzo Musetti is just the THIRD Italian Monte-Carlo finalist in the Open Era Rolex Monte-Carlo Masters | #RolexMonteCarloMasters
0:39

Soak it in Lorenzo 🥹🇮🇹 Lorenzo Musetti is just the THIRD Italian Monte-Carlo finalist in the Open Era Rolex Monte-Carlo Masters | #RolexMonteCarloMasters

ATP Tour

34,080 Aufrufe • vor 1 Jahr

Monaco: A Muslim prays in the middle of the Café de Paris in Monte Carlo.
0:29

Monaco: A Muslim prays in the middle of the Café de Paris in Monte Carlo.

Yossi BenYakar

15,805 Aufrufe • vor 8 Monaten

33 years ago today, Michael Jackson received multiple awards At the 1993 World Music Awards in Monte Carlo, Monaco, presented by Michael Douglas, Prince Albert and Princess Stephanie of Monaco.
6:35

33 years ago today, Michael Jackson received multiple awards At the 1993 World Music Awards in Monte Carlo, Monaco, presented by Michael Douglas, Prince Albert and Princess Stephanie of Monaco.

Michael Jackson Crave

91,626 Aufrufe • vor 1 Monat

Destination: Rolex Monte-Carlo Masters 🧡 Monte-Carlo, but on 8mm film…
0:50

Destination: Rolex Monte-Carlo Masters 🧡 Monte-Carlo, but on 8mm film…

ATP Tour

26,462 Aufrufe • vor 2 Monaten

Carlos Alcaraz vs Jannik Sinner, Monte-Carlo 2026 Final. The first chapter of tennis' greatest rivalry in 2026. Sinner and Alcaraz met for the first time that season in a blockbuster Monte-Carlo final.
6:09

Carlos Alcaraz vs Jannik Sinner, Monte-Carlo 2026 Final. The first chapter of tennis' greatest rivalry in 2026. Sinner and Alcaraz met for the first time that season in a blockbuster Monte-Carlo final.

Tennis Performances

15,693 Aufrufe • vor 1 Tag

Next Gen in Monte Carlo 🤩🇲🇨 #RolexMonteCarloMasters
0:21

Next Gen in Monte Carlo 🤩🇲🇨 #RolexMonteCarloMasters

Tennis TV

70,822 Aufrufe • vor 2 Monaten

Good vibes only from Stef ✌️ Rolex Monte-Carlo Masters | #RolexMonteCarloMasters
0:24

Good vibes only from Stef ✌️ Rolex Monte-Carlo Masters | #RolexMonteCarloMasters

ATP Tour

20,718 Aufrufe • vor 1 Jahr

Option 2: Carlos Alcaraz in Monte-Carlo!
0:31

Option 2: Carlos Alcaraz in Monte-Carlo!

Tennis TV

13,620 Aufrufe • vor 6 Monaten