Загрузка видео...

Не удалось загрузить видео

На главную

Sequential Monte Carlo speculative decoding from Makora keeps multiple draft tokens alive in parallel instead of rewinding failed matches.

SemiAnalysis_'s profile picture

SemiAnalysis

117,028 subscribers

24,216 просмотров • 25 дней назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Speculative decoding speeds up generation from LLMs significantly by computing several potential tokens in parallel. Learn about this technique and how it has been utilized to achieve 2–3x speed-ups at inference:
0:32

Speculative decoding speeds up generation from LLMs significantly by computing several potential tokens in parallel. Learn about this technique and how it has been utilized to achieve 2–3x speed-ups at inference:

Google AI

38,566 просмотров • 1 год назад

The example below is using prompt-based speculative decoding. Specifically, ngram hashing is utilized to suggest drafts of up to 64 tokens. The hasher keeps track of ngrams in the observed contexts, so mostly effective for coding tasks. Here is another demo:
0:47

The example below is using prompt-based speculative decoding. Specifically, ngram hashing is utilized to suggest drafts of up to 64 tokens. The hasher keeps track of ngrams in the observed contexts, so mostly effective for coding tasks. Here is another demo:

Georgi Gerganov

29,798 просмотров • 2 месяцев назад

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 · M5 Max · greedy exact match ▸ 85 tok/s, 3.3× at 1024 tokens (runtime) ▸ ~70 tok/s, 2.6× in the video (terminal I/O overhead) ▸ 80 tok/s, 3.1× at 2048 tokens (runtime) Currently working on: → Long context (speedup degrades past 4K tokens, KV cache growth) → Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.
0:39

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 · M5 Max · greedy exact match ▸ 85 tok/s, 3.3× at 1024 tokens (runtime) ▸ ~70 tok/s, 2.6× in the video (terminal I/O overhead) ▸ 80 tok/s, 3.1× at 2048 tokens (runtime) Currently working on: → Long context (speedup degrades past 4K tokens, KV cache growth) → Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.

bstn 👁️

36,942 просмотров • 2 месяцев назад

Unbelievable shot from Musetti in Monte-Carlo
0:29

Unbelievable shot from Musetti in Monte-Carlo

José Morgado

45,846 просмотров • 1 год назад

What a shot from Altmaier in Monte-Carlo.
0:29

What a shot from Altmaier in Monte-Carlo.

José Morgado

32,290 просмотров • 1 год назад

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?
0:26

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Avi Chawla

157,390 просмотров • 1 месяц назад

Is thumbnail A/B testing sequential or in parallel? Here’s the answer!
0:23

Is thumbnail A/B testing sequential or in parallel? Here’s the answer!

YouTube Creator Insider

27,158 просмотров • 1 год назад

With love, from Monte Carlo 🇲🇨🇫🇷
1:05

With love, from Monte Carlo 🇲🇨🇫🇷

Thomas Casanova ✈️🏇🏑🏌️🏂

19,250 просмотров • 1 год назад

Getting warm in Monte Carlo with last year’s finalists 💪 🔥 Carlos Alcaraz Rolex Monte-Carlo Masters
0:42

Getting warm in Monte Carlo with last year’s finalists 💪 🔥 Carlos Alcaraz Rolex Monte-Carlo Masters

Tennis TV

27,372 просмотров • 2 месяцев назад

Lorenzo Musetti. In the fading evening light of Monte-Carlo. Stunning.
0:30

Lorenzo Musetti. In the fading evening light of Monte-Carlo. Stunning.

Rolex Monte-Carlo Masters

32,852 просмотров • 1 год назад

Learning from the best 🤩 Rolex Monte-Carlo Masters #RolexMonteCarloMasters
0:40

Learning from the best 🤩 Rolex Monte-Carlo Masters #RolexMonteCarloMasters

Tennis TV

13,103 просмотров • 2 месяцев назад

Having some fun with the new speculative generation feature in LM Studio: -MLX 4-bit Qwen 32B/0.5B draft runs a lot faster for coding tasks than the 32B model alone -Nice to visualizase the draft tokens generated:
0:25

Having some fun with the new speculative generation feature in LM Studio: -MLX 4-bit Qwen 32B/0.5B draft runs a lot faster for coding tasks than the 32B model alone -Nice to visualizase the draft tokens generated:

Awni Hannun

12,642 просмотров • 1 год назад

Soak it in Lorenzo 🥹🇮🇹 Lorenzo Musetti is just the THIRD Italian Monte-Carlo finalist in the Open Era Rolex Monte-Carlo Masters | #RolexMonteCarloMasters
0:39

Soak it in Lorenzo 🥹🇮🇹 Lorenzo Musetti is just the THIRD Italian Monte-Carlo finalist in the Open Era Rolex Monte-Carlo Masters | #RolexMonteCarloMasters

ATP Tour

34,080 просмотров • 1 год назад

Monaco: A Muslim prays in the middle of the Café de Paris in Monte Carlo.
0:29

Monaco: A Muslim prays in the middle of the Café de Paris in Monte Carlo.

Yossi BenYakar

15,805 просмотров • 8 месяцев назад

33 years ago today, Michael Jackson received multiple awards At the 1993 World Music Awards in Monte Carlo, Monaco, presented by Michael Douglas, Prince Albert and Princess Stephanie of Monaco.
6:35

33 years ago today, Michael Jackson received multiple awards At the 1993 World Music Awards in Monte Carlo, Monaco, presented by Michael Douglas, Prince Albert and Princess Stephanie of Monaco.

Michael Jackson Crave

91,626 просмотров • 1 месяц назад

Destination: Rolex Monte-Carlo Masters 🧡 Monte-Carlo, but on 8mm film…
0:50

Destination: Rolex Monte-Carlo Masters 🧡 Monte-Carlo, but on 8mm film…

ATP Tour

26,462 просмотров • 2 месяцев назад

Carlos Alcaraz vs Jannik Sinner, Monte-Carlo 2026 Final. The first chapter of tennis' greatest rivalry in 2026. Sinner and Alcaraz met for the first time that season in a blockbuster Monte-Carlo final.
6:09

Carlos Alcaraz vs Jannik Sinner, Monte-Carlo 2026 Final. The first chapter of tennis' greatest rivalry in 2026. Sinner and Alcaraz met for the first time that season in a blockbuster Monte-Carlo final.

Tennis Performances

15,693 просмотров • 1 день назад

Next Gen in Monte Carlo 🤩🇲🇨 #RolexMonteCarloMasters
0:21

Next Gen in Monte Carlo 🤩🇲🇨 #RolexMonteCarloMasters

Tennis TV

70,822 просмотров • 2 месяцев назад

Good vibes only from Stef ✌️ Rolex Monte-Carlo Masters | #RolexMonteCarloMasters
0:24

Good vibes only from Stef ✌️ Rolex Monte-Carlo Masters | #RolexMonteCarloMasters

ATP Tour

20,718 просмотров • 1 год назад

Option 2: Carlos Alcaraz in Monte-Carlo!
0:31

Option 2: Carlos Alcaraz in Monte-Carlo!

Tennis TV

13,620 просмотров • 6 месяцев назад