Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Sequential Monte Carlo speculative decoding from Makora keeps multiple draft tokens alive in parallel instead of rewinding failed matches.

SemiAnalysis_'s profile picture

SemiAnalysis

167,228 subscribers

24,418 görüntüleme • 4 ay önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

3 Yorum

Sakura Yuki profil fotoğrafı
Sakura Yuki4 ay önce

@makora_ai LLM inference is so memory-bound that drafting and scoring parallel particles is basically free compute-wise. Since it's on SGLang, it must be using tree KV caches to keep memory overhead sane.

itchy profil fotoğrafı
itchy4 ay önce

@makora_ai @no_stp_on_snek

Jonas Hernlund profil fotoğrafı
Jonas Hernlund4 ay önce

@makora_ai Nice gain on tokens per GPU. But in the field the ceiling is not the decoding trick, it is power and cooling per rack. You can double efficiency and still wait 3 years for interconnect. Algorithms scale faster than substations.

Benzer Videolar

The example below is using prompt-based speculative decoding. Specifically, ngram hashing is utilized to suggest drafts of up to 64 tokens. The hasher keeps track of ngrams in the observed contexts, so mostly effective for coding tasks. Here is another demo:
0:47

The example below is using prompt-based speculative decoding. Specifically, ngram hashing is utilized to suggest drafts of up to 64 tokens. The hasher keeps track of ngrams in the observed contexts, so mostly effective for coding tasks. Here is another demo:

Georgi Gerganov

29,890 görüntüleme • 6 ay önce

I asked 6 agents in two conditions (parallel v/s sequential) to create the same tower defence game 👇 LLMs are known to have mode collapse, i.e. if you have multiple parallel runs on the same prompt, you get very similar responses. Is there a way to increase diversity? Yes, it turns out that if you do such multiple runs **sequentially**, instead of in parallel. (Plus for each sequential run, append results from previous runs and instruct the LLM to create something different from before.) I did this quick experiment to explore parallel v/s sequential runs for diversity. Given the same prompt to build a tower defence game, sequential runs had a much higher diversity and innovation v/s parallel runs. Check out this (AI-generated) video for gameplay clips. The sequential cohort created interesting new dynamics and visual style, while parallel ones all collapsed to similar colors and orbital style dynamics.
1:11

I asked 6 agents in two conditions (parallel v/s sequential) to create the same tower defence game 👇 LLMs are known to have mode collapse, i.e. if you have multiple parallel runs on the same prompt, you get very similar responses. Is there a way to increase diversity? Yes, it turns out that if you do such multiple runs **sequentially**, instead of in parallel. (Plus for each sequential run, append results from previous runs and instruct the LLM to create something different from before.) I did this quick experiment to explore parallel v/s sequential runs for diversity. Given the same prompt to build a tower defence game, sequential runs had a much higher diversity and innovation v/s parallel runs. Check out this (AI-generated) video for gameplay clips. The sequential cohort created interesting new dynamics and visual style, while parallel ones all collapsed to similar colors and orbital style dynamics.

Paras Chopra

11,592 görüntüleme • 2 ay önce

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 · M5 Max · greedy exact match ▸ 85 tok/s, 3.3× at 1024 tokens (runtime) ▸ ~70 tok/s, 2.6× in the video (terminal I/O overhead) ▸ 80 tok/s, 3.1× at 2048 tokens (runtime) Currently working on: → Long context (speedup degrades past 4K tokens, KV cache growth) → Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.
0:39

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 · M5 Max · greedy exact match ▸ 85 tok/s, 3.3× at 1024 tokens (runtime) ▸ ~70 tok/s, 2.6× in the video (terminal I/O overhead) ▸ 80 tok/s, 3.1× at 2048 tokens (runtime) Currently working on: → Long context (speedup degrades past 4K tokens, KV cache growth) → Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.

bstn 👁️

37,049 görüntüleme • 6 ay önce

Unbelievable shot from Musetti in Monte-Carlo
0:29

Unbelievable shot from Musetti in Monte-Carlo

José Morgado

46,022 görüntüleme • 1 yıl önce

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?
0:26

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Avi Chawla

158,352 görüntüleme • 5 ay önce

With love, from Monte Carlo 🇲🇨🇫🇷
1:05

With love, from Monte Carlo 🇲🇨🇫🇷

Thomas Casanova

19,905 görüntüleme • 2 yıl önce

Is thumbnail A/B testing sequential or in parallel? Here’s the answer!
0:23

Is thumbnail A/B testing sequential or in parallel? Here’s the answer!

YouTube Creator Insider

27,161 görüntüleme • 2 yıl önce

Getting warm in Monte Carlo with last year’s finalists 💪 🔥 Carlos Alcaraz Rolex Monte-Carlo Masters
0:42

Getting warm in Monte Carlo with last year’s finalists 💪 🔥 Carlos Alcaraz Rolex Monte-Carlo Masters

Tennis TV

27,372 görüntüleme • 6 ay önce

Lorenzo Musetti. In the fading evening light of Monte-Carlo. Stunning.
0:30

Lorenzo Musetti. In the fading evening light of Monte-Carlo. Stunning.

Rolex Monte-Carlo Masters

32,852 görüntüleme • 1 yıl önce

Say goodbye to sequential bottlenecks. In the stable release of Android Studio Quail 2, you can run multiple agent tasks in parallel with Gemini! 💬🔄 Watch how you can initiate and manage multiple agent tasks simultaneously across your chats. Read more in the launch blog post:
0:53

Say goodbye to sequential bottlenecks. In the stable release of Android Studio Quail 2, you can run multiple agent tasks in parallel with Gemini! 💬🔄 Watch how you can initiate and manage multiple agent tasks simultaneously across your chats. Read more in the launch blog post:

Google for Developers

14,716 görüntüleme • 3 ay önce

Can you spot the difference? 👀 Speculative decoding helps accelerate generative AI at the edge, especially on NVIDIA Jetson. Qwen 3.8 27B went from 13 to 35 tokens/sec, while Nemotron 3.5 Lightning went from 65 to 115. See how it works:
0:27

Can you spot the difference? 👀 Speculative decoding helps accelerate generative AI at the edge, especially on NVIDIA Jetson. Qwen 3.8 27B went from 13 to 35 tokens/sec, while Nemotron 3.5 Lightning went from 65 to 115. See how it works:

NVIDIA Robotics

44,587 görüntüleme • 2 ay önce

Learning from the best 🤩 Rolex Monte-Carlo Masters #RolexMonteCarloMasters
0:40

Learning from the best 🤩 Rolex Monte-Carlo Masters #RolexMonteCarloMasters

Tennis TV

13,103 görüntüleme • 6 ay önce

Soak it in Lorenzo 🥹🇮🇹 Lorenzo Musetti is just the THIRD Italian Monte-Carlo finalist in the Open Era Rolex Monte-Carlo Masters | #RolexMonteCarloMasters
0:39

Soak it in Lorenzo 🥹🇮🇹 Lorenzo Musetti is just the THIRD Italian Monte-Carlo finalist in the Open Era Rolex Monte-Carlo Masters | #RolexMonteCarloMasters

ATP Tour

34,097 görüntüleme • 1 yıl önce

Monaco: A Muslim prays in the middle of the Café de Paris in Monte Carlo.
0:29

Monaco: A Muslim prays in the middle of the Café de Paris in Monte Carlo.

Yossi BenYakar

15,805 görüntüleme • 11 ay önce

Carlos Alcaraz vs Jannik Sinner, Monte-Carlo 2026 Final. The first chapter of tennis' greatest rivalry in 2026. Sinner and Alcaraz met for the first time that season in a blockbuster Monte-Carlo final.
6:09

Carlos Alcaraz vs Jannik Sinner, Monte-Carlo 2026 Final. The first chapter of tennis' greatest rivalry in 2026. Sinner and Alcaraz met for the first time that season in a blockbuster Monte-Carlo final.

Tennis Performances

16,055 görüntüleme • 3 ay önce

33 years ago today, Michael Jackson received multiple awards At the 1993 World Music Awards in Monte Carlo, Monaco, presented by Michael Douglas, Prince Albert and Princess Stephanie of Monaco.
6:35

33 years ago today, Michael Jackson received multiple awards At the 1993 World Music Awards in Monte Carlo, Monaco, presented by Michael Douglas, Prince Albert and Princess Stephanie of Monaco.

Michael Jackson Crave

91,626 görüntüleme • 5 ay önce

Another cool stuff from Alibaba! MiniMax-H3 Acc LoRAs. Parallel Decoding Distillation for efficient video gen in a few steps. FL2VA + Ref2VA
0:41

Another cool stuff from Alibaba! MiniMax-H3 Acc LoRAs. Parallel Decoding Distillation for efficient video gen in a few steps. FL2VA + Ref2VA

Wildminder

36,958 görüntüleme • 1 ay önce