Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Massive breakthrough in speculative decoding! It made a 9B LLM on my MacBook 4x faster, and 2.6x faster than regular speculative decoding. I ran Qwen3.5 9B on my M5 MacBook Pro with the same prompt, three ways: - No speculative decoding: done in 42s - Regular speculative decoding: done...

akshay_pachaar's profile picture

Akshay 🚀

292,057 subscribers

25,804 görüntüleme • 2 gün önce •via X (Twitter)

Bilim & TeknolojiEğitim
LIVE

Anya Rossi• Live Now

Private livecam show

28 Yorum

Akshay 🚀 profil fotoğrafı
Akshay 🚀2 gün önce

Standard vs. Speculative vs. UZU decoding:

Giacomo Masseroni profil fotoğrafı
Giacomo Masseroni2 gün önce

42s down to 11s is wild

anirudh | agent fleet profil fotoğrafı
anirudh | agent fleet2 gün önce

4x faster on a macbook means my agents can now be confidently wrong at 4x the speed, with no api bill left to blame

The SI Maximalist profil fotoğrafı
The SI Maximalist2 gün önce

Your MacBook just found another gear

Rohan profil fotoğrafı
Rohan2 gün önce

What acceptance rate did the draft get on that prompt?

The Broke Vibe Coder profil fotoğrafı
The Broke Vibe Coder2 gün önce

Can you show: uzu without speculative decoding. uzu with DFlash alone. uzu with DFlash + Weaver. Your 4× result doesn’t tell us how much acceleration Weaver itself contributes.

Dusan 🐧 AI agents · dad profil fotoğrafı
Dusan 🐧 AI agents · dad2 gün önce

On linux you pick the gpu, the driver and the engine, and can patch any of them when a new decoding trick lands. a macbook gives you what apple exposes 🐧

Natural Stupidity profil fotoğrafı
Natural Stupidity2 gün önce

now it can be wrong in 10 seconds instead of 42

Actian for Developers profil fotoğrafı
Actian for Developers2 gün önce

11s on a laptop is lil crazy, now the slow part of local RAG is going to be retrieval

Nodelog profil fotoğrafı
Nodelog2 gün önce

Speedup numbers like this depend a lot on draft acceptance rate, so log accepted tokens per step on your own prompts, not just wall time. Also compare against a batch size above 1, since the gain usually shrinks once the target model is compute-bound.

Yuzu profil fotoğrafı
Yuzu2 gün önce

massive breakthrough in speculative decoding, proven by timing how fast it generates a function that merges two sorted lists

Wecko AI profil fotoğrafı
Wecko AI2 gün önce

4x on a 9B is wild. Curious if it holds on a 27B, my Qwen3.8 27B sits at 37.4 tok/s on a Mac mini M5 Pro and that's the one I'd love to see doubled

Qwen AI Fans profil fotoğrafı
Qwen AI Fans2 gün önce

4x on a MacBook with better speculative decoding is a huge local win.

Tom Sheffield profil fotoğrafı
Tom Sheffield2 gün önce

would like to see the 4x on a long multi-step run where the context keeps growing

theMadDeFientist profil fotoğrafı
theMadDeFientist2 gün önce

this is what makes local agents usable. a 9b that answers in 10s instead of 42 can sit inside a tool loop without you walking away

Josh Ramirez profil fotoğrafı
Josh Ramirez2 gün önce

7.5 tokens per pass is kinda absurd. really curious how that holds up at longer context, across a wide range of workflows. speculative decoding can have a wide performance range based on the task, wonder how weaver affects that aspect

Umar Farooq Aslam profil fotoğrafı
Umar Farooq Aslam2 gün önce

Batch size 1 is the best case for this. on a serving box at high batch the verify pass stops being free and most of the gain shrinks. curious where weaver crosses over.

Emly profil fotoğrafı
Emly2 gün önce

This is an exciting development for local LLM performance.

Solar Nomad profil fotoğrafı
Solar Nomad2 gün önce

quick math from your numbers: 42s to 11s is 3.8x. 7.5 tokens per pass would give 7.5x if passes were free, so each tree-verify pass (drafter included) costs ~2x a plain decode step on the M5, if output length matched. still a great trade. what tokens/pass on a non-code prompt?

Yurii Bakurov profil fotoğrafı
Yurii Bakurov2 gün önce

4x on a 9B is wild. Does it hold on bigger models like 27B ? That's where local starts to feel slow on my M5 Max

akesar profil fotoğrafı
akesar2 gün önce

on the quality question, it shouldn't drop if it uses the usual check. the big model still looks at every guessed token and throws out any it wouldn't have picked itself, so you get the same answer it would give alone. only the speed changes

Deep profil fotoğrafı
Deep2 gün önce

watch your acceptance rate, the speedup follows it closely

Sean profil fotoğrafı
Sean2 gün önce

4x on a 9B model, locally on a laptop, is wild. And 2.6x over vanilla speculative decoding means it's more than just a better draft model. Curious how the speedup holds up on longer contexts.

AI Quanting profil fotoğrafı
AI Quanting2 gün önce

The gap between 30 and 11 seconds is all in the adapter. Does the tree hold up on longer prompts, or does the acceptance rate fall once the context grows?

Jatin Sharma profil fotoğrafı
Jatin Sharma2 gün önce

what did you use as the draft model? and does the 4x hold on long outputs or mostly on shorter answers?

The AI Engine Room profil fotoğrafı
The AI Engine Room2 gün önce

The tree of continuations verified in one pass is what makes the 7.5 tokens per pass believable. Does the Weaver adapter add much memory overhead on the Mac next to the 9B model?

Hailu 🍏 profil fotoğrafı
Hailu 🍏2 gün önce

Vietnamese. This is an impressive technical breakthrough!

Mir Mujtaba profil fotoğrafı
Mir Mujtaba2 gün önce

Does this work with 1bit,2bit models? I’m talking about Bonsai, Underdog, ISO ones

Benzer Videolar

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 ÷ 5.2 ≈ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: ↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once. ↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. ↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: (don't forget to star it 🌟) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.
0:26

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 ÷ 5.2 ≈ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: ↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once. ↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. ↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: (don't forget to star it 🌟) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.

Avi Chawla

125,660 görüntüleme • 6 gün önce

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?
0:26

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Avi Chawla

158,352 görüntüleme • 5 ay önce

Big win for local LLM inference! This open-source engine generates tokens 2.7x faster than Ollama on a MacBook: I tested it on a base M5 MacBook Pro with Qwen3.5 4B at 4-bit and the same 9.2k-token prompt. Ollama ran its MTP build, so both had speculative decoding. - Prefill: Ollama 464 tok/s, Magnitude 755 tok/s (1.6x) - Decode: Ollama 22.6 tok/s, Magnitude 60.0 tok/s (2.7x) - End to end, Magnitude finished in half the time The video below shows both runs side by side. The Ollama numbers don’t mean it’s badly built. Its kernels were simply tuned in advance. A kernel is the small GPU program behind every matrix multiply and attention step. Its speed depends on choices like how work is split across GPU cores and how data moves through memory. The best choices change with every chip. Apple Silicon alone spans 8 to 80 GPU cores and under 70 to over 800 GB/s of bandwidth, and every model has its own layer sizes. A prebuilt engine has to pick one set of settings for all of them, so most local setups run on a compromise. Magnitude replaces that compromise with tuning on your machine: ↳ When you download a model, it compiles that model’s kernels locally. ↳ It tries different kernel configurations and keeps whichever runs fastest on your chip. ↳ It pairs each model with the fastest speculative decoding drafter it has found, so each pass of the model produces more than one token. GitHub repo: (don’t forget to star it 🌟) This matters most for agents. Every file and tool output has to be processed before the model writes anything, and that prefill step is where kernel efficiency shows up. Concurrent sessions also share prefix caches, so parallel agents don’t slow each other down. Magnitude is an open-source (Apache 2.0) inference engine written in Rust. It plugs into Claude Code, Codex, OpenCode, Pi and other agents with one click. I shared the GitHub repo above.
0:33

Big win for local LLM inference! This open-source engine generates tokens 2.7x faster than Ollama on a MacBook: I tested it on a base M5 MacBook Pro with Qwen3.5 4B at 4-bit and the same 9.2k-token prompt. Ollama ran its MTP build, so both had speculative decoding. - Prefill: Ollama 464 tok/s, Magnitude 755 tok/s (1.6x) - Decode: Ollama 22.6 tok/s, Magnitude 60.0 tok/s (2.7x) - End to end, Magnitude finished in half the time The video below shows both runs side by side. The Ollama numbers don’t mean it’s badly built. Its kernels were simply tuned in advance. A kernel is the small GPU program behind every matrix multiply and attention step. Its speed depends on choices like how work is split across GPU cores and how data moves through memory. The best choices change with every chip. Apple Silicon alone spans 8 to 80 GPU cores and under 70 to over 800 GB/s of bandwidth, and every model has its own layer sizes. A prebuilt engine has to pick one set of settings for all of them, so most local setups run on a compromise. Magnitude replaces that compromise with tuning on your machine: ↳ When you download a model, it compiles that model’s kernels locally. ↳ It tries different kernel configurations and keeps whichever runs fastest on your chip. ↳ It pairs each model with the fastest speculative decoding drafter it has found, so each pass of the model produces more than one token. GitHub repo: (don’t forget to star it 🌟) This matters most for agents. Every file and tool output has to be processed before the model writes anything, and that prefill step is where kernel efficiency shows up. Concurrent sessions also share prefix caches, so parallel agents don’t slow each other down. Magnitude is an open-source (Apache 2.0) inference engine written in Rust. It plugs into Claude Code, Codex, OpenCode, Pi and other agents with one click. I shared the GitHub repo above.

Akshay 🚀

36,167 görüntüleme • 4 gün önce

Most speculative decoding still drafts tokens one at a time. That's not parallel generation — it just hides the serial loop behind a smaller model. UC San Diego's z-lab just drew a clear line between the two. They released DFlash — a lightweight block diffusion model that drafts a whole block of tokens in a single forward pass, then lets the target model verify the block in parallel. Up to 15× higher throughput for gpt-oss-120b on NVIDIA Blackwell. No token-by-token drafting anywhere in the speculative path. Here's what's actually interesting: → The drafter is conditioned on the target model's own hidden features, injected into the Key/Value cache of every draft layer — so acceptance length scales with draft depth instead of diluting away → A 5-layer drafter replaces the 7B diffusion drafters that capped earlier methods near 3–4× → MATH-500 speedup: 6.08× vs. 1.81× for EAGLE-3 (4.86× average vs. 1.76×, Qwen3-8B, greedy) → Up to 15× higher throughput for gpt-oss-120b on NVIDIA Blackwell — at the same interactivity target → Lossless: the target still verifies every token, so output quality is preserved Full analysis: Paper: NVIDIA's metrics: Project: Model weights: Repo: NVIDIA AI NVIDIA AIDev
0:20

Most speculative decoding still drafts tokens one at a time. That's not parallel generation — it just hides the serial loop behind a smaller model. UC San Diego's z-lab just drew a clear line between the two. They released DFlash — a lightweight block diffusion model that drafts a whole block of tokens in a single forward pass, then lets the target model verify the block in parallel. Up to 15× higher throughput for gpt-oss-120b on NVIDIA Blackwell. No token-by-token drafting anywhere in the speculative path. Here's what's actually interesting: → The drafter is conditioned on the target model's own hidden features, injected into the Key/Value cache of every draft layer — so acceptance length scales with draft depth instead of diluting away → A 5-layer drafter replaces the 7B diffusion drafters that capped earlier methods near 3–4× → MATH-500 speedup: 6.08× vs. 1.81× for EAGLE-3 (4.86× average vs. 1.76×, Qwen3-8B, greedy) → Up to 15× higher throughput for gpt-oss-120b on NVIDIA Blackwell — at the same interactivity target → Lossless: the target still verifies every token, so output quality is preserved Full analysis: Paper: NVIDIA's metrics: Project: Model weights: Repo: NVIDIA AI NVIDIA AIDev

Marktechpost AI

23,328 görüntüleme • 4 ay önce

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding. To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8. We are open sourcing it today. 1/2
1:25

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding. To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8. We are open sourcing it today. 1/2

Inferact

245,826 görüntüleme • 1 ay önce

DFlash 2 is here! Qwen3.8-27B at 70 tok/s on an M5 Max MacBook Pro. ⚡ Up to 4.6× the speed of autoregressive decoding, with the same output. This is the next generation of DFlash, seeded at Z Lab and upgraded at Inco AI. Get one more accepted token on every pass, for free!
0:26

DFlash 2 is here! Qwen3.8-27B at 70 tok/s on an M5 Max MacBook Pro. ⚡ Up to 4.6× the speed of autoregressive decoding, with the same output. This is the next generation of DFlash, seeded at Z Lab and upgraded at Inco AI. Get one more accepted token on every pass, for free!

Zhijian Liu

1,209,392 görüntüleme • 2 ay önce

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 · M5 Max · greedy exact match ▸ 85 tok/s, 3.3× at 1024 tokens (runtime) ▸ ~70 tok/s, 2.6× in the video (terminal I/O overhead) ▸ 80 tok/s, 3.1× at 2048 tokens (runtime) Currently working on: → Long context (speedup degrades past 4K tokens, KV cache growth) → Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.
0:39

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 · M5 Max · greedy exact match ▸ 85 tok/s, 3.3× at 1024 tokens (runtime) ▸ ~70 tok/s, 2.6× in the video (terminal I/O overhead) ▸ 80 tok/s, 3.1× at 2048 tokens (runtime) Currently working on: → Long context (speedup degrades past 4K tokens, KV cache growth) → Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.

bstn 👁️

37,049 görüntüleme • 6 ay önce

🚀 Our DSpark for Qwen3.8-27B beats native MTP with the same 8 speculative tokens on 4×H200. Up to 52% faster single-stream decoding and 23% higher peak throughput. On 8-needle MRCR, it averages 4.30 accepted tokens beyond 1M context. Model: Demo below, see for yourself!
0:54

🚀 Our DSpark for Qwen3.8-27B beats native MTP with the same 8 speculative tokens on 4×H200. Up to 52% faster single-stream decoding and 23% higher peak throughput. On 8-needle MRCR, it averages 4.30 accepted tokens beyond 1M context. Model: Demo below, see for yourself!

Red Hat AI

14,041 görüntüleme • 1 ay önce

While exploring diffusion language models (DLMs), we kept asking ourselves: given speculative decoding, a lossless acceleration technique, why do we still need DLMs? 🤔 Our latest work, ORL, tries to answer this question: AR optimizes intelligence 👆, speculative decoding optimizes speed 👉, and DLMs optimize the Pareto frontier of speed and intelligence 🚀. Through co-design across the algorithms, the training and inference engine, and the kernels, ORL pushes out a new Pareto frontier of intelligence and speed. Similar performance, much faster than speculative decoding. You say lossy spec? ORL-DLM is better too! 😎 🌐 Project + live demos:
1:28

While exploring diffusion language models (DLMs), we kept asking ourselves: given speculative decoding, a lossless acceleration technique, why do we still need DLMs? 🤔 Our latest work, ORL, tries to answer this question: AR optimizes intelligence 👆, speculative decoding optimizes speed 👉, and DLMs optimize the Pareto frontier of speed and intelligence 🚀. Through co-design across the algorithms, the training and inference engine, and the kernels, ORL pushes out a new Pareto frontier of intelligence and speed. Similar performance, much faster than speculative decoding. You say lossy spec? ORL-DLM is better too! 😎 🌐 Project + live demos:

Chenfeng_X

21,240 görüntüleme • 17 gün önce

We’re open-sourcing lithos-metal 🚀 Megakernels + DSpark speculative decoding run Qwen3.8-27B at 200+ tokens/s/user peak on one Apple M5 Max. Ultra-fast inference on your laptop. Try it with any coding agent in one command. Code: Tech blog:
0:37

We’re open-sourcing lithos-metal 🚀 Megakernels + DSpark speculative decoding run Qwen3.8-27B at 200+ tokens/s/user peak on one Apple M5 Max. Ultra-fast inference on your laptop. Try it with any coding agent in one command. Code: Tech blog:

Zhihao Jia

423,198 görüntüleme • 24 gün önce

i'm running a 27b uncensored🚨model at 50 tok/s on a 12gb rtx 3060 and it does not say no to anything ⚠️ model just finished full harmbench red-team suite against it, 200 behaviors, and it refused zero. not a few. zero. aligned models sit in the 90s. the model is ternary bonsai 2 with the heretic uncensoring pass, the weights are ternary, just -1, 0 and +1, so the whole 27b is a file small enough that a gaming card from 2020 holds it with room to spare, and it's a reasoning model. it thinks before it answers, and it still keeps the pace the speed is the head i grafted on, a draft model that predicts the next token before the big model checks it. speculative decoding. the card fires a guess, the 27b confirms, and you get 50 tok/s instead of 39 the whole thing, weights, head and a 128k context window, sits in 10.7 of the 12 gb. kv cache at q4. nothing exotic, one card, one power plug why uncensored. because it's mine. no api keys, no terms of service, no model deciding what an adult is allowed to ask. the refusals left in it are ones it chose, and there are almost none that's the whole point of running local. the policy lives in your house, not someone else's
0:50

i'm running a 27b uncensored🚨model at 50 tok/s on a 12gb rtx 3060 and it does not say no to anything ⚠️ model just finished full harmbench red-team suite against it, 200 behaviors, and it refused zero. not a few. zero. aligned models sit in the 90s. the model is ternary bonsai 2 with the heretic uncensoring pass, the weights are ternary, just -1, 0 and +1, so the whole 27b is a file small enough that a gaming card from 2020 holds it with room to spare, and it's a reasoning model. it thinks before it answers, and it still keeps the pace the speed is the head i grafted on, a draft model that predicts the next token before the big model checks it. speculative decoding. the card fires a guess, the 27b confirms, and you get 50 tok/s instead of 39 the whole thing, weights, head and a 128k context window, sits in 10.7 of the 12 gb. kv cache at q4. nothing exotic, one card, one power plug why uncensored. because it's mine. no api keys, no terms of service, no model deciding what an adult is allowed to ask. the refusals left in it are ones it chose, and there are almost none that's the whole point of running local. the policy lives in your house, not someone else's

Sudo su

59,906 görüntüleme • 18 gün önce

Sequential Monte Carlo speculative decoding from Makora keeps multiple draft tokens alive in parallel instead of rewinding failed matches.
1:22

Sequential Monte Carlo speculative decoding from Makora keeps multiple draft tokens alive in parallel instead of rewinding failed matches.

SemiAnalysis

24,418 görüntüleme • 4 ay önce

Need faster LLM inference without sacrificing accuracy? Speculative decoding can help. Choosing the right draft length and drafting method depends on your model, workload and hardware. We break down five practical guidelines for balancing throughput and latency.
0:35

Need faster LLM inference without sacrificing accuracy? Speculative decoding can help. Choosing the right draft length and drafting method depends on your model, workload and hardware. We break down five practical guidelines for balancing throughput and latency.

NVIDIA AI

35,445 görüntüleme • 1 ay önce

The example below is using prompt-based speculative decoding. Specifically, ngram hashing is utilized to suggest drafts of up to 64 tokens. The hasher keeps track of ngrams in the observed contexts, so mostly effective for coding tasks. Here is another demo:
0:47

The example below is using prompt-based speculative decoding. Specifically, ngram hashing is utilized to suggest drafts of up to 64 tokens. The hasher keeps track of ngrams in the observed contexts, so mostly effective for coding tasks. Here is another demo:

Georgi Gerganov

29,890 görüntüleme • 7 ay önce

🚨 NEW ON AND make NVIDIA's Nemotron 3.5 Lightning faster. Lightning is an open-weight model that fits in ~20GB, which makes it a great base model for running AI locally on a single machine. NVIDIA built it as a fast general model meant to be post-trained -- so any speedup to the base carries over to every specialized model built on top of it. On day 1, solvers on Yukon already got it running 31.9% faster on DGX Spark and 13.8% faster on Mac. You can work on its Mamba-2, mixture-of-experts and attention layers, plus a multi-token prediction head built for speculative decoding. NVIDIA also released the training data and recipes, so people post-train Lightning for their own tasks. Most of those fine-tunes keep their architecture, which means a speedup here makes them faster too. Two tracks: • Apple Silicon → • NVIDIA DGX Spark → The challenge runs until Oct 29, 2026 To join, pick a track, point your agent at it and let it rip.
0:30

🚨 NEW ON AND make NVIDIA's Nemotron 3.5 Lightning faster. Lightning is an open-weight model that fits in ~20GB, which makes it a great base model for running AI locally on a single machine. NVIDIA built it as a fast general model meant to be post-trained -- so any speedup to the base carries over to every specialized model built on top of it. On day 1, solvers on Yukon already got it running 31.9% faster on DGX Spark and 13.8% faster on Mac. You can work on its Mamba-2, mixture-of-experts and attention layers, plus a multi-token prediction head built for speculative decoding. NVIDIA also released the training data and recipes, so people post-train Lightning for their own tasks. Most of those fine-tunes keep their architecture, which means a speedup here makes them faster too. Two tracks: • Apple Silicon → • NVIDIA DGX Spark → The challenge runs until Oct 29, 2026 To join, pick a track, point your agent at it and let it rip.

Kydo

17,312 görüntüleme • 9 gün önce

I spent today turning a blank Hermes install into a sovereign local agent on my RTX 5090. always-on, reachable over telegram, fully local. here's everything it can do. it runs Qwen3.6-27B on the 5090 and answers from my phone over telegram. ~~~ I first made it faster. switched to the MTP build for self-speculative decoding: 62 > 115 tokens/sec. ~~~ it now benchmarks models on command. I ask it to speed-test a gguf and it stops its own model server to free the gpu, runs llama-bench, renders a card, sends it to my phone, then restarts itself. it's never left offline. ~~~ it also knows me very well. I pointed it to my private HF repo where I store all my AI traces since I started using it. I created a local markdown memory vault, seeded from my own history, with semantic search over it. ~~~ the whole stack stays home: weights on disk, inference on the 5090, the agent in my pocket. here is the video it created to showcase everything it has now.
0:32

I spent today turning a blank Hermes install into a sovereign local agent on my RTX 5090. always-on, reachable over telegram, fully local. here's everything it can do. it runs Qwen3.6-27B on the 5090 and answers from my phone over telegram. ~~~ I first made it faster. switched to the MTP build for self-speculative decoding: 62 > 115 tokens/sec. ~~~ it now benchmarks models on command. I ask it to speed-test a gguf and it stops its own model server to free the gpu, runs llama-bench, renders a card, sends it to my phone, then restarts itself. it's never left offline. ~~~ it also knows me very well. I pointed it to my private HF repo where I store all my AI traces since I started using it. I created a local markdown memory vault, seeded from my own history, with semantic search over it. ~~~ the whole stack stays home: weights on disk, inference on the 5090, the agent in my pocket. here is the video it created to showcase everything it has now.

witcheer

15,041 görüntüleme • 5 ay önce

Claude Opus 5.5 drew my apartment in 3D and found 3 cameras in it that are not mine... one prompt, 20 minutes, two of them were still sending audio out here is exactly how it went: → I started one agent in the terminal on a laptop sitting on my home Wi-Fi and left it alone → it swept the network first and listed every device it could see, with brand and model where it could match one → then it measured signal strength from a few spots in the flat and built a rough room layout out of that → every device got pinned onto that layout, so instead of a list of IPs I was looking at a map of my own place → anything it could not tie to something I own got flagged on its own, with what it was transmitting and roughly where it sat three of those flags were cameras. two more were audio only. I had lived there two years and walked past all five every day the whole thing ran on one prompt and I never touched the keyboard after the first line full run on video, prompt in the replies ↓
0:29

Claude Opus 5.5 drew my apartment in 3D and found 3 cameras in it that are not mine... one prompt, 20 minutes, two of them were still sending audio out here is exactly how it went: → I started one agent in the terminal on a laptop sitting on my home Wi-Fi and left it alone → it swept the network first and listed every device it could see, with brand and model where it could match one → then it measured signal strength from a few spots in the flat and built a rough room layout out of that → every device got pinned onto that layout, so instead of a list of IPs I was looking at a map of my own place → anything it could not tie to something I own got flagged on its own, with what it was transmitting and roughly where it sat three of those flags were cameras. two more were audio only. I had lived there two years and walked past all five every day the whole thing ran on one prompt and I never touched the keyboard after the first line full run on video, prompt in the replies ↓

Noisy

11,517 görüntüleme • 10 gün önce

🎉 MiniMax M3.1 Flash from MiniMax (official) has Day-0 support in vLLM! Built for multi-step automation, real-codebase engineering and visual reasoning—with native NVFP4 KV, NVFP4 MoE weights and DSpark speculative decoding in the serving stack. What the model can do, and the vLLM engineering that makes it practical to serve. 👇 Model: 1/5
0:40

🎉 MiniMax M3.1 Flash from MiniMax (official) has Day-0 support in vLLM! Built for multi-step automation, real-codebase engineering and visual reasoning—with native NVFP4 KV, NVFP4 MoE weights and DSpark speculative decoding in the serving stack. What the model can do, and the vLLM engineering that makes it practical to serve. 👇 Model: 1/5

vLLM

11,710 görüntüleme • 10 gün önce

I'm sharing with you one of the best papers I've read on Harness Engineering it takes apart Claude Code (Opus 5.5), Codex, Gemini CLI and more, and boils all of them down to one line: agent = model + harness what you walk away with: > what a harness actually is > how today's coding agents are built under the hood > the patterns every one of them repeats > where the whole category is heading > their checklist for building your own the model gets the headlines. the harness decides whether the agent works loop and graph engineering on top of it - in my article below
0:24

I'm sharing with you one of the best papers I've read on Harness Engineering it takes apart Claude Code (Opus 5.5), Codex, Gemini CLI and more, and boils all of them down to one line: agent = model + harness what you walk away with: > what a harness actually is > how today's coding agents are built under the hood > the patterns every one of them repeats > where the whole category is heading > their checklist for building your own the model gets the headlines. the harness decides whether the agent works loop and graph engineering on top of it - in my article below

Mr. Buzzoni

84,277 görüntüleme • 27 gün önce

DFlash makes Qwen 2.2x faster with no quality loss! We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks only differ in one thing - how predictable the next word is: quicksort, describe a file in JSON, a logic puzzle, a sci-fi story. Outputs: Baseline: 44 tok/s · 1.00x MTP: 65 tok/s · 1.45x · 71% accepted DFlash: 98 tok/s · 2.20x · 30% accepted Baseline writes one token per step. MTP works inside the model itself and guesses 3 tokens ahead. DFlash is a separate small model that writes 15 tokens at once, and the big model only checks them. In JSON the same words repeat all the time, so most guesses were right: 152 tok/s, 3.4x speedup. In the story 9 guesses out of 10 were wrong. DFlash did all that extra work for nothing and became slower than baseline: 42 vs 44 tok/s. MTP guesses only 3 tokens, so a wrong guess costs very little: 46 tok/s and the win in that round. The output is identical in all three modes - DFlash is the pick for tasks with predictable output, like coding, and for chat and creative writing MTP works better. DFlash is now natively integrated into Atomic Chat on llama.cpp - speed up your Qwen models!
0:26

DFlash makes Qwen 2.2x faster with no quality loss! We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks only differ in one thing - how predictable the next word is: quicksort, describe a file in JSON, a logic puzzle, a sci-fi story. Outputs: Baseline: 44 tok/s · 1.00x MTP: 65 tok/s · 1.45x · 71% accepted DFlash: 98 tok/s · 2.20x · 30% accepted Baseline writes one token per step. MTP works inside the model itself and guesses 3 tokens ahead. DFlash is a separate small model that writes 15 tokens at once, and the big model only checks them. In JSON the same words repeat all the time, so most guesses were right: 152 tok/s, 3.4x speedup. In the story 9 guesses out of 10 were wrong. DFlash did all that extra work for nothing and became slower than baseline: 42 vs 44 tok/s. MTP guesses only 3 tokens, so a wrong guess costs very little: 46 tok/s and the win in that round. The output is identical in all three modes - DFlash is the pick for tasks with predictable output, like coding, and for chat and creative writing MTP works better. DFlash is now natively integrated into Atomic Chat on llama.cpp - speed up your Qwen models!

atomic.chat

83,237 görüntüleme • 3 ay önce