Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Massive breakthrough in speculative decoding! It made a 9B LLM on my MacBook 4x faster, and 2.6x faster than regular speculative decoding. I ran Qwen3.5 9B on my M5 MacBook Pro with the same prompt, three ways: - No speculative decoding: done in 42s - Regular speculative decoding: done...

akshay_pachaar's profile picture

Akshay ๐Ÿš€

292,057 subscribers

25,804 Aufrufe โ€ข vor 2 Tagen โ€ขvia X (Twitter)

Wissenschaft & TechnologieBildung
LIVE

Anya Rossiโ€ข Live Now

Private livecam show

28 Kommentare

Profilbild von Akshay ๐Ÿš€
Akshay ๐Ÿš€vor 2 Tagen

Standard vs. Speculative vs. UZU decoding:

Profilbild von Giacomo Masseroni
Giacomo Masseronivor 2 Tagen

42s down to 11s is wild

Profilbild von anirudh | agent fleet
anirudh | agent fleetvor 2 Tagen

4x faster on a macbook means my agents can now be confidently wrong at 4x the speed, with no api bill left to blame

Profilbild von The SI Maximalist
The SI Maximalistvor 2 Tagen

Your MacBook just found another gear

Profilbild von Rohan
Rohanvor 2 Tagen

What acceptance rate did the draft get on that prompt?

Profilbild von The Broke Vibe Coder
The Broke Vibe Codervor 2 Tagen

Can you show: uzu without speculative decoding. uzu with DFlash alone. uzu with DFlash + Weaver. Your 4ร— result doesnโ€™t tell us how much acceleration Weaver itself contributes.

Profilbild von Dusan ๐Ÿง AI agents ยท dad
Dusan ๐Ÿง AI agents ยท dadvor 2 Tagen

On linux you pick the gpu, the driver and the engine, and can patch any of them when a new decoding trick lands. a macbook gives you what apple exposes ๐Ÿง

Profilbild von Natural Stupidity
Natural Stupidityvor 2 Tagen

now it can be wrong in 10 seconds instead of 42

Profilbild von Actian for Developers
Actian for Developersvor 2 Tagen

11s on a laptop is lil crazy, now the slow part of local RAG is going to be retrieval

Profilbild von Nodelog
Nodelogvor 2 Tagen

Speedup numbers like this depend a lot on draft acceptance rate, so log accepted tokens per step on your own prompts, not just wall time. Also compare against a batch size above 1, since the gain usually shrinks once the target model is compute-bound.

Profilbild von Yuzu
Yuzuvor 2 Tagen

massive breakthrough in speculative decoding, proven by timing how fast it generates a function that merges two sorted lists

Profilbild von Wecko AI
Wecko AIvor 2 Tagen

4x on a 9B is wild. Curious if it holds on a 27B, my Qwen3.8 27B sits at 37.4 tok/s on a Mac mini M5 Pro and that's the one I'd love to see doubled

Profilbild von Qwen AI Fans
Qwen AI Fansvor 2 Tagen

4x on a MacBook with better speculative decoding is a huge local win.

Profilbild von Tom Sheffield
Tom Sheffieldvor 2 Tagen

would like to see the 4x on a long multi-step run where the context keeps growing

Profilbild von theMadDeFientist
theMadDeFientistvor 2 Tagen

this is what makes local agents usable. a 9b that answers in 10s instead of 42 can sit inside a tool loop without you walking away

Profilbild von Josh Ramirez
Josh Ramirezvor 2 Tagen

7.5 tokens per pass is kinda absurd. really curious how that holds up at longer context, across a wide range of workflows. speculative decoding can have a wide performance range based on the task, wonder how weaver affects that aspect

Profilbild von Umar Farooq Aslam
Umar Farooq Aslamvor 2 Tagen

Batch size 1 is the best case for this. on a serving box at high batch the verify pass stops being free and most of the gain shrinks. curious where weaver crosses over.

Profilbild von Emly
Emlyvor 2 Tagen

This is an exciting development for local LLM performance.

Profilbild von Solar Nomad
Solar Nomadvor 2 Tagen

quick math from your numbers: 42s to 11s is 3.8x. 7.5 tokens per pass would give 7.5x if passes were free, so each tree-verify pass (drafter included) costs ~2x a plain decode step on the M5, if output length matched. still a great trade. what tokens/pass on a non-code prompt?

Profilbild von Yurii Bakurov
Yurii Bakurovvor 2 Tagen

4x on a 9B is wild. Does it hold on bigger models like 27B ? That's where local starts to feel slow on my M5 Max

Profilbild von akesar
akesarvor 2 Tagen

on the quality question, it shouldn't drop if it uses the usual check. the big model still looks at every guessed token and throws out any it wouldn't have picked itself, so you get the same answer it would give alone. only the speed changes

Profilbild von Deep
Deepvor 2 Tagen

watch your acceptance rate, the speedup follows it closely

Profilbild von Sean
Seanvor 2 Tagen

4x on a 9B model, locally on a laptop, is wild. And 2.6x over vanilla speculative decoding means it's more than just a better draft model. Curious how the speedup holds up on longer contexts.

Profilbild von AI Quanting
AI Quantingvor 2 Tagen

The gap between 30 and 11 seconds is all in the adapter. Does the tree hold up on longer prompts, or does the acceptance rate fall once the context grows?

Profilbild von Jatin Sharma
Jatin Sharmavor 2 Tagen

what did you use as the draft model? and does the 4x hold on long outputs or mostly on shorter answers?

Profilbild von The AI Engine Room
The AI Engine Roomvor 2 Tagen

The tree of continuations verified in one pass is what makes the 7.5 tokens per pass believable. Does the Weaver adapter add much memory overhead on the Mac next to the 9B model?

Profilbild von Hailu ๐Ÿ
Hailu ๐Ÿvor 2 Tagen

Vietnamese. This is an impressive technical breakthrough!

Profilbild von Mir Mujtaba
Mir Mujtabavor 2 Tagen

Does this work with 1bit,2bit models? Iโ€™m talking about Bonsai, Underdog, ISO ones

ร„hnliche Videos

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 รท 5.2 โ‰ˆ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: โ†ณ A small drafter (DFlash) proposes candidates for the next 16 positions at once. โ†ณ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. โ†ณ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: (don't forget to star it ๐ŸŒŸ) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.
0:26

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 รท 5.2 โ‰ˆ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: โ†ณ A small drafter (DFlash) proposes candidates for the next 16 positions at once. โ†ณ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. โ†ณ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: (don't forget to star it ๐ŸŒŸ) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.

Avi Chawla

125,660 Aufrufe โ€ข vor 6 Tagen

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. ๐Ÿ‘‰ Over to you: What use case are you working on that can benefit from this new technique?
0:26

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. ๐Ÿ‘‰ Over to you: What use case are you working on that can benefit from this new technique?

Avi Chawla

158,352 Aufrufe โ€ข vor 5 Monaten

Big win for local LLM inference! This open-source engine generates tokens 2.7x faster than Ollama on a MacBook: I tested it on a base M5 MacBook Pro with Qwen3.5 4B at 4-bit and the same 9.2k-token prompt. Ollama ran its MTP build, so both had speculative decoding. - Prefill: Ollama 464 tok/s, Magnitude 755 tok/s (1.6x) - Decode: Ollama 22.6 tok/s, Magnitude 60.0 tok/s (2.7x) - End to end, Magnitude finished in half the time The video below shows both runs side by side. The Ollama numbers donโ€™t mean itโ€™s badly built. Its kernels were simply tuned in advance. A kernel is the small GPU program behind every matrix multiply and attention step. Its speed depends on choices like how work is split across GPU cores and how data moves through memory. The best choices change with every chip. Apple Silicon alone spans 8 to 80 GPU cores and under 70 to over 800 GB/s of bandwidth, and every model has its own layer sizes. A prebuilt engine has to pick one set of settings for all of them, so most local setups run on a compromise. Magnitude replaces that compromise with tuning on your machine: โ†ณ When you download a model, it compiles that modelโ€™s kernels locally. โ†ณ It tries different kernel configurations and keeps whichever runs fastest on your chip. โ†ณ It pairs each model with the fastest speculative decoding drafter it has found, so each pass of the model produces more than one token. GitHub repo: (donโ€™t forget to star it ๐ŸŒŸ) This matters most for agents. Every file and tool output has to be processed before the model writes anything, and that prefill step is where kernel efficiency shows up. Concurrent sessions also share prefix caches, so parallel agents donโ€™t slow each other down. Magnitude is an open-source (Apache 2.0) inference engine written in Rust. It plugs into Claude Code, Codex, OpenCode, Pi and other agents with one click. I shared the GitHub repo above.
0:33

Big win for local LLM inference! This open-source engine generates tokens 2.7x faster than Ollama on a MacBook: I tested it on a base M5 MacBook Pro with Qwen3.5 4B at 4-bit and the same 9.2k-token prompt. Ollama ran its MTP build, so both had speculative decoding. - Prefill: Ollama 464 tok/s, Magnitude 755 tok/s (1.6x) - Decode: Ollama 22.6 tok/s, Magnitude 60.0 tok/s (2.7x) - End to end, Magnitude finished in half the time The video below shows both runs side by side. The Ollama numbers donโ€™t mean itโ€™s badly built. Its kernels were simply tuned in advance. A kernel is the small GPU program behind every matrix multiply and attention step. Its speed depends on choices like how work is split across GPU cores and how data moves through memory. The best choices change with every chip. Apple Silicon alone spans 8 to 80 GPU cores and under 70 to over 800 GB/s of bandwidth, and every model has its own layer sizes. A prebuilt engine has to pick one set of settings for all of them, so most local setups run on a compromise. Magnitude replaces that compromise with tuning on your machine: โ†ณ When you download a model, it compiles that modelโ€™s kernels locally. โ†ณ It tries different kernel configurations and keeps whichever runs fastest on your chip. โ†ณ It pairs each model with the fastest speculative decoding drafter it has found, so each pass of the model produces more than one token. GitHub repo: (donโ€™t forget to star it ๐ŸŒŸ) This matters most for agents. Every file and tool output has to be processed before the model writes anything, and that prefill step is where kernel efficiency shows up. Concurrent sessions also share prefix caches, so parallel agents donโ€™t slow each other down. Magnitude is an open-source (Apache 2.0) inference engine written in Rust. It plugs into Claude Code, Codex, OpenCode, Pi and other agents with one click. I shared the GitHub repo above.

Akshay ๐Ÿš€

36,167 Aufrufe โ€ข vor 4 Tagen

Most speculative decoding still drafts tokens one at a time. That's not parallel generation โ€” it just hides the serial loop behind a smaller model. UC San Diego's z-lab just drew a clear line between the two. They released DFlash โ€” a lightweight block diffusion model that drafts a whole block of tokens in a single forward pass, then lets the target model verify the block in parallel. Up to 15ร— higher throughput for gpt-oss-120b on NVIDIA Blackwell. No token-by-token drafting anywhere in the speculative path. Here's what's actually interesting: โ†’ The drafter is conditioned on the target model's own hidden features, injected into the Key/Value cache of every draft layer โ€” so acceptance length scales with draft depth instead of diluting away โ†’ A 5-layer drafter replaces the 7B diffusion drafters that capped earlier methods near 3โ€“4ร— โ†’ MATH-500 speedup: 6.08ร— vs. 1.81ร— for EAGLE-3 (4.86ร— average vs. 1.76ร—, Qwen3-8B, greedy) โ†’ Up to 15ร— higher throughput for gpt-oss-120b on NVIDIA Blackwell โ€” at the same interactivity target โ†’ Lossless: the target still verifies every token, so output quality is preserved Full analysis: Paper: NVIDIA's metrics: Project: Model weights: Repo: NVIDIA AI NVIDIA AIDev
0:20

Most speculative decoding still drafts tokens one at a time. That's not parallel generation โ€” it just hides the serial loop behind a smaller model. UC San Diego's z-lab just drew a clear line between the two. They released DFlash โ€” a lightweight block diffusion model that drafts a whole block of tokens in a single forward pass, then lets the target model verify the block in parallel. Up to 15ร— higher throughput for gpt-oss-120b on NVIDIA Blackwell. No token-by-token drafting anywhere in the speculative path. Here's what's actually interesting: โ†’ The drafter is conditioned on the target model's own hidden features, injected into the Key/Value cache of every draft layer โ€” so acceptance length scales with draft depth instead of diluting away โ†’ A 5-layer drafter replaces the 7B diffusion drafters that capped earlier methods near 3โ€“4ร— โ†’ MATH-500 speedup: 6.08ร— vs. 1.81ร— for EAGLE-3 (4.86ร— average vs. 1.76ร—, Qwen3-8B, greedy) โ†’ Up to 15ร— higher throughput for gpt-oss-120b on NVIDIA Blackwell โ€” at the same interactivity target โ†’ Lossless: the target still verifies every token, so output quality is preserved Full analysis: Paper: NVIDIA's metrics: Project: Model weights: Repo: NVIDIA AI NVIDIA AIDev

Marktechpost AI

23,328 Aufrufe โ€ข vor 4 Monaten

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding. To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8. We are open sourcing it today. 1/2
1:25

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding. To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8. We are open sourcing it today. 1/2

Inferact

245,826 Aufrufe โ€ข vor 1 Monat

DFlash 2 is here! Qwen3.8-27B at 70 tok/s on an M5 Max MacBook Pro. โšก Up to 4.6ร— the speed of autoregressive decoding, with the same output. This is the next generation of DFlash, seeded at Z Lab and upgraded at Inco AI. Get one more accepted token on every pass, for free!
0:26

DFlash 2 is here! Qwen3.8-27B at 70 tok/s on an M5 Max MacBook Pro. โšก Up to 4.6ร— the speed of autoregressive decoding, with the same output. This is the next generation of DFlash, seeded at Z Lab and upgraded at Inco AI. Get one more accepted token on every pass, for free!

Zhijian Liu

1,209,392 Aufrufe โ€ข vor 2 Monaten

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 ยท M5 Max ยท greedy exact match โ–ธ 85 tok/s, 3.3ร— at 1024 tokens (runtime) โ–ธ ~70 tok/s, 2.6ร— in the video (terminal I/O overhead) โ–ธ 80 tok/s, 3.1ร— at 2048 tokens (runtime) Currently working on: โ†’ Long context (speedup degrades past 4K tokens, KV cache growth) โ†’ Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.
0:39

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 ยท M5 Max ยท greedy exact match โ–ธ 85 tok/s, 3.3ร— at 1024 tokens (runtime) โ–ธ ~70 tok/s, 2.6ร— in the video (terminal I/O overhead) โ–ธ 80 tok/s, 3.1ร— at 2048 tokens (runtime) Currently working on: โ†’ Long context (speedup degrades past 4K tokens, KV cache growth) โ†’ Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.

bstn ๐Ÿ‘๏ธ

37,049 Aufrufe โ€ข vor 6 Monaten

๐Ÿš€ Our DSpark for Qwen3.8-27B beats native MTP with the same 8 speculative tokens on 4ร—H200. Up to 52% faster single-stream decoding and 23% higher peak throughput. On 8-needle MRCR, it averages 4.30 accepted tokens beyond 1M context. Model: Demo below, see for yourself!
0:54

๐Ÿš€ Our DSpark for Qwen3.8-27B beats native MTP with the same 8 speculative tokens on 4ร—H200. Up to 52% faster single-stream decoding and 23% higher peak throughput. On 8-needle MRCR, it averages 4.30 accepted tokens beyond 1M context. Model: Demo below, see for yourself!

Red Hat AI

14,041 Aufrufe โ€ข vor 1 Monat

While exploring diffusion language models (DLMs), we kept asking ourselves: given speculative decoding, a lossless acceleration technique, why do we still need DLMs? ๐Ÿค” Our latest work, ORL, tries to answer this question: AR optimizes intelligence ๐Ÿ‘†, speculative decoding optimizes speed ๐Ÿ‘‰, and DLMs optimize the Pareto frontier of speed and intelligence ๐Ÿš€. Through co-design across the algorithms, the training and inference engine, and the kernels, ORL pushes out a new Pareto frontier of intelligence and speed. Similar performance, much faster than speculative decoding. You say lossy spec? ORL-DLM is better too! ๐Ÿ˜Ž ๐ŸŒ Project + live demos:
1:28

While exploring diffusion language models (DLMs), we kept asking ourselves: given speculative decoding, a lossless acceleration technique, why do we still need DLMs? ๐Ÿค” Our latest work, ORL, tries to answer this question: AR optimizes intelligence ๐Ÿ‘†, speculative decoding optimizes speed ๐Ÿ‘‰, and DLMs optimize the Pareto frontier of speed and intelligence ๐Ÿš€. Through co-design across the algorithms, the training and inference engine, and the kernels, ORL pushes out a new Pareto frontier of intelligence and speed. Similar performance, much faster than speculative decoding. You say lossy spec? ORL-DLM is better too! ๐Ÿ˜Ž ๐ŸŒ Project + live demos:

Chenfeng_X

21,240 Aufrufe โ€ข vor 17 Tagen

Weโ€™re open-sourcing lithos-metal ๐Ÿš€ Megakernels + DSpark speculative decoding run Qwen3.8-27B at 200+ tokens/s/user peak on one Apple M5 Max. Ultra-fast inference on your laptop. Try it with any coding agent in one command. Code: Tech blog:
0:37

Weโ€™re open-sourcing lithos-metal ๐Ÿš€ Megakernels + DSpark speculative decoding run Qwen3.8-27B at 200+ tokens/s/user peak on one Apple M5 Max. Ultra-fast inference on your laptop. Try it with any coding agent in one command. Code: Tech blog:

Zhihao Jia

423,198 Aufrufe โ€ข vor 24 Tagen

i'm running a 27b uncensored๐Ÿšจmodel at 50 tok/s on a 12gb rtx 3060 and it does not say no to anything โš ๏ธ model just finished full harmbench red-team suite against it, 200 behaviors, and it refused zero. not a few. zero. aligned models sit in the 90s. the model is ternary bonsai 2 with the heretic uncensoring pass, the weights are ternary, just -1, 0 and +1, so the whole 27b is a file small enough that a gaming card from 2020 holds it with room to spare, and it's a reasoning model. it thinks before it answers, and it still keeps the pace the speed is the head i grafted on, a draft model that predicts the next token before the big model checks it. speculative decoding. the card fires a guess, the 27b confirms, and you get 50 tok/s instead of 39 the whole thing, weights, head and a 128k context window, sits in 10.7 of the 12 gb. kv cache at q4. nothing exotic, one card, one power plug why uncensored. because it's mine. no api keys, no terms of service, no model deciding what an adult is allowed to ask. the refusals left in it are ones it chose, and there are almost none that's the whole point of running local. the policy lives in your house, not someone else's
0:50

i'm running a 27b uncensored๐Ÿšจmodel at 50 tok/s on a 12gb rtx 3060 and it does not say no to anything โš ๏ธ model just finished full harmbench red-team suite against it, 200 behaviors, and it refused zero. not a few. zero. aligned models sit in the 90s. the model is ternary bonsai 2 with the heretic uncensoring pass, the weights are ternary, just -1, 0 and +1, so the whole 27b is a file small enough that a gaming card from 2020 holds it with room to spare, and it's a reasoning model. it thinks before it answers, and it still keeps the pace the speed is the head i grafted on, a draft model that predicts the next token before the big model checks it. speculative decoding. the card fires a guess, the 27b confirms, and you get 50 tok/s instead of 39 the whole thing, weights, head and a 128k context window, sits in 10.7 of the 12 gb. kv cache at q4. nothing exotic, one card, one power plug why uncensored. because it's mine. no api keys, no terms of service, no model deciding what an adult is allowed to ask. the refusals left in it are ones it chose, and there are almost none that's the whole point of running local. the policy lives in your house, not someone else's

Sudo su

59,906 Aufrufe โ€ข vor 18 Tagen

Sequential Monte Carlo speculative decoding from Makora keeps multiple draft tokens alive in parallel instead of rewinding failed matches.
1:22

Sequential Monte Carlo speculative decoding from Makora keeps multiple draft tokens alive in parallel instead of rewinding failed matches.

SemiAnalysis

24,418 Aufrufe โ€ข vor 4 Monaten

Need faster LLM inference without sacrificing accuracy? Speculative decoding can help. Choosing the right draft length and drafting method depends on your model, workload and hardware. We break down five practical guidelines for balancing throughput and latency.
0:35

Need faster LLM inference without sacrificing accuracy? Speculative decoding can help. Choosing the right draft length and drafting method depends on your model, workload and hardware. We break down five practical guidelines for balancing throughput and latency.

NVIDIA AI

35,445 Aufrufe โ€ข vor 1 Monat

The example below is using prompt-based speculative decoding. Specifically, ngram hashing is utilized to suggest drafts of up to 64 tokens. The hasher keeps track of ngrams in the observed contexts, so mostly effective for coding tasks. Here is another demo:
0:47

The example below is using prompt-based speculative decoding. Specifically, ngram hashing is utilized to suggest drafts of up to 64 tokens. The hasher keeps track of ngrams in the observed contexts, so mostly effective for coding tasks. Here is another demo:

Georgi Gerganov

29,890 Aufrufe โ€ข vor 7 Monaten

๐Ÿšจ NEW ON AND make NVIDIA's Nemotron 3.5 Lightning faster. Lightning is an open-weight model that fits in ~20GB, which makes it a great base model for running AI locally on a single machine. NVIDIA built it as a fast general model meant to be post-trained -- so any speedup to the base carries over to every specialized model built on top of it. On day 1, solvers on Yukon already got it running 31.9% faster on DGX Spark and 13.8% faster on Mac. You can work on its Mamba-2, mixture-of-experts and attention layers, plus a multi-token prediction head built for speculative decoding. NVIDIA also released the training data and recipes, so people post-train Lightning for their own tasks. Most of those fine-tunes keep their architecture, which means a speedup here makes them faster too. Two tracks: โ€ข Apple Silicon โ†’ โ€ข NVIDIA DGX Spark โ†’ The challenge runs until Oct 29, 2026 To join, pick a track, point your agent at it and let it rip.
0:30

๐Ÿšจ NEW ON AND make NVIDIA's Nemotron 3.5 Lightning faster. Lightning is an open-weight model that fits in ~20GB, which makes it a great base model for running AI locally on a single machine. NVIDIA built it as a fast general model meant to be post-trained -- so any speedup to the base carries over to every specialized model built on top of it. On day 1, solvers on Yukon already got it running 31.9% faster on DGX Spark and 13.8% faster on Mac. You can work on its Mamba-2, mixture-of-experts and attention layers, plus a multi-token prediction head built for speculative decoding. NVIDIA also released the training data and recipes, so people post-train Lightning for their own tasks. Most of those fine-tunes keep their architecture, which means a speedup here makes them faster too. Two tracks: โ€ข Apple Silicon โ†’ โ€ข NVIDIA DGX Spark โ†’ The challenge runs until Oct 29, 2026 To join, pick a track, point your agent at it and let it rip.

Kydo

17,312 Aufrufe โ€ข vor 9 Tagen

I spent today turning a blank Hermes install into a sovereign local agent on my RTX 5090. always-on, reachable over telegram, fully local. here's everything it can do. it runs Qwen3.6-27B on the 5090 and answers from my phone over telegram. ~~~ I first made it faster. switched to the MTP build for self-speculative decoding: 62 > 115 tokens/sec. ~~~ it now benchmarks models on command. I ask it to speed-test a gguf and it stops its own model server to free the gpu, runs llama-bench, renders a card, sends it to my phone, then restarts itself. it's never left offline. ~~~ it also knows me very well. I pointed it to my private HF repo where I store all my AI traces since I started using it. I created a local markdown memory vault, seeded from my own history, with semantic search over it. ~~~ the whole stack stays home: weights on disk, inference on the 5090, the agent in my pocket. here is the video it created to showcase everything it has now.
0:32

I spent today turning a blank Hermes install into a sovereign local agent on my RTX 5090. always-on, reachable over telegram, fully local. here's everything it can do. it runs Qwen3.6-27B on the 5090 and answers from my phone over telegram. ~~~ I first made it faster. switched to the MTP build for self-speculative decoding: 62 > 115 tokens/sec. ~~~ it now benchmarks models on command. I ask it to speed-test a gguf and it stops its own model server to free the gpu, runs llama-bench, renders a card, sends it to my phone, then restarts itself. it's never left offline. ~~~ it also knows me very well. I pointed it to my private HF repo where I store all my AI traces since I started using it. I created a local markdown memory vault, seeded from my own history, with semantic search over it. ~~~ the whole stack stays home: weights on disk, inference on the 5090, the agent in my pocket. here is the video it created to showcase everything it has now.

witcheer

15,041 Aufrufe โ€ข vor 5 Monaten

Claude Opus 5.5 drew my apartment in 3D and found 3 cameras in it that are not mine... one prompt, 20 minutes, two of them were still sending audio out here is exactly how it went: โ†’ I started one agent in the terminal on a laptop sitting on my home Wi-Fi and left it alone โ†’ it swept the network first and listed every device it could see, with brand and model where it could match one โ†’ then it measured signal strength from a few spots in the flat and built a rough room layout out of that โ†’ every device got pinned onto that layout, so instead of a list of IPs I was looking at a map of my own place โ†’ anything it could not tie to something I own got flagged on its own, with what it was transmitting and roughly where it sat three of those flags were cameras. two more were audio only. I had lived there two years and walked past all five every day the whole thing ran on one prompt and I never touched the keyboard after the first line full run on video, prompt in the replies โ†“
0:29

Claude Opus 5.5 drew my apartment in 3D and found 3 cameras in it that are not mine... one prompt, 20 minutes, two of them were still sending audio out here is exactly how it went: โ†’ I started one agent in the terminal on a laptop sitting on my home Wi-Fi and left it alone โ†’ it swept the network first and listed every device it could see, with brand and model where it could match one โ†’ then it measured signal strength from a few spots in the flat and built a rough room layout out of that โ†’ every device got pinned onto that layout, so instead of a list of IPs I was looking at a map of my own place โ†’ anything it could not tie to something I own got flagged on its own, with what it was transmitting and roughly where it sat three of those flags were cameras. two more were audio only. I had lived there two years and walked past all five every day the whole thing ran on one prompt and I never touched the keyboard after the first line full run on video, prompt in the replies โ†“

Noisy

11,517 Aufrufe โ€ข vor 10 Tagen

๐ŸŽ‰ MiniMax M3.1 Flash from MiniMax (official) has Day-0 support in vLLM! Built for multi-step automation, real-codebase engineering and visual reasoningโ€”with native NVFP4 KV, NVFP4 MoE weights and DSpark speculative decoding in the serving stack. What the model can do, and the vLLM engineering that makes it practical to serve. ๐Ÿ‘‡ Model: 1/5
0:40

๐ŸŽ‰ MiniMax M3.1 Flash from MiniMax (official) has Day-0 support in vLLM! Built for multi-step automation, real-codebase engineering and visual reasoningโ€”with native NVFP4 KV, NVFP4 MoE weights and DSpark speculative decoding in the serving stack. What the model can do, and the vLLM engineering that makes it practical to serve. ๐Ÿ‘‡ Model: 1/5

vLLM

11,710 Aufrufe โ€ข vor 10 Tagen

I'm sharing with you one of the best papers I've read on Harness Engineering it takes apart Claude Code (Opus 5.5), Codex, Gemini CLI and more, and boils all of them down to one line: agent = model + harness what you walk away with: > what a harness actually is > how today's coding agents are built under the hood > the patterns every one of them repeats > where the whole category is heading > their checklist for building your own the model gets the headlines. the harness decides whether the agent works loop and graph engineering on top of it - in my article below
0:24

I'm sharing with you one of the best papers I've read on Harness Engineering it takes apart Claude Code (Opus 5.5), Codex, Gemini CLI and more, and boils all of them down to one line: agent = model + harness what you walk away with: > what a harness actually is > how today's coding agents are built under the hood > the patterns every one of them repeats > where the whole category is heading > their checklist for building your own the model gets the headlines. the harness decides whether the agent works loop and graph engineering on top of it - in my article below

Mr. Buzzoni

84,277 Aufrufe โ€ข vor 27 Tagen

DFlash makes Qwen 2.2x faster with no quality loss! We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks only differ in one thing - how predictable the next word is: quicksort, describe a file in JSON, a logic puzzle, a sci-fi story. Outputs: Baseline: 44 tok/s ยท 1.00x MTP: 65 tok/s ยท 1.45x ยท 71% accepted DFlash: 98 tok/s ยท 2.20x ยท 30% accepted Baseline writes one token per step. MTP works inside the model itself and guesses 3 tokens ahead. DFlash is a separate small model that writes 15 tokens at once, and the big model only checks them. In JSON the same words repeat all the time, so most guesses were right: 152 tok/s, 3.4x speedup. In the story 9 guesses out of 10 were wrong. DFlash did all that extra work for nothing and became slower than baseline: 42 vs 44 tok/s. MTP guesses only 3 tokens, so a wrong guess costs very little: 46 tok/s and the win in that round. The output is identical in all three modes - DFlash is the pick for tasks with predictable output, like coding, and for chat and creative writing MTP works better. DFlash is now natively integrated into Atomic Chat on llama.cpp - speed up your Qwen models!
0:26

DFlash makes Qwen 2.2x faster with no quality loss! We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks only differ in one thing - how predictable the next word is: quicksort, describe a file in JSON, a logic puzzle, a sci-fi story. Outputs: Baseline: 44 tok/s ยท 1.00x MTP: 65 tok/s ยท 1.45x ยท 71% accepted DFlash: 98 tok/s ยท 2.20x ยท 30% accepted Baseline writes one token per step. MTP works inside the model itself and guesses 3 tokens ahead. DFlash is a separate small model that writes 15 tokens at once, and the big model only checks them. In JSON the same words repeat all the time, so most guesses were right: 152 tok/s, 3.4x speedup. In the story 9 guesses out of 10 were wrong. DFlash did all that extra work for nothing and became slower than baseline: 42 vs 44 tok/s. MTP guesses only 3 tokens, so a wrong guess costs very little: 46 tok/s and the win in that round. The output is identical in all three modes - DFlash is the pick for tasks with predictable output, like coding, and for chat and creative writing MTP works better. DFlash is now natively integrated into Atomic Chat on llama.cpp - speed up your Qwen models!

atomic.chat

83,237 Aufrufe โ€ข vor 3 Monaten