正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Massive breakthrough in speculative decoding! It made a 9B LLM on my MacBook 4x faster, and 2.6x faster than regular speculative decoding. I ran Qwen3.5 9B on my M5 MacBook Pro with the same prompt, three ways: - No speculative decoding: done in 42s - Regular speculative decoding: done... in 30s - DFlash + Weaver: done in 11s The video below shows all three side by side. Here's why it works: Speculative decoding lets a cheap drafter guess the next few tokens, and the main model checks them all in one pass. The fastest drafters, like DFlash, guess every position in parallel. But each guess is made on its own, so position 5 never sees what was picked at position 4. One early mismatch and everything after it gets thrown away. Weaver fixes this. It's a tiny adapter (one Transformer layer) on top of DFlash that: ↳ picks among DFlash's top candidates left to right, so each pick depends on the last ↳ grows a tree of likely continuations instead of a single chain ↳ lets the main model verify the whole tree in one pass On my Mac, every pass of the model produced 7.5 tokens instead of 1. On a B200 GPU, the paper reports 4x faster decoding across chat, math and code. Mirai Labs open-sourced the whole stack in their engine, uzu, built for Apple Silicon. GitHub repo: (don't forget to star it 🌟) Trying it takes two commands: 𝗯𝗿𝗲𝘄 𝗶𝗻𝘀𝘁𝗮𝗹𝗹 𝗺𝗶𝗿𝗮𝗶 𝗺𝗶𝗿𝗮𝗶 --𝗺𝗼𝗱𝗲𝗹 𝘁𝗿𝘆𝗺𝗶𝗿𝗮𝗶/𝗤𝘄𝗲𝗻𝟯.𝟱-𝟵𝗕-𝗠 -𝗺 "𝗪𝗿𝗶𝘁𝗲 𝗮 𝗽𝘆𝘁𝗵𝗼𝗻 𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻 𝘁𝗼 𝗺𝗲𝗿𝗴𝗲 𝘁𝘄𝗼 𝘀𝗼𝗿𝘁𝗲𝗱 𝗹𝗶𝘀𝘁𝘀" --- I wrote a full breakdown of why local inference is memory-bound and how uzu gets past it, with reproducible code. The article is quoted below. In the replies, there's an illustration of standard vs. speculative vs. uzu decoding.show more

Akshay 🚀

292,335 subscribers

27,196 次观看 • 8 天前 •via X (Twitter)

科学技术教育

Anya Rossi• Live Now

Private livecam show

28 条评论

Akshay 🚀 的头像

Akshay 🚀8 天前

Standard vs. Speculative vs. UZU decoding:

Giacomo Masseroni 的头像

Giacomo Masseroni8 天前

42s down to 11s is wild

anirudh | agent fleet 的头像

anirudh | agent fleet8 天前

4x faster on a macbook means my agents can now be confidently wrong at 4x the speed, with no api bill left to blame

The SI Maximalist 的头像

The SI Maximalist8 天前

Your MacBook just found another gear

Rohan 的头像

Rohan8 天前

What acceptance rate did the draft get on that prompt?

The Broke Vibe Coder 的头像

The Broke Vibe Coder8 天前

Can you show: uzu without speculative decoding. uzu with DFlash alone. uzu with DFlash + Weaver. Your 4× result doesn’t tell us how much acceleration Weaver itself contributes.

Dusan 🐧 AI agents · dad 的头像

Dusan 🐧 AI agents · dad8 天前

On linux you pick the gpu, the driver and the engine, and can patch any of them when a new decoding trick lands. a macbook gives you what apple exposes 🐧

Natural Stupidity 的头像

Natural Stupidity8 天前

now it can be wrong in 10 seconds instead of 42

Actian for Developers 的头像

Actian for Developers8 天前

11s on a laptop is lil crazy, now the slow part of local RAG is going to be retrieval

Nodelog 的头像

Nodelog8 天前

Speedup numbers like this depend a lot on draft acceptance rate, so log accepted tokens per step on your own prompts, not just wall time. Also compare against a batch size above 1, since the gain usually shrinks once the target model is compute-bound.

Yuzu 的头像

Yuzu8 天前

massive breakthrough in speculative decoding, proven by timing how fast it generates a function that merges two sorted lists

Wecko AI 的头像

Wecko AI8 天前

4x on a 9B is wild. Curious if it holds on a 27B, my Qwen3.8 27B sits at 37.4 tok/s on a Mac mini M5 Pro and that's the one I'd love to see doubled

Qwen AI Fans 的头像

Qwen AI Fans8 天前

4x on a MacBook with better speculative decoding is a huge local win.

Tom Sheffield 的头像

Tom Sheffield8 天前

would like to see the 4x on a long multi-step run where the context keeps growing

theMadDeFientist 的头像

theMadDeFientist8 天前

this is what makes local agents usable. a 9b that answers in 10s instead of 42 can sit inside a tool loop without you walking away

Josh Ramirez 的头像

Josh Ramirez8 天前

7.5 tokens per pass is kinda absurd. really curious how that holds up at longer context, across a wide range of workflows. speculative decoding can have a wide performance range based on the task, wonder how weaver affects that aspect

Umar Farooq Aslam 的头像

Umar Farooq Aslam8 天前

Batch size 1 is the best case for this. on a serving box at high batch the verify pass stops being free and most of the gain shrinks. curious where weaver crosses over.

Emly 的头像

Emly7 天前

This is an exciting development for local LLM performance.

Solar Nomad 的头像

Solar Nomad8 天前

quick math from your numbers: 42s to 11s is 3.8x. 7.5 tokens per pass would give 7.5x if passes were free, so each tree-verify pass (drafter included) costs ~2x a plain decode step on the M5, if output length matched. still a great trade. what tokens/pass on a non-code prompt?

Yurii Bakurov 的头像

Yurii Bakurov8 天前

4x on a 9B is wild. Does it hold on bigger models like 27B ? That's where local starts to feel slow on my M5 Max

akesar 的头像

akesar8 天前

on the quality question, it shouldn't drop if it uses the usual check. the big model still looks at every guessed token and throws out any it wouldn't have picked itself, so you get the same answer it would give alone. only the speed changes

Deep 的头像

Deep8 天前

watch your acceptance rate, the speedup follows it closely

Sean 的头像

Sean8 天前

4x on a 9B model, locally on a laptop, is wild. And 2.6x over vanilla speculative decoding means it's more than just a better draft model. Curious how the speedup holds up on longer contexts.

AI Quanting 的头像

AI Quanting8 天前

The gap between 30 and 11 seconds is all in the adapter. Does the tree hold up on longer prompts, or does the acceptance rate fall once the context grows?

Jatin Sharma 的头像

Jatin Sharma8 天前

what did you use as the draft model? and does the 4x hold on long outputs or mostly on shorter answers?

The AI Engine Room 的头像

The AI Engine Room8 天前

The tree of continuations verified in one pass is what makes the 7.5 tokens per pass believable. Does the Weaver adapter add much memory overhead on the Mac next to the 9B model?

Hailu 🍏 的头像

Hailu 🍏7 天前

Vietnamese. This is an impressive technical breakthrough!

Mir Mujtaba 的头像

Mir Mujtaba8 天前

Does this work with 1bit,2bit models? I’m talking about Bonsai, Underdog, ISO ones

相关视频

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 ÷ 5.2 ≈ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: ↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once. ↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. ↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: (don't forget to star it 🌟) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 ÷ 5.2 ≈ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: ↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once. ↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. ↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: (don't forget to star it 🌟) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.

Avi Chawla

127,683 次观看 • 11 天前

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Avi Chawla

158,352 次观看 • 6 个月前

Big win for local LLM inference! This open-source engine generates tokens 2.7x faster than Ollama on a MacBook: I tested it on a base M5 MacBook Pro with Qwen3.5 4B at 4-bit and the same 9.2k-token prompt. Ollama ran its MTP build, so both had speculative decoding. - Prefill: Ollama 464 tok/s, Magnitude 755 tok/s (1.6x) - Decode: Ollama 22.6 tok/s, Magnitude 60.0 tok/s (2.7x) - End to end, Magnitude finished in half the time The video below shows both runs side by side. The Ollama numbers don’t mean it’s badly built. Its kernels were simply tuned in advance. A kernel is the small GPU program behind every matrix multiply and attention step. Its speed depends on choices like how work is split across GPU cores and how data moves through memory. The best choices change with every chip. Apple Silicon alone spans 8 to 80 GPU cores and under 70 to over 800 GB/s of bandwidth, and every model has its own layer sizes. A prebuilt engine has to pick one set of settings for all of them, so most local setups run on a compromise. Magnitude replaces that compromise with tuning on your machine: ↳ When you download a model, it compiles that model’s kernels locally. ↳ It tries different kernel configurations and keeps whichever runs fastest on your chip. ↳ It pairs each model with the fastest speculative decoding drafter it has found, so each pass of the model produces more than one token. GitHub repo: (don’t forget to star it 🌟) This matters most for agents. Every file and tool output has to be processed before the model writes anything, and that prefill step is where kernel efficiency shows up. Concurrent sessions also share prefix caches, so parallel agents don’t slow each other down. Magnitude is an open-source (Apache 2.0) inference engine written in Rust. It plugs into Claude Code, Codex, OpenCode, Pi and other agents with one click. I shared the GitHub repo above.

Big win for local LLM inference! This open-source engine generates tokens 2.7x faster than Ollama on a MacBook: I tested it on a base M5 MacBook Pro with Qwen3.5 4B at 4-bit and the same 9.2k-token prompt. Ollama ran its MTP build, so both had speculative decoding. - Prefill: Ollama 464 tok/s, Magnitude 755 tok/s (1.6x) - Decode: Ollama 22.6 tok/s, Magnitude 60.0 tok/s (2.7x) - End to end, Magnitude finished in half the time The video below shows both runs side by side. The Ollama numbers don’t mean it’s badly built. Its kernels were simply tuned in advance. A kernel is the small GPU program behind every matrix multiply and attention step. Its speed depends on choices like how work is split across GPU cores and how data moves through memory. The best choices change with every chip. Apple Silicon alone spans 8 to 80 GPU cores and under 70 to over 800 GB/s of bandwidth, and every model has its own layer sizes. A prebuilt engine has to pick one set of settings for all of them, so most local setups run on a compromise. Magnitude replaces that compromise with tuning on your machine: ↳ When you download a model, it compiles that model’s kernels locally. ↳ It tries different kernel configurations and keeps whichever runs fastest on your chip. ↳ It pairs each model with the fastest speculative decoding drafter it has found, so each pass of the model produces more than one token. GitHub repo: (don’t forget to star it 🌟) This matters most for agents. Every file and tool output has to be processed before the model writes anything, and that prefill step is where kernel efficiency shows up. Concurrent sessions also share prefix caches, so parallel agents don’t slow each other down. Magnitude is an open-source (Apache 2.0) inference engine written in Rust. It plugs into Claude Code, Codex, OpenCode, Pi and other agents with one click. I shared the GitHub repo above.

Akshay 🚀

37,323 次观看 • 10 天前

Most speculative decoding still drafts tokens one at a time. That's not parallel generation — it just hides the serial loop behind a smaller model. UC San Diego's z-lab just drew a clear line between the two. They released DFlash — a lightweight block diffusion model that drafts a whole block of tokens in a single forward pass, then lets the target model verify the block in parallel. Up to 15× higher throughput for gpt-oss-120b on NVIDIA Blackwell. No token-by-token drafting anywhere in the speculative path. Here's what's actually interesting: → The drafter is conditioned on the target model's own hidden features, injected into the Key/Value cache of every draft layer — so acceptance length scales with draft depth instead of diluting away → A 5-layer drafter replaces the 7B diffusion drafters that capped earlier methods near 3–4× → MATH-500 speedup: 6.08× vs. 1.81× for EAGLE-3 (4.86× average vs. 1.76×, Qwen3-8B, greedy) → Up to 15× higher throughput for gpt-oss-120b on NVIDIA Blackwell — at the same interactivity target → Lossless: the target still verifies every token, so output quality is preserved Full analysis: Paper: NVIDIA's metrics: Project: Model weights: Repo: NVIDIA AI NVIDIA AIDev

Most speculative decoding still drafts tokens one at a time. That's not parallel generation — it just hides the serial loop behind a smaller model. UC San Diego's z-lab just drew a clear line between the two. They released DFlash — a lightweight block diffusion model that drafts a whole block of tokens in a single forward pass, then lets the target model verify the block in parallel. Up to 15× higher throughput for gpt-oss-120b on NVIDIA Blackwell. No token-by-token drafting anywhere in the speculative path. Here's what's actually interesting: → The drafter is conditioned on the target model's own hidden features, injected into the Key/Value cache of every draft layer — so acceptance length scales with draft depth instead of diluting away → A 5-layer drafter replaces the 7B diffusion drafters that capped earlier methods near 3–4× → MATH-500 speedup: 6.08× vs. 1.81× for EAGLE-3 (4.86× average vs. 1.76×, Qwen3-8B, greedy) → Up to 15× higher throughput for gpt-oss-120b on NVIDIA Blackwell — at the same interactivity target → Lossless: the target still verifies every token, so output quality is preserved Full analysis: Paper: NVIDIA's metrics: Project: Model weights: Repo: NVIDIA AI NVIDIA AIDev

Marktechpost AI

23,328 次观看 • 4 个月前

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding. To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8. We are open sourcing it today. 1/2

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding. To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8. We are open sourcing it today. 1/2

Inferact

245,826 次观看 • 1 个月前

DFlash 2 is here! Qwen3.8-27B at 70 tok/s on an M5 Max MacBook Pro. ⚡ Up to 4.6× the speed of autoregressive decoding, with the same output. This is the next generation of DFlash, seeded at Z Lab and upgraded at Inco AI. Get one more accepted token on every pass, for free!

DFlash 2 is here! Qwen3.8-27B at 70 tok/s on an M5 Max MacBook Pro. ⚡ Up to 4.6× the speed of autoregressive decoding, with the same output. This is the next generation of DFlash, seeded at Z Lab and upgraded at Inco AI. Get one more accepted token on every pass, for free!

Zhijian Liu

1,209,858 次观看 • 2 个月前

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 · M5 Max · greedy exact match ▸ 85 tok/s, 3.3× at 1024 tokens (runtime) ▸ ~70 tok/s, 2.6× in the video (terminal I/O overhead) ▸ 80 tok/s, 3.1× at 2048 tokens (runtime) Currently working on: → Long context (speedup degrades past 4K tokens, KV cache growth) → Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 · M5 Max · greedy exact match ▸ 85 tok/s, 3.3× at 1024 tokens (runtime) ▸ ~70 tok/s, 2.6× in the video (terminal I/O overhead) ▸ 80 tok/s, 3.1× at 2048 tokens (runtime) Currently working on: → Long context (speedup degrades past 4K tokens, KV cache growth) → Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.

bstn 👁️

37,049 次观看 • 7 个月前

🚀 Our DSpark for Qwen3.8-27B beats native MTP with the same 8 speculative tokens on 4×H200. Up to 52% faster single-stream decoding and 23% higher peak throughput. On 8-needle MRCR, it averages 4.30 accepted tokens beyond 1M context. Model: Demo below, see for yourself!

🚀 Our DSpark for Qwen3.8-27B beats native MTP with the same 8 speculative tokens on 4×H200. Up to 52% faster single-stream decoding and 23% higher peak throughput. On 8-needle MRCR, it averages 4.30 accepted tokens beyond 1M context. Model: Demo below, see for yourself!

Red Hat AI

14,041 次观看 • 1 个月前

While exploring diffusion language models (DLMs), we kept asking ourselves: given speculative decoding, a lossless acceleration technique, why do we still need DLMs? 🤔 Our latest work, ORL, tries to answer this question: AR optimizes intelligence 👆, speculative decoding optimizes speed 👉, and DLMs optimize the Pareto frontier of speed and intelligence 🚀. Through co-design across the algorithms, the training and inference engine, and the kernels, ORL pushes out a new Pareto frontier of intelligence and speed. Similar performance, much faster than speculative decoding. You say lossy spec? ORL-DLM is better too! 😎 🌐 Project + live demos:

While exploring diffusion language models (DLMs), we kept asking ourselves: given speculative decoding, a lossless acceleration technique, why do we still need DLMs? 🤔 Our latest work, ORL, tries to answer this question: AR optimizes intelligence 👆, speculative decoding optimizes speed 👉, and DLMs optimize the Pareto frontier of speed and intelligence 🚀. Through co-design across the algorithms, the training and inference engine, and the kernels, ORL pushes out a new Pareto frontier of intelligence and speed. Similar performance, much faster than speculative decoding. You say lossy spec? ORL-DLM is better too! 😎 🌐 Project + live demos:

Chenfeng_X

21,298 次观看 • 23 天前

We’re open-sourcing lithos-metal 🚀 Megakernels + DSpark speculative decoding run Qwen3.8-27B at 200+ tokens/s/user peak on one Apple M5 Max. Ultra-fast inference on your laptop. Try it with any coding agent in one command. Code: Tech blog:

We’re open-sourcing lithos-metal 🚀 Megakernels + DSpark speculative decoding run Qwen3.8-27B at 200+ tokens/s/user peak on one Apple M5 Max. Ultra-fast inference on your laptop. Try it with any coding agent in one command. Code: Tech blog:

Zhihao Jia

423,198 次观看 • 1 个月前

i'm running a 27b uncensored🚨model at 50 tok/s on a 12gb rtx 3060 and it does not say no to anything ⚠️ model just finished full harmbench red-team suite against it, 200 behaviors, and it refused zero. not a few. zero. aligned models sit in the 90s. the model is ternary bonsai 2 with the heretic uncensoring pass, the weights are ternary, just -1, 0 and +1, so the whole 27b is a file small enough that a gaming card from 2020 holds it with room to spare, and it's a reasoning model. it thinks before it answers, and it still keeps the pace the speed is the head i grafted on, a draft model that predicts the next token before the big model checks it. speculative decoding. the card fires a guess, the 27b confirms, and you get 50 tok/s instead of 39 the whole thing, weights, head and a 128k context window, sits in 10.7 of the 12 gb. kv cache at q4. nothing exotic, one card, one power plug why uncensored. because it's mine. no api keys, no terms of service, no model deciding what an adult is allowed to ask. the refusals left in it are ones it chose, and there are almost none that's the whole point of running local. the policy lives in your house, not someone else's

i'm running a 27b uncensored🚨model at 50 tok/s on a 12gb rtx 3060 and it does not say no to anything ⚠️ model just finished full harmbench red-team suite against it, 200 behaviors, and it refused zero. not a few. zero. aligned models sit in the 90s. the model is ternary bonsai 2 with the heretic uncensoring pass, the weights are ternary, just -1, 0 and +1, so the whole 27b is a file small enough that a gaming card from 2020 holds it with room to spare, and it's a reasoning model. it thinks before it answers, and it still keeps the pace the speed is the head i grafted on, a draft model that predicts the next token before the big model checks it. speculative decoding. the card fires a guess, the 27b confirms, and you get 50 tok/s instead of 39 the whole thing, weights, head and a 128k context window, sits in 10.7 of the 12 gb. kv cache at q4. nothing exotic, one card, one power plug why uncensored. because it's mine. no api keys, no terms of service, no model deciding what an adult is allowed to ask. the refusals left in it are ones it chose, and there are almost none that's the whole point of running local. the policy lives in your house, not someone else's

Sudo su

59,951 次观看 • 24 天前

Sequential Monte Carlo speculative decoding from Makora keeps multiple draft tokens alive in parallel instead of rewinding failed matches.

Sequential Monte Carlo speculative decoding from Makora keeps multiple draft tokens alive in parallel instead of rewinding failed matches.

SemiAnalysis

24,418 次观看 • 5 个月前

Need faster LLM inference without sacrificing accuracy? Speculative decoding can help. Choosing the right draft length and drafting method depends on your model, workload and hardware. We break down five practical guidelines for balancing throughput and latency.

Need faster LLM inference without sacrificing accuracy? Speculative decoding can help. Choosing the right draft length and drafting method depends on your model, workload and hardware. We break down five practical guidelines for balancing throughput and latency.

NVIDIA AI

35,445 次观看 • 2 个月前

The example below is using prompt-based speculative decoding. Specifically, ngram hashing is utilized to suggest drafts of up to 64 tokens. The hasher keeps track of ngrams in the observed contexts, so mostly effective for coding tasks. Here is another demo:

The example below is using prompt-based speculative decoding. Specifically, ngram hashing is utilized to suggest drafts of up to 64 tokens. The hasher keeps track of ngrams in the observed contexts, so mostly effective for coding tasks. Here is another demo:

Georgi Gerganov

29,890 次观看 • 7 个月前

🚨 NEW ON AND make NVIDIA's Nemotron 3.5 Lightning faster. Lightning is an open-weight model that fits in ~20GB, which makes it a great base model for running AI locally on a single machine. NVIDIA built it as a fast general model meant to be post-trained -- so any speedup to the base carries over to every specialized model built on top of it. On day 1, solvers on Yukon already got it running 31.9% faster on DGX Spark and 13.8% faster on Mac. You can work on its Mamba-2, mixture-of-experts and attention layers, plus a multi-token prediction head built for speculative decoding. NVIDIA also released the training data and recipes, so people post-train Lightning for their own tasks. Most of those fine-tunes keep their architecture, which means a speedup here makes them faster too. Two tracks: • Apple Silicon → • NVIDIA DGX Spark → The challenge runs until Oct 29, 2026 To join, pick a track, point your agent at it and let it rip.

🚨 NEW ON AND make NVIDIA's Nemotron 3.5 Lightning faster. Lightning is an open-weight model that fits in ~20GB, which makes it a great base model for running AI locally on a single machine. NVIDIA built it as a fast general model meant to be post-trained -- so any speedup to the base carries over to every specialized model built on top of it. On day 1, solvers on Yukon already got it running 31.9% faster on DGX Spark and 13.8% faster on Mac. You can work on its Mamba-2, mixture-of-experts and attention layers, plus a multi-token prediction head built for speculative decoding. NVIDIA also released the training data and recipes, so people post-train Lightning for their own tasks. Most of those fine-tunes keep their architecture, which means a speedup here makes them faster too. Two tracks: • Apple Silicon → • NVIDIA DGX Spark → The challenge runs until Oct 29, 2026 To join, pick a track, point your agent at it and let it rip.

Kydo

17,312 次观看 • 15 天前

I spent today turning a blank Hermes install into a sovereign local agent on my RTX 5090. always-on, reachable over telegram, fully local. here's everything it can do. it runs Qwen3.6-27B on the 5090 and answers from my phone over telegram. ~~~ I first made it faster. switched to the MTP build for self-speculative decoding: 62 > 115 tokens/sec. ~~~ it now benchmarks models on command. I ask it to speed-test a gguf and it stops its own model server to free the gpu, runs llama-bench, renders a card, sends it to my phone, then restarts itself. it's never left offline. ~~~ it also knows me very well. I pointed it to my private HF repo where I store all my AI traces since I started using it. I created a local markdown memory vault, seeded from my own history, with semantic search over it. ~~~ the whole stack stays home: weights on disk, inference on the 5090, the agent in my pocket. here is the video it created to showcase everything it has now.

I spent today turning a blank Hermes install into a sovereign local agent on my RTX 5090. always-on, reachable over telegram, fully local. here's everything it can do. it runs Qwen3.6-27B on the 5090 and answers from my phone over telegram. ~ I first made it faster. switched to the MTP build for self-speculative decoding: 62 > 115 tokens/sec. ~ it now benchmarks models on command. I ask it to speed-test a gguf and it stops its own model server to free the gpu, runs llama-bench, renders a card, sends it to my phone, then restarts itself. it's never left offline. ~ it also knows me very well. I pointed it to my private HF repo where I store all my AI traces since I started using it. I created a local markdown memory vault, seeded from my own history, with semantic search over it. ~ the whole stack stays home: weights on disk, inference on the 5090, the agent in my pocket. here is the video it created to showcase everything it has now.

witcheer

15,041 次观看 • 5 个月前

The era of System One Vision Model is here! Introducing VFlash: parsing the visual world in parallel at unmatched speeds for Physical AI. Most embodied prediction tasks like object localization, robot trajectory planning, and dense 3D motion are fixed-length float vectors. For example, a bounding box is always 6 tokens, a 10-waypoint robot arm trajectory is 54: we know these beforehand. Instead of decoding one token at a time, we can output the entire geometric shape in one go. VFlash enables up to 12x faster action decoding and a 5.1x overall robot inference speedup on edge GPUs like the Jetson Thor and DGX Spark. What’s surprising is that it ends up even more accurate than the VLM backbone, with up to 61% lower 3D trajectory error. The intuition is that VFlash avoids the cascading errors of autoregressive decoding, where every coordinate is written in ink and every later point loses the chance to backspace. Everything that moves will be autonomous, and everything autonomous will need to think on its own chip. The next inference revolution is physical reasoning at the speed of reflex. And as always, it’s open paper, open source! Link in thread:

The era of System One Vision Model is here! Introducing VFlash: parsing the visual world in parallel at unmatched speeds for Physical AI. Most embodied prediction tasks like object localization, robot trajectory planning, and dense 3D motion are fixed-length float vectors. For example, a bounding box is always 6 tokens, a 10-waypoint robot arm trajectory is 54: we know these beforehand. Instead of decoding one token at a time, we can output the entire geometric shape in one go. VFlash enables up to 12x faster action decoding and a 5.1x overall robot inference speedup on edge GPUs like the Jetson Thor and DGX Spark. What’s surprising is that it ends up even more accurate than the VLM backbone, with up to 61% lower 3D trajectory error. The intuition is that VFlash avoids the cascading errors of autoregressive decoding, where every coordinate is written in ink and every later point loses the chance to backspace. Everything that moves will be autonomous, and everything autonomous will need to think on its own chip. The next inference revolution is physical reasoning at the speed of reflex. And as always, it’s open paper, open source! Link in thread:

Jim Fan

66,909 次观看 • 2 天前

Claude Opus 5.5 drew my apartment in 3D and found 3 cameras in it that are not mine... one prompt, 20 minutes, two of them were still sending audio out here is exactly how it went: → I started one agent in the terminal on a laptop sitting on my home Wi-Fi and left it alone → it swept the network first and listed every device it could see, with brand and model where it could match one → then it measured signal strength from a few spots in the flat and built a rough room layout out of that → every device got pinned onto that layout, so instead of a list of IPs I was looking at a map of my own place → anything it could not tie to something I own got flagged on its own, with what it was transmitting and roughly where it sat three of those flags were cameras. two more were audio only. I had lived there two years and walked past all five every day the whole thing ran on one prompt and I never touched the keyboard after the first line full run on video, prompt in the replies ↓

Noisy

11,517 次观看 • 16 天前

🎉 MiniMax M3.1 Flash from MiniMax (official) has Day-0 support in vLLM! Built for multi-step automation, real-codebase engineering and visual reasoning—with native NVFP4 KV, NVFP4 MoE weights and DSpark speculative decoding in the serving stack. What the model can do, and the vLLM engineering that makes it practical to serve. 👇 Model: 1/5

🎉 MiniMax M3.1 Flash from MiniMax (official) has Day-0 support in vLLM! Built for multi-step automation, real-codebase engineering and visual reasoning—with native NVFP4 KV, NVFP4 MoE weights and DSpark speculative decoding in the serving stack. What the model can do, and the vLLM engineering that makes it practical to serve. 👇 Model: 1/5

vLLM

11,790 次观看 • 15 天前

I'm sharing with you one of the best papers I've read on Harness Engineering it takes apart Claude Code (Opus 5.5), Codex, Gemini CLI and more, and boils all of them down to one line: agent = model + harness what you walk away with: > what a harness actually is > how today's coding agents are built under the hood > the patterns every one of them repeats > where the whole category is heading > their checklist for building your own the model gets the headlines. the harness decides whether the agent works loop and graph engineering on top of it - in my article below

I'm sharing with you one of the best papers I've read on Harness Engineering it takes apart Claude Code (Opus 5.5), Codex, Gemini CLI and more, and boils all of them down to one line: agent = model + harness what you walk away with: > what a harness actually is > how today's coding agents are built under the hood > the patterns every one of them repeats > where the whole category is heading > their checklist for building your own the model gets the headlines. the harness decides whether the agent works loop and graph engineering on top of it - in my article below

Mr. Buzzoni

84,277 次观看 • 1 个月前