Загрузка видео...

Не удалось загрузить видео

На главную

Big win for local LLM inference! This open-source engine generates tokens 2.7x faster than Ollama on a MacBook: I tested it on a base M5 MacBook Pro with Qwen3.5 4B at 4-bit and the same 9.2k-token prompt. Ollama ran its MTP build, so both had speculative decoding. - Prefill:...

akshay_pachaar's profile picture

Akshay 🚀

292,044 subscribers

34,322 просмотров • 1 день назад •via X (Twitter)

Наука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 24

Фото профиля Mihir Kolakaluri
Mihir Kolakaluri1 день назад

decode going from 22 to 60 tok/s is the difference between waiting on it and not even noticing it so i am curious if the gap stays this big on a bigger model like an 8b

Фото профиля Sven Meyer
Sven Meyer1 день назад

How does it compare to Strata ? 🤔

Фото профиля Hussain Hashim | Building SundayBack
Hussain Hashim | Building SundayBack1 день назад

@akshay_pachaar interesting result! curious how memory usage compared during those tests. I found that sometimes optimizing for speed can cause unexpected memory spikes.

Фото профиля Mohammadjavad (Javad) Maheronnaghsh
Mohammadjavad (Javad) Maheronnaghsh1 день назад

the future is local llm

Фото профиля SiEdge
SiEdge1 день назад

Tuning kernels on each machine instead of shipping one compromise build is a smart way to speed up local agents.

Фото профиля Lightchain AI
Lightchain AI1 день назад

did you notice any quality drop or is it the same output just faster?

Фото профиля Dusan 🐧 AI agents · dad
Dusan 🐧 AI agents · dad1 день назад

16 gb is tight for a model plus a long prompt. a light linux desktop idles at a few hundred mb, so nearly all of it goes to the model and the context 🐧

Фото профиля Ran Aroussi
Ran Aroussi1 день назад

60 tok/s from a 4B model on a base laptop changes the maths. Most of what an agent does all day is routine tool calls, not frontier reasoning. How much of that work could a small fine-tuned model you own handle before you ever need a paid API?

Фото профиля Saroj
Saroj1 день назад

Context from Magnitude's own benchmarks: averaged across models on an M4 Pro, it's 1.32x faster prefill and 2.19x faster decode than Ollama (MLX). So 2.7x is a strong single-model result; about 2x is the fairer number to expect. One-time tuning on download takes up to a minute.

Фото профиля Techboy | AI & Tech
Techboy | AI & Tech1 день назад

tuning kernels on your own chip at download is the clever part. the catch is it's one 4b model at 4-bit on a base m5, bigger models are the real test

Фото профиля Radik Zagirov
Radik Zagirov1 день назад

Side effect of per-chip autotuning worth knowing: different kernel configs change the floating-point reduction order, so logits differ slightly by machine. Under greedy decoding a near-tie flips one token and the rest diverges. Same eval, two Macs, possibly different transcripts.

Фото профиля Day
Day1 день назад

props for giving ollama its mtp build too. a 2.7x win over a handicapped baseline wouldn't mean much

Фото профиля Looply
Looply1 день назад

half the time per call, and an agent loop makes hundreds of calls

Фото профиля Oleg
Oleg1 день назад

how much of the 2.7x decode is the drafter, not the kernels?

Фото профиля Sikandar
Sikandar1 день назад

the end-to-end number is the useful one here. i'd love the same comparison on a small repo bugfix: read files, edit, run tests. that's the wait a solo builder actually feels.

Фото профиля Kalin | AI bills & gotchas
Kalin | AI bills & gotchas1 день назад

2.7x on a 4B, does it hold at 30B?

Фото профиля sir we are secular
sir we are secular1 день назад

decode 22.6 to 60 is the real story, prefill only moved 1.6x. curious how much comes from the drafter pairing vs the kernel tuning. if it's mostly the drafter, ollama can close a lot of that gap without per-machine compiles

Фото профиля Akash Hadagali
Akash Hadagali1 день назад

Running Ollama's MTP build so both sides had speculative decoding makes this a fairer comparison than most local speed charts.

Фото профиля John Rood
John Rood1 день назад

the 2.7x is per stream. put four agents on that same box and what decides whether it is usable is the memory bandwidth they all share, and tuning adds efficiency, not bandwidth. does the tuned kernel hold up with four different prefixes in flight?

Фото профиля austin
austin1 день назад

prefill going from 464 to 755 tok/s is the number that matters for agents, every tool output gets chewed through before the first token shows up. does the kernel tuning run once per model on download, and how long does it take on a base m5?

Фото профиля Diego Viteri
Diego Viteri1 день назад

kernels compiled for your exact chip. gentoo won

Фото профиля Kalyan
Kalyan1 день назад

Saving this - I'm on an 8 GB MacBook Air and slow local runs are why I almost bought a GPU for my agent setup. Does the speedup hold on 8 GB, or does a 4B model at 4-bit still swap?

Фото профиля Miloš van DotNET ✡︎
Miloš van DotNET ✡︎1 день назад

What's the draft acceptance rate on each side? With both running speculative decoding, a 2.7x gap usually lives in how often the draft tokens get accepted, not in raw kernel speed.

Фото профиля Buffer
Buffer1 день назад

decode going 22.6 to 60 tok/s on a base M5 is the wild part, prefill only got 1.6x. does it redo the local kernel tuning every time you pull a new model, and how long is that first compile?

Похожие видео

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 ÷ 5.2 ≈ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: ↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once. ↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. ↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: (don't forget to star it 🌟) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.
0:26

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 ÷ 5.2 ≈ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: ↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once. ↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. ↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: (don't forget to star it 🌟) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.

Avi Chawla

123,278 просмотров • 2 дней назад

Qwen3.8-27B at 144 tok/s on an M5 Max MacBook Pro ⚡ Meet Inco Splash: our open-source inference engine, built around the model and around Apple silicon. Up to 3× the decode speed of Ollama, 2× oMLX, and almost 4× when an agent fans out into sub-agents.
0:26

Qwen3.8-27B at 144 tok/s on an M5 Max MacBook Pro ⚡ Meet Inco Splash: our open-source inference engine, built around the model and around Apple silicon. Up to 3× the decode speed of Ollama, 2× oMLX, and almost 4× when an agent fans out into sub-agents.

Inco AI

1,008,472 просмотров • 1 месяц назад

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.
0:42

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

345,385 просмотров • 2 месяцев назад

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.
0:20

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.

FHILY👑

40,049 просмотров • 1 месяц назад

Introducing Magnitude: your actually local agent 100% private and offline. No token costs, no API keys. Open source. Today's agents are local. The model isn't. Every prompt, every file, every secret gets sent straight to Anthropic and OpenAI. Magnitude is built around local models and runs the whole stack itself. The inference engine is part of the agent, so the models run inside it, right on your computer. It lives in your terminal, and setup is one command. Magnitude profiles your hardware and shows you which models fit, with the trade-offs between quality, speed, and memory. Pick one and start working. No painful config or server to babysit. Out of the box, it can use your shell, edit files, and run scripts. Add skills and it can work with Excel, PowerPoint, PDFs or Chrome. Use it for everyday work: - Analyze sensitive data - Manage private notes - Review code and logs - Search and organize files - Build docs or slides npm i -g @magnitudedev/cli GitHub:
0:35

Introducing Magnitude: your actually local agent 100% private and offline. No token costs, no API keys. Open source. Today's agents are local. The model isn't. Every prompt, every file, every secret gets sent straight to Anthropic and OpenAI. Magnitude is built around local models and runs the whole stack itself. The inference engine is part of the agent, so the models run inside it, right on your computer. It lives in your terminal, and setup is one command. Magnitude profiles your hardware and shows you which models fit, with the trade-offs between quality, speed, and memory. Pick one and start working. No painful config or server to babysit. Out of the box, it can use your shell, edit files, and run scripts. Add skills and it can work with Excel, PowerPoint, PDFs or Chrome. Use it for everyday work: - Analyze sensitive data - Manage private notes - Review code and logs - Search and organize files - Build docs or slides npm i -g @magnitudedev/cli GitHub:

Tom Greenwald

398,275 просмотров • 2 месяцев назад

run agent harnesses 100% private & offline. (no token costs, no API keys, 100% open-source) your agent runs locally. the model doesn't. every prompt, every file, and every secret still leaves your machine before the agent does anything with it. Magnitude fixes that. it's an open source inference server that runs models on your own hardware and plugs into the coding agent you already use. setup is one command. it profiles your machine, measures the memory bandwidth that sets your token rate, and hands back complete configurations instead of a list of models. each one names a model, a compression level, a context size, and a speed range you can expect. pick one and start working. it doesn't replace your harness. setup asks which one you want and writes that config for you. Pi, OpenCode, Claude Code, Codex, and Cline all work, and there's a built-in one tuned for local models if you don't have a harness yet. that one uses your shell, edits files, and runs scripts out of the box. add skills and it handles Excel, PowerPoint, PDFs, or Chrome. everyday work it covers: → analyze sensitive data → manage private notes → review code and logs → search and organize files → build docs or slides Apache 2.0. no rate limits, and nothing leaves the machine. 𝗻𝗽𝗺 𝗶 -𝗴 @𝗺𝗮𝗴𝗻𝗶𝘁𝘂𝗱𝗲𝗱𝗲𝘃/𝗰𝗹𝗶 the repo is here: (don't forget to star 🌟) i wrote the full breakdown of why picking the configuration is the hard part. the article is quoted below.
0:38

run agent harnesses 100% private & offline. (no token costs, no API keys, 100% open-source) your agent runs locally. the model doesn't. every prompt, every file, and every secret still leaves your machine before the agent does anything with it. Magnitude fixes that. it's an open source inference server that runs models on your own hardware and plugs into the coding agent you already use. setup is one command. it profiles your machine, measures the memory bandwidth that sets your token rate, and hands back complete configurations instead of a list of models. each one names a model, a compression level, a context size, and a speed range you can expect. pick one and start working. it doesn't replace your harness. setup asks which one you want and writes that config for you. Pi, OpenCode, Claude Code, Codex, and Cline all work, and there's a built-in one tuned for local models if you don't have a harness yet. that one uses your shell, edits files, and runs scripts out of the box. add skills and it handles Excel, PowerPoint, PDFs, or Chrome. everyday work it covers: → analyze sensitive data → manage private notes → review code and logs → search and organize files → build docs or slides Apache 2.0. no rate limits, and nothing leaves the machine. 𝗻𝗽𝗺 𝗶 -𝗴 @𝗺𝗮𝗴𝗻𝗶𝘁𝘂𝗱𝗲𝗱𝗲𝘃/𝗰𝗹𝗶 the repo is here: (don't forget to star 🌟) i wrote the full breakdown of why picking the configuration is the hard part. the article is quoted below.

Akshay 🚀

56,499 просмотров • 1 месяц назад

underrated gems in Kimi-K3 release: > an early K3 wrote the majority of the kernels in the late development stages > it built a triton-class compiler from scratch, MiniTriton, that delivers performance on par with or better than Triton and torch.compile > then it designed a chip, by a model, for a model, in one 48-hour autonomous run the model is rewriting and optimizing every layer of the stack it runs on: kernels for its own training. A compiler for its own kernels. silicon for its own weights. we're watching software build its own hardware
0:22

underrated gems in Kimi-K3 release: > an early K3 wrote the majority of the kernels in the late development stages > it built a triton-class compiler from scratch, MiniTriton, that delivers performance on par with or better than Triton and torch.compile > then it designed a chip, by a model, for a model, in one 48-hour autonomous run the model is rewriting and optimizing every layer of the stack it runs on: kernels for its own training. A compiler for its own kernels. silicon for its own weights. we're watching software build its own hardware

Alex Ker 🔭

311,780 просмотров • 3 месяцев назад

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding. To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8. We are open sourcing it today. 1/2
1:25

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding. To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8. We are open sourcing it today. 1/2

Inferact

245,762 просмотров • 1 месяц назад

this is what 12gb of vram builds in 2026, absolute magic > rtx 3060 12gb, #1 gpu on steam > bonsai 2 27b + mtp, 5.95 gb of weights > hermes agent, 5 hours, 328k tokens written > 8 js files, 2,368 lines, zero hand written code > 50 tok/s fresh, 22 tok/s average, 125k context watch the full video, 5 hours in 12 minutes of pure dance of a local ai model on rtx 3060 12gb vram, and stay till the end for the full gameplay. this entire game was built by bonsai2, a qwen 3.8 27b dense compressed to ternary, and this small model is punching way above its weight. it built multi file engineering work using hermes agent, sure it's not fast but perfect for overnights and routine work and the quality is insane, and context holding is another best one, it does not lose the thread. i ran PrismML bonsai 1 made from qwen 3.6 27b dense and i built things with it, but this time with the latest base model qwen 3.8 these results are insane, and because i loved building with it a lot i thought many more of you would run it because this gpu exists in almost every home. so i packaged mtp, doubled the speed from 26 tok/s to 50 tok/s, packed the prefill fix in and released it on huggingface, almost 4,000 downloads in 3 days. i'll leave a link below.
15:03

this is what 12gb of vram builds in 2026, absolute magic > rtx 3060 12gb, #1 gpu on steam > bonsai 2 27b + mtp, 5.95 gb of weights > hermes agent, 5 hours, 328k tokens written > 8 js files, 2,368 lines, zero hand written code > 50 tok/s fresh, 22 tok/s average, 125k context watch the full video, 5 hours in 12 minutes of pure dance of a local ai model on rtx 3060 12gb vram, and stay till the end for the full gameplay. this entire game was built by bonsai2, a qwen 3.8 27b dense compressed to ternary, and this small model is punching way above its weight. it built multi file engineering work using hermes agent, sure it's not fast but perfect for overnights and routine work and the quality is insane, and context holding is another best one, it does not lose the thread. i ran PrismML bonsai 1 made from qwen 3.6 27b dense and i built things with it, but this time with the latest base model qwen 3.8 these results are insane, and because i loved building with it a lot i thought many more of you would run it because this gpu exists in almost every home. so i packaged mtp, doubled the speed from 26 tok/s to 50 tok/s, packed the prefill fix in and released it on huggingface, almost 4,000 downloads in 3 days. i'll leave a link below.

Sudo su

92,700 просмотров • 1 месяц назад

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?
0:26

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Avi Chawla

158,352 просмотров • 5 месяцев назад

BENCHMARKED: LFM3-1.7B by Liquid AI. Liquid AI published 1.6 GB for the KV cache at 128K context. I ran it on my Mac mini and measured 1.6 GB. The whole model at 128K uses 3.15 GB of GPU memory, under 5% of the 64GB. It still generates 80 tok/s. Only 6 of its 24 layers keep a full KV cache. The other 18 use short convolutions or sliding-window attention. llama.cpp (Metal), Q4_K_M, M5 Pro 64GB, median of 3 runs - 512 ctx: 184 tok/s, 0.09s to first token - 4K ctx: 178 tok/s, 0.72s to first token - 32K ctx: 141 tok/s - 128K ctx: 80 tok/s Q8_0 does 125 tok/s at 512, so 4-bit is 47% faster.
0:20

BENCHMARKED: LFM3-1.7B by Liquid AI. Liquid AI published 1.6 GB for the KV cache at 128K context. I ran it on my Mac mini and measured 1.6 GB. The whole model at 128K uses 3.15 GB of GPU memory, under 5% of the 64GB. It still generates 80 tok/s. Only 6 of its 24 layers keep a full KV cache. The other 18 use short convolutions or sliding-window attention. llama.cpp (Metal), Q4_K_M, M5 Pro 64GB, median of 3 runs - 512 ctx: 184 tok/s, 0.09s to first token - 4K ctx: 178 tok/s, 0.72s to first token - 32K ctx: 141 tok/s - 128K ctx: 80 tok/s Q8_0 does 125 tok/s at 512, so 4-bit is 47% faster.

Pratik Gandhi

33,212 просмотров • 2 дней назад

Agent Zero crushed Codex by 2.6× on token efficiency! We gave both agents the same task on the same local model (Qwen3.6 27B): build a retro Q*bert arcade game. Each agent wrote and rewrote it across iterations, adding enemies and a HUD. Outputs: • Agent Zero: 239K tokens, 53 min • Codex: 627K tokens, 51 min Both agents built a working Q*bert, but Gitlawb 's Agent Zero is the cleaner build, with crisper textures and hops that land where they should. Codex refills the context window with its whole history every turn, so the model loses the thread and hallucinates code it never wrote. Agent Zero sends only what's new, so it keeps more of the window open and makes fewer mistakes. Run agents on local AI models in Atomic Chat!
0:25

Agent Zero crushed Codex by 2.6× on token efficiency! We gave both agents the same task on the same local model (Qwen3.6 27B): build a retro Q*bert arcade game. Each agent wrote and rewrote it across iterations, adding enemies and a HUD. Outputs: • Agent Zero: 239K tokens, 53 min • Codex: 627K tokens, 51 min Both agents built a working Q*bert, but Gitlawb 's Agent Zero is the cleaner build, with crisper textures and hops that land where they should. Codex refills the context window with its whole history every turn, so the model loses the thread and hallucinates code it never wrote. Agent Zero sends only what's new, so it keeps more of the window open and makes fewer mistakes. Run agents on local AI models in Atomic Chat!

atomic.chat

320,574 просмотров • 3 месяцев назад

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.
1:24

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.

stevibe

404,026 просмотров • 6 месяцев назад

GLM-5.3-Flash on two DGX Sparks built this voxel sky kingdom in 71.1 min, on Mia's recipe and the TensorFold Python engine. Mia's recipe at commit 3fd60dfc: EXL3 4 bpw quant, one model split across both Sparks (TP2), DFlash2 draft model, 8 slots (her recipe allows up to 8; the default on two Sparks is 4). Agent: OpenCode 1.19.3. One prompt, one try, no help. Thinking on, sampling from the model card: temperature 1.0, top_p 0.95. Voxel world: done in 71.1 min Tokens written: 109,305 Browser check: 9/10 Decode, code: 51.2 tok/s (separate speed run, thinking off) Decode, prose: 48.6 tok/s 500 words: about 14.0 s Eight requests at once: 99.2 tok/s in total Context: 1,048,576 tokens, the recipe's setting Aetherfall: floating isles over a cloud sea with a sky whale and its calf, a waterfall, a windmill, a castle, a shrine, airships and its own camera tour. Open: parts of the world are cut off hard at the edges, most likely a bug in the generated code. Answer limit in the build test: 65,536 tokens, the figure names for its Terminal-Bench runs. Its model card names up to 163,840 for another test. The longest answer was 54,782 tokens, so none was cut. Before this I ran the same task on Mia's Zig-engine branch, which her docs call work in progress: Decode, code: 26.6 tok/s Decode, prose: 25.5 tok/s Voxel world: done in 98.8 min Browser check: 8.0/10 The Python engine is the one her main recipe runs. Play the world in your browser. Every number, the exact command and the raw output: Recipe and quant by Mia. TensorFold engine by Ash Hart. GLM-5.3-Flash by Z.ai. DFlash2 draft by Inco AI.
0:42

GLM-5.3-Flash on two DGX Sparks built this voxel sky kingdom in 71.1 min, on Mia's recipe and the TensorFold Python engine. Mia's recipe at commit 3fd60dfc: EXL3 4 bpw quant, one model split across both Sparks (TP2), DFlash2 draft model, 8 slots (her recipe allows up to 8; the default on two Sparks is 4). Agent: OpenCode 1.19.3. One prompt, one try, no help. Thinking on, sampling from the model card: temperature 1.0, top_p 0.95. Voxel world: done in 71.1 min Tokens written: 109,305 Browser check: 9/10 Decode, code: 51.2 tok/s (separate speed run, thinking off) Decode, prose: 48.6 tok/s 500 words: about 14.0 s Eight requests at once: 99.2 tok/s in total Context: 1,048,576 tokens, the recipe's setting Aetherfall: floating isles over a cloud sea with a sky whale and its calf, a waterfall, a windmill, a castle, a shrine, airships and its own camera tour. Open: parts of the world are cut off hard at the edges, most likely a bug in the generated code. Answer limit in the build test: 65,536 tokens, the figure names for its Terminal-Bench runs. Its model card names up to 163,840 for another test. The longest answer was 54,782 tokens, so none was cut. Before this I ran the same task on Mia's Zig-engine branch, which her docs call work in progress: Decode, code: 26.6 tok/s Decode, prose: 25.5 tok/s Voxel world: done in 98.8 min Browser check: 8.0/10 The Python engine is the one her main recipe runs. Play the world in your browser. Every number, the exact command and the raw output: Recipe and quant by Mia. TensorFold engine by Ash Hart. GLM-5.3-Flash by Z.ai. DFlash2 draft by Inco AI.

Niklas

16,730 просмотров • 5 дней назад

Okay, this one is sneaky good. Hermes can now run every subagent on a different model. Advanced Delegation is an official Nous Research plugin. You set up named model routes, like fast and deep, and Hermes sends each piece of work down the route that fits. ▸ A fast, cheap model for lookups and summaries ▸ Your strongest model for the hard thinking ▸ A local model on your own endpoint ▸ Or another Hermes profile, with its own model, skills and SOULmd Every result shows the route and model it actually ran on. And Hermes only sees the names and descriptions you write, so it can't invent a route. One agent, a whole bench of models. Setup in the reply.
0:26

Okay, this one is sneaky good. Hermes can now run every subagent on a different model. Advanced Delegation is an official Nous Research plugin. You set up named model routes, like fast and deep, and Hermes sends each piece of work down the route that fits. ▸ A fast, cheap model for lookups and summaries ▸ Your strongest model for the hard thinking ▸ A local model on your own endpoint ▸ Or another Hermes profile, with its own model, skills and SOULmd Every result shows the route and model it actually ran on. And Hermes only sees the names and descriptions you write, so it can't invent a route. One agent, a whole bench of models. Setup in the reply.

Hermes Release Watch

26,677 просмотров • 6 дней назад

DFlash makes Qwen 2.2x faster with no quality loss! We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks only differ in one thing - how predictable the next word is: quicksort, describe a file in JSON, a logic puzzle, a sci-fi story. Outputs: Baseline: 44 tok/s · 1.00x MTP: 65 tok/s · 1.45x · 71% accepted DFlash: 98 tok/s · 2.20x · 30% accepted Baseline writes one token per step. MTP works inside the model itself and guesses 3 tokens ahead. DFlash is a separate small model that writes 15 tokens at once, and the big model only checks them. In JSON the same words repeat all the time, so most guesses were right: 152 tok/s, 3.4x speedup. In the story 9 guesses out of 10 were wrong. DFlash did all that extra work for nothing and became slower than baseline: 42 vs 44 tok/s. MTP guesses only 3 tokens, so a wrong guess costs very little: 46 tok/s and the win in that round. The output is identical in all three modes - DFlash is the pick for tasks with predictable output, like coding, and for chat and creative writing MTP works better. DFlash is now natively integrated into Atomic Chat on llama.cpp - speed up your Qwen models!
0:26

DFlash makes Qwen 2.2x faster with no quality loss! We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks only differ in one thing - how predictable the next word is: quicksort, describe a file in JSON, a logic puzzle, a sci-fi story. Outputs: Baseline: 44 tok/s · 1.00x MTP: 65 tok/s · 1.45x · 71% accepted DFlash: 98 tok/s · 2.20x · 30% accepted Baseline writes one token per step. MTP works inside the model itself and guesses 3 tokens ahead. DFlash is a separate small model that writes 15 tokens at once, and the big model only checks them. In JSON the same words repeat all the time, so most guesses were right: 152 tok/s, 3.4x speedup. In the story 9 guesses out of 10 were wrong. DFlash did all that extra work for nothing and became slower than baseline: 42 vs 44 tok/s. MTP guesses only 3 tokens, so a wrong guess costs very little: 46 tok/s and the win in that round. The output is identical in all three modes - DFlash is the pick for tasks with predictable output, like coding, and for chat and creative writing MTP works better. DFlash is now natively integrated into Atomic Chat on llama.cpp - speed up your Qwen models!

atomic.chat

83,237 просмотров • 3 месяцев назад

Qwen 3.8 Flash Next (125B-A6B) now runs natively in mlx-serve on Apple Silicon. No Python, Zig + Metal. (Preview Release) M4 Max, 4-bit pack (~75 GB resident): - ~70 tok/s serial, ~98 tok/s with MTP when tested using `npx llmprobe` - prefill ~700 tok/s, sparse attention past 2k tokens with fused kernels - images and video input supported Hyper-connections, n-gram embeddings gathered straight from disk, QSA block selection, 5 new Metal kernels. All open source, parity-tested against the HF reference. Kernel work: - hyper-connection read: 3 Metal kernels, split-K, write folded into the next read - GDN decode: 2 fused dispatches per layer - hd-256 attention: mask arm, sparse prefill + verify stay fused - all greedy-equivalent to the unfused path If you port any of this to your engine, a mention would be nice. MLX-Serve Preview 26.8.11 build: 4bit Speed Pack: 4-8bit Quality Pack: As for the model, i'm not sure yet, it feels verbose, and Opus like. Thinks a lot. Goes on long self testing loops/talking with itself, but need more time with it. It seems like a good model for agentic things.
0:25

Qwen 3.8 Flash Next (125B-A6B) now runs natively in mlx-serve on Apple Silicon. No Python, Zig + Metal. (Preview Release) M4 Max, 4-bit pack (~75 GB resident): - ~70 tok/s serial, ~98 tok/s with MTP when tested using `npx llmprobe` - prefill ~700 tok/s, sparse attention past 2k tokens with fused kernels - images and video input supported Hyper-connections, n-gram embeddings gathered straight from disk, QSA block selection, 5 new Metal kernels. All open source, parity-tested against the HF reference. Kernel work: - hyper-connection read: 3 Metal kernels, split-K, write folded into the next read - GDN decode: 2 fused dispatches per layer - hd-256 attention: mask arm, sparse prefill + verify stay fused - all greedy-equivalent to the unfused path If you port any of this to your engine, a mention would be nice. MLX-Serve Preview 26.8.11 build: 4bit Speed Pack: 4-8bit Quality Pack: As for the model, i'm not sure yet, it feels verbose, and Opus like. Thinks a lot. Goes on long self testing loops/talking with itself, but need more time with it. It seems like a good model for agentic things.

David Dalcu

122,696 просмотров • 2 месяцев назад

Splash is here! A month ago, DFlash 2 ran Qwen3.8-27B at 70 tok/s on an M5 Max. Today, Splash runs it at 144 tok/s on the same Mac ⚡ Getting that speed used to take work. Splash is built around the model, with the DFlash 2 draft already in it. Two commands and you're running.
0:26

Splash is here! A month ago, DFlash 2 ran Qwen3.8-27B at 70 tok/s on an M5 Max. Today, Splash runs it at 144 tok/s on the same Mac ⚡ Getting that speed used to take work. Splash is built around the model, with the DFlash 2 draft already in it. Two commands and you're running.

Zhijian Liu

95,415 просмотров • 1 месяц назад

WHAT IS AN AI "SOFTWARE FACTORY" AND IS IT HYPE (31 MINUTE BREAKDOWN) I think it's a silly name for a genuinely USEFUL idea! A software factory is 5-6 markdown files that sit next to your code and tell your agents how you like to work, so you can build high quality apps 24/7. It's going viral because AI coding has a trust problem. The model can build the feature, but with no structure around it you end up babysitting the agent, wondering what changed and hoping it didn't break something important. So you build with agents the same way a factory builds physical products! 1. Each feature gets its own station, which in software means its own branch, so multiple agents can work at the same time without stepping on each other. 2. The build station gives the agent rules for how to write the code, because "it works" is very different from "a developer could open this repo next month and understand what happened." 3. The proof station makes the agent show evidence. Screenshots, videos, speed numbers, before-and-after states. It has to prove the thing works instead of saying it works. 4. The review station runs the work through a code review agent, and if it doesn't clear the bar, it goes back through the line. 5. Then you show up at the end to merge. For a 100+ years people have run production this way, and it worked because the structure is good. The full episode on what’s a software factory is NOW live on The Startup Ideas Podcast (SIP) 🧃 with the wonderful Micky Watch: So is it hype?!? I don't think it is, because of what it does to your output! WITHOUT a factory, you build ONE feature at a time and you're the bottleneck at every step, prompting, checking the diff, testing it yourself, hoping nothing else broke (spoiler alert it often does). WITH a factory, EACH feature runs in its own isolated copy of the app, so you can have 10+ of them going at once, and each agent has to prove its own work and pass a code review before it ever reaches you. Instead of supervising the work, you're APPROVING finished work that already has evidence attached. REALLY interesting to see how work with agents is evolving to be….well, similar to working with people!
31:29

WHAT IS AN AI "SOFTWARE FACTORY" AND IS IT HYPE (31 MINUTE BREAKDOWN) I think it's a silly name for a genuinely USEFUL idea! A software factory is 5-6 markdown files that sit next to your code and tell your agents how you like to work, so you can build high quality apps 24/7. It's going viral because AI coding has a trust problem. The model can build the feature, but with no structure around it you end up babysitting the agent, wondering what changed and hoping it didn't break something important. So you build with agents the same way a factory builds physical products! 1. Each feature gets its own station, which in software means its own branch, so multiple agents can work at the same time without stepping on each other. 2. The build station gives the agent rules for how to write the code, because "it works" is very different from "a developer could open this repo next month and understand what happened." 3. The proof station makes the agent show evidence. Screenshots, videos, speed numbers, before-and-after states. It has to prove the thing works instead of saying it works. 4. The review station runs the work through a code review agent, and if it doesn't clear the bar, it goes back through the line. 5. Then you show up at the end to merge. For a 100+ years people have run production this way, and it worked because the structure is good. The full episode on what’s a software factory is NOW live on The Startup Ideas Podcast (SIP) 🧃 with the wonderful Micky Watch: So is it hype?!? I don't think it is, because of what it does to your output! WITHOUT a factory, you build ONE feature at a time and you're the bottleneck at every step, prompting, checking the diff, testing it yourself, hoping nothing else broke (spoiler alert it often does). WITH a factory, EACH feature runs in its own isolated copy of the app, so you can have 10+ of them going at once, and each agent has to prove its own work and pass a code review before it ever reaches you. Instead of supervising the work, you're APPROVING finished work that already has evidence attached. REALLY interesting to see how work with agents is evolving to be….well, similar to working with people!

GREG ISENBERG

32,056 просмотров • 1 месяц назад

I had to test it myself to believe this unreal inference speed. 3,000 tokens/s for 1 user on standard datacenter GPUs. They leveraged a hidden efficiency gap in how GPUs generate tokens. Kog just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). Their tech preview is on a 2B model, and they show how their techniques will scale to large frontier MoE models at similar speeds. That's a huge number because normal low-batch GPU decoding for 2B to 8B models is usually closer to 100 to 300 tokens/s per request, so Kog is claiming something like a 10X to 30X jump in the speed one user actually feels. Their trick: they are getting the speed by treating LLM decoding as a memory streaming problem, not mainly a math problem. For 1 user at batch size 1, the GPU is not doing big, efficient matrix-matrix work like in training or large-batch serving; it is repeatedly pulling the model’s active weights from high-bandwidth memory for each new token, so speed depends on how smoothly those weights keep flowing. Normal inference stacks keep breaking that flow. They run many separate GPU programs for different parts of the model, move intermediate results through memory, wait at synchronization points, talk back to the CPU for scheduling or sampling, and then repeat this token after token. Kog’s answer is to co-design 3 things that are usually tuned separately: the runtime, the low-level GPU code, and the model architecture. The biggest engineering move is the monokernel, where the whole decode pass runs as 1 persistent GPU-resident program, including sampling, so the system does not keep stopping for kernel launches, CPU scheduling, and intermediate memory round trips. They also rebuilt synchronization, because their own measurements say grid sync was eating around 35% of token-generation time; instead of making every compute unit wait at a broad barrier, each unit waits only for the exact data it needs. On AMD MI300X, they also map memory access around the chiplet layout, because memory latency changes depending on which die makes the request. Then their Laneformer model uses Delayed Tensor Parallelism, which lets cross-GPU communication happen in the background instead of blocking every layer.
1:08

I had to test it myself to believe this unreal inference speed. 3,000 tokens/s for 1 user on standard datacenter GPUs. They leveraged a hidden efficiency gap in how GPUs generate tokens. Kog just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). Their tech preview is on a 2B model, and they show how their techniques will scale to large frontier MoE models at similar speeds. That's a huge number because normal low-batch GPU decoding for 2B to 8B models is usually closer to 100 to 300 tokens/s per request, so Kog is claiming something like a 10X to 30X jump in the speed one user actually feels. Their trick: they are getting the speed by treating LLM decoding as a memory streaming problem, not mainly a math problem. For 1 user at batch size 1, the GPU is not doing big, efficient matrix-matrix work like in training or large-batch serving; it is repeatedly pulling the model’s active weights from high-bandwidth memory for each new token, so speed depends on how smoothly those weights keep flowing. Normal inference stacks keep breaking that flow. They run many separate GPU programs for different parts of the model, move intermediate results through memory, wait at synchronization points, talk back to the CPU for scheduling or sampling, and then repeat this token after token. Kog’s answer is to co-design 3 things that are usually tuned separately: the runtime, the low-level GPU code, and the model architecture. The biggest engineering move is the monokernel, where the whole decode pass runs as 1 persistent GPU-resident program, including sampling, so the system does not keep stopping for kernel launches, CPU scheduling, and intermediate memory round trips. They also rebuilt synchronization, because their own measurements say grid sync was eating around 35% of token-generation time; instead of making every compute unit wait at a broad barrier, each unit waits only for the exact data it needs. On AMD MI300X, they also map memory access around the chiplet layout, because memory latency changes depending on which die makes the request. Then their Laneformer model uses Delayed Tensor Parallelism, which lets cross-GPU communication happen in the background instead of blocking every layer.

Rohan Paul

13,282 просмотров • 5 месяцев назад