Two models. One DGX Spark One identical voxel Eiffel... Tower prompt. Qwen3.8-27B xhigh: 83K tokens, 41m29s, 33.4 tok/s Ornith 1.5 35B-A3B: 47K tokens, 10m35s, 74.0 tok/s Thinking on, no token ceiling, matched camera. Ornith finished much faster; Qwen spent far more tokens.show more

Ornith-1.0 35B - MLX Device - 128gb m5 max... mbp Harness - PocketAI Hub 4 bit - 11,504 tokens, 95.98 tok/s, 21.20 GB peak RAM 8 bit - 13,128 tokens, 64.96 tok/s, 38.92 GB peak RAM BF16 - 12,491 tokens, 47.72 tok/s, 70.59 GB peak RAM prompt and link to model belowshow more

Day-0 Qwen3.8-27B vs Qwen3.6-27B: the voxel pagoda test. Same... prompt, one attempt each, single DGX Spark, both NVFP4. 3.8: 75,291 tok · 52m 56s · 23.7 tok/s (vLLM, native MTP) 3.6: 12,117 tok · 5m 13s · 38.8 tok/s (llama.cpp, DFlash K10) 3.8 thought for 160K characters before writing a line of code.show more

Local showdown: Qwen vs Deepseek fireworks one-shot: "build a... fireworks show over a city, single HTML file, no libraries." two locals on DGX Sparks, temp 0.6, one attempt, no edits. DeepSeek-V4-Flash (2x GB10): 1m20s, 5,214 tokens, Qwen3.8-27B NVFP4 (1x GB10, xhigh): thought for 21 minutes before typing one character of the answer. 38,269 tokens total, ~29K of them thinkingshow more

Compared Qwen3.6 35B and 27B in the same conditions... with Google TurboQuant Device: MacBook Pro M5Max 64GB RAM Outputs characteristics: Qwen3.6 35B: 6672 tokens, 2m 10s, 65 tok/s Qwen3.6 27B: 7344 tokens, 5m 22s, 24 tok/s Conclusion: Both models were asked to draw waves using HTML, 35B responded quickly but the result feels weak and messy, while 27B took more time and delivered a much cleaner and more consistent result, because it is built for thinking and planning, so it works better on tasks that need structure, overall 27B is a better choice for tasks where planning matters, while 35B is more suitable for everyday use when you just need a fast responseshow more

I think the CMP 170HX just became one of... the weirdest local-LLM value plays. 👀 I got Qwen3.8-27B W8A16 running on a single 170HX 64GB with vLLM 0.27.1 + DFlash2: ⚡ ~120 tok/s decode 🚀 ~1.6–1.9K tok/s prefill 🧠 262K context with BF16 KV 📦 342K-token KV capacity 🎯 DFlash2 averaging 3.66 tokens/draft 💾 60.5 / 63.5 GiB VRAM used 27B model is doing ~120 tokens/sec on one GPU. Software optimization is getting ridiculous. Those numbers are from your measured W8A16 + DFlash2 run: ~120 tok/s decode, ~1,580–1,940 tok/s prefill, 342,729-token KV capacity, and 60.5/63.5 GiB VRAM usage. #170hx #LocalAI #localllmshow more

One task, two models. GLM 5.3 Flash running slower... at 25 tok/s prose Qwen3.8 Flash running faster at 47 tok/s prose Who will finish the task first?show more

Laguna XS 2.1 just matched Qwen 3.6 35B on... a real coding task & ran twice as fast. > Same RTX 3090 > Laguna: 158 tok/s. Qwen: 81 tok/s The local model race isn't about size anymore. It's about who moves faster.show more

GEMMA 4 FINISHED 14 MINUTES FASTER THAN QWEN 3.6... DESPITE LOWER TOKENS PER SECOND BY USING 5X FEWER TOKENSshow more

167 tok/s on a single RTX 4090. FreeToken just... pushed Qwen3.6-35B-A3B NVFP4 into ridiculous territory. → RTX 4090 24GB → 35B total parameters → ~3B active per token → NVFP4 → 167 tok/s decode → 23.55GB peak VRAM → No speculative decoding → No MTP → No draft model That last part is what makes this interesting. The speed isn’t coming from guessing future tokens. FreeToken is exploiting the MoE architecture directly, keeping the active experts on the GPU while using bandwidth-aware execution for the rest. A 35B model with only ~3B active parameters running at 167 tok/s on a consumer 4090 is exactly why MoE models are becoming so compelling for local AI.show more

454 tok/s on Qwen3.8 27B on single RTX 6000... with +6000 mem overclock Mia’s Qwen3.8-27B NVFP4 W4A4 • SGLang + DFlash2 block-16 drafter • FP8 KVshow more

Qwen3.8 27B is now available on Cerebras Shared Tier... at ~1500 tok/s. 🚀 It's so fast that the time I spent on writing the prompt was longer than the time it spent to summarize my codebase.show more

Achieving 3,000 tokens/s with 16-parallel inference of Qwen3.6 35B-A3B... NVFP4 on a single RTX PRO 6000 WorkStation Edition. I am using ParallelHue—a tool I am developing—to color-code the tokens generated simultaneously via speculative decoding.show more

Grok-1 support in gpt-fast at faster(?) than anyone else... has reported so far. 75 tok/s for a 300B+ parameter model on an 8xA100 node. If I understand correctly, ColossalAI reported 15 seconds to generate 100 tokens. gpt-fast takes 4.2 seconds to generate *400* tokens.show more

4 DGX Sparks running DeepSeek V4.1 Flash 6 concurrent... sessions, 170.73 tok/s coding throughput. 🔥 1,200 output tokens in 7.028 seconds. Warm single-stream: 💻 Code: 77.61 tok/s 🧮 Math: 74.08 🧠 Reasoning: 48.75 ✍️ Prose: 33.02 6-stream aggregate: 💻 Code: 170.73 tok/s 🔢 Counting: 291.64 tok/s 552B MoE · MXFP4 · TP4 · vLLM + DSpark + CUDA graphs ✅ Tool calling + vision ✅ 127,480-token retrieval test passed ✅ 300K configured context Video replays the actual recorded streams at 1× speed. Thinking OFF, 200-token cap per coding request. Short-prompt coding throughput Credit to Tech2Wild for the Boot9 recipe. NVIDIA AIshow more

/1 Gemma 4 31B just crushed Qwen 3.6 27B... in a local LLM gamedev contest inside atomic.chat (prompt is below) Device: MacBook Pro M5 Max, 64GB RAM Results: Qwen 3.6 27B: 32 tokens/sec · 18m 04s · 33,946 tokens Gemma 4 31B: 27 tokens/sec · 3m 51s · 6,209 tokens So what is more important: tokens per second, or the quality of the final answer? Qwen made a very long response and showed more creativity and visual style. But Gemma gave a shorter, clearer, and more logical answer in much less time. In this one-shot Pac-Man gamedev contest, Gemma 4 31B was the clear winner. Its game logic was stronger: click reactions were smoother, and it handled interactions with elements like walls, ghosts, and particle effects better. But this was only one test. Maybe Qwen 3.6 27B can show better results with better settings. Open the comments, try our prompt, and share your result below.show more

Qwen3.8-Flash-Next now reaches ~43 tok/s after a 122,902-token prompt... on ONE DGX Spark. ⚡🚀 MTP k=2 won my draft-depth sweep, with +42.5% mean decode over no draft. The PLE table stays fully on-device. I promised the deeper MTP tests. Here are the results, and now you can explore them in an interactive benchmark page too. 𝗧𝗪𝗢 𝗗𝗥𝗔𝗙𝗧 𝗧𝗢𝗞𝗘𝗡𝗦 𝗪𝗢𝗡 Mean single-request decode with 32K context configured: MTP k=2: 39.21 tok/s MTP k=3: 36.42 tok/s MTP k=1: 35.18 tok/s No draft: 27.51 tok/s k=2 also produced the fastest individual sweep run: 41.34 tok/s. Four runs each for no draft, k=1 and k=2. Seven for k=3. Decode excludes time to first token. Here, k means speculative draft depth, not quantization bits. k=3 produced more tokens per step, but the extra drafting work did not pay off in throughput. k=2 is my current pick for this setup. 𝗧𝗛𝗘 𝟭𝟮𝟯𝗞-𝗧𝗢𝗞𝗘𝗡 𝗣𝗥𝗢𝗠𝗣𝗧 𝗧𝗘𝗦𝗧 I then ran a separate long-prompt comparison: Actual input: 122,902 tokens Configured context: 262,144 Requested output: 128 tokens One request at a time MTP k=2: ~43 tok/s No draft: 26.2 tok/s Time to first token: 110.6 seconds with MTP 107.0 seconds without it The win here is generation speed, not faster prefill. To keep the scope clear: 256K was the configured limit. This was a real ~123K input, not a completely filled 256K window or a full k sweep at that depth. 𝗣𝗟𝗘 𝗦𝗧𝗔𝗬𝗦 𝗢𝗡 𝗧𝗛𝗘 𝗦𝗣𝗔𝗥𝗞 Whole model on-device: 78.57 GiB Packed 5-bit PLE table: 30.4 GiB, included in that total No NVMe PLE offload in this build. This is still turboderp’s 3.05bpw_h5_ng5 EXL3 pack, served through my vllm-exl3 integration. My work here is the serving integration and testing. These are preliminary performance measurements, not a quality evaluation or a claim of bit-exact full-output parity. 𝗘𝗫𝗣𝗟𝗢𝗥𝗘 𝗧𝗛𝗘 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 The benchmark page has the individual sweep values, long-prompt comparison, and measurement scope. You can play the animation, export the charts, or download the HTML and data to render them yourself. No Spark needed to view the results. Credit to turboderp / ExLlamaV3 for the pack and kernels, vLLM for the serving engine, and Qwen Qwen Developers for the model. Recipe + reproduction: Interactive benchmark:show more

MTP speedup Qwen by 2.5x in Atomic Chat Dense... vs MoE models on 2x RTX 5090 Qwen3.6 27B: 51 → 117 tps +137% Qwen3.6 35B-A3B: 218 → 267 tps +25% MTP drafts several tokens ahead and verifies them in one pass. The speedup depends on memory moved per pass. Dense 27B reads all 27B params per token, MoE 35B-A3B only reads 3B active. Dense had way more to save by batching. The baseline tps also differ (218 vs 51) for the same reason from the other side. Token generation is memory-bandwidth bound, and MoE moves ~8x less memory per token, so its baseline is already 4x ahead. ~80% draft acceptance. Zero accuracy loss. ~1 GB extra VRAM. Open-source code and local AI app – in the comments 👇show more

This is the worst model I have ever tested,... even 9b models outperform it. What even is this pagoda? Same prompt I always use. - Total time: 9 min 21 s (560.9s) - Tokens: 6,891 completion (710 reasoning + ~6,181 answer) | 84 prompt - Speed: ~12.3 tok/s end-to-end - First token: 0.15s - Output: 22,094 chars answer / 17.2KB HTML, natural stop (finish_reason=stop, no cap hit)show more

5 days ago it took 2 GPUs to build... this. today it takes 1. same prompt. same particle simulation. completely different model. Qwen-Coder-Next (80B) on 2x 3090s. 46 tok/s. 564 lines. 2 iterations to get it working. 48GB VRAM across two cards just to hold it. Qwen3.5-35B-A3B on a single 3090. 112 tok/s. 461 lines. first try. cleaner code, fewer lines, better structured. 19.7GB on disk with 4GB VRAM to spare. half the parameters. one GPU instead of two. 2.4x faster. and the output actually improved. this is what happens when architecture catches up to ambition. Gated Delta Networks(Mamba2 variant) hybrid with sparse MoE. 3B active params out of 35B per token. efficiency at the architecture level, not just quantization. the curve isn't flattening. it's steepening.show more

Bonsai-27B ships with a little friend: a dspark drafter... checkpoint 🌱⚡ Same model, same prompt, one L40S GPU: 72 → 135 tok/s Flip it on and go zoom 🏎️💨show more