Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

4 DGX Sparks running DeepSeek V4.1 Flash 6 concurrent sessions, 170.73 tok/s coding throughput. 🔥 1,200 output tokens in 7.028 seconds. Warm single-stream: 💻 Code: 77.61 tok/s 🧮 Math: 74.08 🧠 Reasoning: 48.75 ✍️ Prose: 33.02 6-stream aggregate: 💻 Code: 170.73 tok/s 🔢 Counting: 291.64 tok/s 552B MoE ·...

WescheNex1q's profile picture

Wësche

2,878 subscribers

24,282 Aufrufe • vor 1 Monat •via X (Twitter)

Wissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Okay, I guess I didn't waste all that money after all Deepseek V4.1 Flash 304 tok/s code, 135 tok/s prose Zero optimization, will have it max optimized by the morning.
0:17

Okay, I guess I didn't waste all that money after all Deepseek V4.1 Flash 304 tok/s code, 135 tok/s prose Zero optimization, will have it max optimized by the morning.

Killy

44,653 Aufrufe • vor 1 Monat

Day-0 Qwen3.8-27B vs Qwen3.6-27B: the voxel pagoda test. Same prompt, one attempt each, single DGX Spark, both NVFP4. 3.8: 75,291 tok · 52m 56s · 23.7 tok/s (vLLM, native MTP) 3.6: 12,117 tok · 5m 13s · 38.8 tok/s (llama.cpp, DFlash K10) 3.8 thought for 160K characters before writing a line of code.
0:15

Day-0 Qwen3.8-27B vs Qwen3.6-27B: the voxel pagoda test. Same prompt, one attempt each, single DGX Spark, both NVFP4. 3.8: 75,291 tok · 52m 56s · 23.7 tok/s (vLLM, native MTP) 3.6: 12,117 tok · 5m 13s · 38.8 tok/s (llama.cpp, DFlash K10) 3.8 thought for 160K characters before writing a line of code.

Wësche

102,648 Aufrufe • vor 2 Monaten

One task, two models. GLM 5.3 Flash running slower at 25 tok/s prose Qwen3.8 Flash running faster at 47 tok/s prose Who will finish the task first?
0:14

One task, two models. GLM 5.3 Flash running slower at 25 tok/s prose Qwen3.8 Flash running faster at 47 tok/s prose Who will finish the task first?

Mia

27,481 Aufrufe • vor 1 Monat

vllm-exl3 v0.3.0 is LIVE with custom native CUDA kernels for 2-bit EXL3 on NVIDIA DGX Spark GB10. GLM-5.3-Flash-EXL3-K2 jumped from 16.9 → 24.6 tok/s average single-stream decode, a +45.6% gain. Coding hit 27.6 tok/s, +85.6%. 🚀 The previous ExLlamaV3-backed path inside vLLM was leaving a lot of GB10 bandwidth on the table. So I rewrote the hot path specifically for EXL3 on Blackwell sm_121: → in-register Trellis dequantization → native fused MoE decode → power-of-two chunked prefill GEMM → parallel NVMe pre-warm Then I tested it side-by-side on physical DGX Spark hardware using my GLM-5.3-Flash-EXL3-K2 pack and live vLLM HTTP streaming. 🚀 𝗗𝗘𝗖𝗢𝗗𝗘 𝗧𝗛𝗥𝗢𝗨𝗚𝗛𝗣𝗨𝗧 Single-stream C1: Coding 14.9 → 27.6 tok/s +85.6% Prose 13.7 → 24.6 tok/s +79.3% Reasoning 18.9 → 25.1 tok/s +32.7% Summary 17.1 → 25.6 tok/s +50.0% Format 16.3 → 24.0 tok/s +47.7% Average: 16.9 → 24.6 tok/s 𝗡𝗘𝗧 𝗚𝗔𝗜𝗡: +45.6% ⏱️ 𝗙𝗜𝗥𝗦𝗧-𝗧𝗢𝗞𝗘𝗡 𝗥𝗘𝗦𝗣𝗢𝗡𝗦𝗜𝗩𝗘𝗡𝗘𝗦𝗦 Coding TTFT: 2,344 ms → 859 ms That is a 63.3% reduction, or about 2.7× faster to first token. Follow-up turn with prefix cache hit: 5,608 ms → 3,588 ms 1.56× faster. ⚡ 𝗪𝗛𝗔𝗧 𝗖𝗛𝗔𝗡𝗚𝗘𝗗 𝗢𝗡 𝗧𝗛𝗘 𝗚𝗣𝗨 40 routed-MoE layers: 19.9 ms → 11.5 ms per token Per-layer MoE compute: 497 μs → 287.8 μs That removes 8.4 ms of MoE compute from every generated token. Total per-step wall time: 59.2 ms → 40.6 ms -31.4% The key is `p2b_fused_moe`. Instead of expanding EXL3 weights through a traditional intermediate path, the new kernel performs Trellis dequantization in-register while executing the routed expert computation. The weights stay compressed until the GPU actually needs them. 🔥 𝗣𝗥𝗘𝗙𝗜𝗟𝗟 𝗚𝗢𝗧 𝗔 𝗡𝗔𝗧𝗜𝗩𝗘 𝗣𝗔𝗧𝗛 𝗧𝗢𝗢 The new `exl3_gemm` uses power-of-two chunked prefill GEMM. Measured: 7.85 TFLOPS 13.0× faster than the legacy prefill kernel 1,875 tok/s cold prefill sustained across 65K context 💾 𝗧𝗛𝗘 𝗕𝗢𝗢𝗧 𝗣𝗔𝗧𝗛 𝗡𝗘𝗘𝗗𝗘𝗗 𝗪𝗢𝗥𝗞 𝗧𝗢𝗢 Loading a ~91 GiB model is part of the user experience. Standard shard loading is mostly serial. The updated recipe parallelizes NVMe pre-warm across 8 workers so the storage controller gets used properly instead of feeding a ~100 GiB model one shard at a time. That turns boot-time storage into another optimization target instead of something we simply accept. 💡 𝗧𝗪𝗢 𝗦𝗘𝗥𝗩𝗜𝗡𝗚 𝗙𝗟𝗔𝗚𝗦 𝗪𝗢𝗥𝗧𝗛 𝗞𝗡𝗢𝗪𝗜𝗡𝗚 `--long-prefill-token-threshold 1024` Prevents giant prefill chunks from monopolizing step budgets and starving parallel decode sessions. `--enable-prefix-caching` Avoids paying for the same conversational prefix again on follow-up turns. 📦 𝗘𝗩𝗘𝗥𝗬𝗧𝗛𝗜𝗡𝗚 𝗜𝗦 𝗢𝗣𝗘𝗡 vllm-exl3: GLM-5.3-Flash one-Spark recipe: Model: This is why I like working at the kernel level. The model did not change. The quant did not change. The hardware did not change. The execution path did. 16.9 → 24.6 tok/s. 🛠️ vLLM turboderp
0:11

vllm-exl3 v0.3.0 is LIVE with custom native CUDA kernels for 2-bit EXL3 on NVIDIA DGX Spark GB10. GLM-5.3-Flash-EXL3-K2 jumped from 16.9 → 24.6 tok/s average single-stream decode, a +45.6% gain. Coding hit 27.6 tok/s, +85.6%. 🚀 The previous ExLlamaV3-backed path inside vLLM was leaving a lot of GB10 bandwidth on the table. So I rewrote the hot path specifically for EXL3 on Blackwell sm_121: → in-register Trellis dequantization → native fused MoE decode → power-of-two chunked prefill GEMM → parallel NVMe pre-warm Then I tested it side-by-side on physical DGX Spark hardware using my GLM-5.3-Flash-EXL3-K2 pack and live vLLM HTTP streaming. 🚀 𝗗𝗘𝗖𝗢𝗗𝗘 𝗧𝗛𝗥𝗢𝗨𝗚𝗛𝗣𝗨𝗧 Single-stream C1: Coding 14.9 → 27.6 tok/s +85.6% Prose 13.7 → 24.6 tok/s +79.3% Reasoning 18.9 → 25.1 tok/s +32.7% Summary 17.1 → 25.6 tok/s +50.0% Format 16.3 → 24.0 tok/s +47.7% Average: 16.9 → 24.6 tok/s 𝗡𝗘𝗧 𝗚𝗔𝗜𝗡: +45.6% ⏱️ 𝗙𝗜𝗥𝗦𝗧-𝗧𝗢𝗞𝗘𝗡 𝗥𝗘𝗦𝗣𝗢𝗡𝗦𝗜𝗩𝗘𝗡𝗘𝗦𝗦 Coding TTFT: 2,344 ms → 859 ms That is a 63.3% reduction, or about 2.7× faster to first token. Follow-up turn with prefix cache hit: 5,608 ms → 3,588 ms 1.56× faster. ⚡ 𝗪𝗛𝗔𝗧 𝗖𝗛𝗔𝗡𝗚𝗘𝗗 𝗢𝗡 𝗧𝗛𝗘 𝗚𝗣𝗨 40 routed-MoE layers: 19.9 ms → 11.5 ms per token Per-layer MoE compute: 497 μs → 287.8 μs That removes 8.4 ms of MoE compute from every generated token. Total per-step wall time: 59.2 ms → 40.6 ms -31.4% The key is `p2b_fused_moe`. Instead of expanding EXL3 weights through a traditional intermediate path, the new kernel performs Trellis dequantization in-register while executing the routed expert computation. The weights stay compressed until the GPU actually needs them. 🔥 𝗣𝗥𝗘𝗙𝗜𝗟𝗟 𝗚𝗢𝗧 𝗔 𝗡𝗔𝗧𝗜𝗩𝗘 𝗣𝗔𝗧𝗛 𝗧𝗢𝗢 The new `exl3_gemm` uses power-of-two chunked prefill GEMM. Measured: 7.85 TFLOPS 13.0× faster than the legacy prefill kernel 1,875 tok/s cold prefill sustained across 65K context 💾 𝗧𝗛𝗘 𝗕𝗢𝗢𝗧 𝗣𝗔𝗧𝗛 𝗡𝗘𝗘𝗗𝗘𝗗 𝗪𝗢𝗥𝗞 𝗧𝗢𝗢 Loading a ~91 GiB model is part of the user experience. Standard shard loading is mostly serial. The updated recipe parallelizes NVMe pre-warm across 8 workers so the storage controller gets used properly instead of feeding a ~100 GiB model one shard at a time. That turns boot-time storage into another optimization target instead of something we simply accept. 💡 𝗧𝗪𝗢 𝗦𝗘𝗥𝗩𝗜𝗡𝗚 𝗙𝗟𝗔𝗚𝗦 𝗪𝗢𝗥𝗧𝗛 𝗞𝗡𝗢𝗪𝗜𝗡𝗚 `--long-prefill-token-threshold 1024` Prevents giant prefill chunks from monopolizing step budgets and starving parallel decode sessions. `--enable-prefix-caching` Avoids paying for the same conversational prefix again on follow-up turns. 📦 𝗘𝗩𝗘𝗥𝗬𝗧𝗛𝗜𝗡𝗚 𝗜𝗦 𝗢𝗣𝗘𝗡 vllm-exl3: GLM-5.3-Flash one-Spark recipe: Model: This is why I like working at the kernel level. The model did not change. The quant did not change. The hardware did not change. The execution path did. 16.9 → 24.6 tok/s. 🛠️ vLLM turboderp

Cruz

21,720 Aufrufe • vor 2 Monaten

nvidia/Qwen3.6-35B-A3B-NVFP4 running in vLLM nightly on my Nvidia GB10 is actually insane 50 tok/s, 4 concurrent generations. total 200 tok/s. ideal for spawning subagents or working in parallel its tool calling behavior is very good as well. I will be giving it test drive on an openclaw instance, and keep you posted More details on NVIDIA forum:
0:12

nvidia/Qwen3.6-35B-A3B-NVFP4 running in vLLM nightly on my Nvidia GB10 is actually insane 50 tok/s, 4 concurrent generations. total 200 tok/s. ideal for spawning subagents or working in parallel its tool calling behavior is very good as well. I will be giving it test drive on an openclaw instance, and keep you posted More details on NVIDIA forum:

Onur Solmaz

27,887 Aufrufe • vor 4 Monaten

Two models. One DGX Spark One identical voxel Eiffel Tower prompt. Qwen3.8-27B xhigh: 83K tokens, 41m29s, 33.4 tok/s Ornith 1.5 35B-A3B: 47K tokens, 10m35s, 74.0 tok/s Thinking on, no token ceiling, matched camera. Ornith finished much faster; Qwen spent far more tokens.
0:18

Two models. One DGX Spark One identical voxel Eiffel Tower prompt. Qwen3.8-27B xhigh: 83K tokens, 41m29s, 33.4 tok/s Ornith 1.5 35B-A3B: 47K tokens, 10m35s, 74.0 tok/s Thinking on, no token ceiling, matched camera. Ornith finished much faster; Qwen spent far more tokens.

Wësche

51,527 Aufrufe • vor 2 Monaten

BENCHMARKED: LFM3-1.7B by Liquid AI. Liquid AI published 1.6 GB for the KV cache at 128K context. I ran it on my Mac mini and measured 1.6 GB. The whole model at 128K uses 3.15 GB of GPU memory, under 5% of the 64GB. It still generates 80 tok/s. Only 6 of its 24 layers keep a full KV cache. The other 18 use short convolutions or sliding-window attention. llama.cpp (Metal), Q4_K_M, M5 Pro 64GB, median of 3 runs - 512 ctx: 184 tok/s, 0.09s to first token - 4K ctx: 178 tok/s, 0.72s to first token - 32K ctx: 141 tok/s - 128K ctx: 80 tok/s Q8_0 does 125 tok/s at 512, so 4-bit is 47% faster.
0:20

BENCHMARKED: LFM3-1.7B by Liquid AI. Liquid AI published 1.6 GB for the KV cache at 128K context. I ran it on my Mac mini and measured 1.6 GB. The whole model at 128K uses 3.15 GB of GPU memory, under 5% of the 64GB. It still generates 80 tok/s. Only 6 of its 24 layers keep a full KV cache. The other 18 use short convolutions or sliding-window attention. llama.cpp (Metal), Q4_K_M, M5 Pro 64GB, median of 3 runs - 512 ctx: 184 tok/s, 0.09s to first token - 4K ctx: 178 tok/s, 0.72s to first token - 32K ctx: 141 tok/s - 128K ctx: 80 tok/s Q8_0 does 125 tok/s at 512, so 4-bit is 47% faster.

Pratik Gandhi

39,576 Aufrufe • vor 7 Tagen

Qwen3.8-Flash-Next now reaches ~43 tok/s after a 122,902-token prompt on ONE DGX Spark. ⚡🚀 MTP k=2 won my draft-depth sweep, with +42.5% mean decode over no draft. The PLE table stays fully on-device. I promised the deeper MTP tests. Here are the results, and now you can explore them in an interactive benchmark page too. 𝗧𝗪𝗢 𝗗𝗥𝗔𝗙𝗧 𝗧𝗢𝗞𝗘𝗡𝗦 𝗪𝗢𝗡 Mean single-request decode with 32K context configured: MTP k=2: 39.21 tok/s MTP k=3: 36.42 tok/s MTP k=1: 35.18 tok/s No draft: 27.51 tok/s k=2 also produced the fastest individual sweep run: 41.34 tok/s. Four runs each for no draft, k=1 and k=2. Seven for k=3. Decode excludes time to first token. Here, k means speculative draft depth, not quantization bits. k=3 produced more tokens per step, but the extra drafting work did not pay off in throughput. k=2 is my current pick for this setup. 𝗧𝗛𝗘 𝟭𝟮𝟯𝗞-𝗧𝗢𝗞𝗘𝗡 𝗣𝗥𝗢𝗠𝗣𝗧 𝗧𝗘𝗦𝗧 I then ran a separate long-prompt comparison: Actual input: 122,902 tokens Configured context: 262,144 Requested output: 128 tokens One request at a time MTP k=2: ~43 tok/s No draft: 26.2 tok/s Time to first token: 110.6 seconds with MTP 107.0 seconds without it The win here is generation speed, not faster prefill. To keep the scope clear: 256K was the configured limit. This was a real ~123K input, not a completely filled 256K window or a full k sweep at that depth. 𝗣𝗟𝗘 𝗦𝗧𝗔𝗬𝗦 𝗢𝗡 𝗧𝗛𝗘 𝗦𝗣𝗔𝗥𝗞 Whole model on-device: 78.57 GiB Packed 5-bit PLE table: 30.4 GiB, included in that total No NVMe PLE offload in this build. This is still turboderp’s 3.05bpw_h5_ng5 EXL3 pack, served through my vllm-exl3 integration. My work here is the serving integration and testing. These are preliminary performance measurements, not a quality evaluation or a claim of bit-exact full-output parity. 𝗘𝗫𝗣𝗟𝗢𝗥𝗘 𝗧𝗛𝗘 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 The benchmark page has the individual sweep values, long-prompt comparison, and measurement scope. You can play the animation, export the charts, or download the HTML and data to render them yourself. No Spark needed to view the results. Credit to turboderp / ExLlamaV3 for the pack and kernels, vLLM for the serving engine, and Qwen Qwen Developers for the model. Recipe + reproduction: Interactive benchmark:
0:19

Qwen3.8-Flash-Next now reaches ~43 tok/s after a 122,902-token prompt on ONE DGX Spark. ⚡🚀 MTP k=2 won my draft-depth sweep, with +42.5% mean decode over no draft. The PLE table stays fully on-device. I promised the deeper MTP tests. Here are the results, and now you can explore them in an interactive benchmark page too. 𝗧𝗪𝗢 𝗗𝗥𝗔𝗙𝗧 𝗧𝗢𝗞𝗘𝗡𝗦 𝗪𝗢𝗡 Mean single-request decode with 32K context configured: MTP k=2: 39.21 tok/s MTP k=3: 36.42 tok/s MTP k=1: 35.18 tok/s No draft: 27.51 tok/s k=2 also produced the fastest individual sweep run: 41.34 tok/s. Four runs each for no draft, k=1 and k=2. Seven for k=3. Decode excludes time to first token. Here, k means speculative draft depth, not quantization bits. k=3 produced more tokens per step, but the extra drafting work did not pay off in throughput. k=2 is my current pick for this setup. 𝗧𝗛𝗘 𝟭𝟮𝟯𝗞-𝗧𝗢𝗞𝗘𝗡 𝗣𝗥𝗢𝗠𝗣𝗧 𝗧𝗘𝗦𝗧 I then ran a separate long-prompt comparison: Actual input: 122,902 tokens Configured context: 262,144 Requested output: 128 tokens One request at a time MTP k=2: ~43 tok/s No draft: 26.2 tok/s Time to first token: 110.6 seconds with MTP 107.0 seconds without it The win here is generation speed, not faster prefill. To keep the scope clear: 256K was the configured limit. This was a real ~123K input, not a completely filled 256K window or a full k sweep at that depth. 𝗣𝗟𝗘 𝗦𝗧𝗔𝗬𝗦 𝗢𝗡 𝗧𝗛𝗘 𝗦𝗣𝗔𝗥𝗞 Whole model on-device: 78.57 GiB Packed 5-bit PLE table: 30.4 GiB, included in that total No NVMe PLE offload in this build. This is still turboderp’s 3.05bpw_h5_ng5 EXL3 pack, served through my vllm-exl3 integration. My work here is the serving integration and testing. These are preliminary performance measurements, not a quality evaluation or a claim of bit-exact full-output parity. 𝗘𝗫𝗣𝗟𝗢𝗥𝗘 𝗧𝗛𝗘 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 The benchmark page has the individual sweep values, long-prompt comparison, and measurement scope. You can play the animation, export the charts, or download the HTML and data to render them yourself. No Spark needed to view the results. Credit to turboderp / ExLlamaV3 for the pack and kernels, vLLM for the serving engine, and Qwen Qwen Developers for the model. Recipe + reproduction: Interactive benchmark:

Cruz

12,258 Aufrufe • vor 1 Monat

Got Aleph Alpha's new Kolibri 1 running on a single DGX Spark the day it came out. Their container is x86 only, so I built vLLM 0.29.0 (arm64) + their aleph-alpha-inference plugin. FP8 weights (74 GB) and an FP8 KV cache, 164K context. The FP8 kernels worked on GB10 even though torch doesn't list sm_121. Speed: 48 tok/s solo, 157 tok/s with 8 streams, 5.2K tok/s prompt read. Then I gave it the pagoda prompt 👇 Kolibri 1, 5 tries, thinking on: 3 blank pages from THREE.OrbitControls used the old way 1 blank page, Three.js never loaded 1 runs: floating cubes 👇 Qwen3.6 27B: first try, 12K tokens. Qwen3.8 27B: first try, 75K.
0:15

Got Aleph Alpha's new Kolibri 1 running on a single DGX Spark the day it came out. Their container is x86 only, so I built vLLM 0.29.0 (arm64) + their aleph-alpha-inference plugin. FP8 weights (74 GB) and an FP8 KV cache, 164K context. The FP8 kernels worked on GB10 even though torch doesn't list sm_121. Speed: 48 tok/s solo, 157 tok/s with 8 streams, 5.2K tok/s prompt read. Then I gave it the pagoda prompt 👇 Kolibri 1, 5 tries, thinking on: 3 blank pages from THREE.OrbitControls used the old way 1 blank page, Three.js never loaded 1 runs: floating cubes 👇 Qwen3.6 27B: first try, 12K tokens. Qwen3.8 27B: first try, 75K.

Wësche

14,468 Aufrufe • vor 1 Monat

I think the CMP 170HX just became one of the weirdest local-LLM value plays. 👀 I got Qwen3.8-27B W8A16 running on a single 170HX 64GB with vLLM 0.27.1 + DFlash2: ⚡ ~120 tok/s decode 🚀 ~1.6–1.9K tok/s prefill 🧠 262K context with BF16 KV 📦 342K-token KV capacity 🎯 DFlash2 averaging 3.66 tokens/draft 💾 60.5 / 63.5 GiB VRAM used 27B model is doing ~120 tokens/sec on one GPU. Software optimization is getting ridiculous. Those numbers are from your measured W8A16 + DFlash2 run: ~120 tok/s decode, ~1,580–1,940 tok/s prefill, 342,729-token KV capacity, and 60.5/63.5 GiB VRAM usage. #170hx #LocalAI #localllm
0:11

I think the CMP 170HX just became one of the weirdest local-LLM value plays. 👀 I got Qwen3.8-27B W8A16 running on a single 170HX 64GB with vLLM 0.27.1 + DFlash2: ⚡ ~120 tok/s decode 🚀 ~1.6–1.9K tok/s prefill 🧠 262K context with BF16 KV 📦 342K-token KV capacity 🎯 DFlash2 averaging 3.66 tokens/draft 💾 60.5 / 63.5 GiB VRAM used 27B model is doing ~120 tokens/sec on one GPU. Software optimization is getting ridiculous. Those numbers are from your measured W8A16 + DFlash2 run: ~120 tok/s decode, ~1,580–1,940 tok/s prefill, 342,729-token KV capacity, and 60.5/63.5 GiB VRAM usage. #170hx #LocalAI #localllm

Own Your Compute

19,494 Aufrufe • vor 2 Monaten

This is awesome. Thanks Mia for this Was able to run with the RTX 3090 use this .env config DRAFT=mtp CONTEXT_SIZE=262144 CACHE_QUANT=8,4 GPU_MEM_GB=22 262K context: - Run 1: 65.98 tok/s - Run 2: 63.23 tok/s - Run 3: 64.41 tok/s Average: ~64.5 tok/s.
0:12

This is awesome. Thanks Mia for this Was able to run with the RTX 3090 use this .env config DRAFT=mtp CONTEXT_SIZE=262144 CACHE_QUANT=8,4 GPU_MEM_GB=22 262K context: - Run 1: 65.98 tok/s - Run 2: 63.23 tok/s - Run 3: 64.41 tok/s Average: ~64.5 tok/s.

Melvin Vivas

99,403 Aufrufe • vor 2 Monaten

DeepSeek V4 Flash Running 12 GB phone CPU only. - Model: 284B parameters - Speed: 1 tok/s - Just clever expert streaming from flash storage. -
0:12

DeepSeek V4 Flash Running 12 GB phone CPU only. - Model: 284B parameters - Speed: 1 tok/s - Just clever expert streaming from flash storage. -

Md Ismail Šojal 🕷️

28,301 Aufrufe • vor 2 Monaten

Ornith-1.0 35B - MLX Device - 128gb m5 max mbp Harness - PocketAI Hub 4 bit - 11,504 tokens, 95.98 tok/s, 21.20 GB peak RAM 8 bit - 13,128 tokens, 64.96 tok/s, 38.92 GB peak RAM BF16 - 12,491 tokens, 47.72 tok/s, 70.59 GB peak RAM prompt and link to model below
0:15

Ornith-1.0 35B - MLX Device - 128gb m5 max mbp Harness - PocketAI Hub 4 bit - 11,504 tokens, 95.98 tok/s, 21.20 GB peak RAM 8 bit - 13,128 tokens, 64.96 tok/s, 38.92 GB peak RAM BF16 - 12,491 tokens, 47.72 tok/s, 70.59 GB peak RAM prompt and link to model below

Trevor Wood

11,765 Aufrufe • vor 3 Monaten

This is the worst model I have ever tested, even 9b models outperform it. What even is this pagoda? Same prompt I always use. - Total time: 9 min 21 s (560.9s) - Tokens: 6,891 completion (710 reasoning + ~6,181 answer) | 84 prompt - Speed: ~12.3 tok/s end-to-end - First token: 0.15s - Output: 22,094 chars answer / 17.2KB HTML, natural stop (finish_reason=stop, no cap hit)
0:14

This is the worst model I have ever tested, even 9b models outperform it. What even is this pagoda? Same prompt I always use. - Total time: 9 min 21 s (560.9s) - Tokens: 6,891 completion (710 reasoning + ~6,181 answer) | 84 prompt - Speed: ~12.3 tok/s end-to-end - First token: 0.15s - Output: 22,094 chars answer / 17.2KB HTML, natural stop (finish_reason=stop, no cap hit)

Wësche

18,306 Aufrufe • vor 2 Monaten

1,000 tok/s vs 85 tok/s visualized
0:11

1,000 tok/s vs 85 tok/s visualized

nader dabit

212,315 Aufrufe • vor 3 Monaten

Qwen 3.8 Flash decisively crushes the M5 Ultra with this new recipe by vr8vr8 . I think I found the fastest dual DGX Spark recipe for Qwen3.8-Flash-Next. 138 tok/s structured, 133 code, 383 structured aggregate at four streams, and now it talks to Hermes. It feels like a cloud API, running on your desk. This should torch any current M5 run right now. This is part two. Same pair of Sparks, vr8vr8's recipe, his new v4.1, plus a patch of mine that's now an open PR against his repo. WHAT INITIALLY BROKE Through Hermes, the recipe answered in garbled mixed Chinese, then looped on "function86 to90 turn91" until it hit the token cap. Most people would blame the quant. The quant is fine. Three things in the serving stack were stacking up: Hermes doesn't send a temperature for this model. When a request omits it, vLLM 0.30 applies the checkpoint's own generation config: temperature 1.0, top_p 0.95, top_k 20, sampling on. The documented opt-out flag still leaves the server default at 1.0. Every Hermes turn was sampling hot. Hermes sends reasoning as a top-level reasoning object. vLLM 0.30 only maps a field called reasoning_effort onto thinking. So "thinking off" never arrived. Thinking is on by default in the chat template, and with no effort set it defaults to xhigh. A request for no thinking became the longest think the model can do, at temperature 1.0. This is why benchmarks alone do not tell you if a recipe is ready. A real harness that leaves temperature out and names the reasoning field differently gets a different model. Mia's older image never applied the generation config and drafts with argmax, so omitted temperature never sampled hot there. This recipe samples the draft. THE PATCH Two changes to vLLM's chat protocol, applied on both ranks by The Hermes reasoning object now sets thinking. none turns it off. low, medium and xhigh pass through to the template. An explicit enable_thinking still wins. If a request omits temperature, the chat path uses 0. An explicit temperature still passes through. Receipts: three table requests with no temperature and thinking off came back in English. A medium request closed its think and answered in English. The PR is open, not merged. Until it lands, a stock Hermes custom provider still samples at 1.0. HIS V4.1 vr8vr8 shipped recipe v4.1 in the meantime, same v6 image: FlashInfer GDN prefill. His card claims about +5% prefill with decode unchanged. On my depth ladder, prefill came in +6 to +7% from 128k to 900k. Claim holds. THE SECOND GRID Same pair, same protocols, thinking off, temperature 0. My runners already send temperature 0, so this is v4.1 against part one, not a test of the patch. Prose 600: 81.6 tok/s (part one 78.4) Structured count 200: 138.3 (131.8) Code clamp: 133.2 (126.4) JSON catalog: 118.6 (111.8) That's up 4 to 6% across the board, which is run-to-run band. I'm not crediting FlashInfer for it. At four streams: structured 383.1 aggregate, code 330.5, JSON 290.4, prose 200.5. Single-stream structured costs 0.52 J per token of GPU power across both boxes. Tool-call smoke test clean, finish_reason tool_calls, no XML in content. DEPTH Needle retrieval 18/18, from 8k out to the 900k rung (880k server-measured). Decode after fill holds between 65 and 77 tok/s the whole way, 70.6 at the top. Prefill runs 3,245 tok/s at 128k and still 2,577 at 880k. AGENT TURNS The OMP agent lane now runs a full 2,048-token write_file turn at 35k context, replacing part one's 35-token window: 107.4 tok/s decode. The Hermes research turn, two web_search calls at 35k context: 81.0 tok/s, up from 74.7. STILL NOT LIKE-FOR-LIKE Against Mia's d2f54b7 recipe on this same pair: prose 1.6x, structured 2.0x, code 2.0x, JSON 1.9x, Hermes turn 2.2x. Different quant, K=5 vs K=3 drafting, bf16 KV vs fp8. My own version is also my own customized recipe, but it felt better to just serve a PR to the creator, since I didn't invent anything new. The biggest single lever is his output head: 1.18 GiB down to 0.33 GiB, read about 5.4 times every decode step, worth +30% engine steps on its own. SUMMARY Every clean lane faster than the mainstream recipe on this rig: checked. Retrieval 18/18 to 900k: checked. Decode holds to 900k: checked. Agent lane measured over a full 2,048-token window: checked. Hermes out of the box: PR open, work in progress. The speed was always the hard part, and vr8vr8 had it in part one. Sources , Benchmarks and Recipe in Reply 👇
0:18

Qwen 3.8 Flash decisively crushes the M5 Ultra with this new recipe by vr8vr8 . I think I found the fastest dual DGX Spark recipe for Qwen3.8-Flash-Next. 138 tok/s structured, 133 code, 383 structured aggregate at four streams, and now it talks to Hermes. It feels like a cloud API, running on your desk. This should torch any current M5 run right now. This is part two. Same pair of Sparks, vr8vr8's recipe, his new v4.1, plus a patch of mine that's now an open PR against his repo. WHAT INITIALLY BROKE Through Hermes, the recipe answered in garbled mixed Chinese, then looped on "function86 to90 turn91" until it hit the token cap. Most people would blame the quant. The quant is fine. Three things in the serving stack were stacking up: Hermes doesn't send a temperature for this model. When a request omits it, vLLM 0.30 applies the checkpoint's own generation config: temperature 1.0, top_p 0.95, top_k 20, sampling on. The documented opt-out flag still leaves the server default at 1.0. Every Hermes turn was sampling hot. Hermes sends reasoning as a top-level reasoning object. vLLM 0.30 only maps a field called reasoning_effort onto thinking. So "thinking off" never arrived. Thinking is on by default in the chat template, and with no effort set it defaults to xhigh. A request for no thinking became the longest think the model can do, at temperature 1.0. This is why benchmarks alone do not tell you if a recipe is ready. A real harness that leaves temperature out and names the reasoning field differently gets a different model. Mia's older image never applied the generation config and drafts with argmax, so omitted temperature never sampled hot there. This recipe samples the draft. THE PATCH Two changes to vLLM's chat protocol, applied on both ranks by The Hermes reasoning object now sets thinking. none turns it off. low, medium and xhigh pass through to the template. An explicit enable_thinking still wins. If a request omits temperature, the chat path uses 0. An explicit temperature still passes through. Receipts: three table requests with no temperature and thinking off came back in English. A medium request closed its think and answered in English. The PR is open, not merged. Until it lands, a stock Hermes custom provider still samples at 1.0. HIS V4.1 vr8vr8 shipped recipe v4.1 in the meantime, same v6 image: FlashInfer GDN prefill. His card claims about +5% prefill with decode unchanged. On my depth ladder, prefill came in +6 to +7% from 128k to 900k. Claim holds. THE SECOND GRID Same pair, same protocols, thinking off, temperature 0. My runners already send temperature 0, so this is v4.1 against part one, not a test of the patch. Prose 600: 81.6 tok/s (part one 78.4) Structured count 200: 138.3 (131.8) Code clamp: 133.2 (126.4) JSON catalog: 118.6 (111.8) That's up 4 to 6% across the board, which is run-to-run band. I'm not crediting FlashInfer for it. At four streams: structured 383.1 aggregate, code 330.5, JSON 290.4, prose 200.5. Single-stream structured costs 0.52 J per token of GPU power across both boxes. Tool-call smoke test clean, finish_reason tool_calls, no XML in content. DEPTH Needle retrieval 18/18, from 8k out to the 900k rung (880k server-measured). Decode after fill holds between 65 and 77 tok/s the whole way, 70.6 at the top. Prefill runs 3,245 tok/s at 128k and still 2,577 at 880k. AGENT TURNS The OMP agent lane now runs a full 2,048-token write_file turn at 35k context, replacing part one's 35-token window: 107.4 tok/s decode. The Hermes research turn, two web_search calls at 35k context: 81.0 tok/s, up from 74.7. STILL NOT LIKE-FOR-LIKE Against Mia's d2f54b7 recipe on this same pair: prose 1.6x, structured 2.0x, code 2.0x, JSON 1.9x, Hermes turn 2.2x. Different quant, K=5 vs K=3 drafting, bf16 KV vs fp8. My own version is also my own customized recipe, but it felt better to just serve a PR to the creator, since I didn't invent anything new. The biggest single lever is his output head: 1.18 GiB down to 0.33 GiB, read about 5.4 times every decode step, worth +30% engine steps on its own. SUMMARY Every clean lane faster than the mainstream recipe on this rig: checked. Retrieval 18/18 to 900k: checked. Decode holds to 900k: checked. Agent lane measured over a full 2,048-token window: checked. Hermes out of the box: PR open, work in progress. The speed was always the hard part, and vr8vr8 had it in part one. Sources , Benchmarks and Recipe in Reply 👇

Yume_X

21,745 Aufrufe • vor 1 Monat

Im calling everyone with 1 RTX 3090. Qwen3.8 Flash-Next, 177B. This is absolutely insane model for 1 card. I swear. Here is what 1 used 3090 does with it on Strata: IQ3_S: 70 tok/s, and still 66 tok/s with 261K of context IQ2_XS: 87 tok/s prompts are read at about 1,900 tok/s The card holds the experts it uses most, the rest stays in RAM. So you need RAM for this. AJ says 64GB. My rig has 128GB, so I didnt test the limit. It doesnt have to be fast memory: mine is old DDR4-2400 on 2 channels. What you get for that: a model that scores 61.6 on LiveBench agentic coding, above every GPT, every Gemini and every Grok. 262K context, vision, and I published a version that doesnt refuse. 1 card from 2020 and 64GB of old RAM. If thats the PC under your desk and you still pay for a coding model every month, try it tonight. Links in the reply.
0:15

Im calling everyone with 1 RTX 3090. Qwen3.8 Flash-Next, 177B. This is absolutely insane model for 1 card. I swear. Here is what 1 used 3090 does with it on Strata: IQ3_S: 70 tok/s, and still 66 tok/s with 261K of context IQ2_XS: 87 tok/s prompts are read at about 1,900 tok/s The card holds the experts it uses most, the rest stays in RAM. So you need RAM for this. AJ says 64GB. My rig has 128GB, so I didnt test the limit. It doesnt have to be fast memory: mine is old DDR4-2400 on 2 channels. What you get for that: a model that scores 61.6 on LiveBench agentic coding, above every GPT, every Gemini and every Grok. 262K context, vision, and I published a version that doesnt refuse. 1 card from 2020 and 64GB of old RAM. If thats the PC under your desk and you still pay for a coding model every month, try it tonight. Links in the reply.

Alexey Fateev

28,086 Aufrufe • vor 23 Tagen

Laguna XS 2.1 just matched Qwen 3.6 35B on a real coding task & ran twice as fast. > Same RTX 3090 > Laguna: 158 tok/s. Qwen: 81 tok/s The local model race isn't about size anymore. It's about who moves faster.
0:19

Laguna XS 2.1 just matched Qwen 3.6 35B on a real coding task & ran twice as fast. > Same RTX 3090 > Laguna: 158 tok/s. Qwen: 81 tok/s The local model race isn't about size anymore. It's about who moves faster.

0xMarioNawfal

63,839 Aufrufe • vor 3 Monaten

Qwen3.8-Flash-Next is starting to feel like the local model Opus fans have been waiting for. Someone ran the NVFP4 176B-class Flash-Next on 2× DGX Sparks, and the results are wild. Real measured scaling → C1: 44.2 tok/s → C2: 64.6 tok/s → C4: 86.8 tok/s aggregate The per-stream speed drops with concurrency, but total throughput keeps climbing. Long-context behavior was even more impressive: → 5K: needle retrieved → 21K: needle retrieved → 84K: needle retrieved → 167K: needle retrieved → 262K: prefill succeeded, but the window was saturated That 167K retrieval test is the one I care about. Long agent runs are where models usually start losing the plot. Flash-Next didn’t. It also held up surprisingly well on physics-heavy reasoning, artifact generation, research workflows, evidence checking, and long-horizon planning. The personality is interesting too. DeepSeek V4 Flash feels like the dependable workhorse. GLM-5.2 feels like the problem-solving machine. Qwen3.8-Flash-Next feels more insightful. It has that rare ability to understand what you’re actually asking rather than just following the surface pattern. The main weakness I’ve noticed is instruction following. It can occasionally drift between prose turns where DeepSeek and GLM stay tighter. And this is why the 256GB M5 Ultra conversation gets interesting. If Apple can pair that huge unified-memory pool with enough bandwidth, this model class becomes genuinely practical for long-running local agents. We’re talking about frontier-class reasoning on hardware sitting on a desk.
0:18

Qwen3.8-Flash-Next is starting to feel like the local model Opus fans have been waiting for. Someone ran the NVFP4 176B-class Flash-Next on 2× DGX Sparks, and the results are wild. Real measured scaling → C1: 44.2 tok/s → C2: 64.6 tok/s → C4: 86.8 tok/s aggregate The per-stream speed drops with concurrency, but total throughput keeps climbing. Long-context behavior was even more impressive: → 5K: needle retrieved → 21K: needle retrieved → 84K: needle retrieved → 167K: needle retrieved → 262K: prefill succeeded, but the window was saturated That 167K retrieval test is the one I care about. Long agent runs are where models usually start losing the plot. Flash-Next didn’t. It also held up surprisingly well on physics-heavy reasoning, artifact generation, research workflows, evidence checking, and long-horizon planning. The personality is interesting too. DeepSeek V4 Flash feels like the dependable workhorse. GLM-5.2 feels like the problem-solving machine. Qwen3.8-Flash-Next feels more insightful. It has that rare ability to understand what you’re actually asking rather than just following the surface pattern. The main weakness I’ve noticed is instruction following. It can occasionally drift between prose turns where DeepSeek and GLM stay tighter. And this is why the 256GB M5 Ultra conversation gets interesting. If Apple can pair that huge unified-memory pool with enough bandwidth, this model class becomes genuinely practical for long-running local agents. We’re talking about frontier-class reasoning on hardware sitting on a desk.

FHILY👑

20,253 Aufrufe • vor 2 Monaten

Deepseek v4.1 Flash on 24GB Macbook. I think I reached my limit, 1.36 tok/s. It is far from being usable but the feeling of having a SOTA model on my consumer machine is awesome. I started from 0.74 tok/s and almost double it. It was a great journey.
0:15

Deepseek v4.1 Flash on 24GB Macbook. I think I reached my limit, 1.36 tok/s. It is far from being usable but the feeling of having a SOTA model on my consumer machine is awesome. I started from 0.74 tok/s and almost double it. It was a great journey.

Marco Franzon

90,955 Aufrufe • vor 1 Monat