Cruz's banner
Cruz's profile picture

Cruz

@ViC305 • 2,931 subscribers

God-fearing husband & father. AI Engineer. Fine-tuning + day-zero EXL3 & GGUF quants + benchmarks • Local LLMs • honest tok/s • fitting big models on small GPUs

Shorts

vllm-exl3 v0.3.0 is LIVE with custom native CUDA kernels for 2-bit EXL3 on NVIDIA DGX Spark GB10. GLM-5.3-Flash-EXL3-K2 jumped from 16.9 → 24.6 tok/s average single-stream decode, a +45.6% gain. Coding hit 27.6 tok/s, +85.6%. 🚀 The previous ExLlamaV3-backed path inside vLLM was leaving a lot of GB10 bandwidth on the table. So I rewrote the hot path specifically for EXL3 on Blackwell sm_121: → in-register Trellis dequantization → native fused MoE decode → power-of-two chunked prefill GEMM → parallel NVMe pre-warm Then I tested it side-by-side on physical DGX Spark hardware using my GLM-5.3-Flash-EXL3-K2 pack and live vLLM HTTP streaming. 🚀 𝗗𝗘𝗖𝗢𝗗𝗘 𝗧𝗛𝗥𝗢𝗨𝗚𝗛𝗣𝗨𝗧 Single-stream C1: Coding 14.9 → 27.6 tok/s +85.6% Prose 13.7 → 24.6 tok/s +79.3% Reasoning 18.9 → 25.1 tok/s +32.7% Summary 17.1 → 25.6 tok/s +50.0% Format 16.3 → 24.0 tok/s +47.7% Average: 16.9 → 24.6 tok/s 𝗡𝗘𝗧 𝗚𝗔𝗜𝗡: +45.6% ⏱️ 𝗙𝗜𝗥𝗦𝗧-𝗧𝗢𝗞𝗘𝗡 𝗥𝗘𝗦𝗣𝗢𝗡𝗦𝗜𝗩𝗘𝗡𝗘𝗦𝗦 Coding TTFT: 2,344 ms → 859 ms That is a 63.3% reduction, or about 2.7× faster to first token. Follow-up turn with prefix cache hit: 5,608 ms → 3,588 ms 1.56× faster. ⚡ 𝗪𝗛𝗔𝗧 𝗖𝗛𝗔𝗡𝗚𝗘𝗗 𝗢𝗡 𝗧𝗛𝗘 𝗚𝗣𝗨 40 routed-MoE layers: 19.9 ms → 11.5 ms per token Per-layer MoE compute: 497 μs → 287.8 μs That removes 8.4 ms of MoE compute from every generated token. Total per-step wall time: 59.2 ms → 40.6 ms -31.4% The key is `p2b_fused_moe`. Instead of expanding EXL3 weights through a traditional intermediate path, the new kernel performs Trellis dequantization in-register while executing the routed expert computation. The weights stay compressed until the GPU actually needs them. 🔥 𝗣𝗥𝗘𝗙𝗜𝗟𝗟 𝗚𝗢𝗧 𝗔 𝗡𝗔𝗧𝗜𝗩𝗘 𝗣𝗔𝗧𝗛 𝗧𝗢𝗢 The new `exl3_gemm` uses power-of-two chunked prefill GEMM. Measured: 7.85 TFLOPS 13.0× faster than the legacy prefill kernel 1,875 tok/s cold prefill sustained across 65K context 💾 𝗧𝗛𝗘 𝗕𝗢𝗢𝗧 𝗣𝗔𝗧𝗛 𝗡𝗘𝗘𝗗𝗘𝗗 𝗪𝗢𝗥𝗞 𝗧𝗢𝗢 Loading a ~91 GiB model is part of the user experience. Standard shard loading is mostly serial. The updated recipe parallelizes NVMe pre-warm across 8 workers so the storage controller gets used properly instead of feeding a ~100 GiB model one shard at a time. That turns boot-time storage into another optimization target instead of something we simply accept. 💡 𝗧𝗪𝗢 𝗦𝗘𝗥𝗩𝗜𝗡𝗚 𝗙𝗟𝗔𝗚𝗦 𝗪𝗢𝗥𝗧𝗛 𝗞𝗡𝗢𝗪𝗜𝗡𝗚 `--long-prefill-token-threshold 1024` Prevents giant prefill chunks from monopolizing step budgets and starving parallel decode sessions. `--enable-prefix-caching` Avoids paying for the same conversational prefix again on follow-up turns. 📦 𝗘𝗩𝗘𝗥𝗬𝗧𝗛𝗜𝗡𝗚 𝗜𝗦 𝗢𝗣𝗘𝗡 vllm-exl3: GLM-5.3-Flash one-Spark recipe: Model: This is why I like working at the kernel level. The model did not change. The quant did not change. The hardware did not change. The execution path did. 16.9 → 24.6 tok/s. 🛠️ vLLM turboderp

vllm-exl3 v0.3.0 is LIVE with custom native CUDA kernels for 2-bit EXL3 on NVIDIA DGX Spark GB10. GLM-5.3-Flash-EXL3-K2 jumped from 16.9 → 24.6 tok/s average single-stream decode, a +45.6% gain. Coding hit 27.6 tok/s, +85.6%. 🚀 The previous ExLlamaV3-backed path inside vLLM was leaving a lot of GB10 bandwidth on the table. So I rewrote the hot path specifically for EXL3 on Blackwell sm_121: → in-register Trellis dequantization → native fused MoE decode → power-of-two chunked prefill GEMM → parallel NVMe pre-warm Then I tested it side-by-side on physical DGX Spark hardware using my GLM-5.3-Flash-EXL3-K2 pack and live vLLM HTTP streaming. 🚀 𝗗𝗘𝗖𝗢𝗗𝗘 𝗧𝗛𝗥𝗢𝗨𝗚𝗛𝗣𝗨𝗧 Single-stream C1: Coding 14.9 → 27.6 tok/s +85.6% Prose 13.7 → 24.6 tok/s +79.3% Reasoning 18.9 → 25.1 tok/s +32.7% Summary 17.1 → 25.6 tok/s +50.0% Format 16.3 → 24.0 tok/s +47.7% Average: 16.9 → 24.6 tok/s 𝗡𝗘𝗧 𝗚𝗔𝗜𝗡: +45.6% ⏱️ 𝗙𝗜𝗥𝗦𝗧-𝗧𝗢𝗞𝗘𝗡 𝗥𝗘𝗦𝗣𝗢𝗡𝗦𝗜𝗩𝗘𝗡𝗘𝗦𝗦 Coding TTFT: 2,344 ms → 859 ms That is a 63.3% reduction, or about 2.7× faster to first token. Follow-up turn with prefix cache hit: 5,608 ms → 3,588 ms 1.56× faster. ⚡ 𝗪𝗛𝗔𝗧 𝗖𝗛𝗔𝗡𝗚𝗘𝗗 𝗢𝗡 𝗧𝗛𝗘 𝗚𝗣𝗨 40 routed-MoE layers: 19.9 ms → 11.5 ms per token Per-layer MoE compute: 497 μs → 287.8 μs That removes 8.4 ms of MoE compute from every generated token. Total per-step wall time: 59.2 ms → 40.6 ms -31.4% The key is `p2b_fused_moe`. Instead of expanding EXL3 weights through a traditional intermediate path, the new kernel performs Trellis dequantization in-register while executing the routed expert computation. The weights stay compressed until the GPU actually needs them. 🔥 𝗣𝗥𝗘𝗙𝗜𝗟𝗟 𝗚𝗢𝗧 𝗔 𝗡𝗔𝗧𝗜𝗩𝗘 𝗣𝗔𝗧𝗛 𝗧𝗢𝗢 The new `exl3_gemm` uses power-of-two chunked prefill GEMM. Measured: 7.85 TFLOPS 13.0× faster than the legacy prefill kernel 1,875 tok/s cold prefill sustained across 65K context 💾 𝗧𝗛𝗘 𝗕𝗢𝗢𝗧 𝗣𝗔𝗧𝗛 𝗡𝗘𝗘𝗗𝗘𝗗 𝗪𝗢𝗥𝗞 𝗧𝗢𝗢 Loading a ~91 GiB model is part of the user experience. Standard shard loading is mostly serial. The updated recipe parallelizes NVMe pre-warm across 8 workers so the storage controller gets used properly instead of feeding a ~100 GiB model one shard at a time. That turns boot-time storage into another optimization target instead of something we simply accept. 💡 𝗧𝗪𝗢 𝗦𝗘𝗥𝗩𝗜𝗡𝗚 𝗙𝗟𝗔𝗚𝗦 𝗪𝗢𝗥𝗧𝗛 𝗞𝗡𝗢𝗪𝗜𝗡𝗚 `--long-prefill-token-threshold 1024` Prevents giant prefill chunks from monopolizing step budgets and starving parallel decode sessions. `--enable-prefix-caching` Avoids paying for the same conversational prefix again on follow-up turns. 📦 𝗘𝗩𝗘𝗥𝗬𝗧𝗛𝗜𝗡𝗚 𝗜𝗦 𝗢𝗣𝗘𝗡 vllm-exl3: GLM-5.3-Flash one-Spark recipe: Model: This is why I like working at the kernel level. The model did not change. The quant did not change. The hardware did not change. The execution path did. 16.9 → 24.6 tok/s. 🛠️ vLLM turboderp

21,720 Aufrufe

Qwen3.8-Flash-Next now reaches ~43 tok/s after a 122,902-token prompt on ONE DGX Spark. ⚡🚀 MTP k=2 won my draft-depth sweep, with +42.5% mean decode over no draft. The PLE table stays fully on-device. I promised the deeper MTP tests. Here are the results, and now you can explore them in an interactive benchmark page too. 𝗧𝗪𝗢 𝗗𝗥𝗔𝗙𝗧 𝗧𝗢𝗞𝗘𝗡𝗦 𝗪𝗢𝗡 Mean single-request decode with 32K context configured: MTP k=2: 39.21 tok/s MTP k=3: 36.42 tok/s MTP k=1: 35.18 tok/s No draft: 27.51 tok/s k=2 also produced the fastest individual sweep run: 41.34 tok/s. Four runs each for no draft, k=1 and k=2. Seven for k=3. Decode excludes time to first token. Here, k means speculative draft depth, not quantization bits. k=3 produced more tokens per step, but the extra drafting work did not pay off in throughput. k=2 is my current pick for this setup. 𝗧𝗛𝗘 𝟭𝟮𝟯𝗞-𝗧𝗢𝗞𝗘𝗡 𝗣𝗥𝗢𝗠𝗣𝗧 𝗧𝗘𝗦𝗧 I then ran a separate long-prompt comparison: Actual input: 122,902 tokens Configured context: 262,144 Requested output: 128 tokens One request at a time MTP k=2: ~43 tok/s No draft: 26.2 tok/s Time to first token: 110.6 seconds with MTP 107.0 seconds without it The win here is generation speed, not faster prefill. To keep the scope clear: 256K was the configured limit. This was a real ~123K input, not a completely filled 256K window or a full k sweep at that depth. 𝗣𝗟𝗘 𝗦𝗧𝗔𝗬𝗦 𝗢𝗡 𝗧𝗛𝗘 𝗦𝗣𝗔𝗥𝗞 Whole model on-device: 78.57 GiB Packed 5-bit PLE table: 30.4 GiB, included in that total No NVMe PLE offload in this build. This is still turboderp’s 3.05bpw_h5_ng5 EXL3 pack, served through my vllm-exl3 integration. My work here is the serving integration and testing. These are preliminary performance measurements, not a quality evaluation or a claim of bit-exact full-output parity. 𝗘𝗫𝗣𝗟𝗢𝗥𝗘 𝗧𝗛𝗘 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 The benchmark page has the individual sweep values, long-prompt comparison, and measurement scope. You can play the animation, export the charts, or download the HTML and data to render them yourself. No Spark needed to view the results. Credit to turboderp / ExLlamaV3 for the pack and kernels, vLLM for the serving engine, and Qwen Qwen Developers for the model. Recipe + reproduction: Interactive benchmark:

Qwen3.8-Flash-Next now reaches ~43 tok/s after a 122,902-token prompt on ONE DGX Spark. ⚡🚀 MTP k=2 won my draft-depth sweep, with +42.5% mean decode over no draft. The PLE table stays fully on-device. I promised the deeper MTP tests. Here are the results, and now you can explore them in an interactive benchmark page too. 𝗧𝗪𝗢 𝗗𝗥𝗔𝗙𝗧 𝗧𝗢𝗞𝗘𝗡𝗦 𝗪𝗢𝗡 Mean single-request decode with 32K context configured: MTP k=2: 39.21 tok/s MTP k=3: 36.42 tok/s MTP k=1: 35.18 tok/s No draft: 27.51 tok/s k=2 also produced the fastest individual sweep run: 41.34 tok/s. Four runs each for no draft, k=1 and k=2. Seven for k=3. Decode excludes time to first token. Here, k means speculative draft depth, not quantization bits. k=3 produced more tokens per step, but the extra drafting work did not pay off in throughput. k=2 is my current pick for this setup. 𝗧𝗛𝗘 𝟭𝟮𝟯𝗞-𝗧𝗢𝗞𝗘𝗡 𝗣𝗥𝗢𝗠𝗣𝗧 𝗧𝗘𝗦𝗧 I then ran a separate long-prompt comparison: Actual input: 122,902 tokens Configured context: 262,144 Requested output: 128 tokens One request at a time MTP k=2: ~43 tok/s No draft: 26.2 tok/s Time to first token: 110.6 seconds with MTP 107.0 seconds without it The win here is generation speed, not faster prefill. To keep the scope clear: 256K was the configured limit. This was a real ~123K input, not a completely filled 256K window or a full k sweep at that depth. 𝗣𝗟𝗘 𝗦𝗧𝗔𝗬𝗦 𝗢𝗡 𝗧𝗛𝗘 𝗦𝗣𝗔𝗥𝗞 Whole model on-device: 78.57 GiB Packed 5-bit PLE table: 30.4 GiB, included in that total No NVMe PLE offload in this build. This is still turboderp’s 3.05bpw_h5_ng5 EXL3 pack, served through my vllm-exl3 integration. My work here is the serving integration and testing. These are preliminary performance measurements, not a quality evaluation or a claim of bit-exact full-output parity. 𝗘𝗫𝗣𝗟𝗢𝗥𝗘 𝗧𝗛𝗘 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 The benchmark page has the individual sweep values, long-prompt comparison, and measurement scope. You can play the animation, export the charts, or download the HTML and data to render them yourself. No Spark needed to view the results. Credit to turboderp / ExLlamaV3 for the pack and kernels, vLLM for the serving engine, and Qwen Qwen Developers for the model. Recipe + reproduction: Interactive benchmark:

12,258 Aufrufe

8GB RTX 4060 Laptop. ~16GB Ornith IQ3_S GGUF. 44.5 tok/s streaming locally. FreeToken didn’t have the qwen35moe GGUF path needed for this model, so I implemented it and submitted PR #131 upstream. 👀 I also exposed the K/I GGUF quant types its CUDA kernels already supported through the Python loader. This is the result. Setup: Ornith-1.5-35B-A3B 35B MoE / ~3B active IQ3_S GGUF · ~16GB RTX 4060 Laptop · 8GB VRAM FreeToken · FlashInfer MoE backend: offload On the attached full-prompt run, I measured ~44.5 tok/s streaming on screen. In the same runs, FreeToken’s server decode counter reported 46.7 → 50.1 tok/s. VRAM stayed at 6,879 / 8,188 MiB, with 20GB host RAM and 98% GPU utilization during generation. Load time: 65s. My llama.cpp CPU reference on the same model was 11.1 tok/s. The interesting part isn’t just “35B on an 8GB GPU.” The code path itself was missing. FreeToken’s CUDA side already supported more GGUF quant types than its Python loader exposed, including the K/I quants needed for this IQ3_S model. And qwen35moe models like Ornith needed their own GGUF architecture/loading path. I implemented both pieces in my fork and opened them upstream in PR #131. Once loaded, FreeToken can use GPU + CPU + host RAM as an elastic MoE inference system instead of requiring the entire checkpoint to live in VRAM. That fits Ornith well because only a subset of its experts are active per token. Important caveat: PR #131 is still open. This is not merged upstream yet. I’m not calling this official Ornith support. I’m showing a working implementation, the code is public, and the attached terminal video is the actual run. PR #131: My fork: Next I’m going to test where this path holds up across more quants, longer context and real agent workloads.

8GB RTX 4060 Laptop. ~16GB Ornith IQ3_S GGUF. 44.5 tok/s streaming locally. FreeToken didn’t have the qwen35moe GGUF path needed for this model, so I implemented it and submitted PR #131 upstream. 👀 I also exposed the K/I GGUF quant types its CUDA kernels already supported through the Python loader. This is the result. Setup: Ornith-1.5-35B-A3B 35B MoE / ~3B active IQ3_S GGUF · ~16GB RTX 4060 Laptop · 8GB VRAM FreeToken · FlashInfer MoE backend: offload On the attached full-prompt run, I measured ~44.5 tok/s streaming on screen. In the same runs, FreeToken’s server decode counter reported 46.7 → 50.1 tok/s. VRAM stayed at 6,879 / 8,188 MiB, with 20GB host RAM and 98% GPU utilization during generation. Load time: 65s. My llama.cpp CPU reference on the same model was 11.1 tok/s. The interesting part isn’t just “35B on an 8GB GPU.” The code path itself was missing. FreeToken’s CUDA side already supported more GGUF quant types than its Python loader exposed, including the K/I quants needed for this IQ3_S model. And qwen35moe models like Ornith needed their own GGUF architecture/loading path. I implemented both pieces in my fork and opened them upstream in PR #131. Once loaded, FreeToken can use GPU + CPU + host RAM as an elastic MoE inference system instead of requiring the entire checkpoint to live in VRAM. That fits Ornith well because only a subset of its experts are active per token. Important caveat: PR #131 is still open. This is not merged upstream yet. I’m not calling this official Ornith support. I’m showing a working implementation, the code is public, and the attached terminal video is the actual run. PR #131: My fork: Next I’m going to test where this path holds up across more quants, longer context and real agent workloads.

12,257 Aufrufe