Yume_X's banner
Yume_X's profile picture

Yume_X

@yume_arasaki • 2,468 subscribers

夢 . Chasing free AI. Mapping the AI era. Your hardware. Your Intelligence. Local Models + Hermes Fleet RTX 4090 | Mac Mini M4 64GB | 2x DGX Spark online

Shorts

Qwen 3.8 Flash decisively crushes the M5 Ultra with this new recipe by vr8vr8 . I think I found the fastest dual DGX Spark recipe for Qwen3.8-Flash-Next. 138 tok/s structured, 133 code, 383 structured aggregate at four streams, and now it talks to Hermes. It feels like a cloud API, running on your desk. This should torch any current M5 run right now. This is part two. Same pair of Sparks, vr8vr8's recipe, his new v4.1, plus a patch of mine that's now an open PR against his repo. WHAT INITIALLY BROKE Through Hermes, the recipe answered in garbled mixed Chinese, then looped on "function86 to90 turn91" until it hit the token cap. Most people would blame the quant. The quant is fine. Three things in the serving stack were stacking up: Hermes doesn't send a temperature for this model. When a request omits it, vLLM 0.30 applies the checkpoint's own generation config: temperature 1.0, top_p 0.95, top_k 20, sampling on. The documented opt-out flag still leaves the server default at 1.0. Every Hermes turn was sampling hot. Hermes sends reasoning as a top-level reasoning object. vLLM 0.30 only maps a field called reasoning_effort onto thinking. So "thinking off" never arrived. Thinking is on by default in the chat template, and with no effort set it defaults to xhigh. A request for no thinking became the longest think the model can do, at temperature 1.0. This is why benchmarks alone do not tell you if a recipe is ready. A real harness that leaves temperature out and names the reasoning field differently gets a different model. Mia's older image never applied the generation config and drafts with argmax, so omitted temperature never sampled hot there. This recipe samples the draft. THE PATCH Two changes to vLLM's chat protocol, applied on both ranks by The Hermes reasoning object now sets thinking. none turns it off. low, medium and xhigh pass through to the template. An explicit enable_thinking still wins. If a request omits temperature, the chat path uses 0. An explicit temperature still passes through. Receipts: three table requests with no temperature and thinking off came back in English. A medium request closed its think and answered in English. The PR is open, not merged. Until it lands, a stock Hermes custom provider still samples at 1.0. HIS V4.1 vr8vr8 shipped recipe v4.1 in the meantime, same v6 image: FlashInfer GDN prefill. His card claims about +5% prefill with decode unchanged. On my depth ladder, prefill came in +6 to +7% from 128k to 900k. Claim holds. THE SECOND GRID Same pair, same protocols, thinking off, temperature 0. My runners already send temperature 0, so this is v4.1 against part one, not a test of the patch. Prose 600: 81.6 tok/s (part one 78.4) Structured count 200: 138.3 (131.8) Code clamp: 133.2 (126.4) JSON catalog: 118.6 (111.8) That's up 4 to 6% across the board, which is run-to-run band. I'm not crediting FlashInfer for it. At four streams: structured 383.1 aggregate, code 330.5, JSON 290.4, prose 200.5. Single-stream structured costs 0.52 J per token of GPU power across both boxes. Tool-call smoke test clean, finish_reason tool_calls, no XML in content. DEPTH Needle retrieval 18/18, from 8k out to the 900k rung (880k server-measured). Decode after fill holds between 65 and 77 tok/s the whole way, 70.6 at the top. Prefill runs 3,245 tok/s at 128k and still 2,577 at 880k. AGENT TURNS The OMP agent lane now runs a full 2,048-token write_file turn at 35k context, replacing part one's 35-token window: 107.4 tok/s decode. The Hermes research turn, two web_search calls at 35k context: 81.0 tok/s, up from 74.7. STILL NOT LIKE-FOR-LIKE Against Mia's d2f54b7 recipe on this same pair: prose 1.6x, structured 2.0x, code 2.0x, JSON 1.9x, Hermes turn 2.2x. Different quant, K=5 vs K=3 drafting, bf16 KV vs fp8. My own version is also my own customized recipe, but it felt better to just serve a PR to the creator, since I didn't invent anything new. The biggest single lever is his output head: 1.18 GiB down to 0.33 GiB, read about 5.4 times every decode step, worth +30% engine steps on its own. SUMMARY Every clean lane faster than the mainstream recipe on this rig: checked. Retrieval 18/18 to 900k: checked. Decode holds to 900k: checked. Agent lane measured over a full 2,048-token window: checked. Hermes out of the box: PR open, work in progress. The speed was always the hard part, and vr8vr8 had it in part one. Sources , Benchmarks and Recipe in Reply 👇

Qwen 3.8 Flash decisively crushes the M5 Ultra with this new recipe by vr8vr8 . I think I found the fastest dual DGX Spark recipe for Qwen3.8-Flash-Next. 138 tok/s structured, 133 code, 383 structured aggregate at four streams, and now it talks to Hermes. It feels like a cloud API, running on your desk. This should torch any current M5 run right now. This is part two. Same pair of Sparks, vr8vr8's recipe, his new v4.1, plus a patch of mine that's now an open PR against his repo. WHAT INITIALLY BROKE Through Hermes, the recipe answered in garbled mixed Chinese, then looped on "function86 to90 turn91" until it hit the token cap. Most people would blame the quant. The quant is fine. Three things in the serving stack were stacking up: Hermes doesn't send a temperature for this model. When a request omits it, vLLM 0.30 applies the checkpoint's own generation config: temperature 1.0, top_p 0.95, top_k 20, sampling on. The documented opt-out flag still leaves the server default at 1.0. Every Hermes turn was sampling hot. Hermes sends reasoning as a top-level reasoning object. vLLM 0.30 only maps a field called reasoning_effort onto thinking. So "thinking off" never arrived. Thinking is on by default in the chat template, and with no effort set it defaults to xhigh. A request for no thinking became the longest think the model can do, at temperature 1.0. This is why benchmarks alone do not tell you if a recipe is ready. A real harness that leaves temperature out and names the reasoning field differently gets a different model. Mia's older image never applied the generation config and drafts with argmax, so omitted temperature never sampled hot there. This recipe samples the draft. THE PATCH Two changes to vLLM's chat protocol, applied on both ranks by The Hermes reasoning object now sets thinking. none turns it off. low, medium and xhigh pass through to the template. An explicit enable_thinking still wins. If a request omits temperature, the chat path uses 0. An explicit temperature still passes through. Receipts: three table requests with no temperature and thinking off came back in English. A medium request closed its think and answered in English. The PR is open, not merged. Until it lands, a stock Hermes custom provider still samples at 1.0. HIS V4.1 vr8vr8 shipped recipe v4.1 in the meantime, same v6 image: FlashInfer GDN prefill. His card claims about +5% prefill with decode unchanged. On my depth ladder, prefill came in +6 to +7% from 128k to 900k. Claim holds. THE SECOND GRID Same pair, same protocols, thinking off, temperature 0. My runners already send temperature 0, so this is v4.1 against part one, not a test of the patch. Prose 600: 81.6 tok/s (part one 78.4) Structured count 200: 138.3 (131.8) Code clamp: 133.2 (126.4) JSON catalog: 118.6 (111.8) That's up 4 to 6% across the board, which is run-to-run band. I'm not crediting FlashInfer for it. At four streams: structured 383.1 aggregate, code 330.5, JSON 290.4, prose 200.5. Single-stream structured costs 0.52 J per token of GPU power across both boxes. Tool-call smoke test clean, finish_reason tool_calls, no XML in content. DEPTH Needle retrieval 18/18, from 8k out to the 900k rung (880k server-measured). Decode after fill holds between 65 and 77 tok/s the whole way, 70.6 at the top. Prefill runs 3,245 tok/s at 128k and still 2,577 at 880k. AGENT TURNS The OMP agent lane now runs a full 2,048-token write_file turn at 35k context, replacing part one's 35-token window: 107.4 tok/s decode. The Hermes research turn, two web_search calls at 35k context: 81.0 tok/s, up from 74.7. STILL NOT LIKE-FOR-LIKE Against Mia's d2f54b7 recipe on this same pair: prose 1.6x, structured 2.0x, code 2.0x, JSON 1.9x, Hermes turn 2.2x. Different quant, K=5 vs K=3 drafting, bf16 KV vs fp8. My own version is also my own customized recipe, but it felt better to just serve a PR to the creator, since I didn't invent anything new. The biggest single lever is his output head: 1.18 GiB down to 0.33 GiB, read about 5.4 times every decode step, worth +30% engine steps on its own. SUMMARY Every clean lane faster than the mainstream recipe on this rig: checked. Retrieval 18/18 to 900k: checked. Decode holds to 900k: checked. Agent lane measured over a full 2,048-token window: checked. Hermes out of the box: PR open, work in progress. The speed was always the hard part, and vr8vr8 had it in part one. Sources , Benchmarks and Recipe in Reply 👇

21,745 Aufrufe

Shipped. Minimax H3 EZLaunch. One install script. Full optimized stack for RTX 3090 and RTX 4090. Windows AND Linux. Ships with every model you need. T2V, I2V, and Ref2V all supported. 5 second clips in about 90 seconds. 15 second clips in under 8 minutes. Zero OOMs across 80 minutes of continuous generation. Same stack that held clean across all 10 space battle renders. What it actually does: - Downloads and configures the correct weight set automatically: pruned INT8 UNET with ConvRot, Turbo LoRA (8-step quality band), Qwen3-VL text encoder in NVFP4, MiniMax video + audio VAEs. No hunting through HuggingFace for the right quants. - Full mode coverage. Text-to-video out of the box. Image-to-video with first/last frame. Reference-to-video for style-guided generation. O - Patches SageAttention v2 sm89 to Triton INT8. - Enables kitchen CUDA with FORCE_CUDA on torch cu128 + driver 580 - Full flag set applied: lowvram, disable-smart-memory, disable-pinned-memory, disable-cuda-malloc, fp16-intermediates, CLIP on CPU - Timed smoke test runner included. Verify your stack without guessing. - Cross-platform installer. Same script, same results, Windows or Linux. The weight selection alone saves people hours. The Sage patch is the difference between launch-fail and sub-90 second renders on Ada. Nobody had either documented publicly. Now it is one command. Try it on similar cards and tweak if you want. MVP. More GPU configurations coming. Stay Tuned. Repo link in reply.

Shipped. Minimax H3 EZLaunch. One install script. Full optimized stack for RTX 3090 and RTX 4090. Windows AND Linux. Ships with every model you need. T2V, I2V, and Ref2V all supported. 5 second clips in about 90 seconds. 15 second clips in under 8 minutes. Zero OOMs across 80 minutes of continuous generation. Same stack that held clean across all 10 space battle renders. What it actually does: - Downloads and configures the correct weight set automatically: pruned INT8 UNET with ConvRot, Turbo LoRA (8-step quality band), Qwen3-VL text encoder in NVFP4, MiniMax video + audio VAEs. No hunting through HuggingFace for the right quants. - Full mode coverage. Text-to-video out of the box. Image-to-video with first/last frame. Reference-to-video for style-guided generation. O - Patches SageAttention v2 sm89 to Triton INT8. - Enables kitchen CUDA with FORCE_CUDA on torch cu128 + driver 580 - Full flag set applied: lowvram, disable-smart-memory, disable-pinned-memory, disable-cuda-malloc, fp16-intermediates, CLIP on CPU - Timed smoke test runner included. Verify your stack without guessing. - Cross-platform installer. Same script, same results, Windows or Linux. The weight selection alone saves people hours. The Sage patch is the difference between launch-fail and sub-90 second renders on Ada. Nobody had either documented publicly. Now it is one command. Try it on similar cards and tweak if you want. MVP. More GPU configurations coming. Stay Tuned. Repo link in reply.

13,279 Aufrufe