Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

DeepSeek-V4.1-Flash-1017 decodes code at 80.5 tok/s on one Mac Studio M5 Ultra 256 GB with BobClawblaw's recipe on TensorFold. Setup: – Recipe and TensorFold branch deepseek-v41-main (72a552ed), his own tool and settings – Context 524,288 tokens, 4 streams, DSpark drafts on, temperature 0 – Speed and quality with thinking...

niklaslenz_ai's profile picture

Niklas

2,214 subscribers

11,028 görüntüleme • 4 gün önce •via X (Twitter)

Bilim & TeknolojiOyun
LIVE

Anya Rossi• Live Now

Private livecam show

16 Yorum

Ash Hart profil fotoğrafı
Ash Hart3 gün önce

Cheers for all the testing Niklas!

Niklas profil fotoğrafı
Niklas3 gün önce

🙏🏽

Dangwon profil fotoğrafı
Dangwon3 gün önce

@ksuniri What a great result! Thanks for sharing.

Martin Kessler profil fotoğrafı
Martin Kessler3 gün önce

There’s an upstream PR that almost doubles this.

Niklas profil fotoğrafı
Niklas3 gün önce

Very nice, thank you. I will have a look at it.

Marc Grau profil fotoğrafı
Marc Grau3 gün önce

@TechMDAI

Alizeh profil fotoğrafı
Alizeh3 gün önce

80 tok/s on mac studio crazy fast

try.works profil fotoğrafı
try.works3 gün önce

Those are actually really bad scores on mmlu pro and LCB. The API gets like 80+

杉山靖彦/AI&IT×税務会計×海外リモワ profil fotoğrafı
杉山靖彦/AI&IT×税務会計×海外リモワ3 gün önce

おお！Mac Studioが届くのが楽しみだ😊❗️

Gabriel Garrett profil fotoğrafı
Gabriel Garrett3 gün önce

Which quant level?

Jay profil fotoğrafı
Jay3 gün önce

How much mem does it use?

Laurent Marques profil fotoğrafı
Laurent Marques3 gün önce

what quant version 1, 4 bits ?

MrUniverse profil fotoğrafı
MrUniverse3 gün önce

@rez0__

Sean Cowan profil fotoğrafı
Sean Cowan3 gün önce

These evals and your site are such a great resource for the community! Any chance you'll be adding a few other tiers for Mac like 64/128 GB or a harness shootout?

汉卿 | hanqing profil fotoğrafı
汉卿 | hanqing3 gün önce

本地 80 tok/s，DeepSeek 在 Mac Studio 上跑得真快。

Elliot profil fotoğrafı
Elliot3 gün önce

What quant is this?

Benzer Videolar

Incredible! I have never seen a better music machine for my prompt, Opus and Sonnet included. 9/10. Turn your sound on. DeepSeek-V4.1-Flash-1017 built it on four DGX Sparks in 93.9 min, with Mia's TensorFold recipe. I checked it myself: this is not an Opus output. It ran on my own Sparks. Setup: – Recipe: Mia's, commit 6beaad9, with the 8 TF_DS_* switches her README table sets on; one model split across all four Sparks (TP4), 16 streams – Context: 1,048,576 tokens – Agent: DeepSeek Harness 0.2.1-rc.2, one prompt, one try, no help, thinking on – Sampling from the DeepSeek model card: temperature 1.0, top_p 1.0, top_k 20 – The sandbox had no internet; all 71 requests went to my local server Speed: – Decode, code: 69.5 tok/s (separate speed run, thinking off) – Decode, prose: 70.2 tok/s – 500 words: about 10.0 s – Eight requests at once: 139.0 tok/s in total The machine: – Built in 93.9 min, 272,026 tokens – Browser check: 9.0/10 – 18 voices, a wheel with 720 sockets, 6 marble types, 4 pieces – All sound is synthesized live in the browser, no samples – Sound on: the music in the video is what the page plays itself Good to know: – After 90 minutes the harness told the model to finish, and it was done at 93.9. – The browser check took 1 point off for the frame-rate test (under 3 fps, it renders in software). – The recipe's default answer budget is 32,768 tokens, the model card allows 262,144, so I raised the limit to that. – On her own README method my ring gets 119.9 tok/s on code, hers is 117.4. Her README lists 165.3 tok/s at eight requests (prose, thinking on, temperature 0.7); my speed test runs thinking off at the model card's sampling. – The ranking card at the end is the class ranking from my voxel-world test. Play the music machine in your browser, every number and the raw output: Mia's recipe and the exact setup: Recipe by Mia. TensorFold engine by Ash Hart. Model by DeepSeek.
0:35

Incredible! I have never seen a better music machine for my prompt, Opus and Sonnet included. 9/10. Turn your sound on. DeepSeek-V4.1-Flash-1017 built it on four DGX Sparks in 93.9 min, with Mia's TensorFold recipe. I checked it myself: this is not an Opus output. It ran on my own Sparks. Setup: – Recipe: Mia's, commit 6beaad9, with the 8 TF_DS_* switches her README table sets on; one model split across all four Sparks (TP4), 16 streams – Context: 1,048,576 tokens – Agent: DeepSeek Harness 0.2.1-rc.2, one prompt, one try, no help, thinking on – Sampling from the DeepSeek model card: temperature 1.0, top_p 1.0, top_k 20 – The sandbox had no internet; all 71 requests went to my local server Speed: – Decode, code: 69.5 tok/s (separate speed run, thinking off) – Decode, prose: 70.2 tok/s – 500 words: about 10.0 s – Eight requests at once: 139.0 tok/s in total The machine: – Built in 93.9 min, 272,026 tokens – Browser check: 9.0/10 – 18 voices, a wheel with 720 sockets, 6 marble types, 4 pieces – All sound is synthesized live in the browser, no samples – Sound on: the music in the video is what the page plays itself Good to know: – After 90 minutes the harness told the model to finish, and it was done at 93.9. – The browser check took 1 point off for the frame-rate test (under 3 fps, it renders in software). – The recipe's default answer budget is 32,768 tokens, the model card allows 262,144, so I raised the limit to that. – On her own README method my ring gets 119.9 tok/s on code, hers is 117.4. Her README lists 165.3 tok/s at eight requests (prose, thinking on, temperature 0.7); my speed test runs thinking off at the model card's sampling. – The ranking card at the end is the class ranking from my voxel-world test. Play the music machine in your browser, every number and the raw output: Mia's recipe and the exact setup: Recipe by Mia. TensorFold engine by Ash Hart. Model by DeepSeek.

Niklas

23,772 görüntüleme • 3 gün önce

GLM-5.3-Flash on two DGX Sparks built this voxel sky kingdom in 71.1 min, on Mia's recipe and the TensorFold Python engine. Mia's recipe at commit 3fd60dfc: EXL3 4 bpw quant, one model split across both Sparks (TP2), DFlash2 draft model, 8 slots (her recipe allows up to 8; the default on two Sparks is 4). Agent: OpenCode 1.19.3. One prompt, one try, no help. Thinking on, sampling from the model card: temperature 1.0, top_p 0.95. Voxel world: done in 71.1 min Tokens written: 109,305 Browser check: 9/10 Decode, code: 51.2 tok/s (separate speed run, thinking off) Decode, prose: 48.6 tok/s 500 words: about 14.0 s Eight requests at once: 99.2 tok/s in total Context: 1,048,576 tokens, the recipe's setting Aetherfall: floating isles over a cloud sea with a sky whale and its calf, a waterfall, a windmill, a castle, a shrine, airships and its own camera tour. Open: parts of the world are cut off hard at the edges, most likely a bug in the generated code. Answer limit in the build test: 65,536 tokens, the figure names for its Terminal-Bench runs. Its model card names up to 163,840 for another test. The longest answer was 54,782 tokens, so none was cut. Before this I ran the same task on Mia's Zig-engine branch, which her docs call work in progress: Decode, code: 26.6 tok/s Decode, prose: 25.5 tok/s Voxel world: done in 98.8 min Browser check: 8.0/10 The Python engine is the one her main recipe runs. Play the world in your browser. Every number, the exact command and the raw output: Recipe and quant by Mia. TensorFold engine by Ash Hart. GLM-5.3-Flash by Z.ai. DFlash2 draft by Inco AI.
0:42

GLM-5.3-Flash on two DGX Sparks built this voxel sky kingdom in 71.1 min, on Mia's recipe and the TensorFold Python engine. Mia's recipe at commit 3fd60dfc: EXL3 4 bpw quant, one model split across both Sparks (TP2), DFlash2 draft model, 8 slots (her recipe allows up to 8; the default on two Sparks is 4). Agent: OpenCode 1.19.3. One prompt, one try, no help. Thinking on, sampling from the model card: temperature 1.0, top_p 0.95. Voxel world: done in 71.1 min Tokens written: 109,305 Browser check: 9/10 Decode, code: 51.2 tok/s (separate speed run, thinking off) Decode, prose: 48.6 tok/s 500 words: about 14.0 s Eight requests at once: 99.2 tok/s in total Context: 1,048,576 tokens, the recipe's setting Aetherfall: floating isles over a cloud sea with a sky whale and its calf, a waterfall, a windmill, a castle, a shrine, airships and its own camera tour. Open: parts of the world are cut off hard at the edges, most likely a bug in the generated code. Answer limit in the build test: 65,536 tokens, the figure names for its Terminal-Bench runs. Its model card names up to 163,840 for another test. The longest answer was 54,782 tokens, so none was cut. Before this I ran the same task on Mia's Zig-engine branch, which her docs call work in progress: Decode, code: 26.6 tok/s Decode, prose: 25.5 tok/s Voxel world: done in 98.8 min Browser check: 8.0/10 The Python engine is the one her main recipe runs. Play the world in your browser. Every number, the exact command and the raw output: Recipe and quant by Mia. TensorFold engine by Ash Hart. GLM-5.3-Flash by Z.ai. DFlash2 draft by Inco AI.

Niklas

16,730 görüntüleme • 5 gün önce

DeepSeek-V4.1-Flash-1017 on 4 DGX Sparks vs 2 DGX Sparks, same voxel-world prompt: the 2-Spark run finished in 59.7 minutes. Both recipes are by Mia. 4 Sparks runs her TP4 recipe on SGLang. 2 Sparks runs her recipe on Ash Hart's TensorFold. Not just a box count: 4 Sparks: official checkpoint, sampling from the model card. 2 Sparks: EXL3 2.9 bpw quant, sampling from Mia's recipe. The answer limit was raised to 262,144 tokens on both. 4 Sparks, world Lumenfall: Agent time: 90.8 min Tokens written: 196,854 Browser check: 9/10 Decode (speed test): 51 tok/s 2 Sparks, world Aetherfall: Agent time: 59.7 min Tokens written: 131,268 Browser check: 8/10 Decode (speed test): 67 tok/s Context tested on both: 128K. Open: only the 4-Spark run reached the 90 minute mark and got the automatic finish-now note. Open: 4-Spark decode on the code prompt swung between two runs (36 and 65 tok/s), cause not found. The 2-Spark world is almost as good. Really good. Play both worlds in your browser: 4 Sparks: 2 Sparks:
0:38

DeepSeek-V4.1-Flash-1017 on 4 DGX Sparks vs 2 DGX Sparks, same voxel-world prompt: the 2-Spark run finished in 59.7 minutes. Both recipes are by Mia. 4 Sparks runs her TP4 recipe on SGLang. 2 Sparks runs her recipe on Ash Hart's TensorFold. Not just a box count: 4 Sparks: official checkpoint, sampling from the model card. 2 Sparks: EXL3 2.9 bpw quant, sampling from Mia's recipe. The answer limit was raised to 262,144 tokens on both. 4 Sparks, world Lumenfall: Agent time: 90.8 min Tokens written: 196,854 Browser check: 9/10 Decode (speed test): 51 tok/s 2 Sparks, world Aetherfall: Agent time: 59.7 min Tokens written: 131,268 Browser check: 8/10 Decode (speed test): 67 tok/s Context tested on both: 128K. Open: only the 4-Spark run reached the 90 minute mark and got the automatic finish-now note. Open: 4-Spark decode on the code prompt swung between two runs (36 and 65 tok/s), cause not found. The 2-Spark world is almost as good. Really good. Play both worlds in your browser: 4 Sparks: 2 Sparks:

Niklas

15,876 görüntüleme • 9 gün önce

DeepSeek V4.1 Flash 1017 on 4x DGX Sparks ⚡️ - Official weights on TensorFold - 8 concurrent streams with thinking on
0:35

DeepSeek V4.1 Flash 1017 on 4x DGX Sparks ⚡️ - Official weights on TensorFold - 8 concurrent streams with thinking on

Mia

22,649 görüntüleme • 6 gün önce

Same Mac. Same model. Same words. My 512 GB M3 Ultra ran GLM-5.3-Flash on MLX last week: 27 tok/s, first token in 0.46 s. Today it runs TensorFold 0.6.2: 61 tok/s, first token in 0.20 s. In the clip it laps the reply twice while the MLX side is still typing. Same reply, byte for byte. Then eight requests at once, 78 tok/s combined, every answer identical to its solo run. The old engine took one at a time. Context stays at 1,048,576 on the Mac. Every number measured on my own machine with the same probe, one week apart. Nothing estimated. Ash Hart built it. Apache 2.0.
0:32

Same Mac. Same model. Same words. My 512 GB M3 Ultra ran GLM-5.3-Flash on MLX last week: 27 tok/s, first token in 0.46 s. Today it runs TensorFold 0.6.2: 61 tok/s, first token in 0.20 s. In the clip it laps the reply twice while the MLX side is still typing. Same reply, byte for byte. Then eight requests at once, 78 tok/s combined, every answer identical to its solo run. The old engine took one at a time. Context stays at 1,048,576 on the Mac. Every number measured on my own machine with the same probe, one week apart. Nothing estimated. Ash Hart built it. Apache 2.0.

Chad Hurley is Volatile Markets

28,183 görüntüleme • 27 gün önce

DeepSeek V4.1 Flash 1017 on TensorFold ⚡️ - 16 concurrent session - Each with full native 1M context - Non-greedy, thinking on. OFFICIAL WEIGHTS. Recipe dropping very soon!
0:53

DeepSeek V4.1 Flash 1017 on TensorFold ⚡️ - 16 concurrent session - Each with full native 1M context - Non-greedy, thinking on. OFFICIAL WEIGHTS. Recipe dropping very soon!

Mia

21,711 görüntüleme • 4 gün önce

Agreed DeepSeek-v4.1-flash is an amazing model. and we fixed the speed, it runs at 615+ tok/s on Wally here it is against DeepSeek's own api making changes to the 2048 game -:
0:40

Agreed DeepSeek-v4.1-flash is an amazing model. and we fixed the speed, it runs at 615+ tok/s on Wally here it is against DeepSeek's own api making changes to the 2048 game -:

RunAnywhere

30,457 görüntüleme • 9 gün önce

GLM-5.3-Flash streaming at 32 tok/s on a single Spark via Tensorfold ⚡️ i will share the recipe later today
0:30

GLM-5.3-Flash streaming at 32 tok/s on a single Spark via Tensorfold ⚡️ i will share the recipe later today

filipe

15,230 görüntüleme • 27 gün önce

Final results for DeepSeek-V4-Flash 284B 3107 on 4x3090 (2-bit quant, my llama.cpp fork) after all optimizations. This morning: 433 tok/s prefill. Right now: 1906 tok/s. At 98K depth, not empty context. Yes I was not mistaken. Its x4.3 Decode untouched, 37-40 tok/s. We did not change a single model weight. One day of removing three walls between the CPU and the cards. Chronology in the thread, raw server log in the video.
1:00

Final results for DeepSeek-V4-Flash 284B 3107 on 4x3090 (2-bit quant, my llama.cpp fork) after all optimizations. This morning: 433 tok/s prefill. Right now: 1906 tok/s. At 98K depth, not empty context. Yes I was not mistaken. Its x4.3 Decode untouched, 37-40 tok/s. We did not change a single model weight. One day of removing three walls between the CPU and the cards. Chronology in the thread, raw server log in the video.

Alexey Fateev

19,246 görüntüleme • 2 ay önce

I didn't expect DeepSeek v4 PRO (not Flash) to run well on the Mac Studio M3 Ultra with 512GB of RAM. This is 2 bit quantized with the same DwarfStar recipe used for Flash. 433GB GGUF file. 130 t/s prefill, 13 t/s generation. Prefill in the video is low because small prompt.
1:24

I didn't expect DeepSeek v4 PRO (not Flash) to run well on the Mac Studio M3 Ultra with 512GB of RAM. This is 2 bit quantized with the same DwarfStar recipe used for Flash. 433GB GGUF file. 130 t/s prefill, 13 t/s generation. Prefill in the video is low because small prompt.

antirez

173,075 görüntüleme • 5 ay önce

Splash is here! A month ago, DFlash 2 ran Qwen3.8-27B at 70 tok/s on an M5 Max. Today, Splash runs it at 144 tok/s on the same Mac ⚡ Getting that speed used to take work. Splash is built around the model, with the DFlash 2 draft already in it. Two commands and you're running.
0:26

Splash is here! A month ago, DFlash 2 ran Qwen3.8-27B at 70 tok/s on an M5 Max. Today, Splash runs it at 144 tok/s on the same Mac ⚡ Getting that speed used to take work. Splash is built around the model, with the DFlash 2 draft already in it. Two commands and you're running.

Zhijian Liu

95,415 görüntüleme • 1 ay önce

DeepSeek V4.1 Flash 1017 runs at 1,521+ tok/s on Fireworks Ultrafast. Fast enough to ship Tetris in 11 seconds, 7x faster than Claude Opus 5.5. Join the Ultrafast waitlist:
1:31

DeepSeek V4.1 Flash 1017 runs at 1,521+ tok/s on Fireworks Ultrafast. Fast enough to ship Tetris in 11 seconds, 7x faster than Claude Opus 5.5. Join the Ultrafast waitlist:

Fireworks

1,046,134 görüntüleme • 8 gün önce

DeepSeek V4 Flash & V4 Pro Official API Speed Test V4 Flash: 80.63 tok/s V4 Pro: 36.72 tok/s
0:22

DeepSeek V4 Flash & V4 Pro Official API Speed Test V4 Flash: 80.63 tok/s V4 Pro: 36.72 tok/s

stevibe

17,555 görüntüleme • 6 ay önce

27B model in 8.45 GB. 60 tok/s on a single 3090. And it built this voxel pagoda from a single prompt. Qwen3.8-27B in 2.4 bits from Mirai Labs is on a completely different level compared to the mess from PrismML. Yeah, it's slightly larger in size, but these guys don't claim 98% accuracy either, unlike some others. Pagoda: 1 prompt, medium reasoning, pi agent, 10 turns, 32.5K output tokens, 12 minutes. A single HTML file on three.js. Speed on my rig, GPU at 290 W, vLLM: 1x 3090: 60 tok/s with MTP, 44 without, prefill ~1190 tok/s on a 7K prompt, 159K context 2x 3090 (pipeline parallel): 66 tok/s, prefill ~1890 tok/s, full 262K context Huge thanks to Artur Chakhvadze and ryan mathieu for the opportunity to test this model before release and help the team with testing.
0:32

27B model in 8.45 GB. 60 tok/s on a single 3090. And it built this voxel pagoda from a single prompt. Qwen3.8-27B in 2.4 bits from Mirai Labs is on a completely different level compared to the mess from PrismML. Yeah, it's slightly larger in size, but these guys don't claim 98% accuracy either, unlike some others. Pagoda: 1 prompt, medium reasoning, pi agent, 10 turns, 32.5K output tokens, 12 minutes. A single HTML file on three.js. Speed on my rig, GPU at 290 W, vLLM: 1x 3090: 60 tok/s with MTP, 44 without, prefill ~1190 tok/s on a 7K prompt, 159K context 2x 3090 (pipeline parallel): 66 tok/s, prefill ~1890 tok/s, full 262K context Huge thanks to Artur Chakhvadze and ryan mathieu for the opportunity to test this model before release and help the team with testing.

Alexey Fateev

29,548 görüntüleme • 1 ay önce

New LFM2.5-2.6B hits DeepSeek-V4 level on tool calling and runs 3.7x faster! We ran Liquid AI 's new LFM2.5-2.6B against DeepSeek-V4-Flash on one box with 4x RTX 5090. Both got the same three jobs, and each one only completes if the model fires every tool call Topics: -weather and local time in six cities -one budget into six currencies -four hotels checked and booked for one date Outputs: LFM2.5-2.6B: 35/35 tool-calls, 366 tok/s, 19s DeepSeek-V4-Flash: 35/35 tool-calls, 77 tok/s, 70s Both models made all 35 calls, and both fired twelve of them in one turn. LFM was trained for agents, and tool calling is where that shows. Strong result for a model you can run right on your phone
0:25

New LFM2.5-2.6B hits DeepSeek-V4 level on tool calling and runs 3.7x faster! We ran Liquid AI 's new LFM2.5-2.6B against DeepSeek-V4-Flash on one box with 4x RTX 5090. Both got the same three jobs, and each one only completes if the model fires every tool call Topics: -weather and local time in six cities -one budget into six currencies -four hotels checked and booked for one date Outputs: LFM2.5-2.6B: 35/35 tool-calls, 366 tok/s, 19s DeepSeek-V4-Flash: 35/35 tool-calls, 77 tok/s, 70s Both models made all 35 calls, and both fired twelve of them in one turn. LFM was trained for agents, and tool calling is where that shows. Strong result for a model you can run right on your phone

atomic.chat

113,185 görüntüleme • 2 ay önce

DeepSeek V4.1 Flash 1017 vs Minimax M3.1 Flash Same prompt. Same task. Deepseek Time: 1 h 28 min 27 s Tokens: 89.487.454 Cost: $0,54 Minimax Time: 1 h 6 min 51 s Tokens: 8.120.000 Cost: $0,22 Which one is better?
0:30

DeepSeek V4.1 Flash 1017 vs Minimax M3.1 Flash Same prompt. Same task. Deepseek Time: 1 h 28 min 27 s Tokens: 89.487.454 Cost: $0,54 Minimax Time: 1 h 6 min 51 s Tokens: 8.120.000 Cost: $0,22 Which one is better?

Rubens Soto | AI & SaaS

44,222 görüntüleme • 10 gün önce

bonsai 2 27b on an rtx 3060 12gb, the full receipt sheet. save this one, the 12gb row of the small gpu guide is built from it. speed by depth, then what context costs, live server, thinking on, real sessions > 7k deep: 24.4 tok/s > 12k deep: 21.9 tok/s > 35k deep: 17.8 tok/s > 77k deep: 13.0 tok/s > 64k window: 7.3gb resident > 128k window: 8.8gb resident > 192k window: 10.2gb resident > 262k window: 11.7gb resident, 0.6gb to spare, the whole native window on a 12gb card > every 64k of context costs 1.47gb, so 327k would not fit > a 41,312 token build session from 35k to 77k of context averaged 15.0 tok/s across 46 minutes > prefill 295 tok/s at 2k of context, 243 tok/s at 35k, first token in 0.6 seconds, fresh decode 26.1 tok/s > the card pinned 149 of 150 w the entire time, 78c, fan at 80%, power bound, not heat bound > 0.158 tok/s per watt at the fresh end the setup > model: ternary bonsai 2 27b, PTQ1_0, 1.75 bits per weight, 5.95gb on disk, base qwen 3.8 27b, apache 2.0 > runtime: prismml llama.cpp fork, prebuilt cuda 12.4 binary, no compile > serve: full 262k native window resident, q4 kv cache, flash attention, one slot, 11.7 of 12gb in use for anyone who followed bonsai 1 in july, that was the 3.9gb 1bit file at 42 tok/s on a 3060 ti, a faster card and a smaller file, so the same card comparison is not on the table yet, it comes with the 8gb test. what changed is the base, qwen 3.8 instead of 3.6, and the retention, 98.2% on their suite instead of 95%, and the whole 262k window fitting on 12gb.
2:03

bonsai 2 27b on an rtx 3060 12gb, the full receipt sheet. save this one, the 12gb row of the small gpu guide is built from it. speed by depth, then what context costs, live server, thinking on, real sessions > 7k deep: 24.4 tok/s > 12k deep: 21.9 tok/s > 35k deep: 17.8 tok/s > 77k deep: 13.0 tok/s > 64k window: 7.3gb resident > 128k window: 8.8gb resident > 192k window: 10.2gb resident > 262k window: 11.7gb resident, 0.6gb to spare, the whole native window on a 12gb card > every 64k of context costs 1.47gb, so 327k would not fit > a 41,312 token build session from 35k to 77k of context averaged 15.0 tok/s across 46 minutes > prefill 295 tok/s at 2k of context, 243 tok/s at 35k, first token in 0.6 seconds, fresh decode 26.1 tok/s > the card pinned 149 of 150 w the entire time, 78c, fan at 80%, power bound, not heat bound > 0.158 tok/s per watt at the fresh end the setup > model: ternary bonsai 2 27b, PTQ1_0, 1.75 bits per weight, 5.95gb on disk, base qwen 3.8 27b, apache 2.0 > runtime: prismml llama.cpp fork, prebuilt cuda 12.4 binary, no compile > serve: full 262k native window resident, q4 kv cache, flash attention, one slot, 11.7 of 12gb in use for anyone who followed bonsai 1 in july, that was the 3.9gb 1bit file at 42 tok/s on a 3060 ti, a faster card and a smaller file, so the same card comparison is not on the table yet, it comes with the 8gb test. what changed is the base, qwen 3.8 instead of 3.6, and the retention, 98.2% on their suite instead of 95%, and the whole 262k window fitting on 12gb.

Sudo su

31,578 görüntüleme • 1 ay önce

I put MCDMA through its paces today and tested Qwen 3.8 Flash Next on my dual Spark / Mac Studio cluster. My first test was disaggregated prefill across my two Sparks, then decode onto my Studio. PP 2,100 tok/s Decode 80 tok/s at 20k context, higher on short replies. That's faster than either machine manages on its own: the Studio alone prefills at about 1,200 tok/s, and the Sparks alone decode at 35 to 65. 🤯 Mac prefill has always been way behind the Sparks; now I get the best of both worlds with the hardware I currently have.
1:10

I put MCDMA through its paces today and tested Qwen 3.8 Flash Next on my dual Spark / Mac Studio cluster. My first test was disaggregated prefill across my two Sparks, then decode onto my Studio. PP 2,100 tok/s Decode 80 tok/s at 20k context, higher on short replies. That's faster than either machine manages on its own: the Studio alone prefills at about 1,200 tok/s, and the Sparks alone decode at 35 to 65. 🤯 Mac prefill has always been way behind the Sparks; now I get the best of both worlds with the hardware I currently have.

Ash Hart

13,240 görüntüleme • 1 ay önce

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.
0:20

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.

FHILY👑

40,049 görüntüleme • 1 ay önce

Mirai Labs (Mirai Labs) compressed Qwen3.8-27B to 2.4 bits. I ported it to llama.cpp and now it runs on a 12 GB card with 128K context. With q4 KV cache the 128K window takes 11.3 GB of VRAM, with q8 KV you get 74K in 11.1 GB. Decode is 40 tok/s, still 34-35 tok/s at 62K deep, and prefill around 1000 tok/s. I measured all of it on 1 RTX 3090 at 300 W, capped to what fits in 12 GB. No requant anywhere. The GGUF carries Mirai's compressed codes bit for bit, and greedy output matches their own vLLM plugin. Everything in the video was built by this same model as an agent in pi, 11 builds from 1 prompt each, a few needed a follow up and 1 is a blooper. That run was vLLM on my 3090s. Today the same weights run in llama.cpp. Big thread, all links and recipes 👇
2:37

Mirai Labs (Mirai Labs) compressed Qwen3.8-27B to 2.4 bits. I ported it to llama.cpp and now it runs on a 12 GB card with 128K context. With q4 KV cache the 128K window takes 11.3 GB of VRAM, with q8 KV you get 74K in 11.1 GB. Decode is 40 tok/s, still 34-35 tok/s at 62K deep, and prefill around 1000 tok/s. I measured all of it on 1 RTX 3090 at 300 W, capped to what fits in 12 GB. No requant anywhere. The GGUF carries Mirai's compressed codes bit for bit, and greedy output matches their own vLLM plugin. Everything in the video was built by this same model as an agent in pi, 11 builds from 1 prompt each, a few needed a follow up and 1 is a blooper. That run was vLLM on my 3090s. Today the same weights run in llama.cpp. Big thread, all links and recipes 👇

Alexey Fateev

31,893 görüntüleme • 1 ay önce