Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Finally pulled it off! Hit 47.57 tok/s (decode) on a single DGX Spark with all experts loaded. Prefill takes a hit, sure, but for a solo box, not too shabby at all, right?

shi3z's profile picture

shi3z

25,528 subscribers

20,361 Aufrufe • vor 1 Tag •via X (Twitter)

Wissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

1 Kommentare

Profilbild von TN
TNvor 23 Stunden

👀

Ähnliche Videos

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.
0:20

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.

FHILY👑

38,386 Aufrufe • vor 10 Tagen

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.
1:04

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.

stevibe

34,141 Aufrufe • vor 4 Monaten

16 people chatting with Qwen3.6-35B at once ONE DGX Spark. This is a real capture, not a mockup: every token you see replays at its actual measured arrival time. Peaks ~440 tok/s total. A single user gets 105 tok/s. NVFP4 + MTP-3 on vLLM.
0:34

16 people chatting with Qwen3.6-35B at once ONE DGX Spark. This is a real capture, not a mockup: every token you see replays at its actual measured arrival time. Peaks ~440 tok/s total. A single user gets 105 tok/s. NVFP4 + MTP-3 on vLLM.

Wësche

26,094 Aufrufe • vor 2 Monaten

DGX Spark hit 228ms time-to-first-token on Qwen3.6 35B-A3B. That is 4x faster than an RTX 3090 and 45% ahead of the RTX 5090 Raw throughput still belongs to the 5090 at 160 tok/s. Full numbers, same Ollama backend, Q4_K_M across all four: RTX 3090: 49.78 tok/s, 852ms RTX 4090: 118.93 tok/s, 686ms RTX 5090: 160.37 tok/s, 409ms DGX Spark: 59.98 tok/s, 228ms
1:24

DGX Spark hit 228ms time-to-first-token on Qwen3.6 35B-A3B. That is 4x faster than an RTX 3090 and 45% ahead of the RTX 5090 Raw throughput still belongs to the 5090 at 160 tok/s. Full numbers, same Ollama backend, Q4_K_M across all four: RTX 3090: 49.78 tok/s, 852ms RTX 4090: 118.93 tok/s, 686ms RTX 5090: 160.37 tok/s, 409ms DGX Spark: 59.98 tok/s, 228ms

NO1ennn

22,497 Aufrufe • vor 1 Monat

Nemotron-3-Ultra running on 4x 6000s edits my latest demo video.. - 75 tok/s decode - 8x concurrency - 256k context - 899 tok/s prefill - 20k tok/s prefill cache - NVFP4 Setting it up to be my Hermes driver. It's good enough at most things and doesn't talk like a moron.
1:26

Nemotron-3-Ultra running on 4x 6000s edits my latest demo video.. - 75 tok/s decode - 8x concurrency - 256k context - 899 tok/s prefill - 20k tok/s prefill cache - NVFP4 Setting it up to be my Hermes driver. It's good enough at most things and doesn't talk like a moron.

0xSero

15,674 Aufrufe • vor 3 Monaten

Final results for DeepSeek-V4-Flash 284B 3107 on 4x3090 (2-bit quant, my llama.cpp fork) after all optimizations. This morning: 433 tok/s prefill. Right now: 1906 tok/s. At 98K depth, not empty context. Yes I was not mistaken. Its x4.3 Decode untouched, 37-40 tok/s. We did not change a single model weight. One day of removing three walls between the CPU and the cards. Chronology in the thread, raw server log in the video.
1:00

Final results for DeepSeek-V4-Flash 284B 3107 on 4x3090 (2-bit quant, my llama.cpp fork) after all optimizations. This morning: 433 tok/s prefill. Right now: 1906 tok/s. At 98K depth, not empty context. Yes I was not mistaken. Its x4.3 Decode untouched, 37-40 tok/s. We did not change a single model weight. One day of removing three walls between the CPU and the cards. Chronology in the thread, raw server log in the video.

Alexey Fateev

19,190 Aufrufe • vor 1 Monat

sniping feels okay in xdefiant, definitely some work to be done around the clunkiness behind it all. hit a split 5 on first day of release so not too shabby 🤠
0:33

sniping feels okay in xdefiant, definitely some work to be done around the clunkiness behind it all. hit a split 5 on first day of release so not too shabby 🤠

100T Kalei

344,417 Aufrufe • vor 2 Jahren

"Why are you benchmarking DGX Spark? It's a training box." Yeah. Low bandwidth, but 128GB of unified memory is just sitting there. Plenty of room to optimize. DGX Spark + Qwen3.6 27B. Four backend/quant combos: 🔴 llama.cpp + UD_Q4_K_XL > 11.0 tok/s (baseline), TTFT 297ms 🟢 llama.cpp + DFlash > 20.4 tok/s (peaks at 97 tok/s), TTFT 320ms 🟡 vLLM FP8 + MTP > 13.1 tok/s, TTFT 540ms 🟣 vLLM NVFP4 + MTP > 24.2 tok/s, TTFT 376ms NVFP4+MTP is the winner for me, rock stable around 24 tok/s, no wild swings. DFlash is the wildcard: massive peaks, but fluctuates a lot. FP8+MTP barely beats baseline, and it's FP8. Love my Spark.
3:01

"Why are you benchmarking DGX Spark? It's a training box." Yeah. Low bandwidth, but 128GB of unified memory is just sitting there. Plenty of room to optimize. DGX Spark + Qwen3.6 27B. Four backend/quant combos: 🔴 llama.cpp + UD_Q4_K_XL > 11.0 tok/s (baseline), TTFT 297ms 🟢 llama.cpp + DFlash > 20.4 tok/s (peaks at 97 tok/s), TTFT 320ms 🟡 vLLM FP8 + MTP > 13.1 tok/s, TTFT 540ms 🟣 vLLM NVFP4 + MTP > 24.2 tok/s, TTFT 376ms NVFP4+MTP is the winner for me, rock stable around 24 tok/s, no wild swings. DFlash is the wildcard: massive peaks, but fluctuates a lot. FP8+MTP barely beats baseline, and it's FP8. Love my Spark.

stevibe

40,223 Aufrufe • vor 4 Monaten

ending your era with 8th single being a smash hit too with all this budget..
2:13

ending your era with 8th single being a smash hit too with all this budget..

kanishk

34,882 Aufrufe • vor 9 Monaten

DS4 running on DGX Spark (GB10 / CUDA), private branch for now. 12 tokens/sec, the memory bandwidth is limited in this system, at 270GB/sec. But prefill is ways more alighed to M3 Max at ~200 t/s. I'll release when more mature, but it is almost sure that it will get merged.
1:34

DS4 running on DGX Spark (GB10 / CUDA), private branch for now. 12 tokens/sec, the memory bandwidth is limited in this system, at 270GB/sec. But prefill is ways more alighed to M3 Max at ~200 t/s. I'll release when more mature, but it is almost sure that it will get merged.

antirez

84,127 Aufrufe • vor 4 Monaten

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.
1:24

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.

stevibe

402,323 Aufrufe • vor 4 Monaten

You can now run Qwen3.8-27B in full BF16 on a free Kaggle TPU. - with about 130 tok/s decode, - 10k tok/s prefill, - Native 262k context. - No quantization. - No paid GPU. - Measured on TPU v5e-8 It also exposes an OpenAI-compatible endpoint, so it plugs into Claude Code, Codex, and OpenCode.
0:48

You can now run Qwen3.8-27B in full BF16 on a free Kaggle TPU. - with about 130 tok/s decode, - 10k tok/s prefill, - Native 262k context. - No quantization. - No paid GPU. - Measured on TPU v5e-8 It also exposes an OpenAI-compatible endpoint, so it plugs into Claude Code, Codex, and OpenCode.

Md Ismail Šojal 🕷️

50,394 Aufrufe • vor 6 Tagen

RUN OUT. Green sets off on a single but Carse gets him well short with a direct hit! #Ashes
0:41

RUN OUT. Green sets off on a single but Carse gets him well short with a direct hit! #Ashes

7Cricket

98,701 Aufrufe • vor 8 Monaten

AI at Meta's Muse Glimmer (30B dense, open-weights) launches with SGLang day-0 support. We got ~230 tok/s on a single RTX 5090, with NVFP4 + DFlash on. It also works out of the box on NVIDIA AI Developer RTX Pro 6000, DGX Spark, and MLX for Mac. Speed and reliability have always been the blockers for a local agent. SGLang and Muse Glimmer make a local agent that's blazing fast, always on, and gets things done. Go build and run a local agent on your own machine!🚀
0:32

AI at Meta's Muse Glimmer (30B dense, open-weights) launches with SGLang day-0 support. We got ~230 tok/s on a single RTX 5090, with NVFP4 + DFlash on. It also works out of the box on NVIDIA AI Developer RTX Pro 6000, DGX Spark, and MLX for Mac. Speed and reliability have always been the blockers for a local agent. SGLang and Muse Glimmer make a local agent that's blazing fast, always on, and gets things done. Go build and run a local agent on your own machine!🚀

LMSYS Org

43,450 Aufrufe • vor 1 Monat

ICE agents in Minnesota pulled up behind a woman in her vehicle and hit it and then drug her out of her car saying she hit them when it was right in front of another ladies home with a ring camera that captured it all! The woman is an American citizen too!
0:38

ICE agents in Minnesota pulled up behind a woman in her vehicle and hit it and then drug her out of her car saying she hit them when it was right in front of another ladies home with a ring camera that captured it all! The woman is an American citizen too!

Suzie rizzio

111,506 Aufrufe • vor 6 Monaten

Not sure if this was a hit or not but it damn sure should’ve been. She TORE!!! “BREAK ME OFF / and gitchi gitchi YA YA” 🔥
1:55

Not sure if this was a hit or not but it damn sure should’ve been. She TORE!!! “BREAK ME OFF / and gitchi gitchi YA YA” 🔥

ᴍᴀʀᴄ ᴅᴇsʜᴀᴜɴ

1,111,597 Aufrufe • vor 1 Monat

I did a piece awhile back on an LNG tanker getting hit and all the internet experts challenged replied with “But LNG isn’t flammable” 😂💀😂
1:02

I did a piece awhile back on an LNG tanker getting hit and all the internet experts challenged replied with “But LNG isn’t flammable” 😂💀😂

Mr Global

14,878 Aufrufe • vor 6 Monaten

"Your Mac is useless for AI without CUDA, buy the $4K DGX." Half right — and the half that's wrong costs you $4,000. CUDA matters for training and some frameworks. Real. But "running local models"? I benchmarked a Mac Mini against the DGX on the same 30B: generation was 56 vs 84 t/s. Not useless — usable, faster than you read, no CUDA required. llama.cpp and Ollama don't care what logo is on the chip. Here's the honest split the video skips: - Commercial fine-tuning, CUDA-only pipelines, big-context prefill → yes, the DGX earns its price - Running and chatting with local models → your Mac already does this fine The DGX isn't overpriced. It's mis-pitched. It's a prefill-and-training machine, not a "your Mac is trash" machine. Buy it for the workload that needs it — not out of CUDA FOMO. Full 3-way benchmark (DGX vs Strix Halo vs Mac Mini, prefill vs generation) pinned.
0:42

"Your Mac is useless for AI without CUDA, buy the $4K DGX." Half right — and the half that's wrong costs you $4,000. CUDA matters for training and some frameworks. Real. But "running local models"? I benchmarked a Mac Mini against the DGX on the same 30B: generation was 56 vs 84 t/s. Not useless — usable, faster than you read, no CUDA required. llama.cpp and Ollama don't care what logo is on the chip. Here's the honest split the video skips: - Commercial fine-tuning, CUDA-only pipelines, big-context prefill → yes, the DGX earns its price - Running and chatting with local models → your Mac already does this fine The DGX isn't overpriced. It's mis-pitched. It's a prefill-and-training machine, not a "your Mac is trash" machine. Buy it for the workload that needs it — not out of CUDA FOMO. Full 3-way benchmark (DGX vs Strix Halo vs Mac Mini, prefill vs generation) pinned.

KyzoroX

263,861 Aufrufe • vor 2 Monaten

Welcome to Mamdanistan… For 15 hours, not a single plow went by, and when one finally did, it hit someone’s car.
0:44

Welcome to Mamdanistan… For 15 hours, not a single plow went by, and when one finally did, it hit someone’s car.

I Meme Therefore I Am 🇺🇸

690,981 Aufrufe • vor 6 Monaten