Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Qwen3.5-35B with only 3B active parameters This MoE model runs FASTER than most 7B dense models. Tested on 3 generations of NVIDIA: - 5090: 137 tok/s - 4090: 112 tok/s - 3090: 78 tok/s The surprise? The 4090 <> 5090 gap is only 22%. With a 3B active MoE,... show more

stevibe

27,366 subscribers

69,532 Aufrufe • vor 5 Monaten •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

The RTX 3090 is a 5-year-old GPU and it still runs a 27B model at 20 tok/s I tested Qwen3.5:27b across 3 generations of NVIDIA: 5090 → ~60 tok/s 4090 → ~40 tok/s 3090 → ~20 tok/s Perfectly linear scaling. Double the generation, double the speed.

The RTX 3090 is a 5-year-old GPU and it still runs a 27B model at 20 tok/s I tested Qwen3.5:27b across 3 generations of NVIDIA: 5090 → ~60 tok/s 4090 → ~40 tok/s 3090 → ~20 tok/s Perfectly linear scaling. Double the generation, double the speed.

stevibe

142,493 Aufrufe • vor 5 Monaten

Finally got my hands on the big one. Qwen3.5-122B-A10B — 122 billion parameters. Too big for any single consumer GPU. So I rented 4 of each... and then one professional card to see if brute force even matters. - 1x RTX PRO 6000 (96GB): 101.4 tok/s - 4x 5090 (128GB): 87.0 tok/s - 4x 4090 (96GB): 25.1 tok/s - 4x 3090 (96GB): 20.8 tok/s One single $8,500 card beat four RTX 5090s

Finally got my hands on the big one. Qwen3.5-122B-A10B — 122 billion parameters. Too big for any single consumer GPU. So I rented 4 of each... and then one professional card to see if brute force even matters. - 1x RTX PRO 6000 (96GB): 101.4 tok/s - 4x 5090 (128GB): 87.0 tok/s - 4x 4090 (96GB): 25.1 tok/s - 4x 3090 (96GB): 20.8 tok/s One single $8,500 card beat four RTX 5090s

stevibe

195,832 Aufrufe • vor 5 Monaten

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.

stevibe

401,804 Aufrufe • vor 3 Monaten

Laguna S 2.1 running native NVFP4 Picked Blackwell hardware for this side-by-side: 🟢 DGX Spark | 19.44 tok/s | 162ms TTFT 🔵 RTX PRO 6000 | 108.54 tok/s | 266ms TTFT 🟣 4× RTX 5090 (TP4+EP4) | 145.81 tok/s | 521ms TTFT 117.6B MoE (8.5B active), quantized to FP4, on your own box.

Laguna S 2.1 running native NVFP4 Picked Blackwell hardware for this side-by-side: 🟢 DGX Spark | 19.44 tok/s | 162ms TTFT 🔵 RTX PRO 6000 | 108.54 tok/s | 266ms TTFT 🟣 4× RTX 5090 (TP4+EP4) | 145.81 tok/s | 521ms TTFT 117.6B MoE (8.5B active), quantized to FP4, on your own box.

stevibe

23,102 Aufrufe • vor 23 Tagen

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.

stevibe

34,141 Aufrufe • vor 3 Monaten

4×5090 lost badly in my last Ollama test But with vLLM + Nemotron 3 Super NVFP4, it came back swinging. 4×5090 - TTFT: 0.281s - Prompt eval: 608 tok/s - Generate eval: 110 tok/s PRO 6000 - TTFT: 0.123s - Prompt eval: 1390 tok/s - Generate eval: 81 tok/s So who wins? - PRO 6000 is still faster to respond - But 4×5090 now wins on raw generation speed

4×5090 lost badly in my last Ollama test But with vLLM + Nemotron 3 Super NVFP4, it came back swinging. 4×5090 - TTFT: 0.281s - Prompt eval: 608 tok/s - Generate eval: 110 tok/s PRO 6000 - TTFT: 0.123s - Prompt eval: 1390 tok/s - Generate eval: 81 tok/s So who wins? - PRO 6000 is still faster to respond - But 4×5090 now wins on raw generation speed

stevibe

16,984 Aufrufe • vor 5 Monaten

128GB in Framework Desktop means you can run giant models locally like AI at Meta's Llama 4 Scout. This is a 109B MoE model with 17B active parameters. At Q6 this runs at >14 tok/s on LM Studio on Fedora 42!

128GB in Framework Desktop means you can run giant models locally like AI at Meta's Llama 4 Scout. This is a 109B MoE model with 17B active parameters. At Q6 this runs at >14 tok/s on LM Studio on Fedora 42!

Framework

30,053 Aufrufe • vor 1 Jahr

Qwen3.6 27B landed yesterday, so I ran it on 4 setups side-by-side to see how they stack up: 🔴 RTX 4090 — 45.59 tok/s, TTFT 525ms 🟢 RTX 5090 — 51.83 tok/s, TTFT 752ms ⚫️ M2 Ultra — 22.30 tok/s, TTFT 216ms 🟣 DGX Spark — 11.08 tok/s, TTFT 319ms This is a standard test: no tuning, just the out-of-the-box experience. For the NVIDIA cards I used llama.cpp with Unsloth's UD-Q4_K_XL quant. For the M2 Ultra I used MLX with Unsloth's UD-MLX-4bit quant, since MLX is the native path on Apple Silicon. Please consider this as the baseline, you can definitely squeeze more out of every one of these with fine-tuned settings.

Qwen3.6 27B landed yesterday, so I ran it on 4 setups side-by-side to see how they stack up: 🔴 RTX 4090 — 45.59 tok/s, TTFT 525ms 🟢 RTX 5090 — 51.83 tok/s, TTFT 752ms ⚫️ M2 Ultra — 22.30 tok/s, TTFT 216ms 🟣 DGX Spark — 11.08 tok/s, TTFT 319ms This is a standard test: no tuning, just the out-of-the-box experience. For the NVIDIA cards I used llama.cpp with Unsloth's UD-Q4_K_XL quant. For the M2 Ultra I used MLX with Unsloth's UD-MLX-4bit quant, since MLX is the native path on Apple Silicon. Please consider this as the baseline, you can definitely squeeze more out of every one of these with fine-tuned settings.

stevibe

104,479 Aufrufe • vor 3 Monaten

Curious about ollama kimi-k2.6:cloud speed? 3 test runs: > 77.9 tok/s, TTFT 979ms > 114.3 tok/s, TTFT 788ms > 86.3 tok/s, TTFT 1117ms For comparison, OpenRouter stats: > Parasail: 14 tok/s > Moonshot AI: 27 tok/s > NovitaAI: 27 tok/s > Cloudflare: 71 tok/s Obvious caveat: cloud speeds fluctuate with load. Just sharing numbers for the curious.

Curious about ollama kimi-k2.6:cloud speed? 3 test runs: > 77.9 tok/s, TTFT 979ms > 114.3 tok/s, TTFT 788ms > 86.3 tok/s, TTFT 1117ms For comparison, OpenRouter stats: > Parasail: 14 tok/s > Moonshot AI: 27 tok/s > NovitaAI: 27 tok/s > Cloudflare: 71 tok/s Obvious caveat: cloud speeds fluctuate with load. Just sharing numbers for the curious.

stevibe

35,129 Aufrufe • vor 3 Monaten

Step-3.7-Flash Q4_K_S on DGX Spark (GB10, 128GB): > ~27 tok/s generation > 198B sparse MoE, ~11B active > 256K context, native vision > Agentic / tool-calling / reasoning > Apache 2.0 I added a mobile chat screen on the right showing what 27 tok/s actually feels like streaming on a phone.

Step-3.7-Flash Q4_K_S on DGX Spark (GB10, 128GB): > ~27 tok/s generation > 198B sparse MoE, ~11B active > 256K context, native vision > Agentic / tool-calling / reasoning > Apache 2.0 I added a mobile chat screen on the right showing what 27 tok/s actually feels like streaming on a phone.

stevibe

19,658 Aufrufe • vor 2 Monaten

GLM-5.2 Nvidia NVFP4 No pruning, no quantising, exact same model card. 110 tok/s single stream. My friend is a genius, holy fudge. Here's the repo, he got full DS4-Flash on a 5090 + DDR5 at 38.5 tok/s This is a breakthrough.

GLM-5.2 Nvidia NVFP4 No pruning, no quantising, exact same model card. 110 tok/s single stream. My friend is a genius, holy fudge. Here's the repo, he got full DS4-Flash on a 5090 + DDR5 at 38.5 tok/s This is a breakthrough.

0xSero

156,978 Aufrufe • vor 1 Monat

Qwen3.5:9b reasoning head-to-head: Mac Studio M2 Ultra 64GB: 43.08 tok/s Mac Mini M4 16GB: 13.07 tok/s Qwen

Qwen3.5:9b reasoning head-to-head: Mac Studio M2 Ultra 64GB: 43.08 tok/s Mac Mini M4 16GB: 13.07 tok/s Qwen

stevibe

243,343 Aufrufe • vor 5 Monaten

been playing with hermes agent paired with qwen 3.5 dense 27B on my single 3090 since last night. there is something about this harness that caught me and i think i know what it is. i've now run five qwen configs on consumer hardware: 35B MoE (3B active) -- 112 tok/s flat across 262K context, 1x 3090 27B dense -- 35 tok/s, zero degradation across the same range, 1x 3090 qwopus 27B (opus distilled) -- 35.7 tok/s, same architecture, different brain 80B coder -- 46 tok/s on 2x 3090s, oneshotted a 564 line particle sim 80B coder -- 1.3 tok/s on 1x 3090, bleeding through RAM because it didn't fit but it still ran with same benchmarks. same prompts. same quant where possible. every config is documented. i know these models. and hermes agent is the first harness that feels like it respects that work. tool calls show inline with execution time. nvidia-smi 0.2s. write_file 0.7s. you see exactly what the agent is doing and how long each step takes. no mystery. no black box. no tool call failures so far and i've been pushing it. most agent frameworks feel like you're watching a spinner and hoping. hermes shows the work. that transparency changes how you trust the output. once you use it you see the UX decisions are not accidental. Teknium 🪽 and the nous team built this like engineers who actually use their own tools. 80 skills. 29 tools. persistent memory. context compression. runs clean on a single consumer GPU.

been playing with hermes agent paired with qwen 3.5 dense 27B on my single 3090 since last night. there is something about this harness that caught me and i think i know what it is. i've now run five qwen configs on consumer hardware: 35B MoE (3B active) -- 112 tok/s flat across 262K context, 1x 3090 27B dense -- 35 tok/s, zero degradation across the same range, 1x 3090 qwopus 27B (opus distilled) -- 35.7 tok/s, same architecture, different brain 80B coder -- 46 tok/s on 2x 3090s, oneshotted a 564 line particle sim 80B coder -- 1.3 tok/s on 1x 3090, bleeding through RAM because it didn't fit but it still ran with same benchmarks. same prompts. same quant where possible. every config is documented. i know these models. and hermes agent is the first harness that feels like it respects that work. tool calls show inline with execution time. nvidia-smi 0.2s. write_file 0.7s. you see exactly what the agent is doing and how long each step takes. no mystery. no black box. no tool call failures so far and i've been pushing it. most agent frameworks feel like you're watching a spinner and hoping. hermes shows the work. that transparency changes how you trust the output. once you use it you see the UX decisions are not accidental. Teknium 🪽 and the nous team built this like engineers who actually use their own tools. 80 skills. 29 tools. persistent memory. context compression. runs clean on a single consumer GPU.

Sudo su

100,259 Aufrufe • vor 5 Monaten

Update: Qwen3.5:9b head-to-head (MLX, Apple Silicon optimized) Mac Studio M2 Ultra: 89.74 tok/s Mac Mini M4: 20.82 tok/s MLX basically doubles the speed on both machines.

Update: Qwen3.5:9b head-to-head (MLX, Apple Silicon optimized) Mac Studio M2 Ultra: 89.74 tok/s Mac Mini M4: 20.82 tok/s MLX basically doubles the speed on both machines.

stevibe

134,256 Aufrufe • vor 5 Monaten

MiniMax M2.7 is 230B params. Can you actually run it at home? I tested Unsloth's UD-IQ3_XXS (80GB) on 4 different rigs: 🟠 4x RTX 4090 (96GB): 71.52 tok/s, TTFT 1045ms 🟢 4x RTX 5090 (128GB): 120.54 tok/s, TTFT 725ms 🟡 1x RTX PRO 6000 (96GB): 118.74 tok/s, TTFT 765ms 🟣 DGX Spark (128GB) — 24.41 tok/s, TTFT 741ms Backend: llama.cpp. Context: 32k. Max tokens: 4096. I went with IQ3_XXS because it's the biggest quant that fits in 96GB VRAM while still leaving safe headroom for 32k context. Same quant across all four rigs, fairest comparison I could run. Now look at rough peak GPU power draw: 🟠 4x4090 → 1,800W peak (450W × 4) 🟢 4x5090 → 2,300W peak (575W × 4) 🟡 RTX PRO 6000 → 600W peak 🟣 DGX Spark → 240W peak (whole system) The RTX PRO 6000 is the quiet winner. One card, 96GB, matching a 4x5090 rig at roughly a quarter of the power and zero multi-GPU headaches. Best tokens-per-watt by a wide margin. DGX Spark is slow on generation but pulls the least power of any rig here, around 240W for the whole system. Prefill-friendly, memory-rich, wall-socket-friendly. And yes, plenty of people cap their cards. Even then, 4x 4090 or 4x 5090 still pulls well over 1,200W from the GPUs alone.

MiniMax M2.7 is 230B params. Can you actually run it at home? I tested Unsloth's UD-IQ3_XXS (80GB) on 4 different rigs: 🟠 4x RTX 4090 (96GB): 71.52 tok/s, TTFT 1045ms 🟢 4x RTX 5090 (128GB): 120.54 tok/s, TTFT 725ms 🟡 1x RTX PRO 6000 (96GB): 118.74 tok/s, TTFT 765ms 🟣 DGX Spark (128GB) — 24.41 tok/s, TTFT 741ms Backend: llama.cpp. Context: 32k. Max tokens: 4096. I went with IQ3_XXS because it's the biggest quant that fits in 96GB VRAM while still leaving safe headroom for 32k context. Same quant across all four rigs, fairest comparison I could run. Now look at rough peak GPU power draw: 🟠 4x4090 → 1,800W peak (450W × 4) 🟢 4x5090 → 2,300W peak (575W × 4) 🟡 RTX PRO 6000 → 600W peak 🟣 DGX Spark → 240W peak (whole system) The RTX PRO 6000 is the quiet winner. One card, 96GB, matching a 4x5090 rig at roughly a quarter of the power and zero multi-GPU headaches. Best tokens-per-watt by a wide margin. DGX Spark is slow on generation but pulls the least power of any rig here, around 240W for the whole system. Prefill-friendly, memory-rich, wall-socket-friendly. And yes, plenty of people cap their cards. Even then, 4x 4090 or 4x 5090 still pulls well over 1,200W from the GPUs alone.

stevibe

191,782 Aufrufe • vor 3 Monaten

the tiebreaker is done. qwen 3.5 27B dense. single RTX 3090. one prompt. zero steering. zero human edits. 1,827 lines across 10 files. 13 minutes. full thinking mode. runs on first load. hermes 4.3 got the same prompt with 2x 3090s and 5x the context it needed. wrote 1,249 lines, left empty files, needed 3 interventions, game was broken on load. same architecture class. same quant. hermes got double the hardware. completely different result. dense wasn't the problem. hermes was. but here's what got me. this model thinks at 27 tok/s. every single token carries 27 billion parameters of reasoning. MoE hit 112 tok/s but only 3B active per token. the dense model is slower and it doesn't matter. watch 13 minutes of autonomous coding on a consumer GPU with zero intervention and tell me speed is what matters. a year ago this wasn't possible. now it runs on hardware you can buy used for $900. no API. no subscription. no cloud. just a 3090 doing what data centers did 18 months ago. full unedited session in the video. every token, every file, every thinking chain. 16 minutes. hit play.

the tiebreaker is done. qwen 3.5 27B dense. single RTX 3090. one prompt. zero steering. zero human edits. 1,827 lines across 10 files. 13 minutes. full thinking mode. runs on first load. hermes 4.3 got the same prompt with 2x 3090s and 5x the context it needed. wrote 1,249 lines, left empty files, needed 3 interventions, game was broken on load. same architecture class. same quant. hermes got double the hardware. completely different result. dense wasn't the problem. hermes was. but here's what got me. this model thinks at 27 tok/s. every single token carries 27 billion parameters of reasoning. MoE hit 112 tok/s but only 3B active per token. the dense model is slower and it doesn't matter. watch 13 minutes of autonomous coding on a consumer GPU with zero intervention and tell me speed is what matters. a year ago this wasn't possible. now it runs on hardware you can buy used for $900. no API. no subscription. no cloud. just a 3090 doing what data centers did 18 months ago. full unedited session in the video. every token, every file, every thinking chain. 16 minutes. hit play.

Sudo su

91,135 Aufrufe • vor 5 Monaten

Gemma4 12B with Unsloth's Quant on DGX Spark Quants: - UD_Q4_K_XL - UD_Q5_K_XL - UD_Q6_K_XL - UD_Q8_K_XL Summary: - Q4: 25.21 tok/s, TTFT 168ms - Q5: 21.7 tok/s, TTFT 182ms - Q6: 17.68 tok/s, TTFT 193.95ms - Q8: 15.22 tok/s, TTFT 221ms

Gemma4 12B with Unsloth's Quant on DGX Spark Quants: - UD_Q4_K_XL - UD_Q5_K_XL - UD_Q6_K_XL - UD_Q8_K_XL Summary: - Q4: 25.21 tok/s, TTFT 168ms - Q5: 21.7 tok/s, TTFT 182ms - Q6: 17.68 tok/s, TTFT 193.95ms - Q8: 15.22 tok/s, TTFT 221ms

stevibe

18,639 Aufrufe • vor 2 Monaten

$A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓$

A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓

slash1s

35,742 Aufrufe • vor 1 Monat

2.3x faster. Ran Unsloth AI Qwen3.6 MTP variants on a DGX Spark (UD-Q6_K_XL): > 27B → 27B MTP: 8.1 → 18.65 t/s (2.3x faster) > 35B A3B → 35B A3B MTP: 56.91 → 66.52 t/s (+17%) The 27B dense model more than doubled throughput from MTP alone. Free speed is free speed.

2.3x faster. Ran Unsloth AI Qwen3.6 MTP variants on a DGX Spark (UD-Q6_K_XL): > 27B → 27B MTP: 8.1 → 18.65 t/s (2.3x faster) > 35B A3B → 35B A3B MTP: 56.91 → 66.52 t/s (+17%) The 27B dense model more than doubled throughput from MTP alone. Free speed is free speed.

stevibe

24,315 Aufrufe • vor 3 Monaten

nvidia's 3B mamba destroyed alibaba's 3B deltanet on the same RTX 3090. only 24 days between releases. same active parameters, same VRAM tier, completely different architectures. nemotron cascade 2: 187 tok/s. flat from 4K to 625K context. zero speed loss. flags: -ngl 99 -np 1. that's it. no context flags, no KV cache tricks. auto-allocates 625K. qwen 3.5 35B-A3B: 112 tok/s. flat from 4K to 262K context. zero speed loss. flags: -ngl 99 -np 1 -c 262144 --cache-type-k q8_0 --cache-type-v q8_0. needed KV cache quantization to fit 262K. both models held a flat line across every context level. both architectures are context-independent. but nvidia's mamba2 is 67% faster at generating tokens on the exact same hardware and needs fewer flags to get there. same node, same GPU, same everything. the only variable is the model. gold medal math olympiad winner running at 187 tokens per second on single RTX 3090 a card from 6 years ago. nvidia cooked.

nvidia's 3B mamba destroyed alibaba's 3B deltanet on the same RTX 3090. only 24 days between releases. same active parameters, same VRAM tier, completely different architectures. nemotron cascade 2: 187 tok/s. flat from 4K to 625K context. zero speed loss. flags: -ngl 99 -np 1. that's it. no context flags, no KV cache tricks. auto-allocates 625K. qwen 3.5 35B-A3B: 112 tok/s. flat from 4K to 262K context. zero speed loss. flags: -ngl 99 -np 1 -c 262144 --cache-type-k q8_0 --cache-type-v q8_0. needed KV cache quantization to fit 262K. both models held a flat line across every context level. both architectures are context-independent. but nvidia's mamba2 is 67% faster at generating tokens on the exact same hardware and needs fewer flags to get there. same node, same GPU, same everything. the only variable is the model. gold medal math olympiad winner running at 187 tokens per second on single RTX 3090 a card from 6 years ago. nvidia cooked.

Sudo su

187,008 Aufrufe • vor 4 Monaten