正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Opus-level intelligence running fully local at home on dual RTX 4090s. Qwen3.8-27B-GGUF: - 80 tok/s - Full 262k context - MTP on - Only 34 GB of 48 GB VRAM used - Outperforms Claude Opus max on SWE-Pro: 61.7 vs Claude Opus 4.6’s 53.4 GPQA: 89.2 vs 91.3 No... show more

Md Ismail Šojal 🕷️

52,669 subscribers

34,855 次观看 • 2 个月前 •via X (Twitter)

科学技术教育

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

People realizing they can run Opus-4.6 level intelligence on their laptop with Qwen3.8-27B. No subscription. No limits. Completely private.

People realizing they can run Opus-4.6 level intelligence on their laptop with Qwen3.8-27B. No subscription. No limits. Completely private.

Mia

168,081 次观看 • 2 个月前

Qwen 3.8 27B (dense) running on a single RTX 4090 (24GB VRAM) at 65 tokens/sec decode with MTP! 260,000 context window or 65 tokens/sec decode with native MTP. The API cartel should be terrified. We are officially running frontier tier agentic AI (benchmarks comparable to claude opus 4.6 max) on a single consumer gaming GPU. I benchmarked Qwen3.8-27B on a single NVIDIA RTX 4090 (24GB VRAM, Ubuntu 22) using Unsloth’s Dynamic Q4_K_XL GGUF on the latest llama.cpp. Here is the complete benchmark breakdown across both Context Scaling and MTP Overdrive (28k prompt baseline): ### PART 1: The Context Scaling Matrix (Pure Throughput) # 1. Standard FP16 KV Cache (Unquantized): - 80k Context: 2,664.7 t/s prefill | 40.68 t/s decode | 22.36 GB VRAM - 100k Context: 2,678.7 t/s prefill | 40.89 t/s decode | 23.59 GB VRAM (100k is the hard ceiling for unquantized f16 KV in 24GB VRAM) # 2. Q8 Quantized KV Cache (-ctv q8_0 -ctk q8_0): - 130k Context: 2,639.1 t/s prefill | 40.96 t/s decode | 22.18 GB VRAM - 170k Context: 2,653.9 t/s prefill | 40.70 t/s decode | 23.68 GB VRAM (170k is the sweet spot for heavy agentic coding workflows) # 3. Q4 Quantized KV Cache (-ctv q4_0 -ctk q4_0): - 260k Context: 2,659.8 t/s prefill | 40.70 t/s decode | 23.00 GB VRAM Full 262k native context residing entirely in 24GB VRAM. Zero system RAM offload. Stress test with a monster 142k real-world prompt (-c 170000, Q8 KV): - Prefill: 1,829.50 tokens/s - Decode: 31.3 tokens/s - VRAM: 23.7 GB rock solid ### PART 2: Native MTP Overdrive (Trading Context for Speed) Since MTP heads are baked into the architecture, enabling native speculative drafting pushes decode speeds straight to 60 t/s with zero external draft model: # 1. MTP + Q8 KV Cache: - 80k Context: 2,370.66 t/s prefill | 59.25 t/s decode | 23.4 GB VRAM (MTP state buffers eat slightly more memory, making 80k the ceiling for Q8) # 2. MTP + Q4 KV Cache: - 130k Context: 2,391.09 t/s prefill | 60.10 t/s decode | 23.5 GB VRAM (Sweet spot: 130,000 context running at a screaming 60 tps decode) ### Qwen3.8-27B vs Muse Glimmer 30B Two days ago I benched Meta's Muse Glimmer 30B hitting 130k context unquantized (19.3 GB VRAM) pulling 50-75 t/s decode. If you own a single RTX 3090 or RTX 4090, you have zero excuse to burn API credits. ### The Reproduction llama.cpp flags: 1. Max Context Stack (260,000 Context @ 41 tps): ./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -c 260000 -ngl 99 --port 8080 -ctv q4_0 -ctk q4_0 2. MTP Overdrive Stack (130,000 Context @ 60 tps): ./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -c 130000 -ngl 99 --port 8080 -ctv q4_0 -ctk q4_0 --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.7 Unsloth's Hugging Face GGUF links, intelligence/agentic benchmark details, and performance charts are posted in the replies. Local compute is eating the cloud alive. How much monthly API spend does a 24GB setup like this actually replace for you?

Qwen 3.8 27B (dense) running on a single RTX 4090 (24GB VRAM) at 65 tokens/sec decode with MTP! 260,000 context window or 65 tokens/sec decode with native MTP. The API cartel should be terrified. We are officially running frontier tier agentic AI (benchmarks comparable to claude opus 4.6 max) on a single consumer gaming GPU. I benchmarked Qwen3.8-27B on a single NVIDIA RTX 4090 (24GB VRAM, Ubuntu 22) using Unsloth’s Dynamic Q4_K_XL GGUF on the latest llama.cpp. Here is the complete benchmark breakdown across both Context Scaling and MTP Overdrive (28k prompt baseline): ### PART 1: The Context Scaling Matrix (Pure Throughput) # 1. Standard FP16 KV Cache (Unquantized): - 80k Context: 2,664.7 t/s prefill | 40.68 t/s decode | 22.36 GB VRAM - 100k Context: 2,678.7 t/s prefill | 40.89 t/s decode | 23.59 GB VRAM (100k is the hard ceiling for unquantized f16 KV in 24GB VRAM) # 2. Q8 Quantized KV Cache (-ctv q8_0 -ctk q8_0): - 130k Context: 2,639.1 t/s prefill | 40.96 t/s decode | 22.18 GB VRAM - 170k Context: 2,653.9 t/s prefill | 40.70 t/s decode | 23.68 GB VRAM (170k is the sweet spot for heavy agentic coding workflows) # 3. Q4 Quantized KV Cache (-ctv q4_0 -ctk q4_0): - 260k Context: 2,659.8 t/s prefill | 40.70 t/s decode | 23.00 GB VRAM Full 262k native context residing entirely in 24GB VRAM. Zero system RAM offload. Stress test with a monster 142k real-world prompt (-c 170000, Q8 KV): - Prefill: 1,829.50 tokens/s - Decode: 31.3 tokens/s - VRAM: 23.7 GB rock solid ### PART 2: Native MTP Overdrive (Trading Context for Speed) Since MTP heads are baked into the architecture, enabling native speculative drafting pushes decode speeds straight to 60 t/s with zero external draft model: # 1. MTP + Q8 KV Cache: - 80k Context: 2,370.66 t/s prefill | 59.25 t/s decode | 23.4 GB VRAM (MTP state buffers eat slightly more memory, making 80k the ceiling for Q8) # 2. MTP + Q4 KV Cache: - 130k Context: 2,391.09 t/s prefill | 60.10 t/s decode | 23.5 GB VRAM (Sweet spot: 130,000 context running at a screaming 60 tps decode) ### Qwen3.8-27B vs Muse Glimmer 30B Two days ago I benched Meta's Muse Glimmer 30B hitting 130k context unquantized (19.3 GB VRAM) pulling 50-75 t/s decode. If you own a single RTX 3090 or RTX 4090, you have zero excuse to burn API credits. ### The Reproduction llama.cpp flags: 1. Max Context Stack (260,000 Context @ 41 tps): ./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -c 260000 -ngl 99 --port 8080 -ctv q4_0 -ctk q4_0 2. MTP Overdrive Stack (130,000 Context @ 60 tps): ./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -c 130000 -ngl 99 --port 8080 -ctv q4_0 -ctk q4_0 --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.7 Unsloth's Hugging Face GGUF links, intelligence/agentic benchmark details, and performance charts are posted in the replies. Local compute is eating the cloud alive. How much monthly API spend does a 24GB setup like this actually replace for you?

Alok

386,157 次观看 • 2 个月前

Claude Opus 4.6 is no joke 🫠 Just asked for a great F1 animation for Redbull using Claude Opus 4.6 on Capacity . Took like 3 minutes to get it.

Claude Opus 4.6 is no joke 🫠 Just asked for a great F1 animation for Redbull using Claude Opus 4.6 on Capacity . Took like 3 minutes to get it.

Samuel Rondot

315,665 次观看 • 9 个月前

Qwen 3.8 27B Q4_K_M - 90 tokens/sec on a single NVIDIA RTX 4090 (24 GB VRAM) with Dflash2! (MTP 60 tps -> 90 tps Dflash2!!!!) Local AI moves so fast (literally!) it’s terrifying. Z lab just dropped DFlash 2 for Qwen 3.8 27b and Muse Glimmer. I patched llama.cpp (PR #27342) and paired it with Unsloth’s Qwen 3.8 27B UD-Q4_K_XL quant. The result? Lossless 90 tokens/s decode. My last post highlighted native MTP hitting 60 t/s at 130,000 context. But DFlash 2 just completely shattered that ceiling. By using parallel block diffusion drafting (predicting whole blocks of tokens in a single pass using dynamic convolutions), DFlash achieves a massive 5.39 token acceptance rate. THE ALPHA TWEAK: `n-max 7` eats too much VRAM for draft states. But if you drop the draft limit to `--spec-draft-n-max 4`, you slash the VRAM overhead and actually increase the throughput. Here is the new 24GB VRAM Physics Matrix (DFlash 2 @ n-max 4): - 30k Context: 1,725 t/s prefill | 87.05 t/s decode | 22.2 GB VRAM - 80k Context: 1,789 t/s prefill | 84.20 t/s decode | 23.3 GB VRAM - 110k Context: 1,767 t/s prefill | 83.35 t/s decode | 23.96 GB VRAM (110k context at 83+ tokens a second sitting exactly on the 24GB hardware limit is absolute wizardry). How to compile the PR today: git clone cd llama.cpp git fetch origin pull/27342/head:pr-27342 git switch pr-27342 cmake -B build -DGGML_CUDA=ON && cmake --build build -j Llama.cpp flags for Dflash (110k Context Ceiling): ./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -md Qwen3.8-27B-DFlash2-Q4_K_M.gguf --spec-type draft-dflash --spec-draft-n-max 4 -c 110000 -ngl 99 --port 8080 -ctv q4_0 -ctk q4_0 The fact that the open source community is shipping block diffusion drafters so quickly that run entirely locally on a gaming GPU is unbelievable. If you own a single RTX 3090 or 4090, it is officially time to upgrade to qwen 3.8 27b with dflash 2 and cancel your API subscriptions and let local silicon eat the cloud. This model beats GPT 5.6 Terra, GLM 5.2 DeepSeek V4 Pro, Muse Spark 1.2 and Claude Opus 4.8 on the artificial analysis agentic index (details in the replies) Hugging Face GGUF links (Base + DFlash2) and the full visual VRAM scaling and Dflash2 vs MTP graphs are also in the replies below. are you sticking to native MTP for the 130k context, or sacrificing 20k context to redline your decode speed? How many tokens/sec are you pushing on your current local rig?

Qwen 3.8 27B Q4_K_M - 90 tokens/sec on a single NVIDIA RTX 4090 (24 GB VRAM) with Dflash2! (MTP 60 tps -> 90 tps Dflash2!!!!) Local AI moves so fast (literally!) it’s terrifying. Z lab just dropped DFlash 2 for Qwen 3.8 27b and Muse Glimmer. I patched llama.cpp (PR #27342) and paired it with Unsloth’s Qwen 3.8 27B UD-Q4_K_XL quant. The result? Lossless 90 tokens/s decode. My last post highlighted native MTP hitting 60 t/s at 130,000 context. But DFlash 2 just completely shattered that ceiling. By using parallel block diffusion drafting (predicting whole blocks of tokens in a single pass using dynamic convolutions), DFlash achieves a massive 5.39 token acceptance rate. THE ALPHA TWEAK: `n-max 7` eats too much VRAM for draft states. But if you drop the draft limit to `--spec-draft-n-max 4`, you slash the VRAM overhead and actually increase the throughput. Here is the new 24GB VRAM Physics Matrix (DFlash 2 @ n-max 4): - 30k Context: 1,725 t/s prefill | 87.05 t/s decode | 22.2 GB VRAM - 80k Context: 1,789 t/s prefill | 84.20 t/s decode | 23.3 GB VRAM - 110k Context: 1,767 t/s prefill | 83.35 t/s decode | 23.96 GB VRAM (110k context at 83+ tokens a second sitting exactly on the 24GB hardware limit is absolute wizardry). How to compile the PR today: git clone cd llama.cpp git fetch origin pull/27342/head:pr-27342 git switch pr-27342 cmake -B build -DGGML_CUDA=ON && cmake --build build -j Llama.cpp flags for Dflash (110k Context Ceiling): ./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -md Qwen3.8-27B-DFlash2-Q4_K_M.gguf --spec-type draft-dflash --spec-draft-n-max 4 -c 110000 -ngl 99 --port 8080 -ctv q4_0 -ctk q4_0 The fact that the open source community is shipping block diffusion drafters so quickly that run entirely locally on a gaming GPU is unbelievable. If you own a single RTX 3090 or 4090, it is officially time to upgrade to qwen 3.8 27b with dflash 2 and cancel your API subscriptions and let local silicon eat the cloud. This model beats GPT 5.6 Terra, GLM 5.2 DeepSeek V4 Pro, Muse Spark 1.2 and Claude Opus 4.8 on the artificial analysis agentic index (details in the replies) Hugging Face GGUF links (Base + DFlash2) and the full visual VRAM scaling and Dflash2 vs MTP graphs are also in the replies below. are you sticking to native MTP for the 130k context, or sacrificing 20k context to redline your decode speed? How many tokens/sec are you pushing on your current local rig?

Alok

113,748 次观看 • 2 个月前

Claude Opus 5, Qwen 3.8 Max, GLM-5.3-Flash, Tencent Hy4 - frontier models sitting on FREE tiers right now: > HuggingFace: Tencent Hy4-preview, 770B MoE, open weights, self-host > TokenRouter: Qwen 3.8 Max free, up to 1B tokens, no card > ZenMux: GLM-5.3-Flash free tier, rate-limited > JustWoker: $50 free credits for Claude Opus 5, API access Four paths to frontier models at $0. The paid tier is becoming the backup plan.

Claude Opus 5, Qwen 3.8 Max, GLM-5.3-Flash, Tencent Hy4 - frontier models sitting on FREE tiers right now: > HuggingFace: Tencent Hy4-preview, 770B MoE, open weights, self-host > TokenRouter: Qwen 3.8 Max free, up to 1B tokens, no card > ZenMux: GLM-5.3-Flash free tier, rate-limited > JustWoker: $50 free credits for Claude Opus 5, API access Four paths to frontier models at $0. The paid tier is becoming the backup plan.

ＺＥＦＩ

12,875 次观看 • 2 个月前

A 27B model in 12 GB of VRAM. Full 262K context. 70 tok/s. And now it is UNCENSORED. I ran it on my own RTX 3090, held to 12 GB. This is Mirai S Qwen3.8-27B, 2.4 bits per weight, an 11 GB file. With my llama.cpp port the same card did 82K context at 38 tok/s inside 12 GB. Then Palmer took the port and did the hard part. Same card, same file, his serve: - context: 82K -> 262,144 with q8 KV - decode: 38 -> 69.9 tok/s - VRAM peak: 11.7 GB - prefill: 1,124 tok/s He keeps part of the KV cache in VRAM and the rest in pinned RAM, and he drafts with MTP at every depth. He checked every kernel against my fork token for token and wrote down every experiment, the failed ones too. On his real 12 GB RTX 4070 it does 76.6. My part is small. I added abliteration. You cant edit these weights, they are compressed codes, so I measured the refusal direction on the quantized model and the engine removes it while the model runs. A 1.3 MB file and 1 flag: - harmful prompts refused: 64 of 64 -> 0 of 64 - held-out set: 78 of 82 -> 0 of 82 - decode: 69.9 -> 69.7 tok/s Counted by a regex, and with thinking on 1 of 24 still refuses. Deep in the context it gets slower on my card: 27.9 tok/s at 60K. So thats 12 GB of VRAM, 262K context, 70 tok/s, abliterated. Open weights, open code, runs on your desk. Cary did a genius amount of work and almost nobody has seen it. Go follow Palmer right now, I mean it. serve: weights + vector:

A 27B model in 12 GB of VRAM. Full 262K context. 70 tok/s. And now it is UNCENSORED. I ran it on my own RTX 3090, held to 12 GB. This is Mirai S Qwen3.8-27B, 2.4 bits per weight, an 11 GB file. With my llama.cpp port the same card did 82K context at 38 tok/s inside 12 GB. Then Palmer took the port and did the hard part. Same card, same file, his serve: - context: 82K -> 262,144 with q8 KV - decode: 38 -> 69.9 tok/s - VRAM peak: 11.7 GB - prefill: 1,124 tok/s He keeps part of the KV cache in VRAM and the rest in pinned RAM, and he drafts with MTP at every depth. He checked every kernel against my fork token for token and wrote down every experiment, the failed ones too. On his real 12 GB RTX 4070 it does 76.6. My part is small. I added abliteration. You cant edit these weights, they are compressed codes, so I measured the refusal direction on the quantized model and the engine removes it while the model runs. A 1.3 MB file and 1 flag: - harmful prompts refused: 64 of 64 -> 0 of 64 - held-out set: 78 of 82 -> 0 of 82 - decode: 69.9 -> 69.7 tok/s Counted by a regex, and with thinking on 1 of 24 still refuses. Deep in the context it gets slower on my card: 27.9 tok/s at 60K. So thats 12 GB of VRAM, 262K context, 70 tok/s, abliterated. Open weights, open code, runs on your desk. Cary did a genius amount of work and almost nobody has seen it. Go follow Palmer right now, I mean it. serve: weights + vector:

Alexey Fateev

133,823 次观看 • 29 天前

We gave two Claude models the same character sheet Opus 5 vs Opus 4.8 Both prompts run on Seedance 2.0. Our thoughts in thread

We gave two Claude models the same character sheet Opus 5 vs Opus 4.8 Both prompts run on Seedance 2.0. Our thoughts in thread

Magnific

19,032 次观看 • 3 个月前

Claude Opus 5.5 High vs GPT 6 Astra Max > 640×360 pixel art of cheetah running look at the difference

Claude Opus 5.5 High vs GPT 6 Astra Max > 640×360 pixel art of cheetah running look at the difference

Harshith

143,700 次观看 • 1 个月前

Grok 4.1 Fast outperforms all major frontier models on τ²-Bench Telecom for agentic tool use Beating the newest models from Google (Gemini 3 Pro), Anthropic (Claude Opus 4.5), and OpenAI (GPT 5.2 xhigh)

Grok 4.1 Fast outperforms all major frontier models on τ²-Bench Telecom for agentic tool use Beating the newest models from Google (Gemini 3 Pro), Anthropic (Claude Opus 4.5), and OpenAI (GPT 5.2 xhigh)

X Freeze

41,019 次观看 • 10 个月前

Chatgpt Pro vs Claude Max < 200$ plan> Claude: $12,529 of Sonnet. Opus $11,726 , Fable $2,485 ChatGPT: $2,084 of Sol GPT-6 Astra: $2,897

Chatgpt Pro vs Claude Max < 200$ plan> Claude: $12,529 of Sonnet. Opus $11,726 , Fable $2,485 ChatGPT: $2,084 of Sol GPT-6 Astra: $2,897

Chetaslua

127,044 次观看 • 1 个月前

Claude Fable 5.1 + Opus 5 + GPT-5.6 Sol are FREE 😳 Experiential Labs is giving free access to a huge lineup of AI models what you get: - Claude Fable 5.1 + Fable 5 - Claude Opus 5 + Sonnet 5 - GPT-5.6 Sol + Luna - Gemini 3.7 Flash - Kimi K3 - GLM 5.3 + 5.3 Flash - DeepSeek V4 Flash - Qwen3.8 27B most models have 1M+ context check the full list: free access is live 👀

Claude Fable 5.1 + Opus 5 + GPT-5.6 Sol are FREE 😳 Experiential Labs is giving free access to a huge lineup of AI models what you get: - Claude Fable 5.1 + Fable 5 - Claude Opus 5 + Sonnet 5 - GPT-5.6 Sol + Luna - Gemini 3.7 Flash - Kimi K3 - GLM 5.3 + 5.3 Flash - DeepSeek V4 Flash - Qwen3.8 27B most models have 1M+ context check the full list: free access is live 👀

K2S

25,647 次观看 • 1 个月前

Fable 5 vs Opus 4.8 on Arcads MCP Both videos generated with Arcads X Claude

Fable 5 vs Opus 4.8 on Arcads MCP Both videos generated with Arcads X Claude

Romain Torres

450,648 次观看 • 4 个月前

Game dev round: Gemini 4 Argon vs. Claude Opus 5.5. Cyberpunk combat built on Higgsfield.

Game dev round: Gemini 4 Argon vs. Claude Opus 5.5. Cyberpunk combat built on Higgsfield.

Higgsfield AI 🧩

18,875 次观看 • 22 天前

Claude Coders: Pair Haiku 5.5 with an Opus advisor *NOW*. Opus 5.5 checks Haiku's work. Take advantage of the speed and cost-efficiency of Haiku with the intelligence of Opus. Copy this into settings.json (or ask Claude to): {"model":"haiku","advisorModel":"opus"}

Claude Coders: Pair Haiku 5.5 with an Opus advisor NOW. Opus 5.5 checks Haiku's work. Take advantage of the speed and cost-efficiency of Haiku with the intelligence of Opus. Copy this into settings.json (or ask Claude to): {"model":"haiku","advisorModel":"opus"}

Dan McAteer

115,828 次观看 • 28 天前

454 tok/s on Qwen3.8 27B on single RTX 6000 with +6000 mem overclock Mia’s Qwen3.8-27B NVFP4 W4A4 • SGLang + DFlash2 block-16 drafter • FP8 KV

454 tok/s on Qwen3.8 27B on single RTX 6000 with +6000 mem overclock Mia’s Qwen3.8-27B NVFP4 W4A4 • SGLang + DFlash2 block-16 drafter • FP8 KV

Killy

50,467 次观看 • 2 个月前

I think the CMP 170HX just became one of the weirdest local-LLM value plays. 👀 I got Qwen3.8-27B W8A16 running on a single 170HX 64GB with vLLM 0.27.1 + DFlash2: ⚡ ~120 tok/s decode 🚀 ~1.6–1.9K tok/s prefill 🧠 262K context with BF16 KV 📦 342K-token KV capacity 🎯 DFlash2 averaging 3.66 tokens/draft 💾 60.5 / 63.5 GiB VRAM used 27B model is doing ~120 tokens/sec on one GPU. Software optimization is getting ridiculous. Those numbers are from your measured W8A16 + DFlash2 run: ~120 tok/s decode, ~1,580–1,940 tok/s prefill, 342,729-token KV capacity, and 60.5/63.5 GiB VRAM usage. #170hx #LocalAI #localllm

I think the CMP 170HX just became one of the weirdest local-LLM value plays. 👀 I got Qwen3.8-27B W8A16 running on a single 170HX 64GB with vLLM 0.27.1 + DFlash2: ⚡ ~120 tok/s decode 🚀 ~1.6–1.9K tok/s prefill 🧠 262K context with BF16 KV 📦 342K-token KV capacity 🎯 DFlash2 averaging 3.66 tokens/draft 💾 60.5 / 63.5 GiB VRAM used 27B model is doing ~120 tokens/sec on one GPU. Software optimization is getting ridiculous. Those numbers are from your measured W8A16 + DFlash2 run: ~120 tok/s decode, ~1,580–1,940 tok/s prefill, 342,729-token KV capacity, and 60.5/63.5 GiB VRAM usage. #170hx #LocalAI #localllm

Own Your Compute

19,494 次观看 • 2 个月前

Claude Opus 4.8 vs GPT 5.5 Prompt 👇

Claude Opus 4.8 vs GPT 5.5 Prompt 👇

Viktor Oddy

28,799 次观看 • 5 个月前

50% more context unlocked for Qwen 3.8 27b Q4_K_XL dflash 2 on a single RTX 4090 (24 GB VRAM) I found a hidden VRAM tax in llama.cpp. By combining my custom 2 bit DFlash 2 drafter with one overlooked server flag, I just unlocked another +80,000 tokens of context. Qwen3.8-27B is now running a massive 250,000 context at 75 tokens/s on a single RTX 4090. Here is the secret: By default, `llama-server` reserves massive chunks of your VRAM to handle multiple concurrent users (batching). If you are running a single user session, you are bleeding memory for features you aren't using. By passing the `--parallel 1` flag, you force the engine to dedicate 100% of your 24GB VRAM buffer to a single user. When we combine the VRAM saved by our Q2_K 2-bit drafter with the VRAM saved by `--parallel 1`, the context ceilings absolutely explode: Note: all benchmarks carried out with a massive 28k prompt. Ubuntu 22. ### THE NEW 24GB PHYSICAL LIMITS (Single RTX 4090): # 1. The "Repo Swallower" (Q4 KV Cache): - Context: 250,000 tokens (Up from 170k!) - Speed: 73.66 t/s decode | 1,608 t/s prefill - Peak VRAM: 23.8 GB # 2. The "High-Precision SWE" (Q8 KV Cache): - Context: 150,000 tokens (Up from 100k!) - Speed: 75.01 t/s decode | 1,667 t/s prefill - Peak VRAM: 23.9 GB # 3. The "Pristine Attention" (Unquantized FP16 KV): - Context: 90,000 tokens - Speed: 80.58 t/s decode | 1,699 t/s prefill - Peak VRAM: 23.92 GB ### HOW TO RUN THE 250K GOD STACK TODAY: (Requires PR #27342 + my Q2_K Hugging Face drafter) llama.cpp flags: ./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -md Qwen3.8-27B-DFlash2-Q2_K.gguf --spec-type draft-dflash --spec-draft-n-max 3 -c 250000 -ngl 99 --parallel 1 --port 8080 -ctv q4_0 -ctk q4_0 We are pushing a quarter million tokens of context with speculative DFlash 2 decoding at 73 tokens/second on a single consumer gaming GPU. I dropped my custom 2 bit Hugging Face GGUF links, visual performance graphs, and the PR #27342 build instructions in the replies below. If you own a single RTX 3090 or 4090, it is officially time to cancel your API subscriptions and let local silicon eat the cloud. how much monthly API spend does an optimized 4090 rig like this actually replace for you?

50% more context unlocked for Qwen 3.8 27b Q4_K_XL dflash 2 on a single RTX 4090 (24 GB VRAM) I found a hidden VRAM tax in llama.cpp. By combining my custom 2 bit DFlash 2 drafter with one overlooked server flag, I just unlocked another +80,000 tokens of context. Qwen3.8-27B is now running a massive 250,000 context at 75 tokens/s on a single RTX 4090. Here is the secret: By default, `llama-server` reserves massive chunks of your VRAM to handle multiple concurrent users (batching). If you are running a single user session, you are bleeding memory for features you aren't using. By passing the `--parallel 1` flag, you force the engine to dedicate 100% of your 24GB VRAM buffer to a single user. When we combine the VRAM saved by our Q2_K 2-bit drafter with the VRAM saved by `--parallel 1`, the context ceilings absolutely explode: Note: all benchmarks carried out with a massive 28k prompt. Ubuntu 22. ### THE NEW 24GB PHYSICAL LIMITS (Single RTX 4090): # 1. The "Repo Swallower" (Q4 KV Cache): - Context: 250,000 tokens (Up from 170k!) - Speed: 73.66 t/s decode | 1,608 t/s prefill - Peak VRAM: 23.8 GB # 2. The "High-Precision SWE" (Q8 KV Cache): - Context: 150,000 tokens (Up from 100k!) - Speed: 75.01 t/s decode | 1,667 t/s prefill - Peak VRAM: 23.9 GB # 3. The "Pristine Attention" (Unquantized FP16 KV): - Context: 90,000 tokens - Speed: 80.58 t/s decode | 1,699 t/s prefill - Peak VRAM: 23.92 GB ### HOW TO RUN THE 250K GOD STACK TODAY: (Requires PR #27342 + my Q2_K Hugging Face drafter) llama.cpp flags: ./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -md Qwen3.8-27B-DFlash2-Q2_K.gguf --spec-type draft-dflash --spec-draft-n-max 3 -c 250000 -ngl 99 --parallel 1 --port 8080 -ctv q4_0 -ctk q4_0 We are pushing a quarter million tokens of context with speculative DFlash 2 decoding at 73 tokens/second on a single consumer gaming GPU. I dropped my custom 2 bit Hugging Face GGUF links, visual performance graphs, and the PR #27342 build instructions in the replies below. If you own a single RTX 3090 or 4090, it is officially time to cancel your API subscriptions and let local silicon eat the cloud. how much monthly API spend does an optimized 4090 rig like this actually replace for you?

Alok

41,735 次观看 • 2 个月前

GPT 5.6 SOL vs Claude Opus 5 Created with

GPT 5.6 SOL vs Claude Opus 5 Created with

Flowith

60,003 次观看 • 3 个月前

Claude Opus 5 VS GPT 5.6 SOL Created with

Claude Opus 5 VS GPT 5.6 SOL Created with

Flowith

120,365 次观看 • 3 个月前