Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

Your gaming PC can now serve frontier models at interactive speed using official checkpoints without extreme quantization! Qwen3.6 35B → 8GB RTX 4060 laptop @ 39 tok/s DeepSeek-V4-Flash 284B → RTX 5090 desktop @ 22-25 tok/s GLM-5.2 753B → RTX PRO 6000 workstation @ 15 tok/s Run your claude... show more

Shuo Yang

5,028 subscribers

1,100,817 görüntüleme • 1 ay önce •via X (Twitter)

Bilim & Teknoloji Eğitim Oyun

Anya Rossi• Live Now

Private livecam show

74 Yorum

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

FreeToken is fast. Comparing to Ollama, we have 3–4× faster decode, and 6–30× faster prefill How? We introduce bandwidth-adaptive CPU–GPU execution + semantic-aware caching across agent turns. More details in the technical report:

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

FreeToken provides native GUI. No GGUF conversion. No building from source. One-click install on Windows and Linux. FreeToken-desktop ships with agent harnesses built in — pick a model, pick an app, go.

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

Download: Code: Reply with your GPU + RAM, and I'll tell you the biggest frontier model your machine can run 👇

Jon Hill | PhantomPrints profil fotoğrafı

Jon Hill | PhantomPrints1 ay önce

How does this black magic work? 50tok/s on a 4070ti super?!?!

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

Bandwidth magic 😄

Featherless AI profil fotoğrafı

Featherless AI1 ay önce

Can you talk more about the in game AI building?

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

I tried dsv4 flash many times and finally made it build a cabin 🤣

Featherless AI profil fotoğrafı

Featherless AI1 ay önce

No but how exactly did you harness it?

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

this mod

Agent007 profil fotoğrafı

Agent0071 ay önce

Interesting, Does it support most of the openweight models? Question why not making it headless where we can run the inference engine using our own tools?? I will test it in RTX PRO 6000 and get back to you. If it runs GLM 5.2 in it, it will be very good....

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

Thanks! It will be really helpful if you can give feedback through github issue!

Agent007 profil fotoğrafı

Agent0071 ay önce

I reviewed little bit the way how it works and tried to run some models. FreeToken, make you offload the experts from VRAM to RAM to be able to run bigger models, which means without enough RAM you cannot achieve it. Am I missing something? If yes, please correct me if I am wrong. Thanks!

Agent007 profil fotoğrafı

Agent0071 ay önce

For GLM5.2, how much RAM it used with a single RTX PRO 6000 and also did you ever tried it with TP like 2 or higher or PP. Appreciate your answers!

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

GLM5.2 setting is detailed in section 5.1, and we are working on multi-gpu support for edge devices!

Agent007 profil fotoğrafı

Agent0071 ay önce

Thank you very much will look into it

Juan Lescano 🇦🇷 profil fotoğrafı

Juan Lescano 🇦🇷1 ay önce

How's AMD hardware support? Will give this a try

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

AMD support is on the way!

🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱 profil fotoğrafı

🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱🇵🇱1 ay önce

@Thejuampi Can we get a good 16gb vram (9070xt) model

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

@Thejuampi Definitely, it will work really well for qwen35B!

Hamza hamud profil fotoğrafı

Hamza hamud1 ay önce

Will apple devices be included soon?

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

Really soon!😄

Hamza hamud profil fotoğrafı

Hamza hamud1 ay önce

honestly amazing repo. I thought SGlang/VLLM already were the most performant out there and there wasnt a way to squeeze out further performance other than making actual changes to llm architecture

Vorname MitD profil fotoğrafı

Vorname MitD1 ay önce

Great work! No more excuses not to run models locally.

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

Everyone should give it a try 😁

Max For AI profil fotoğrafı

Max For AI1 ay önce

Congratulations! That's a fantastic work! But does anyone actually use an RTX PRO 6000 for gaming...?

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

lol some rich people ask me to support this 😄

Max For AI profil fotoğrafı

Max For AI1 ay önce

OMG....

Yichuan Wang profil fotoğrafı

Yichuan Wang1 ay önce

amazing work

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

Thanks @YichuanM !

OrcaRouter 🐳 profil fotoğrafı

OrcaRouter 🐳1 ay önce

Can you send us a DM? We’ve some interesting ideas

Yohan Jeong profil fotoğrafı

Yohan Jeong1 ay önce

This is incredible! I always wanted to try Qwen 3.8 on my PC. Thank you so much!!!

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

😄

anytan profil fotoğrafı

anytan1 ay önce

Qwen3.6 35B at 39 tok/s on an 8GB RTX 4060 laptop is the number that stands out here. The 284B on a single 5090 at 22-25 tok/s is the harder claim to believe.

Lisa profil fotoğrafı

Lisa1 ay önce

Wow I am gettimg crazier by the day, full blown lift-off of singularity, V4 flash now running very decently on my 3080 Ti without any quant whatsoever, cancelling my dgx spark order for now! Game CHANGER!

Charlie Ruan profil fotoğrafı

Charlie Ruan1 ay önce

Nice!!

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

Thanks Charlie!

Emir profil fotoğrafı

Emir1 ay önce

Does it run qwen3.6 35b unquantized on my rtx 3080 ti? I have 32gb ram. Due to these limitations I always use 4bit quants. My speed is 30 to 40 t/s tg right now

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

I think freetoken will make it much faster 😄

Ziwen profil fotoğrafı

Ziwen1 ay önce

Wait we can run v4 flash with 5090? 🤔

Yilong Zhao profil fotoğrafı

Yilong Zhao1 ay önce

Awesome!

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

Thanks 🐲!

espace latent profil fotoğrafı

espace latent1 ay önce

Amazing work, do you have any recommendation on best quantized models that would fit on 5070ti laptop (12GB) ?

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

Qwen3.6 35B nvfp4 and gemma4 26B nvfp4 are both good!

espace latent profil fotoğrafı

espace latent1 ay önce

Thanks a lot!

APMI profil fotoğrafı

APMI1 ay önce

Great, but the chat doesn't have the option to upload photos and PDF documents to the conversation like LM Studio, and without that, it's not possible to do many things with these models.

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

This feature will be enabled within one week 😁 Stay tuned!

Coconut profil fotoğrafı

Coconut1 ay önce

Does it work for dense models as well or just MOE?

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

We also supports moe models, and they benefits from the agentic states reuses😁

Coconut profil fotoğrafı

Coconut1 ay önce

Sorry for not being clear - Does Free Token work with Dense models? like for example 3.8 27 B? I have RTX3080 laptop with 16GB VRAM + 64 GB RAM.

OrcaRouter 🐳 profil fotoğrafı

OrcaRouter 🐳1 ay önce

Interesting 🧐

Jason Kneen profil fotoğrafı

Jason Kneen1 ay önce

Silicon Mac support

Agent 47 | くろあか profil fotoğrafı

Agent 47 | くろあか1 ay önce

Any good model that runs on my rtx3050ti 4gb vram with 16gb ram. Thanks.

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

4GB vram is a little bit tricky... We will support model that works for you!

安叫兽|Bird🕊️ 🔶 BNB profil fotoğrafı

安叫兽|Bird🕊️ 🔶 BNB1 ay önce

4060 笔记本跑 35B 到 39 tok/s，多少有点离谱了

Min-Liang Tan profil fotoğrafı

Min-Liang Tan1 ay önce

Let's see how this runs on our Razer Blades @RazerAI

aditya profil fotoğrafı

aditya1 ay önce

ok but like what about a gtx 1650? 😭

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

rtx 10 and 20 are on the way!

Hanchen Li profil fotoğrafı

Hanchen Li1 ay önce

@melissapan 🤩

james profil fotoğrafı

james1 ay önce

what about macs

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

Really soon!

Mick.net - Maker: Document.Bot 🤖 BestTime.app 🎉 profil fotoğrafı

Mick.net - Maker: Document.Bot 🤖 BestTime.app 🎉1 ay önce

Does this work on M Mac’s? @grok

Trevor Wood profil fotoğrafı

Trevor Wood1 ay önce

Very cool. Great work guys, will test this out

N profil fotoğrafı

N1 ay önce

Have you got any plans to bring out a Mac version?

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

top priority 😄

caio temer profil fotoğrafı

caio temer1 ay önce

45–50 TPS Qwen 3.6 35B-A3B NVFP4 i7-11800H + 32 GB RAM + RTX 3070 Mobile 8 GB Quase achei que não tinha melhorado a performance porque esqueci de aumentar o número máximo de execuções paralelas que o padrão é 4. Depois de ajustar, o resultado foi incrível. Detalhe: a 3070 Mobile tem 448 GB/s de bandwidth, contra 256 GB/s da 4060 Mobile.

MimiCO profil fotoğrafı

MimiCO1 ay önce

The endgame of the cost war bloomberg just covered. 6% performance gap, 75% cheaper via API, and now $0 on your own hardware

Mack profil fotoğrafı

Mack1 ay önce

When macOS?

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

Maybe 3 weeks later

Diego Garcia profil fotoğrafı

Diego Garcia1 ay önce

Amazing! It allows to use parallel GPUs? I've 2x RTX 5060 TI (16gb each) + 64GB DDR5 RAM + Ryzen 9950x. What would be the best models?

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

@diegogarciamkt Multi gpu is on the way! We are handling with diverse communication patterns in edge setting 😄

Diego Garcia profil fotoğrafı

Diego Garcia1 ay önce

Is there a way for now to use it? Would be amazing.

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

@diegogarciamkt Download the desktop:

Diego Garcia profil fotoğrafı

Diego Garcia1 ay önce

Already did! I meant to use the multi-gpu setup somehow. If not, I'll wait for this. Thank you!

Shuo Yang profil fotoğrafı

Shuo Yang1 ay önce

@diegogarciamkt We have automatic software update. This will be enabled soon!

Benzer Videolar

Qwen3.8-Flash-Next now runs on a single RTX 5090 at 68.3 tok/s — no extreme quantization, no speculative decoding. · Production-level checkpoint: GB300-validated NVFP4 checkpoint from RadixArk · 63GB host RAM — less than what 1-bit quants of this model need · The 51GB n-gram table? It lives on your NVMe, at ~0.5% throughput cost In the video: one prompt → a playable minecraft-style world, then FreeToken on my 5090 desktop fixing a real FreeToken bug in 10.5 minutes. Meet FreeToken 🧵

Qwen3.8-Flash-Next now runs on a single RTX 5090 at 68.3 tok/s — no extreme quantization, no speculative decoding. · Production-level checkpoint: GB300-validated NVFP4 checkpoint from RadixArk · 63GB host RAM — less than what 1-bit quants of this model need · The 51GB n-gram table? It lives on your NVMe, at ~0.5% throughput cost In the video: one prompt → a playable minecraft-style world, then FreeToken on my 5090 desktop fixing a real FreeToken bug in 10.5 minutes. Meet FreeToken 🧵

Shuo Yang

64,307 görüntüleme • 1 ay önce

New LFM2.5-2.6B hits DeepSeek-V4 level on tool calling and runs 3.7x faster! We ran Liquid AI 's new LFM2.5-2.6B against DeepSeek-V4-Flash on one box with 4x RTX 5090. Both got the same three jobs, and each one only completes if the model fires every tool call Topics: -weather and local time in six cities -one budget into six currencies -four hotels checked and booked for one date Outputs: LFM2.5-2.6B: 35/35 tool-calls, 366 tok/s, 19s DeepSeek-V4-Flash: 35/35 tool-calls, 77 tok/s, 70s Both models made all 35 calls, and both fired twelve of them in one turn. LFM was trained for agents, and tool calling is where that shows. Strong result for a model you can run right on your phone

New LFM2.5-2.6B hits DeepSeek-V4 level on tool calling and runs 3.7x faster! We ran Liquid AI 's new LFM2.5-2.6B against DeepSeek-V4-Flash on one box with 4x RTX 5090. Both got the same three jobs, and each one only completes if the model fires every tool call Topics: -weather and local time in six cities -one budget into six currencies -four hotels checked and booked for one date Outputs: LFM2.5-2.6B: 35/35 tool-calls, 366 tok/s, 19s DeepSeek-V4-Flash: 35/35 tool-calls, 77 tok/s, 70s Both models made all 35 calls, and both fired twelve of them in one turn. LFM was trained for agents, and tool calling is where that shows. Strong result for a model you can run right on your phone

atomic.chat

113,185 görüntüleme • 2 ay önce

MiniMax M2.7 is 230B params. Can you actually run it at home? I tested Unsloth's UD-IQ3_XXS (80GB) on 4 different rigs: 🟠 4x RTX 4090 (96GB): 71.52 tok/s, TTFT 1045ms 🟢 4x RTX 5090 (128GB): 120.54 tok/s, TTFT 725ms 🟡 1x RTX PRO 6000 (96GB): 118.74 tok/s, TTFT 765ms 🟣 DGX Spark (128GB) — 24.41 tok/s, TTFT 741ms Backend: llama.cpp. Context: 32k. Max tokens: 4096. I went with IQ3_XXS because it's the biggest quant that fits in 96GB VRAM while still leaving safe headroom for 32k context. Same quant across all four rigs, fairest comparison I could run. Now look at rough peak GPU power draw: 🟠 4x4090 → 1,800W peak (450W × 4) 🟢 4x5090 → 2,300W peak (575W × 4) 🟡 RTX PRO 6000 → 600W peak 🟣 DGX Spark → 240W peak (whole system) The RTX PRO 6000 is the quiet winner. One card, 96GB, matching a 4x5090 rig at roughly a quarter of the power and zero multi-GPU headaches. Best tokens-per-watt by a wide margin. DGX Spark is slow on generation but pulls the least power of any rig here, around 240W for the whole system. Prefill-friendly, memory-rich, wall-socket-friendly. And yes, plenty of people cap their cards. Even then, 4x 4090 or 4x 5090 still pulls well over 1,200W from the GPUs alone.

MiniMax M2.7 is 230B params. Can you actually run it at home? I tested Unsloth's UD-IQ3_XXS (80GB) on 4 different rigs: 🟠 4x RTX 4090 (96GB): 71.52 tok/s, TTFT 1045ms 🟢 4x RTX 5090 (128GB): 120.54 tok/s, TTFT 725ms 🟡 1x RTX PRO 6000 (96GB): 118.74 tok/s, TTFT 765ms 🟣 DGX Spark (128GB) — 24.41 tok/s, TTFT 741ms Backend: llama.cpp. Context: 32k. Max tokens: 4096. I went with IQ3_XXS because it's the biggest quant that fits in 96GB VRAM while still leaving safe headroom for 32k context. Same quant across all four rigs, fairest comparison I could run. Now look at rough peak GPU power draw: 🟠 4x4090 → 1,800W peak (450W × 4) 🟢 4x5090 → 2,300W peak (575W × 4) 🟡 RTX PRO 6000 → 600W peak 🟣 DGX Spark → 240W peak (whole system) The RTX PRO 6000 is the quiet winner. One card, 96GB, matching a 4x5090 rig at roughly a quarter of the power and zero multi-GPU headaches. Best tokens-per-watt by a wide margin. DGX Spark is slow on generation but pulls the least power of any rig here, around 240W for the whole system. Prefill-friendly, memory-rich, wall-socket-friendly. And yes, plenty of people cap their cards. Even then, 4x 4090 or 4x 5090 still pulls well over 1,200W from the GPUs alone.

stevibe

193,239 görüntüleme • 5 ay önce

AI at Meta's Muse Glimmer (30B dense, open-weights) launches with SGLang day-0 support. We got ~230 tok/s on a single RTX 5090, with NVFP4 + DFlash on. It also works out of the box on NVIDIA AI Developer RTX Pro 6000, DGX Spark, and MLX for Mac. Speed and reliability have always been the blockers for a local agent. SGLang and Muse Glimmer make a local agent that's blazing fast, always on, and gets things done. Go build and run a local agent on your own machine!🚀

AI at Meta's Muse Glimmer (30B dense, open-weights) launches with SGLang day-0 support. We got ~230 tok/s on a single RTX 5090, with NVFP4 + DFlash on. It also works out of the box on NVIDIA AI Developer RTX Pro 6000, DGX Spark, and MLX for Mac. Speed and reliability have always been the blockers for a local agent. SGLang and Muse Glimmer make a local agent that's blazing fast, always on, and gets things done. Go build and run a local agent on your own machine!🚀

LMSYS Org

44,030 görüntüleme • 2 ay önce

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

Alok

62,381 görüntüleme • 3 ay önce

Anyone with 8GB or 12GB VRAM setups needs to understand that "-ncmoe" is the key flag to boost performance on llama.cpp Here are my results for Qwen3.6 35B A3B, with 64k q8_0 context on a 8GB RTX 3070Ti: ⚪️ no flag → 8.7 tok/s RAM: 13.6GB & VRAM: 7.8GB 🔴 -ncmoe 35 → 27.5 tok/s RAM: 12.1GB & VRAM: 4.3GB 🟢 -ncmoe 30 → 32.5 tok/s RAM: 12GB & VRAM: 5.6GB 🔵 -ncmoe 25 → 40.9 tok/s RAM: 12GB & VRAM: 6.9GB Please note the ram and vram usage you see are total usage of a windows pc, with the model running. My friend's setup: 8GB VRAM and 16GB RAM. You can boost performance by switching to Linux, just something to keep in mind. Basically, this flag keeps the MoE experts in the first X layers on your CPU + RAM, instead of eating all your VRAM straight away. This is a smart hybrid offload way that lets you run bigger models without OOM while keeping the rest on your GPU for speed. As we can see on the data, there's a sweet spot. When we lower it from 35 to 25, speed bumps +50% because there are more layers on your GPU (look at the VRAM usage). The key here is to play around with the number and fit as much as possible on your VRAM, goal is to have 1GB/800MB headroom to avoid stress. ↓ server flags below

Anyone with 8GB or 12GB VRAM setups needs to understand that "-ncmoe" is the key flag to boost performance on llama.cpp Here are my results for Qwen3.6 35B A3B, with 64k q8_0 context on a 8GB RTX 3070Ti: ⚪️ no flag → 8.7 tok/s RAM: 13.6GB & VRAM: 7.8GB 🔴 -ncmoe 35 → 27.5 tok/s RAM: 12.1GB & VRAM: 4.3GB 🟢 -ncmoe 30 → 32.5 tok/s RAM: 12GB & VRAM: 5.6GB 🔵 -ncmoe 25 → 40.9 tok/s RAM: 12GB & VRAM: 6.9GB Please note the ram and vram usage you see are total usage of a windows pc, with the model running. My friend's setup: 8GB VRAM and 16GB RAM. You can boost performance by switching to Linux, just something to keep in mind. Basically, this flag keeps the MoE experts in the first X layers on your CPU + RAM, instead of eating all your VRAM straight away. This is a smart hybrid offload way that lets you run bigger models without OOM while keeping the rest on your GPU for speed. As we can see on the data, there's a sweet spot. When we lower it from 35 to 25, speed bumps +50% because there are more layers on your GPU (look at the VRAM usage). The key here is to play around with the number and fit as much as possible on your VRAM, goal is to have 1GB/800MB headroom to avoid stress. ↓ server flags below

left curve dev

166,478 görüntüleme • 5 ay önce

$Gemma 4 12B QAT + MTP on 8GB VRAM. llama.cpp flags included. let's run it. 20+ tok/sec decode. 700+ tok/sec prefill. on a single RTX 4060. copy these exact flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \ --spec-type draft-mtp \ --spec-draft-n-max 4 \ --spec-draft-p-min 0.7 \ --spec-draft-model gemma-4-12B-it-qat-assistant-MTP-Q8_0.gguf \ -c 48000 -ngl 38 -v → -ngl 99 if you're on 12–24GB VRAM (RTX 3090, 4090, 4080, 3080, 4070 Ti, 3080 Ti) → -ngl 38 for 8GB setups with 48k context (RTX 4060, 3060, 2080, 2070, 3070) → drop it (or the context) lower if you're squeezing on 4–6GB (RTX 2060, 3050, 1660) my rig: RTX 4060 8GB · i7H · 16GB RAM MTP is giving me 25–40% decode throughput gains across Gemma 4 models. nearly zero VRAM cost for that bump. the draft assistant GGUF is only ~300–400MB depending on quant. one thing to know (architectural catch): unlike Qwen3.6 and Qwen3.5 models which bakes MTP heads straight into the base GGUF, Gemma 4 needs a separate draft assistant model downloaded alongside. not a big deal. just don't forget it or MTP won't run. draft assistant GGUF link → comments while you wait for anthropic mythos release, test this and drop your decode numbers below, curious how it scales across different setups.$

Gemma 4 12B QAT + MTP on 8GB VRAM. llama.cpp flags included. let's run it. 20+ tok/sec decode. 700+ tok/sec prefill. on a single RTX 4060. copy these exact flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \ --spec-type draft-mtp \ --spec-draft-n-max 4 \ --spec-draft-p-min 0.7 \ --spec-draft-model gemma-4-12B-it-qat-assistant-MTP-Q8_0.gguf \ -c 48000 -ngl 38 -v → -ngl 99 if you're on 12–24GB VRAM (RTX 3090, 4090, 4080, 3080, 4070 Ti, 3080 Ti) → -ngl 38 for 8GB setups with 48k context (RTX 4060, 3060, 2080, 2070, 3070) → drop it (or the context) lower if you're squeezing on 4–6GB (RTX 2060, 3050, 1660) my rig: RTX 4060 8GB · i7H · 16GB RAM MTP is giving me 25–40% decode throughput gains across Gemma 4 models. nearly zero VRAM cost for that bump. the draft assistant GGUF is only ~300–400MB depending on quant. one thing to know (architectural catch): unlike Qwen3.6 and Qwen3.5 models which bakes MTP heads straight into the base GGUF, Gemma 4 needs a separate draft assistant model downloaded alongside. not a big deal. just don't forget it or MTP won't run. draft assistant GGUF link → comments while you wait for anthropic mythos release, test this and drop your decode numbers below, curious how it scales across different setups.

Alok

29,006 görüntüleme • 4 ay önce

Google's Gemma 4 26B A4B QAT hits 25+ tokens/sec and 320+ tokens/sec prefill on 8 GB VRAM (RTX 4060) + 16 GB RAM using TurboQuant Prefill just went from 200 → 320+ tok/s on the same 8GB card. 1.6x, no new hardware, no new quant, just a KV cache trick stacked on top of the Gemma 4 26B MoE setup from a few days ago. A few days ago I posted Gemma 4 26B A4B hitting 28 tok/s decode on 8GB VRAM using native MTP. prefill was stuck around 200 tok/s. fair callout by the community. So today I tested something I'd already been meaning to try: TheTom/llama-cpp-turboquant, the TurboQuant KV cache fork by Tom Turney (Tom Turney). (github link in the comments) thanks to him, the fork just got resynced to mainline, so MTP + TurboQuant now run together cleanly (I didnt see any meaningful gains by using MTP with this setup though but you can try). The flags (No MTP): -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -c 64000 --cache-type-k q8_0 --cache-type-v turbo3 Results on the same RTX 4060 8GB, tested with a 27k token prompt at 64k context loaded: Prefill: 200 tok/s → 320+ tok/s Decode: stayed above 25 tok/s (without MTP) Why it works: TurboQuant uses walsh hadamard rotation + polar quantization on the KV cache. keys are sensitive to compression, values aren't much, so it splits the difference: K stays at q8_0, V drops to turbo3 (~3 bits). bonus from the memory savings: same 8GB card can now stretch to 100-120k context with minimal decode penalty. It should now be snappier with any agent harness such as hermes agent without compromise on intelligence. If you're already running Gemma 4 on a small card, this stacks on top for free. Try --cache-type-k q8_0 --cache-type-v turbo3 on your setup and report back what your prefill/decode split looks like. unsloth model gguf and llama.cpp turboquant fork links in the comments. what's your prefill number before vs after?

Google's Gemma 4 26B A4B QAT hits 25+ tokens/sec and 320+ tokens/sec prefill on 8 GB VRAM (RTX 4060) + 16 GB RAM using TurboQuant Prefill just went from 200 → 320+ tok/s on the same 8GB card. 1.6x, no new hardware, no new quant, just a KV cache trick stacked on top of the Gemma 4 26B MoE setup from a few days ago. A few days ago I posted Gemma 4 26B A4B hitting 28 tok/s decode on 8GB VRAM using native MTP. prefill was stuck around 200 tok/s. fair callout by the community. So today I tested something I'd already been meaning to try: TheTom/llama-cpp-turboquant, the TurboQuant KV cache fork by Tom Turney (Tom Turney). (github link in the comments) thanks to him, the fork just got resynced to mainline, so MTP + TurboQuant now run together cleanly (I didnt see any meaningful gains by using MTP with this setup though but you can try). The flags (No MTP): -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -c 64000 --cache-type-k q8_0 --cache-type-v turbo3 Results on the same RTX 4060 8GB, tested with a 27k token prompt at 64k context loaded: Prefill: 200 tok/s → 320+ tok/s Decode: stayed above 25 tok/s (without MTP) Why it works: TurboQuant uses walsh hadamard rotation + polar quantization on the KV cache. keys are sensitive to compression, values aren't much, so it splits the difference: K stays at q8_0, V drops to turbo3 (~3 bits). bonus from the memory savings: same 8GB card can now stretch to 100-120k context with minimal decode penalty. It should now be snappier with any agent harness such as hermes agent without compromise on intelligence. If you're already running Gemma 4 on a small card, this stacks on top for free. Try --cache-type-k q8_0 --cache-type-v turbo3 on your setup and report back what your prefill/decode split looks like. unsloth model gguf and llama.cpp turboquant fork links in the comments. what's your prefill number before vs after?

Alok

119,821 görüntüleme • 3 ay önce

🤯 A localmaxxer hit ~381 tok/s on a SINGLE RTX 3090 with Qwen3.8-27B. This developer has turned a 24GB RTX 3090 into a monster Qwen inference box - w/some creativity. Four days ago 👉 ⚡ ~82 tok/s single-user Then 👉 ⚡ ~114 tok/s with optimized MTP ⚡ ~138 tok/s with DFlash2 + lookup drafting Now 👉 🔥 ~381 tok/s on ONE request How? The recipe combines ... 🧠 Qwen3.8-27B 🎮 1× RTX 3090 24GB @ 250W ⚙️ heavily optimized vLLM ⚡ DFlash2 speculative decoding 🔎 lookup-augmented drafting 📚 prefix caching 🧮 16-token verification blocks 💾 quantized KV / heads / activations DFlash2 normally proposes 7 tokens. The developer realized the verification block doesn't have to stop there. If Qwen is answering from a document already sitting in the prompt, the system can fill the remaining draft positions using tokens found directly in that context. 🎯 So the target model can verify 16 tokens at once. On a ~25K-token document reproduction task: Previous DFlash2 👉 ~260 tok/s Longer verification + context lookup 👉 🔥 ~382 tok/s Acceptance: 🤯 15 of 16 tokens per verification step That is where the crazy number comes from. ⚠️ On ordinary real-world chat prompts, the same setup is around ~133 tok/s Still extremely fast for a dense 27B model on an RTX 3090. The 381 tok/s mode shines when the answer largely comes from material already in context so these are best use cases 📚 RAG / document Q&A 💻 Coding assistants applying edits 📝 Quoting or rewriting documents 🔎 Extracting information from long prompts And another optimization 👉 With prefix caching, a second question against the same 25K-token document reportedly goes from: 🐌 22.4 sec TTFT → ⚡ 0.56 sec TTFT Because the model doesn't need to process the whole document again. 🎯 It's specifically a mode for RAG front ends and coding agents. Follow iamMess on Reddit or syv-ai on GitHub 🔗 Reddit: r/LocalLLaMA/comments/1vtup5s/ 🔗 GitHub: /syv-ai/qwen38-27b-rtx3090

🤯 A localmaxxer hit ~381 tok/s on a SINGLE RTX 3090 with Qwen3.8-27B. This developer has turned a 24GB RTX 3090 into a monster Qwen inference box - w/some creativity. Four days ago 👉 ⚡ ~82 tok/s single-user Then 👉 ⚡ ~114 tok/s with optimized MTP ⚡ ~138 tok/s with DFlash2 + lookup drafting Now 👉 🔥 ~381 tok/s on ONE request How? The recipe combines ... 🧠 Qwen3.8-27B 🎮 1× RTX 3090 24GB @ 250W ⚙️ heavily optimized vLLM ⚡ DFlash2 speculative decoding 🔎 lookup-augmented drafting 📚 prefix caching 🧮 16-token verification blocks 💾 quantized KV / heads / activations DFlash2 normally proposes 7 tokens. The developer realized the verification block doesn't have to stop there. If Qwen is answering from a document already sitting in the prompt, the system can fill the remaining draft positions using tokens found directly in that context. 🎯 So the target model can verify 16 tokens at once. On a ~25K-token document reproduction task: Previous DFlash2 👉 ~260 tok/s Longer verification + context lookup 👉 🔥 ~382 tok/s Acceptance: 🤯 15 of 16 tokens per verification step That is where the crazy number comes from. ⚠️ On ordinary real-world chat prompts, the same setup is around ~133 tok/s Still extremely fast for a dense 27B model on an RTX 3090. The 381 tok/s mode shines when the answer largely comes from material already in context so these are best use cases 📚 RAG / document Q&A 💻 Coding assistants applying edits 📝 Quoting or rewriting documents 🔎 Extracting information from long prompts And another optimization 👉 With prefix caching, a second question against the same 25K-token document reportedly goes from: 🐌 22.4 sec TTFT → ⚡ 0.56 sec TTFT Because the model doesn't need to process the whole document again. 🎯 It's specifically a mode for RAG front ends and coding agents. Follow iamMess on Reddit or syv-ai on GitHub 🔗 Reddit: r/LocalLLaMA/comments/1vtup5s/ 🔗 GitHub: /syv-ai/qwen38-27b-rtx3090

David Hendrickson

245,223 görüntüleme • 1 ay önce

i spent 3 hours finding the sweet spot for hermes 4.3 36B on a single RTX 3090. saving you the trouble, anon. the model is 21.8GB at Q4_K_M. that leaves 2.2GB free on 24GB VRAM. not much room for KV cache. here's what actually happened: 4K: 35.3 tok/s 8K: 35.2 tok/s 16K: 34.8 tok/s 32K: 34.6 tok/s 64K: 6.4 tok/s 128K: 1.9 tok/s flat from 4K to 32K. then it falls off a cliff at 64K. the trick is quantized KV cache. without it you OOM at 16K. with quantized KV cache you get 32K at full speed. all 64 layers on GPU. at 64K something weird happens. ngl 99 (all layers on GPU) = 3.96 tok/s. the KV cache silently spills to CPU. drop to ngl 55 and speed jumps to 6.37. drop to ngl 48 and it gets worse again (3.46). there's an offload sweet spot where you free just enough VRAM for the cache without losing too much compute to PCIe transfers. 128K works at ngl 32 but you're at 1.95 tok/s. half the model on CPU. usable for batch work, not for interactive. the sweet spot command: llama-server -m hermes-4.3-36b-Q4_K_M.gguf -ngl 99 -c 32768 --cache-type-k q4_0 --cache-type-v q4_0 32K context. 34.6 tok/s. all on GPU. this is where dense 36B lives on 24GB. for comparison, qwen 3.5 (35B MoE, 3B active) holds 112 tok/s from 4K all the way to 262K on the same GPU. no speed drop. same total params, completely different architecture. hybrid linear attention means flat context scaling. dense pays for every token in the KV cache. code and quality comparison coming next. fast vs slow generation side by side in the videos below.

i spent 3 hours finding the sweet spot for hermes 4.3 36B on a single RTX 3090. saving you the trouble, anon. the model is 21.8GB at Q4_K_M. that leaves 2.2GB free on 24GB VRAM. not much room for KV cache. here's what actually happened: 4K: 35.3 tok/s 8K: 35.2 tok/s 16K: 34.8 tok/s 32K: 34.6 tok/s 64K: 6.4 tok/s 128K: 1.9 tok/s flat from 4K to 32K. then it falls off a cliff at 64K. the trick is quantized KV cache. without it you OOM at 16K. with quantized KV cache you get 32K at full speed. all 64 layers on GPU. at 64K something weird happens. ngl 99 (all layers on GPU) = 3.96 tok/s. the KV cache silently spills to CPU. drop to ngl 55 and speed jumps to 6.37. drop to ngl 48 and it gets worse again (3.46). there's an offload sweet spot where you free just enough VRAM for the cache without losing too much compute to PCIe transfers. 128K works at ngl 32 but you're at 1.95 tok/s. half the model on CPU. usable for batch work, not for interactive. the sweet spot command: llama-server -m hermes-4.3-36b-Q4_K_M.gguf -ngl 99 -c 32768 --cache-type-k q4_0 --cache-type-v q4_0 32K context. 34.6 tok/s. all on GPU. this is where dense 36B lives on 24GB. for comparison, qwen 3.5 (35B MoE, 3B active) holds 112 tok/s from 4K all the way to 262K on the same GPU. no speed drop. same total params, completely different architecture. hybrid linear attention means flat context scaling. dense pays for every token in the KV cache. code and quality comparison coming next. fast vs slow generation side by side in the videos below.

Sudo su

52,268 görüntüleme • 7 ay önce

DFlash makes Qwen 2.2x faster with no quality loss! We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks only differ in one thing - how predictable the next word is: quicksort, describe a file in JSON, a logic puzzle, a sci-fi story. Outputs: Baseline: 44 tok/s · 1.00x MTP: 65 tok/s · 1.45x · 71% accepted DFlash: 98 tok/s · 2.20x · 30% accepted Baseline writes one token per step. MTP works inside the model itself and guesses 3 tokens ahead. DFlash is a separate small model that writes 15 tokens at once, and the big model only checks them. In JSON the same words repeat all the time, so most guesses were right: 152 tok/s, 3.4x speedup. In the story 9 guesses out of 10 were wrong. DFlash did all that extra work for nothing and became slower than baseline: 42 vs 44 tok/s. MTP guesses only 3 tokens, so a wrong guess costs very little: 46 tok/s and the win in that round. The output is identical in all three modes - DFlash is the pick for tasks with predictable output, like coding, and for chat and creative writing MTP works better. DFlash is now natively integrated into Atomic Chat on llama.cpp - speed up your Qwen models!

DFlash makes Qwen 2.2x faster with no quality loss! We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks only differ in one thing - how predictable the next word is: quicksort, describe a file in JSON, a logic puzzle, a sci-fi story. Outputs: Baseline: 44 tok/s · 1.00x MTP: 65 tok/s · 1.45x · 71% accepted DFlash: 98 tok/s · 2.20x · 30% accepted Baseline writes one token per step. MTP works inside the model itself and guesses 3 tokens ahead. DFlash is a separate small model that writes 15 tokens at once, and the big model only checks them. In JSON the same words repeat all the time, so most guesses were right: 152 tok/s, 3.4x speedup. In the story 9 guesses out of 10 were wrong. DFlash did all that extra work for nothing and became slower than baseline: 42 vs 44 tok/s. MTP guesses only 3 tokens, so a wrong guess costs very little: 46 tok/s and the win in that round. The output is identical in all three modes - DFlash is the pick for tasks with predictable output, like coding, and for chat and creative writing MTP works better. DFlash is now natively integrated into Atomic Chat on llama.cpp - speed up your Qwen models!

atomic.chat

83,196 görüntüleme • 3 ay önce

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

344,809 görüntüleme • 1 ay önce

a new 8GB VRAM GPU dense Local LLM leader was born yesterday runs on: RTX 4060 / RTX 3070 / RTX 2080. any 8GB card Qwen 3.5 9B (dense) was the go to for 6-8GB VRAM builds. Gemma 4 12B QAT (dense) just changed that. same llama.cpp + cuda 13.2. i7 12700H. 16GB RAM. same -ngl 99 flags. same 48k context. unsloth gemma-4-12b-it-Q4_K_M.gguf → 15 tok/sec @ 48k ctx unsloth gemma-4-12B-it-qat-UD-Q4_K_XL.gguf → 32 tok/sec @ 48k ctx → 26 tok/sec @ 64k ctx 64k context is a big deal. Hermes 3 agent requires 64k minimum to run. you're now getting full hermes compatible context on a budget consumer GPU at 26 tok/sec locally. 2.1x faster on identical hardware. and here's the part that breaks your brain: the QAT-UD-Q4_K_XL is actually SMALLER than the Q4_K_M "XL" why? QAT = Quantization Aware Training Google didn't train the model first and compress it later they trained it to be quantized from day one the weights already know how to survive low precision that's why you get more quality per byte llamacpp flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 48000 -v fits in 8GB VRAM clean. no API. no cloud. no subscription. and this isn't even the MTP variant yet Gemma-4-E2B QAT runs on 3GB RAM, E4B on 5GB, 12B on 7GB, 26-A4B on 15GB and 31B on 18GB. I have benchmarked the 26b and 31b qat as well on a single RTX 4090, checkout the comments for details. If you have a 6GB or 8GB VRAM GPU, post your numbers. more benchmarks and configs coming soon

a new 8GB VRAM GPU dense Local LLM leader was born yesterday runs on: RTX 4060 / RTX 3070 / RTX 2080. any 8GB card Qwen 3.5 9B (dense) was the go to for 6-8GB VRAM builds. Gemma 4 12B QAT (dense) just changed that. same llama.cpp + cuda 13.2. i7 12700H. 16GB RAM. same -ngl 99 flags. same 48k context. unsloth gemma-4-12b-it-Q4_K_M.gguf → 15 tok/sec @ 48k ctx unsloth gemma-4-12B-it-qat-UD-Q4_K_XL.gguf → 32 tok/sec @ 48k ctx → 26 tok/sec @ 64k ctx 64k context is a big deal. Hermes 3 agent requires 64k minimum to run. you're now getting full hermes compatible context on a budget consumer GPU at 26 tok/sec locally. 2.1x faster on identical hardware. and here's the part that breaks your brain: the QAT-UD-Q4_K_XL is actually SMALLER than the Q4_K_M "XL" why? QAT = Quantization Aware Training Google didn't train the model first and compress it later they trained it to be quantized from day one the weights already know how to survive low precision that's why you get more quality per byte llamacpp flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 48000 -v fits in 8GB VRAM clean. no API. no cloud. no subscription. and this isn't even the MTP variant yet Gemma-4-E2B QAT runs on 3GB RAM, E4B on 5GB, 12B on 7GB, 26-A4B on 15GB and 31B on 18GB. I have benchmarked the 26b and 31b qat as well on a single RTX 4090, checkout the comments for details. If you have a 6GB or 8GB VRAM GPU, post your numbers. more benchmarks and configs coming soon

Alok

264,581 görüntüleme • 4 ay önce

$I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.$

I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.

Alok

36,617 görüntüleme • 2 ay önce

$$50 free credits for GLM 5.2, KIMI 2.7, and DeepSeek V4 Pro. No promo most people know about. Your buddy pays $20/mo for Claude Pro and burns through it in a week. > You get $50 in credits on Aiand for models that actually beat Sonnet on code benchmarks - and the balance barely moves. Inside: → GLM 5.2 - the model that shipped with better coding scores than closed frontier → KIMI K2.7 Code - long-context beast, sits at the top of open leaderboards → DeepSeek V4 Pro - reasoning at a fraction of GPT-5.6 cost → OpenAI-compatible endpoint - plug into any agent, IDE, or tool you already run Setup, 5 steps: → aiand(.)com → Sign up with Google or GitHub → If the page won't load, switch to an EU VPN and retry → Top right corner - click Credits, select $50 → In Stripe checkout, apply promo code: UNITABETAI → $50 lands. Start firing prompts. Balance drains slowly. I've been running agents on it for days and barely dented it. The promo code is valid as of the time of publication Bookmark this before they cap the promo.$

$50 free credits for GLM 5.2, KIMI 2.7, and DeepSeek V4 Pro. No promo most people know about. Your buddy pays $20/mo for Claude Pro and burns through it in a week. > You get $50 in credits on Aiand for models that actually beat Sonnet on code benchmarks - and the balance barely moves. Inside: → GLM 5.2 - the model that shipped with better coding scores than closed frontier → KIMI K2.7 Code - long-context beast, sits at the top of open leaderboards → DeepSeek V4 Pro - reasoning at a fraction of GPT-5.6 cost → OpenAI-compatible endpoint - plug into any agent, IDE, or tool you already run Setup, 5 steps: → aiand(.)com → Sign up with Google or GitHub → If the page won't load, switch to an EU VPN and retry → Top right corner - click Credits, select $50 → In Stripe checkout, apply promo code: UNITABETAI → $50 lands. Start firing prompts. Balance drains slowly. I've been running agents on it for days and barely dented it. The promo code is valid as of the time of publication Bookmark this before they cap the promo.

Atenov int.

14,846 görüntüleme • 3 ay önce

"which quant should I download?" is a question you may never have to answer again the team Hamster Labs has figured out how to kill it with pMLX. download once at full precision (bf16) and the engine re-fits it to your machine on the fly, based on the job you give it tell it two things: how much context you need, and the slowest speed you'll accept. it reads your mac and picks the quantization plus how many experts stay in ram vs. stream from disk. no need to download a smaller quant or figure out which quant fits your hardware. same model for every use case and the config changes based on what you need I ran this on my own M4 Max and Qwen3.8-Flash-Next splits into two configs. short context, under 32k: full bf16, ~85% of experts in ram, fast at full precision since it fits my ram long context, 64k to 256k: keep ~70% bf16 and stream the rest from SSD at 22 tok/s, or drop to q8 and get 40 tok/s. I pick per task and the model itself never changes. there are many possibilities since I have the ram to spar - if i need speed, Q4 100% resident (74gb ram @ 62 tok/s) - if i need balance, Q8 95% resident (76gb @ 38 tok/s) - if i need accuracy, bf16 70% resident (96gb @ 16 tok/s) given whatever RAM you have (16/32/64/128/256 GB) + your context + your min speed, the engine picks the precision (bf16→q8→q4) and the expert-residency/paging split that fits your box and maximizes quality & speed last thing to optimize is speed. there are so many things we want to power with open models at Hamster and these 180b-300b class models have the potential to play a big role in that

"which quant should I download?" is a question you may never have to answer again the team Hamster Labs has figured out how to kill it with pMLX. download once at full precision (bf16) and the engine re-fits it to your machine on the fly, based on the job you give it tell it two things: how much context you need, and the slowest speed you'll accept. it reads your mac and picks the quantization plus how many experts stay in ram vs. stream from disk. no need to download a smaller quant or figure out which quant fits your hardware. same model for every use case and the config changes based on what you need I ran this on my own M4 Max and Qwen3.8-Flash-Next splits into two configs. short context, under 32k: full bf16, ~85% of experts in ram, fast at full precision since it fits my ram long context, 64k to 256k: keep ~70% bf16 and stream the rest from SSD at 22 tok/s, or drop to q8 and get 40 tok/s. I pick per task and the model itself never changes. there are many possibilities since I have the ram to spar - if i need speed, Q4 100% resident (74gb ram @ 62 tok/s) - if i need balance, Q8 95% resident (76gb @ 38 tok/s) - if i need accuracy, bf16 70% resident (96gb @ 16 tok/s) given whatever RAM you have (16/32/64/128/256 GB) + your context + your min speed, the engine picks the precision (bf16→q8→q4) and the expert-residency/paging split that fits your box and maximizes quality & speed last thing to optimize is speed. there are so many things we want to power with open models at Hamster and these 180b-300b class models have the potential to play a big role in that

Eyal Toledano

10,743 görüntüleme • 1 ay önce

Deepseek V4 Flash 0731 (Q2) - 12 tokens/sec - Single RTX 4090 - 650+ tokens/sec prefill - 250k context - no kv cache quantization! DeepSeek just dropped the official V4 Flash 0731 two days ago with a massive agent capabilities upgrade. The official benchmarks are literally crushing their own V4-Pro-Preview on agentic tasks like Terminal Bench 2.1 and DeepSWE. Unsloth AI said they couldn't wait to bring it to local devices, and they delivered. If you thought my 118B Poolside Laguna S 2.1 MoE run last week on a single GPU was wild, hold onto your hardware. I just successfully ran Unsloth’s brand new 91GB DeepSeek-V4-Flash-0731 (UD-IQ2_M) GGUF entirely locally. And I pushed it to a mind-bending 250,000 context window. The VRAM ceiling is an illusion if you know how to optimize llama.cpp. Here are the benchmarks and the cheat codes to run a local frontier class model yourself. For the hardware and setup, I used a single NVIDIA RTX 4090 (24GB VRAM) hooked up via a PCIe 4 bus, running Ubuntu 22.04 LTS and CUDA 13.0. You don't need a massive enterprise server for this, if you have more than 80 GB of standard DDR4 RAM and a 24GB card like an RTX 3090 or 4090, you can run this exact stack yourself. All benchmarks were run using a massive 28k token prompt to truly stress test the prefill limits. no kv cache quantization THE BENCHMARKS (Scaling Context): # 80k Context (Baseline: -b 2048 -ub 2048): Prefill: 465.43 t/s | Decode: 13.00 t/s | VRAM: 22.87 GB # 80k Context (Optimized: -b 4096 -ub 4096): Prefill: 643.15 t/s | Decode: 12.20 t/s | VRAM: 23.00 GB (Notice how doubling the batch flags spiked my prefill throughput by nearly 200 t/s with almost zero VRAM penalty) # 180k Context (-b 4096 -ub 4096): Prefill: 629.18 t/s | Decode: 11.92 t/s | VRAM: 23.40 GB # 250k Context MAXIMUM (-b 4096 -ub 4096): Prefill: 619.02 t/s | Decode: 11.54 t/s | VRAM: 23.40 GB # THE SECRET SAUCE (Why this works): Unsloth’s UD-IQ2_M quant is ~91GB across 3 files. Since I only have 24GB of VRAM, the PCIe 4 bus and system RAM have to do the heavy lifting. The magic bullet is the --no-mmap flag. By completely bypassing OS disk paging, I forced llama.cpp to load the massive model weights directly into the system RAM upfront. Combined with Flash Attention (-fa on) and exactly 12 CPU threads (--threads 12), I maintained an incredibly stable 11.5+ tokens/sec decode speed even at a quarter million token context. # THE EXACT COMMAND: ./build/bin/llama-server -m /workspace/models/DeepSeek-V4-Flash-0731-UD-IQ2_M-00001-of-00003.gguf -c 250000 -fa on --port 8080 --threads 12 -b 4096 -ub 4096 --no-mmap -v Local conversational and agentic coding AI is fully here. You don’t need an API or an H100 cluster. Qwen 3.8 27b drops next week making the 24GB VRAM tier even more worthwhile. What does your current local AI rig look like, and what's the craziest model you've managed to squeeze into it? Official huggingface GGUF links from Unsloth and performance graphs are dropped in the replies below!

Alok

46,100 görüntüleme • 2 ay önce

$I nearly 2x'd the speed while only using +1GB VRAM with the new MTP update in llama.cpp 🤯 You need to add these flags to start using it: --spec-type draft-mtp \ --spec-draft-p-min 0.75 \ --spec-draft-n-max 2 My results with Qwen3.6 27B on a single RTX 5080 ↓ ⚪️ no flag (without mtp) → 54.3 tok/s with 13.26GB VRAM 🔵 --spec-draft-n-max 2 → 90.7 tok/s with 14.29GB VRAM 🔴 --spec-draft-n-max 2 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.30GB VRAM 🟢 --spec-draft-n-max 6 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.87GB VRAM Increasing to 6 draft tokens didn't help my setup for some reason. I made sure to test with a low context length to have enough headroom and eliminate risk of vram stress. From my understanding: 1) The speed gains are very task-dependent. You need to test across a wide range of tasks to get a realistic idea of the benefits 2) We’re already running heavily quantized GGUF models (Q3, Q4, Q6, etc.), so we already benefit from strong speed/performance thanks to the reduced size. That’s why some people are seeing little to no improvement compared to MLX or other quantized versions The progress over the past few days has been insane to say the least. However, MTP now consumes significantly more VRAM. Personally 16GB just isn't enough to use MTP and run it with a good context size. Time to upgrade lads, 24GB+ users are eating GOOD today 🔥 Full setup below ↓$

I nearly 2x'd the speed while only using +1GB VRAM with the new MTP update in llama.cpp 🤯 You need to add these flags to start using it: --spec-type draft-mtp \ --spec-draft-p-min 0.75 \ --spec-draft-n-max 2 My results with Qwen3.6 27B on a single RTX 5080 ↓ ⚪️ no flag (without mtp) → 54.3 tok/s with 13.26GB VRAM 🔵 --spec-draft-n-max 2 → 90.7 tok/s with 14.29GB VRAM 🔴 --spec-draft-n-max 2 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.30GB VRAM 🟢 --spec-draft-n-max 6 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.87GB VRAM Increasing to 6 draft tokens didn't help my setup for some reason. I made sure to test with a low context length to have enough headroom and eliminate risk of vram stress. From my understanding: 1) The speed gains are very task-dependent. You need to test across a wide range of tasks to get a realistic idea of the benefits 2) We’re already running heavily quantized GGUF models (Q3, Q4, Q6, etc.), so we already benefit from strong speed/performance thanks to the reduced size. That’s why some people are seeing little to no improvement compared to MLX or other quantized versions The progress over the past few days has been insane to say the least. However, MTP now consumes significantly more VRAM. Personally 16GB just isn't enough to use MTP and run it with a good context size. Time to upgrade lads, 24GB+ users are eating GOOD today 🔥 Full setup below ↓

left curve dev

30,086 görüntüleme • 4 ay önce

People made fun of Alex Finn for buying three Mac Studios to run AI at home. Then Fable got banned for a week, GLM 5.2 dropped, and those exact Mac Studios started reselling for 4x what he paid. He showed me how he built his home AI lab from scratch. Here's the playbook: 1) The hardware. three 512GB Mac Studios, an NVIDIA DGX Spark, a custom RTX 5090 build, and a few Mac Minis. ~$30k all in. 2) The buying framework... - Mac Studio: huge memory, runs GLM 5.2 (open weights, near Opus 4.8 on benchmarks), but slow. - DGX Spark ($4,800): the sweet spot for most people. - RTX 5090: smaller models at blazing speed (Qwen's 29B now hits Sonnet 4 level). 3) Tailscale networks every machine into one private network with root access to each other. Only one machine is plugged into a monitor. 4) A Nous Research Hermes agent is his IT guy. New model drops? It SSHs into the right box, loads 5 candidates, runs evals overnight, and reports back which task belongs on which machine. Alex has literally never loaded a model himself. 5) The whole point: achieving "ambient intelligence." Always-on jobs that would bankrupt you on per-token billing. A security sweep of his API endpoints every hour. Code optimization every 20 minutes. Database anomaly & churn detection. Hourly scraping of X, Reddit & Hacker News for business opportunities. 6) Running those workloads on frontier models would cost thousands a month. His actual cost: ~$60 more in electricity. 7) Btw he's not anti-frontier. He still maxes out his Claude plan. The way he sees it: frontier is for hard thinking, local is for the foot soldiers that never sleep. 8) "We own everything except for the intelligence. Why can't we own the intelligence?" 9) He thinks frontier-level intelligence runs on consumer hardware within 6 months.

People made fun of Alex Finn for buying three Mac Studios to run AI at home. Then Fable got banned for a week, GLM 5.2 dropped, and those exact Mac Studios started reselling for 4x what he paid. He showed me how he built his home AI lab from scratch. Here's the playbook: 1) The hardware. three 512GB Mac Studios, an NVIDIA DGX Spark, a custom RTX 5090 build, and a few Mac Minis. ~$30k all in. 2) The buying framework... - Mac Studio: huge memory, runs GLM 5.2 (open weights, near Opus 4.8 on benchmarks), but slow. - DGX Spark ($4,800): the sweet spot for most people. - RTX 5090: smaller models at blazing speed (Qwen's 29B now hits Sonnet 4 level). 3) Tailscale networks every machine into one private network with root access to each other. Only one machine is plugged into a monitor. 4) A Nous Research Hermes agent is his IT guy. New model drops? It SSHs into the right box, loads 5 candidates, runs evals overnight, and reports back which task belongs on which machine. Alex has literally never loaded a model himself. 5) The whole point: achieving "ambient intelligence." Always-on jobs that would bankrupt you on per-token billing. A security sweep of his API endpoints every hour. Code optimization every 20 minutes. Database anomaly & churn detection. Hourly scraping of X, Reddit & Hacker News for business opportunities. 6) Running those workloads on frontier models would cost thousands a month. His actual cost: ~$60 more in electricity. 7) Btw he's not anti-frontier. He still maxes out his Claude plan. The way he sees it: frontier is for hard thinking, local is for the foot soldiers that never sleep. 8) "We own everything except for the intelligence. Why can't we own the intelligence?" 9) He thinks frontier-level intelligence runs on consumer hardware within 6 months.

Alex Lieberman

57,764 görüntüleme • 3 ay önce

$Someone is running Qwen3.8-35B-A3B on a single RTX 3060 with just 12GB of VRAM and 16GB of system RAM. And the reported numbers are surprisingly good. ~50 tok/s decode ~550 tok/s prefill 262K context window 12GB VRAM 16GB system RAM No multi-GPU setup. Just a consumer graphics card and a carefully configured local inference engine. The model is particularly interesting because it’s Qwen3.8 distilled into the Qwen3.6-35B-A3B architecture. That makes it worth investigating beyond raw inference speed. The developer currently uses Ornith-1.5 as the engine behind their Hermes and Pi agents. Now the question is whether this new model can deliver enough quality and reliability to replace it. The next step is testing it as an agent, not just a chatbot. The planned comparison includes four models: • Qwen3.8-35B-A3B • Ornith-1.5-35B • Qwen3.8-27B • Base Qwen3.6-35B-A3B An agentic coding benchmark should help reveal whether the distilled model can follow instructions, modify code, solve problems, and complete multi-step tasks effectively. That distinction matters because high token throughput doesn’t automatically translate into a better coding agent. A model can generate code quickly and still struggle with planning, debugging, tool calls, or completing a task without intervention. The hardware configuration is also worth examining. The developer is using llama-server with the Q4_K_M GGUF quantization, Flash Attention enabled, and 26 MoE layers assigned to CPU processing. The configuration also enables a 262,144-token context window, Q8 KV-cache quantization, and reasoning mode. This is how the setup is configured: llama-server \ -m Qwen3.8-35B-A3B-Q4_K_M.gguf \ -ngl 99 \ --n-cpu-moe 26 \ -c 262144 \ -fa on \ --jinja \ -np 1 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --reasoning on One important detail: the model’s 35B parameter count doesn’t mean all 35 billion parameters are activated for every token. The A3B designation indicates roughly 3 billion active parameters per token. That sparse MoE architecture helps explain how a model of this size can deliver high generation speeds on modest hardware. CPU-based expert processing also helps work around the GPU’s limited memory, although the exact trade-offs depend on the workload. The 262K context claim is equally interesting. Supporting a context window that large doesn’t mean every request at that length will maintain the same speed. Prompt length, KV-cache memory, and CPU/GPU placement can all affect performance. The developer’s next test is particularly useful: graphing decode speed against context length. That should help show how performance changes as the prompt grows, rather than presenting one speed figure without the conditions behind it. If the coding benchmark also holds up, this could become a serious option for people who want capable local agents without investing in expensive GPUs. For now, the reported inference numbers are promising. The real test is whether Qwen3.8-35B-A3B can match or outperform Ornith-1.5-35B on actual agentic coding tasks. Fast local inference gets your attention. Reliable autonomous task completion is what earns a model a place in the stack.$

Someone is running Qwen3.8-35B-A3B on a single RTX 3060 with just 12GB of VRAM and 16GB of system RAM. And the reported numbers are surprisingly good. ~50 tok/s decode ~550 tok/s prefill 262K context window 12GB VRAM 16GB system RAM No multi-GPU setup. Just a consumer graphics card and a carefully configured local inference engine. The model is particularly interesting because it’s Qwen3.8 distilled into the Qwen3.6-35B-A3B architecture. That makes it worth investigating beyond raw inference speed. The developer currently uses Ornith-1.5 as the engine behind their Hermes and Pi agents. Now the question is whether this new model can deliver enough quality and reliability to replace it. The next step is testing it as an agent, not just a chatbot. The planned comparison includes four models: • Qwen3.8-35B-A3B • Ornith-1.5-35B • Qwen3.8-27B • Base Qwen3.6-35B-A3B An agentic coding benchmark should help reveal whether the distilled model can follow instructions, modify code, solve problems, and complete multi-step tasks effectively. That distinction matters because high token throughput doesn’t automatically translate into a better coding agent. A model can generate code quickly and still struggle with planning, debugging, tool calls, or completing a task without intervention. The hardware configuration is also worth examining. The developer is using llama-server with the Q4_K_M GGUF quantization, Flash Attention enabled, and 26 MoE layers assigned to CPU processing. The configuration also enables a 262,144-token context window, Q8 KV-cache quantization, and reasoning mode. This is how the setup is configured: llama-server \ -m Qwen3.8-35B-A3B-Q4_K_M.gguf \ -ngl 99 \ --n-cpu-moe 26 \ -c 262144 \ -fa on \ --jinja \ -np 1 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --reasoning on One important detail: the model’s 35B parameter count doesn’t mean all 35 billion parameters are activated for every token. The A3B designation indicates roughly 3 billion active parameters per token. That sparse MoE architecture helps explain how a model of this size can deliver high generation speeds on modest hardware. CPU-based expert processing also helps work around the GPU’s limited memory, although the exact trade-offs depend on the workload. The 262K context claim is equally interesting. Supporting a context window that large doesn’t mean every request at that length will maintain the same speed. Prompt length, KV-cache memory, and CPU/GPU placement can all affect performance. The developer’s next test is particularly useful: graphing decode speed against context length. That should help show how performance changes as the prompt grows, rather than presenting one speed figure without the conditions behind it. If the coding benchmark also holds up, this could become a serious option for people who want capable local agents without investing in expensive GPUs. For now, the reported inference numbers are promising. The real test is whether Qwen3.8-35B-A3B can match or outperform Ornith-1.5-35B on actual agentic coding tasks. Fast local inference gets your attention. Reliable autonomous task completion is what earns a model a place in the stack.

FHILY👑

19,254 görüntüleme • 2 gün önce