Загрузка видео...

Не удалось загрузить видео

На главную

testing Qwen3.5-35B-A3B latest optimized version by UnslothAI on a single RTX 3090. one detailed prompt. zero handholding. watch a 3B model scaffold an entire multifile game project autonomously. the setup: > model: Qwen3.5-35B-A3B (80B total, only 3B active per token) > quant: UD-Q4_K_XL by Unsloth (MXFP4 layers removed in...

167,565 просмотров • 8 месяцев назад •via X (Twitter)

Комментарии: 21

Фото профиля Sunny
Sunny8 месяцев назад

crazy: You are getting late-2025 frontier performance (Sonnet 4.5 / early Opus 4.5 tier) in a model you can run locally for free, uncensored, with zero latency or cost. Opus 4.0 was amazing in May 2025 — this Qwen smokes it in 2026.

Фото профиля The one
The one8 месяцев назад

My 3090 seems to be working even faster (and hotter) with llama-server.exe ^ -m ".\Qwen3.5-355B-A3B-UD-Q4_K_XL.gguf" ^ -ngl 99 ^ -c 131072 ^ -b 2048 ^ -ub 512 ^ -np 1 ^ --cache-type-k q8_0 ^ --cache-type-v q8_0 ^ --host 127.0.0.1 ^ --flash-attn on ^ -t 0 ^ --port 8080

Фото профиля Steve💙🇨🇦
Steve💙🇨🇦8 месяцев назад

I'm getting 148 t/s with the q6 variant of this setup. Rtx 5090.

Фото профиля Mouse&Keyboard
Mouse&Keyboard8 месяцев назад

Ypu get 140 tok/s cause context is not fully in VRAM, also in DRAM. Got 1900 tok/s vram only with 100k context

Фото профиля KleinyME
KleinyME8 месяцев назад

Do you feel like 1 3090 is enough to have significant local models running for an openclaw subagents? Debating if its worth buying or i need need significantly more firepower

Фото профиля M.en
M.en8 месяцев назад

Watching AI build games while my code still has typos. 😅

Фото профиля Vasilis Stergiou
Vasilis Stergiou8 месяцев назад

Any vLLM option for this Qwen version?

Фото профиля IRMC
IRMC8 месяцев назад

Thank you very much. Will try it immediately on my 4090. I have a LangGraph full of Agents in which I'm currently implementing OpenClaw. Intelligence and context length are key succes factors in such 'local' architecture. My llm_router offers pooling, so np 1 is no probs. Thanks.

Фото профиля Subhash Dasyam
Subhash Dasyam8 месяцев назад

The -np 1 callout is the kind of detail that saves someone 2 hours of debugging OOM crashes. 262K context with q8_0 KV on a single 3090 is wild, that would've been unthinkable like a year ago. Curious how the MXFP4 removal in the latest UD quant affected quality vs the earlier version.

Фото профиля sharaff |🦞
sharaff |🦞8 месяцев назад

This on 32GB Mac mini

Фото профиля Hetarth Chopra
Hetarth Chopra8 месяцев назад

I would love for you to try this same setup on spot instances.

Фото профиля HÄCKZLER
HÄCKZLER8 месяцев назад

@grok what does this mean?

Фото профиля random_hero
random_hero8 месяцев назад

@grok AMD 7800X3D, 32GB DDR4 5600 and AMD RX 7800 XT. Any Chance this will run on my machine?

Фото профиля Diesel-O-Saurus 🦖💨
Diesel-O-Saurus 🦖💨8 месяцев назад

Is this all parameters? No Temperature adjustments, top-k for coding etc? For me Claude Code frequently doesn’t finish.

Фото профиля JM
JM8 месяцев назад

I fake back some things about it not running well previously (other chat on twitter 😉). Tweaked my llama.cpp memory parameters a bit, got a context going of 32k or even 64k while having some ram leftover for vscode. 40 tokens/sec. Works in copilot. That's on a 5060ti-16gb.

Фото профиля toni
toni8 месяцев назад

3B active params doing all the heavy lifting shows the real story here. The 80B size is basically marketing. Sparse MoE means you're running a 3B model that learned from a giant one.

Фото профиля Nico
Nico8 месяцев назад

with my 3090, i ve only 50 t/ s , any idea why ?

Фото профиля Torpedo
Torpedo8 месяцев назад

End results

Фото профиля khaled Diab, Ph.D.
khaled Diab, Ph.D.8 месяцев назад

Perfect, how much time did the prompt take compared to running on claude code? and how different are the results?

Фото профиля Subscribe MLM
Subscribe MLM8 месяцев назад

@grok what could I run on a Mac mini?

Фото профиля jowsey 🐱cat emoji
jowsey 🐱cat emoji8 месяцев назад

-np 1 isn't default, it's -1, which is auto

Похожие видео

i spent 3 hours finding the sweet spot for hermes 4.3 36B on a single RTX 3090. saving you the trouble, anon. the model is 21.8GB at Q4_K_M. that leaves 2.2GB free on 24GB VRAM. not much room for KV cache. here's what actually happened: 4K: 35.3 tok/s 8K: 35.2 tok/s 16K: 34.8 tok/s 32K: 34.6 tok/s 64K: 6.4 tok/s 128K: 1.9 tok/s flat from 4K to 32K. then it falls off a cliff at 64K. the trick is quantized KV cache. without it you OOM at 16K. with quantized KV cache you get 32K at full speed. all 64 layers on GPU. at 64K something weird happens. ngl 99 (all layers on GPU) = 3.96 tok/s. the KV cache silently spills to CPU. drop to ngl 55 and speed jumps to 6.37. drop to ngl 48 and it gets worse again (3.46). there's an offload sweet spot where you free just enough VRAM for the cache without losing too much compute to PCIe transfers. 128K works at ngl 32 but you're at 1.95 tok/s. half the model on CPU. usable for batch work, not for interactive. the sweet spot command: llama-server -m hermes-4.3-36b-Q4_K_M.gguf -ngl 99 -c 32768 --cache-type-k q4_0 --cache-type-v q4_0 32K context. 34.6 tok/s. all on GPU. this is where dense 36B lives on 24GB. for comparison, qwen 3.5 (35B MoE, 3B active) holds 112 tok/s from 4K all the way to 262K on the same GPU. no speed drop. same total params, completely different architecture. hybrid linear attention means flat context scaling. dense pays for every token in the KV cache. code and quality comparison coming next. fast vs slow generation side by side in the videos below.

Sudo su

52,268 просмотров • 8 месяцев назад

A single RTX 4090 (24 GB VRAM) can run the updated gemma 4 31B (dense) model with a 190,000 context window at 33 tokens/second. The VRAM barrier is dying. Google quietly updated Gemma 4, and Unsloth immediately compiled the new quants. I built llama.cpp from source on Ubuntu 22 to benchmark it. Google's stealth update 2 days ago enabled uniform Flash Attention 4 on Hopper to boost prefill and patched the chat template to improve tool calling. The agentic reasoning gains on the benchmark charts are massive: TB2 (Agents): +4.5% (to 25.8%) Tau2 (Telecom): +10.1% (to 62.7%) Running on Ubuntu 22, CUDA 13.0 with a single NVIDIA GeForce RTX 4090. Here is the exact step by step benchmarking process with a massive 28k tokens prompt and the commands I used to squeeze out maximum context without killing my throughput: # 1. The Baseline (Unquantized KV Cache) I started with full GPU offload (-ngl 99) and pushed the context to 40k. llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -ngl 99 -c 40000 -fa on --port 8080 -v VRAM: 23.8 GB (maxed out on card) Throughput: Prefill: 2198.81 t/s | Decode: 35.77 t/s (with 28k tokens prompt) # 2. The CPU Split Trap I tried stretching to 80k context by offloading layers to the CPU (-ngl 52). llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 80000 -ngl 52 -fa on --port 8080 -v Throughput: Prefill: 1212.73 t/s | Decode: 5 t/s (with 28k tokens prompt) # 3. The KV Quantization Breakthrough Instead of spilling layers to the CPU, I kept the model fully on card (-ngl 99) but enabled 8-bit KV cache quantization to free up VRAM. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 100000 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 --port 8080 -v VRAM: 23.9 GB Throughput: Prefill: 2139.68 t/s | Decode: 32 t/s (with 28k tokens prompt) Result: 100k tokens of context on a single GPU with practically zero speed loss (and minimal intelligence loss). # 4. The Limit Test (Q4 KV Cache) To find the absolute breaking point, I dropped the KV cache to 4 bit (q4_0) and set -c 190000. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 190000 --cache-type-k q4_0 --cache-type-v q4_0 -ngl 99 --port 8080 -v VRAM: 23.8 GB Throughput: Prefill: 2206.66 t/s | Decode: 33 t/s (with 28k tokens prompt) (Note: Pushing it to 220k required dropping to -ngl 58 again, which immediately penalized decode down to 17 t/s). # The Tradeoff: For Max Reasoning: Keep your KV cache unquantized (f16). You get pristine reasoning but hit a strict 40k context ceiling. For Massive Document Retrieval: If you need to feed the model giant codebases, use --cache-type-k q4_0. Getting 190k context at 33 tokens/second on a consumer desktop with a 31b dense model is a cheat code. If you’re rocking a single 3090 or 4090 and slept on Gemma 4 earlier, this update is your cue to dust off the terminal. Hugging Face links to the Unsloth QAT quants are in the replies below.

Alok

76,069 просмотров • 4 месяцев назад

Google's Gemma 4 26B A4B QAT hits 25+ tokens/sec and 320+ tokens/sec prefill on 8 GB VRAM (RTX 4060) + 16 GB RAM using TurboQuant Prefill just went from 200 → 320+ tok/s on the same 8GB card. 1.6x, no new hardware, no new quant, just a KV cache trick stacked on top of the Gemma 4 26B MoE setup from a few days ago. A few days ago I posted Gemma 4 26B A4B hitting 28 tok/s decode on 8GB VRAM using native MTP. prefill was stuck around 200 tok/s. fair callout by the community. So today I tested something I'd already been meaning to try: TheTom/llama-cpp-turboquant, the TurboQuant KV cache fork by Tom Turney (Tom Turney). (github link in the comments) thanks to him, the fork just got resynced to mainline, so MTP + TurboQuant now run together cleanly (I didnt see any meaningful gains by using MTP with this setup though but you can try). The flags (No MTP): -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -c 64000 --cache-type-k q8_0 --cache-type-v turbo3 Results on the same RTX 4060 8GB, tested with a 27k token prompt at 64k context loaded: Prefill: 200 tok/s → 320+ tok/s Decode: stayed above 25 tok/s (without MTP) Why it works: TurboQuant uses walsh hadamard rotation + polar quantization on the KV cache. keys are sensitive to compression, values aren't much, so it splits the difference: K stays at q8_0, V drops to turbo3 (~3 bits). bonus from the memory savings: same 8GB card can now stretch to 100-120k context with minimal decode penalty. It should now be snappier with any agent harness such as hermes agent without compromise on intelligence. If you're already running Gemma 4 on a small card, this stacks on top for free. Try --cache-type-k q8_0 --cache-type-v turbo3 on your setup and report back what your prefill/decode split looks like. unsloth model gguf and llama.cpp turboquant fork links in the comments. what's your prefill number before vs after?

Alok

119,821 просмотров • 5 месяцев назад

I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.

Alok

36,617 просмотров • 3 месяцев назад