Video wird geladen...
Video konnte nicht geladen werden
testing Qwen3.5-35B-A3B latest optimized version by UnslothAI on a single RTX 3090. one detailed prompt. zero handholding. watch a 3B model scaffold an entire multifile game project autonomously. the setup: > model: Qwen3.5-35B-A3B (80B total, only 3B active per token) > quant: UD-Q4_K_XL by Unsloth (MXFP4 layers removed in... show more
167,565 Aufrufe • vor 8 Monaten •via X (Twitter)
21 Kommentare

crazy: You are getting late-2025 frontier performance (Sonnet 4.5 / early Opus 4.5 tier) in a model you can run locally for free, uncensored, with zero latency or cost. Opus 4.0 was amazing in May 2025 — this Qwen smokes it in 2026.

My 3090 seems to be working even faster (and hotter) with llama-server.exe ^ -m ".\Qwen3.5-355B-A3B-UD-Q4_K_XL.gguf" ^ -ngl 99 ^ -c 131072 ^ -b 2048 ^ -ub 512 ^ -np 1 ^ --cache-type-k q8_0 ^ --cache-type-v q8_0 ^ --host 127.0.0.1 ^ --flash-attn on ^ -t 0 ^ --port 8080

I'm getting 148 t/s with the q6 variant of this setup. Rtx 5090.

Ypu get 140 tok/s cause context is not fully in VRAM, also in DRAM. Got 1900 tok/s vram only with 100k context

Do you feel like 1 3090 is enough to have significant local models running for an openclaw subagents? Debating if its worth buying or i need need significantly more firepower

Watching AI build games while my code still has typos. 😅

Any vLLM option for this Qwen version?

Thank you very much. Will try it immediately on my 4090. I have a LangGraph full of Agents in which I'm currently implementing OpenClaw. Intelligence and context length are key succes factors in such 'local' architecture. My llm_router offers pooling, so np 1 is no probs. Thanks.

The -np 1 callout is the kind of detail that saves someone 2 hours of debugging OOM crashes. 262K context with q8_0 KV on a single 3090 is wild, that would've been unthinkable like a year ago. Curious how the MXFP4 removal in the latest UD quant affected quality vs the earlier version.

This on 32GB Mac mini

I would love for you to try this same setup on spot instances.

@grok what does this mean?

@grok AMD 7800X3D, 32GB DDR4 5600 and AMD RX 7800 XT. Any Chance this will run on my machine?

Is this all parameters? No Temperature adjustments, top-k for coding etc? For me Claude Code frequently doesn’t finish.

I fake back some things about it not running well previously (other chat on twitter 😉). Tweaked my llama.cpp memory parameters a bit, got a context going of 32k or even 64k while having some ram leftover for vscode. 40 tokens/sec. Works in copilot. That's on a 5060ti-16gb.

3B active params doing all the heavy lifting shows the real story here. The 80B size is basically marketing. Sparse MoE means you're running a 3B model that learned from a giant one.

with my 3090, i ve only 50 t/ s , any idea why ?

End results

Perfect, how much time did the prompt take compared to running on claude code? and how different are the results?

@grok what could I run on a Mac mini?

-np 1 isn't default, it's -1, which is auto
