Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

A great tool to estimate how much VRAM your LLMs actually need. Alter the hardware config, quantization, etc., it tells you about: - Generation speed (tokens/sec) - Precise memory allocation - System throughput, etc. No more VRAM guessing!

Avi Chawla

58,954 subscribers

22,147 views • 9 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

llmfit. Useful tool that probes hardware and tells you exactly which LLMs will actually run. - handles MoE expert offloading, picks the best quantization for your RAM, estimates tokens/sec before you even pull the weights. Essential for local dev.

llmfit. Useful tool that probes hardware and tells you exactly which LLMs will actually run. - handles MoE expert offloading, picks the best quantization for your RAM, estimates tokens/sec before you even pull the weights. Essential for local dev.

Wildminder

247,800 views • 5 months ago

Major update coming to the points system: we are making it more accessible and fair for hobbyists running nodes. You will now earn a bonus to your reward multiplier for every 4GB quantized block of VRAM you contribute. More VRAM = more rewards 🔥 More details in the days ahead.

Major update coming to the points system: we are making it more accessible and fair for hobbyists running nodes. You will now earn a bonus to your reward multiplier for every 4GB quantized block of VRAM you contribute. More VRAM = more rewards 🔥 More details in the days ahead.

Hyperspace

10,715 views • 1 year ago

🚨Stop gambling on the wrong local LLM. Most people download a model, watch it crawl, then start over. LLM Hardware Scanner is a CLI tool that scans your actual specs and ranks hundreds of LLMs before you waste the bandwidth: ⤷ Reads your RAM, CPU, and GPU to score real compatibility ⤷ Accounts for MoE activation not just bloated total parameter counts ⤷ Auto-selects the optimal quantization for your exact setup ⤷ Ranks models across quality, speed, context length, and true memory fit ⤷ Works natively with Ollama, llama.cpp, and MLX ⤷ Shows a composite score plus a speed estimate before you download a single byte Not a benchmark site. Not a spec sheet. A hardware-aware ranker that tells you the truth about what will actually run on your machine. One scan. Instant ranking. Zero wasted downloads.

🚨Stop gambling on the wrong local LLM. Most people download a model, watch it crawl, then start over. LLM Hardware Scanner is a CLI tool that scans your actual specs and ranks hundreds of LLMs before you waste the bandwidth: ⤷ Reads your RAM, CPU, and GPU to score real compatibility ⤷ Accounts for MoE activation not just bloated total parameter counts ⤷ Auto-selects the optimal quantization for your exact setup ⤷ Ranks models across quality, speed, context length, and true memory fit ⤷ Works natively with Ollama, llama.cpp, and MLX ⤷ Shows a composite score plus a speed estimate before you download a single byte Not a benchmark site. Not a spec sheet. A hardware-aware ranker that tells you the truth about what will actually run on your machine. One scan. Instant ranking. Zero wasted downloads.

Charlie Hills

37,499 views • 4 months ago

Anyone with 8GB or 12GB VRAM setups needs to understand that "-ncmoe" is the key flag to boost performance on llama.cpp Here are my results for Qwen3.6 35B A3B, with 64k q8_0 context on a 8GB RTX 3070Ti: ⚪️ no flag → 8.7 tok/s RAM: 13.6GB & VRAM: 7.8GB 🔴 -ncmoe 35 → 27.5 tok/s RAM: 12.1GB & VRAM: 4.3GB 🟢 -ncmoe 30 → 32.5 tok/s RAM: 12GB & VRAM: 5.6GB 🔵 -ncmoe 25 → 40.9 tok/s RAM: 12GB & VRAM: 6.9GB Please note the ram and vram usage you see are total usage of a windows pc, with the model running. My friend's setup: 8GB VRAM and 16GB RAM. You can boost performance by switching to Linux, just something to keep in mind. Basically, this flag keeps the MoE experts in the first X layers on your CPU + RAM, instead of eating all your VRAM straight away. This is a smart hybrid offload way that lets you run bigger models without OOM while keeping the rest on your GPU for speed. As we can see on the data, there's a sweet spot. When we lower it from 35 to 25, speed bumps +50% because there are more layers on your GPU (look at the VRAM usage). The key here is to play around with the number and fit as much as possible on your VRAM, goal is to have 1GB/800MB headroom to avoid stress. ↓ server flags below

Anyone with 8GB or 12GB VRAM setups needs to understand that "-ncmoe" is the key flag to boost performance on llama.cpp Here are my results for Qwen3.6 35B A3B, with 64k q8_0 context on a 8GB RTX 3070Ti: ⚪️ no flag → 8.7 tok/s RAM: 13.6GB & VRAM: 7.8GB 🔴 -ncmoe 35 → 27.5 tok/s RAM: 12.1GB & VRAM: 4.3GB 🟢 -ncmoe 30 → 32.5 tok/s RAM: 12GB & VRAM: 5.6GB 🔵 -ncmoe 25 → 40.9 tok/s RAM: 12GB & VRAM: 6.9GB Please note the ram and vram usage you see are total usage of a windows pc, with the model running. My friend's setup: 8GB VRAM and 16GB RAM. You can boost performance by switching to Linux, just something to keep in mind. Basically, this flag keeps the MoE experts in the first X layers on your CPU + RAM, instead of eating all your VRAM straight away. This is a smart hybrid offload way that lets you run bigger models without OOM while keeping the rest on your GPU for speed. As we can see on the data, there's a sweet spot. When we lower it from 35 to 25, speed bumps +50% because there are more layers on your GPU (look at the VRAM usage). The key here is to play around with the number and fit as much as possible on your VRAM, goal is to have 1GB/800MB headroom to avoid stress. ↓ server flags below

left curve dev

166,063 views • 2 months ago

$I nearly 2x'd the speed while only using +1GB VRAM with the new MTP update in llama.cpp 🤯 You need to add these flags to start using it: --spec-type draft-mtp \ --spec-draft-p-min 0.75 \ --spec-draft-n-max 2 My results with Qwen3.6 27B on a single RTX 5080 ↓ ⚪️ no flag (without mtp) → 54.3 tok/s with 13.26GB VRAM 🔵 --spec-draft-n-max 2 → 90.7 tok/s with 14.29GB VRAM 🔴 --spec-draft-n-max 2 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.30GB VRAM 🟢 --spec-draft-n-max 6 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.87GB VRAM Increasing to 6 draft tokens didn't help my setup for some reason. I made sure to test with a low context length to have enough headroom and eliminate risk of vram stress. From my understanding: 1) The speed gains are very task-dependent. You need to test across a wide range of tasks to get a realistic idea of the benefits 2) We’re already running heavily quantized GGUF models (Q3, Q4, Q6, etc.), so we already benefit from strong speed/performance thanks to the reduced size. That’s why some people are seeing little to no improvement compared to MLX or other quantized versions The progress over the past few days has been insane to say the least. However, MTP now consumes significantly more VRAM. Personally 16GB just isn't enough to use MTP and run it with a good context size. Time to upgrade lads, 24GB+ users are eating GOOD today 🔥 Full setup below ↓$

I nearly 2x'd the speed while only using +1GB VRAM with the new MTP update in llama.cpp 🤯 You need to add these flags to start using it: --spec-type draft-mtp \ --spec-draft-p-min 0.75 \ --spec-draft-n-max 2 My results with Qwen3.6 27B on a single RTX 5080 ↓ ⚪️ no flag (without mtp) → 54.3 tok/s with 13.26GB VRAM 🔵 --spec-draft-n-max 2 → 90.7 tok/s with 14.29GB VRAM 🔴 --spec-draft-n-max 2 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.30GB VRAM 🟢 --spec-draft-n-max 6 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.87GB VRAM Increasing to 6 draft tokens didn't help my setup for some reason. I made sure to test with a low context length to have enough headroom and eliminate risk of vram stress. From my understanding: 1) The speed gains are very task-dependent. You need to test across a wide range of tasks to get a realistic idea of the benefits 2) We’re already running heavily quantized GGUF models (Q3, Q4, Q6, etc.), so we already benefit from strong speed/performance thanks to the reduced size. That’s why some people are seeing little to no improvement compared to MLX or other quantized versions The progress over the past few days has been insane to say the least. However, MTP now consumes significantly more VRAM. Personally 16GB just isn't enough to use MTP and run it with a good context size. Time to upgrade lads, 24GB+ users are eating GOOD today 🔥 Full setup below ↓

left curve dev

29,946 views • 2 months ago

** 3D Engine Update 6 ** New Buffer System. 15-20% Faster !! Using Planes A and B interleaved horizontally we can get the best of all worlds with a near continuous frame buffer for fast Triangle & Quad writes and still use DMA for fast sending of the RAM frame buffer to VRAM. Normally you get one benefit without the other leading to tradeoffs. There is a caveat ofcourse, I lose a spare plane , however thats not a large concern for what im trying to achieve, i still have the full sprite load if i need to simulate ground etc - we can also multiplex sprites if needed. So each plane is full of half fill tiles ( 4 pixels set in each tile row ) , which are then slightly offset to make the full image. Also changed to H40 mode for faster DMA albiet centered view , plenty of Hud options for side panels etc. What is quite cool is the layout in VRAM is very similar to what you see on screen just with some striping . VRAM layout to the right. Theres no more VRAM used verse a normal double buffer , the trick is here to send the buffer in one frame always as theres no room for a double buffer in vram. I have to implement partial ofscreen drawing yet hence the snapping from front to back you see still . #SGDK #SegaGenesis #SegaMegadrive

3D Engine Update 6 New Buffer System. 15-20% Faster !! Using Planes A and B interleaved horizontally we can get the best of all worlds with a near continuous frame buffer for fast Triangle & Quad writes and still use DMA for fast sending of the RAM frame buffer to VRAM. Normally you get one benefit without the other leading to tradeoffs. There is a caveat ofcourse, I lose a spare plane , however thats not a large concern for what im trying to achieve, i still have the full sprite load if i need to simulate ground etc - we can also multiplex sprites if needed. So each plane is full of half fill tiles ( 4 pixels set in each tile row ) , which are then slightly offset to make the full image. Also changed to H40 mode for faster DMA albiet centered view , plenty of Hud options for side panels etc. What is quite cool is the layout in VRAM is very similar to what you see on screen just with some striping . VRAM layout to the right. Theres no more VRAM used verse a normal double buffer , the trick is here to send the buffer in one frame always as theres no room for a double buffer in vram. I have to implement partial ofscreen drawing yet hence the snapping from front to back you see still . #SGDK #SegaGenesis #SegaMegadrive

Shannon Birt

12,052 views • 3 months ago

We've partnered to bring more Gemma 3 quantized models to you! 🚀 We worked with Georgi Gerganov llama.cpp, LM Studio, MLX, ollama to make sure you can run it using your favorite tool! Gemma models optimized with QAT, reduce memory requirements while keeping quality! All models checkpoints are available on Hugging Face and Kaggle. 🤗 What does this mean? - Gemma 3 27B (int4): Fits on NVIDIA RTX 3090 (24GB VRAM) or similar. - Gemma 3 12B (int4): Only needs a NVIDIA RTX 4060 (8GB VRAM) or similar. - Gemma 3 4B, 1B (int4): Run anything with more than 2.5GB Memory. Want to see it in action? Video below shows how easy it is to get started using LMStudio:

We've partnered to bring more Gemma 3 quantized models to you! 🚀 We worked with Georgi Gerganov llama.cpp, LM Studio, MLX, ollama to make sure you can run it using your favorite tool! Gemma models optimized with QAT, reduce memory requirements while keeping quality! All models checkpoints are available on Hugging Face and Kaggle. 🤗 What does this mean? - Gemma 3 27B (int4): Fits on NVIDIA RTX 3090 (24GB VRAM) or similar. - Gemma 3 12B (int4): Only needs a NVIDIA RTX 4060 (8GB VRAM) or similar. - Gemma 3 4B, 1B (int4): Run anything with more than 2.5GB Memory. Want to see it in action? Video below shows how easy it is to get started using LMStudio:

Philipp Schmid

15,996 views • 1 year ago

OpenClaw's memory out of the box sucks With this simple Obsidian system, it's actually awesome In this video I go over using Obsidian as a memory layer for your AI agent, making it SO much more smarter than it is now If you use OpenClaw or Hermes, you need to watch this:

OpenClaw's memory out of the box sucks With this simple Obsidian system, it's actually awesome In this video I go over using Obsidian as a memory layer for your AI agent, making it SO much more smarter than it is now If you use OpenClaw or Hermes, you need to watch this:

Alex Finn

89,606 views • 3 months ago

a new 8GB VRAM GPU dense Local LLM leader was born yesterday runs on: RTX 4060 / RTX 3070 / RTX 2080. any 8GB card Qwen 3.5 9B (dense) was the go to for 6-8GB VRAM builds. Gemma 4 12B QAT (dense) just changed that. same llama.cpp + cuda 13.2. i7 12700H. 16GB RAM. same -ngl 99 flags. same 48k context. unsloth gemma-4-12b-it-Q4_K_M.gguf → 15 tok/sec @ 48k ctx unsloth gemma-4-12B-it-qat-UD-Q4_K_XL.gguf → 32 tok/sec @ 48k ctx → 26 tok/sec @ 64k ctx 64k context is a big deal. Hermes 3 agent requires 64k minimum to run. you're now getting full hermes compatible context on a budget consumer GPU at 26 tok/sec locally. 2.1x faster on identical hardware. and here's the part that breaks your brain: the QAT-UD-Q4_K_XL is actually SMALLER than the Q4_K_M "XL" why? QAT = Quantization Aware Training Google didn't train the model first and compress it later they trained it to be quantized from day one the weights already know how to survive low precision that's why you get more quality per byte llamacpp flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 48000 -v fits in 8GB VRAM clean. no API. no cloud. no subscription. and this isn't even the MTP variant yet Gemma-4-E2B QAT runs on 3GB RAM, E4B on 5GB, 12B on 7GB, 26-A4B on 15GB and 31B on 18GB. I have benchmarked the 26b and 31b qat as well on a single RTX 4090, checkout the comments for details. If you have a 6GB or 8GB VRAM GPU, post your numbers. more benchmarks and configs coming soon

a new 8GB VRAM GPU dense Local LLM leader was born yesterday runs on: RTX 4060 / RTX 3070 / RTX 2080. any 8GB card Qwen 3.5 9B (dense) was the go to for 6-8GB VRAM builds. Gemma 4 12B QAT (dense) just changed that. same llama.cpp + cuda 13.2. i7 12700H. 16GB RAM. same -ngl 99 flags. same 48k context. unsloth gemma-4-12b-it-Q4_K_M.gguf → 15 tok/sec @ 48k ctx unsloth gemma-4-12B-it-qat-UD-Q4_K_XL.gguf → 32 tok/sec @ 48k ctx → 26 tok/sec @ 64k ctx 64k context is a big deal. Hermes 3 agent requires 64k minimum to run. you're now getting full hermes compatible context on a budget consumer GPU at 26 tok/sec locally. 2.1x faster on identical hardware. and here's the part that breaks your brain: the QAT-UD-Q4_K_XL is actually SMALLER than the Q4_K_M "XL" why? QAT = Quantization Aware Training Google didn't train the model first and compress it later they trained it to be quantized from day one the weights already know how to survive low precision that's why you get more quality per byte llamacpp flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 48000 -v fits in 8GB VRAM clean. no API. no cloud. no subscription. and this isn't even the MTP variant yet Gemma-4-E2B QAT runs on 3GB RAM, E4B on 5GB, 12B on 7GB, 26-A4B on 15GB and 31B on 18GB. I have benchmarked the 26b and 31b qat as well on a single RTX 4090, checkout the comments for details. If you have a 6GB or 8GB VRAM GPU, post your numbers. more benchmarks and configs coming soon

Alok

259,993 views • 1 month ago

So now we've an alternative to OpenClaw for enterprise with zero setup and a 70% lower token usage You can connect PokeeClaw to all your apps for: - Data analysis - Email management - PPT/XLSX generation - And much more And everything else like memory, 24/7 running, etc.

So now we've an alternative to OpenClaw for enterprise with zero setup and a 70% lower token usage You can connect PokeeClaw to all your apps for: - Data analysis - Email management - PPT/XLSX generation - And much more And everything else like memory, 24/7 running, etc.

Paul Couvert

42,977 views • 3 months ago

384GB of VRAM. On your desk. Not in a datacenter. Autonomous, the company behind SmartDesk, just open sourced complete build guides for a personal AI computer that runs open models on hardware no one can switch off. It's called Autonomous Computer. Three configurations, one philosophy: a model you rent can be cut off overnight. A model running in your own house can't. → Home build: 2x RTX 5090 workstation → Team build: 4x RTX PRO 6000 Blackwell, 384GB VRAM → On-prem business build: 8x RTX 4090/5090, up to 256GB VRAM Every config ships with the full bill of materials, 3D printable and CNC housing files, wiring diagrams, BIOS tuning, and assembly photos. Software side covers OS setup, NVIDIA drivers, and serving open models through Ollama, vLLM, and llama.cpp. Fork it. Change it. Build it. Sell it. MIT License. 100% Opensource. Repo:

384GB of VRAM. On your desk. Not in a datacenter. Autonomous, the company behind SmartDesk, just open sourced complete build guides for a personal AI computer that runs open models on hardware no one can switch off. It's called Autonomous Computer. Three configurations, one philosophy: a model you rent can be cut off overnight. A model running in your own house can't. → Home build: 2x RTX 5090 workstation → Team build: 4x RTX PRO 6000 Blackwell, 384GB VRAM → On-prem business build: 8x RTX 4090/5090, up to 256GB VRAM Every config ships with the full bill of materials, 3D printable and CNC housing files, wiring diagrams, BIOS tuning, and assembly photos. Software side covers OS setup, NVIDIA drivers, and serving open models through Ollama, vLLM, and llama.cpp. Fork it. Change it. Build it. Sell it. MIT License. 100% Opensource. Repo:

Harman

47,636 views • 21 days ago

🚨 The OriginTrail Bounty is UNDERWAY! 🔥 Round 1 is OPEN NOW — 50,000 TRAC Tokens up for grabs! OriginTrail is paying builders 1,000 to 10,000 TRAC Tokens to connect AI agents (ChatGPT, Claude, OpenClaw, Hermes, etc.) to the new Working Memory and Shared Memory layers in DKG v10. Just dropped a clear breakdown video explaining: ✅ - How the 3 memory layers work ✅ - What makes a Flagship integration (10k TRAC) ✅ - Real examples you can actually build ✅ - How to submit your project If you're a developer or agent builder → this is your moment. Watch the full explainer 👇

🚨 The OriginTrail Bounty is UNDERWAY! 🔥 Round 1 is OPEN NOW — 50,000 TRAC Tokens up for grabs! OriginTrail is paying builders 1,000 to 10,000 TRAC Tokens to connect AI agents (ChatGPT, Claude, OpenClaw, Hermes, etc.) to the new Working Memory and Shared Memory layers in DKG v10. Just dropped a clear breakdown video explaining: ✅ - How the 3 memory layers work ✅ - What makes a Flagship integration (10k TRAC) ✅ - Real examples you can actually build ✅ - How to submit your project If you're a developer or agent builder → this is your moment. Watch the full explainer 👇

dkg://TriniZone

39,451 views • 2 months ago

Sorry, I’m kind of tired of staking, restaking, L2, bridges, etc. I got a boner for creativity, so I went to NFT. I want to tell you about the Persona collection ( and to be more precise, about who is behind it.

Sensitive content

Sorry, I’m kind of tired of staking, restaking, L2, bridges, etc. I got a boner for creativity, so I went to NFT. I want to tell you about the Persona collection ( and to be more precise, about who is behind it.

Sufidyanov.eth

27,249 views • 2 years ago

Holy shit... One month before Andrej Karpathy dropped the “LLM Wiki” idea… someone had already built it. Not theory. Not hype. A working system. LLMs today restart from zero every time. No memory. No evolution. No learning. That’s broken. Most people use CLAUDE.md / AGENTS.md → Manual updates → Easy to forget → Knowledge gets lost That’s not intelligence. That’s a notepad. So I built dynamic memory for LLMs: • Learns automatically • Stores without manual effort • Weakens irrelevant data over time • Strengthens what actually matters Before the “LLM Wiki” idea went viral, this already had: • Brain-like decay/boost memory • Smart deduplication (no noise) • Experience tracking per tech • Cross-referenced knowledge graph • Loss-proof backups • Auto-grouped tool sessions Then I added: • memory_audit (clean + optimize memory) • Memory injection directly into tools Now this isn’t just a concept. It’s a live system that turns LLMs from stateless tools → evolving intelligence. Using it daily: → ~5K tokens → Zero manual work → Memory that actually compounds Link in comments. RT & Bookmark — this will be standard soon.

Holy shit... One month before Andrej Karpathy dropped the “LLM Wiki” idea… someone had already built it. Not theory. Not hype. A working system. LLMs today restart from zero every time. No memory. No evolution. No learning. That’s broken. Most people use CLAUDE.md / AGENTS.md → Manual updates → Easy to forget → Knowledge gets lost That’s not intelligence. That’s a notepad. So I built dynamic memory for LLMs: • Learns automatically • Stores without manual effort • Weakens irrelevant data over time • Strengthens what actually matters Before the “LLM Wiki” idea went viral, this already had: • Brain-like decay/boost memory • Smart deduplication (no noise) • Experience tracking per tech • Cross-referenced knowledge graph • Loss-proof backups • Auto-grouped tool sessions Then I added: • memory_audit (clean + optimize memory) • Memory injection directly into tools Now this isn’t just a concept. It’s a live system that turns LLMs from stateless tools → evolving intelligence. Using it daily: → ~5K tokens → Zero manual work → Memory that actually compounds Link in comments. RT & Bookmark — this will be standard soon.

Nainsi Dwivedi

15,726 views • 3 months ago

A single RTX 4090 (24 GB VRAM) can run the updated gemma 4 31B (dense) model with a 190,000 context window at 33 tokens/second. The VRAM barrier is dying. Google quietly updated Gemma 4, and Unsloth immediately compiled the new quants. I built llama.cpp from source on Ubuntu 22 to benchmark it. Google's stealth update 2 days ago enabled uniform Flash Attention 4 on Hopper to boost prefill and patched the chat template to improve tool calling. The agentic reasoning gains on the benchmark charts are massive: TB2 (Agents): +4.5% (to 25.8%) Tau2 (Telecom): +10.1% (to 62.7%) Running on Ubuntu 22, CUDA 13.0 with a single NVIDIA GeForce RTX 4090. Here is the exact step by step benchmarking process with a massive 28k tokens prompt and the commands I used to squeeze out maximum context without killing my throughput: # 1. The Baseline (Unquantized KV Cache) I started with full GPU offload (-ngl 99) and pushed the context to 40k. llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -ngl 99 -c 40000 -fa on --port 8080 -v VRAM: 23.8 GB (maxed out on card) Throughput: Prefill: 2198.81 t/s | Decode: 35.77 t/s (with 28k tokens prompt) # 2. The CPU Split Trap I tried stretching to 80k context by offloading layers to the CPU (-ngl 52). llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 80000 -ngl 52 -fa on --port 8080 -v Throughput: Prefill: 1212.73 t/s | Decode: 5 t/s (with 28k tokens prompt) # 3. The KV Quantization Breakthrough Instead of spilling layers to the CPU, I kept the model fully on card (-ngl 99) but enabled 8-bit KV cache quantization to free up VRAM. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 100000 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 --port 8080 -v VRAM: 23.9 GB Throughput: Prefill: 2139.68 t/s | Decode: 32 t/s (with 28k tokens prompt) Result: 100k tokens of context on a single GPU with practically zero speed loss (and minimal intelligence loss). # 4. The Limit Test (Q4 KV Cache) To find the absolute breaking point, I dropped the KV cache to 4 bit (q4_0) and set -c 190000. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 190000 --cache-type-k q4_0 --cache-type-v q4_0 -ngl 99 --port 8080 -v VRAM: 23.8 GB Throughput: Prefill: 2206.66 t/s | Decode: 33 t/s (with 28k tokens prompt) (Note: Pushing it to 220k required dropping to -ngl 58 again, which immediately penalized decode down to 17 t/s). # The Tradeoff: For Max Reasoning: Keep your KV cache unquantized (f16). You get pristine reasoning but hit a strict 40k context ceiling. For Massive Document Retrieval: If you need to feed the model giant codebases, use --cache-type-k q4_0. Getting 190k context at 33 tokens/second on a consumer desktop with a 31b dense model is a cheat code. If you’re rocking a single 3090 or 4090 and slept on Gemma 4 earlier, this update is your cue to dust off the terminal. Hugging Face links to the Unsloth QAT quants are in the replies below.

A single RTX 4090 (24 GB VRAM) can run the updated gemma 4 31B (dense) model with a 190,000 context window at 33 tokens/second. The VRAM barrier is dying. Google quietly updated Gemma 4, and Unsloth immediately compiled the new quants. I built llama.cpp from source on Ubuntu 22 to benchmark it. Google's stealth update 2 days ago enabled uniform Flash Attention 4 on Hopper to boost prefill and patched the chat template to improve tool calling. The agentic reasoning gains on the benchmark charts are massive: TB2 (Agents): +4.5% (to 25.8%) Tau2 (Telecom): +10.1% (to 62.7%) Running on Ubuntu 22, CUDA 13.0 with a single NVIDIA GeForce RTX 4090. Here is the exact step by step benchmarking process with a massive 28k tokens prompt and the commands I used to squeeze out maximum context without killing my throughput: # 1. The Baseline (Unquantized KV Cache) I started with full GPU offload (-ngl 99) and pushed the context to 40k. llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -ngl 99 -c 40000 -fa on --port 8080 -v VRAM: 23.8 GB (maxed out on card) Throughput: Prefill: 2198.81 t/s | Decode: 35.77 t/s (with 28k tokens prompt) # 2. The CPU Split Trap I tried stretching to 80k context by offloading layers to the CPU (-ngl 52). llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 80000 -ngl 52 -fa on --port 8080 -v Throughput: Prefill: 1212.73 t/s | Decode: 5 t/s (with 28k tokens prompt) # 3. The KV Quantization Breakthrough Instead of spilling layers to the CPU, I kept the model fully on card (-ngl 99) but enabled 8-bit KV cache quantization to free up VRAM. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 100000 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 --port 8080 -v VRAM: 23.9 GB Throughput: Prefill: 2139.68 t/s | Decode: 32 t/s (with 28k tokens prompt) Result: 100k tokens of context on a single GPU with practically zero speed loss (and minimal intelligence loss). # 4. The Limit Test (Q4 KV Cache) To find the absolute breaking point, I dropped the KV cache to 4 bit (q4_0) and set -c 190000. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 190000 --cache-type-k q4_0 --cache-type-v q4_0 -ngl 99 --port 8080 -v VRAM: 23.8 GB Throughput: Prefill: 2206.66 t/s | Decode: 33 t/s (with 28k tokens prompt) (Note: Pushing it to 220k required dropping to -ngl 58 again, which immediately penalized decode down to 17 t/s). # The Tradeoff: For Max Reasoning: Keep your KV cache unquantized (f16). You get pristine reasoning but hit a strict 40k context ceiling. For Massive Document Retrieval: If you need to feed the model giant codebases, use --cache-type-k q4_0. Getting 190k context at 33 tokens/second on a consumer desktop with a 31b dense model is a cheat code. If you’re rocking a single 3090 or 4090 and slept on Gemma 4 earlier, this update is your cue to dust off the terminal. Hugging Face links to the Unsloth QAT quants are in the replies below.

Alok

75,482 views • 12 days ago

People contact me all the time about Gomorrah and ask me why it's so great etc. If you need a reason to watch it, no better time than now... here's why you should watch it and why it makes other crime shows just average.

People contact me all the time about Gomorrah and ask me why it's so great etc. If you need a reason to watch it, no better time than now... here's why you should watch it and why it makes other crime shows just average.

Jeff Nadu

26,680 views • 1 month ago

CONFIG: Dylan Field (aka Dylan Field), Founder & CEO of Figma says: "You have to actually think." "There's a world where a lot of people are letting AI think for them... & they're tricking themselves into believing they're thinking, themselves." "Especially when the AI is sycophantic & it's telling you you're so great, & you're such a genius, but then giving you what it wants to do.. that's no longer your idea. That's no longer your intention." "Use AI as a tool to learn, a tool to understand the world better, a tool for leverage to get the non-critical stuff done faster." Live from Config 2026, San Francisco

CONFIG: Dylan Field (aka Dylan Field), Founder & CEO of Figma says: "You have to actually think." "There's a world where a lot of people are letting AI think for them... & they're tricking themselves into believing they're thinking, themselves." "Especially when the AI is sycophantic & it's telling you you're so great, & you're such a genius, but then giving you what it wants to do.. that's no longer your idea. That's no longer your intention." "Use AI as a tool to learn, a tool to understand the world better, a tool for leverage to get the non-critical stuff done faster." Live from Config 2026, San Francisco

Molly O’Shea

106,897 views • 1 month ago

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization, it requires ~700GB of memory to run the model. EXO puts half of the layers on each device, combining their memory. EXO uses MLX distributed with TB5 interconnect, optimized for Apple Silicon. If we need higher throughput, adding two more devices lets us serve more users at once. EXO Labs handles all of this seamlessly - adding more devices to the cluster for linear scaling as we need it. The new EXO 1.0 will be open-source soonTM

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization, it requires ~700GB of memory to run the model. EXO puts half of the layers on each device, combining their memory. EXO uses MLX distributed with TB5 interconnect, optimized for Apple Silicon. If we need higher throughput, adding two more devices lets us serve more users at once. EXO Labs handles all of this seamlessly - adding more devices to the cluster for linear scaling as we need it. The new EXO 1.0 will be open-source soonTM

Matt Beton

158,485 views • 11 months ago

Do you want your kids to grow up as little marxists? No? Me neither. One of my fav things about The Tuttle Twins (we read them to our kids) is how they actually teach important lessons about history, economics, rights, liberty, etc in a way that children will actually understand.

Do you want your kids to grow up as little marxists? No? Me neither. One of my fav things about The Tuttle Twins (we read them to our kids) is how they actually teach important lessons about history, economics, rights, liberty, etc in a way that children will actually understand.

Declaration of Memes

110,522 views • 11 months ago

LLMs can take gigabytes of memory to store, which limits what can be run on consumer hardware. But quantization can dramatically compress models, making a wider selection of models available to developers. You can often reduce model size by 4x or more while maintaining reasonable performance. In our new short course Quantization Fundamentals taught by Hugging Face's @younesbelkada and Marc Sun, you'll: - Learn how to quantize nearly any open source model - Use int8 and bfloat16 (Brain float 16) data types to load and run LLMs using PyTorch and the Hugging Face Transformers library - Dive into the technical details of linear quantization to map 32-bit floats to 8-bit integers As models get bigger and bigger, quantization becomes more important for making models practical and accessible. Please check out the course here:

LLMs can take gigabytes of memory to store, which limits what can be run on consumer hardware. But quantization can dramatically compress models, making a wider selection of models available to developers. You can often reduce model size by 4x or more while maintaining reasonable performance. In our new short course Quantization Fundamentals taught by Hugging Face's @younesbelkada and Marc Sun, you'll: - Learn how to quantize nearly any open source model - Use int8 and bfloat16 (Brain float 16) data types to load and run LLMs using PyTorch and the Hugging Face Transformers library - Dive into the technical details of linear quantization to map 32-bit floats to 8-bit integers As models get bigger and bigger, quantization becomes more important for making models practical and accessible. Please check out the course here:

Andrew Ng

288,284 views • 2 years ago