Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Digital Foundry discuss #Steam Machine specs, with concerns on performance and 8GB of VRAM. "The GPU is a little bit anemic perhaps, esp relative to current gen consoles, which it's trailing slightly" "8GB of VRAM poses more possibly fundamental limits"

NIB

6,631 subscribers

30,276 views • 9 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Anyone with 8GB or 12GB VRAM setups needs to understand that "-ncmoe" is the key flag to boost performance on llama.cpp Here are my results for Qwen3.6 35B A3B, with 64k q8_0 context on a 8GB RTX 3070Ti: ⚪️ no flag → 8.7 tok/s RAM: 13.6GB & VRAM: 7.8GB 🔴 -ncmoe 35 → 27.5 tok/s RAM: 12.1GB & VRAM: 4.3GB 🟢 -ncmoe 30 → 32.5 tok/s RAM: 12GB & VRAM: 5.6GB 🔵 -ncmoe 25 → 40.9 tok/s RAM: 12GB & VRAM: 6.9GB Please note the ram and vram usage you see are total usage of a windows pc, with the model running. My friend's setup: 8GB VRAM and 16GB RAM. You can boost performance by switching to Linux, just something to keep in mind. Basically, this flag keeps the MoE experts in the first X layers on your CPU + RAM, instead of eating all your VRAM straight away. This is a smart hybrid offload way that lets you run bigger models without OOM while keeping the rest on your GPU for speed. As we can see on the data, there's a sweet spot. When we lower it from 35 to 25, speed bumps +50% because there are more layers on your GPU (look at the VRAM usage). The key here is to play around with the number and fit as much as possible on your VRAM, goal is to have 1GB/800MB headroom to avoid stress. ↓ server flags below

Anyone with 8GB or 12GB VRAM setups needs to understand that "-ncmoe" is the key flag to boost performance on llama.cpp Here are my results for Qwen3.6 35B A3B, with 64k q8_0 context on a 8GB RTX 3070Ti: ⚪️ no flag → 8.7 tok/s RAM: 13.6GB & VRAM: 7.8GB 🔴 -ncmoe 35 → 27.5 tok/s RAM: 12.1GB & VRAM: 4.3GB 🟢 -ncmoe 30 → 32.5 tok/s RAM: 12GB & VRAM: 5.6GB 🔵 -ncmoe 25 → 40.9 tok/s RAM: 12GB & VRAM: 6.9GB Please note the ram and vram usage you see are total usage of a windows pc, with the model running. My friend's setup: 8GB VRAM and 16GB RAM. You can boost performance by switching to Linux, just something to keep in mind. Basically, this flag keeps the MoE experts in the first X layers on your CPU + RAM, instead of eating all your VRAM straight away. This is a smart hybrid offload way that lets you run bigger models without OOM while keeping the rest on your GPU for speed. As we can see on the data, there's a sweet spot. When we lower it from 35 to 25, speed bumps +50% because there are more layers on your GPU (look at the VRAM usage). The key here is to play around with the number and fit as much as possible on your VRAM, goal is to have 1GB/800MB headroom to avoid stress. ↓ server flags below

left curve dev

166,368 views • 3 months ago

a new 8GB VRAM GPU dense Local LLM leader was born yesterday runs on: RTX 4060 / RTX 3070 / RTX 2080. any 8GB card Qwen 3.5 9B (dense) was the go to for 6-8GB VRAM builds. Gemma 4 12B QAT (dense) just changed that. same llama.cpp + cuda 13.2. i7 12700H. 16GB RAM. same -ngl 99 flags. same 48k context. unsloth gemma-4-12b-it-Q4_K_M.gguf → 15 tok/sec @ 48k ctx unsloth gemma-4-12B-it-qat-UD-Q4_K_XL.gguf → 32 tok/sec @ 48k ctx → 26 tok/sec @ 64k ctx 64k context is a big deal. Hermes 3 agent requires 64k minimum to run. you're now getting full hermes compatible context on a budget consumer GPU at 26 tok/sec locally. 2.1x faster on identical hardware. and here's the part that breaks your brain: the QAT-UD-Q4_K_XL is actually SMALLER than the Q4_K_M "XL" why? QAT = Quantization Aware Training Google didn't train the model first and compress it later they trained it to be quantized from day one the weights already know how to survive low precision that's why you get more quality per byte llamacpp flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 48000 -v fits in 8GB VRAM clean. no API. no cloud. no subscription. and this isn't even the MTP variant yet Gemma-4-E2B QAT runs on 3GB RAM, E4B on 5GB, 12B on 7GB, 26-A4B on 15GB and 31B on 18GB. I have benchmarked the 26b and 31b qat as well on a single RTX 4090, checkout the comments for details. If you have a 6GB or 8GB VRAM GPU, post your numbers. more benchmarks and configs coming soon

a new 8GB VRAM GPU dense Local LLM leader was born yesterday runs on: RTX 4060 / RTX 3070 / RTX 2080. any 8GB card Qwen 3.5 9B (dense) was the go to for 6-8GB VRAM builds. Gemma 4 12B QAT (dense) just changed that. same llama.cpp + cuda 13.2. i7 12700H. 16GB RAM. same -ngl 99 flags. same 48k context. unsloth gemma-4-12b-it-Q4_K_M.gguf → 15 tok/sec @ 48k ctx unsloth gemma-4-12B-it-qat-UD-Q4_K_XL.gguf → 32 tok/sec @ 48k ctx → 26 tok/sec @ 64k ctx 64k context is a big deal. Hermes 3 agent requires 64k minimum to run. you're now getting full hermes compatible context on a budget consumer GPU at 26 tok/sec locally. 2.1x faster on identical hardware. and here's the part that breaks your brain: the QAT-UD-Q4_K_XL is actually SMALLER than the Q4_K_M "XL" why? QAT = Quantization Aware Training Google didn't train the model first and compress it later they trained it to be quantized from day one the weights already know how to survive low precision that's why you get more quality per byte llamacpp flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 48000 -v fits in 8GB VRAM clean. no API. no cloud. no subscription. and this isn't even the MTP variant yet Gemma-4-E2B QAT runs on 3GB RAM, E4B on 5GB, 12B on 7GB, 26-A4B on 15GB and 31B on 18GB. I have benchmarked the 26b and 31b qat as well on a single RTX 4090, checkout the comments for details. If you have a 6GB or 8GB VRAM GPU, post your numbers. more benchmarks and configs coming soon

Alok

259,993 views • 2 months ago

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.

dunik

148,663 views • 26 days ago

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

Alok

60,866 views • 1 month ago

The truth is coming out! Digital Foundry: Cyberpunk on Switch 2 has higher graphical quality than Xbox Series S and uses the same texture quality setting as PS5 which the Series S isn’t capable of! They note it’s due to 9GB of RAM the Switch 2 has available for games! Series S only has 8gb RAM for games!

The truth is coming out! Digital Foundry: Cyberpunk on Switch 2 has higher graphical quality than Xbox Series S and uses the same texture quality setting as PS5 which the Series S isn’t capable of! They note it’s due to 9GB of RAM the Switch 2 has available for games! Series S only has 8gb RAM for games!

OBE1plays YouTube

225,511 views • 1 year ago

🚨Digital Foundry tries to find a GPU on the market to compare with PS5 Pro. They mentions Nvidia 3070 Ti ,which BY ITSELF CAN COST MORE than a PS5 Pro and say even then.... it isn't comparable because PS5 Pro will have MORE memory That is just a GPU - 1 component

🚨Digital Foundry tries to find a GPU on the market to compare with PS5 Pro. They mentions Nvidia 3070 Ti ,which BY ITSELF CAN COST MORE than a PS5 Pro and say even then.... it isn't comparable because PS5 Pro will have MORE memory That is just a GPU - 1 component

THE RED DRAGON

130,530 views • 1 year ago

Wan2.2 is now natively supported in ComfyUI on Day 0! 🔹 A next-gen video model with MoE (Mixture of Experts) architecture with dual noise experts, under Apache 2.0 license! - Cinematic-level Aesthetic Control - Large-scale Complex Motion - Precise Semantic Compliance 📚 Versions available: - Wan2.2-TI2V-5B: FP16 - Wan2.2-I2V-14B: FP16/FP8 - Wan2.2-T2V-14B: FP16/FP8 💻 Down to 8GB VRAM requirement for the 5B version with ComfyUI auto-offloading.

Wan2.2 is now natively supported in ComfyUI on Day 0! 🔹 A next-gen video model with MoE (Mixture of Experts) architecture with dual noise experts, under Apache 2.0 license! - Cinematic-level Aesthetic Control - Large-scale Complex Motion - Precise Semantic Compliance 📚 Versions available: - Wan2.2-TI2V-5B: FP16 - Wan2.2-I2V-14B: FP16/FP8 - Wan2.2-T2V-14B: FP16/FP8 💻 Down to 8GB VRAM requirement for the 5B version with ComfyUI auto-offloading.

ComfyUI

84,512 views • 1 year ago

A weird issue that I'm having with the PC version of 007 First Light is some textures don't seem to load properly and are extremely low resolution. I don't think it could be related to low VRAM since I'm running this game on an RX 9070 XT which has 16 GB of VRAM and can't pinpoint what exactly causes this.

A weird issue that I'm having with the PC version of 007 First Light is some textures don't seem to load properly and are extremely low resolution. I don't think it could be related to low VRAM since I'm running this game on an RX 9070 XT which has 16 GB of VRAM and can't pinpoint what exactly causes this.

NikTek

38,454 views • 2 months ago

"Assassin’s Creed: Odyssey" map of Greece on a single 32,768 x 32,768 texture in the Browser. Virtual Texture tech in WebGL 2.0 The source is a 804 Mb PNG. Without Virtual Texture tech you'd need to download that and wait for a few seconds for your browser to decode it before you could use it. Your browser would then crash when it tries to upload it to the GPU :) If we imagine that the upload to the GPU would succeed - it would take 4Gb of VRAM. Virtual Texture loads immediately, because the image is split into small fixed-sized tiles with full LoD chain (mips) and uses 2048x2048, which is just 16 Mb of VRAM. Article:

"Assassin’s Creed: Odyssey" map of Greece on a single 32,768 x 32,768 texture in the Browser. Virtual Texture tech in WebGL 2.0 The source is a 804 Mb PNG. Without Virtual Texture tech you'd need to download that and wait for a few seconds for your browser to decode it before you could use it. Your browser would then crash when it tries to upload it to the GPU :) If we imagine that the upload to the GPU would succeed - it would take 4Gb of VRAM. Virtual Texture loads immediately, because the image is split into small fixed-sized tiles with full LoD chain (mips) and uses 2048x2048, which is just 16 Mb of VRAM. Article:

Alex Goldring

188,343 views • 6 months ago

The free UE5.5 AI motion capture plugin is compatible with Unity, Blender, and VMC. It supports both real-time motion capture via webcam and video uploads to generate 3D motion data. Powered by a 1-billion-parameter motion model running locally on your computer, it requires 8GB of VRAM for real-time processing. Compatible with Nvidia, AMD, and Intel GPUs, it can leverage NVIDIA CUDA to significantly boost graphics computation efficiency.

The free UE5.5 AI motion capture plugin is compatible with Unity, Blender, and VMC. It supports both real-time motion capture via webcam and video uploads to generate 3D motion data. Powered by a 1-billion-parameter motion model running locally on your computer, it requires 8GB of VRAM for real-time processing. Compatible with Nvidia, AMD, and Intel GPUs, it can leverage NVIDIA CUDA to significantly boost graphics computation efficiency.

CYANPUPPETS

17,842 views • 1 year ago

Take a closer look at the new features coming to #Cyberpunk2077 together with the #PhantomLiberty expansion! Some of these features will be also added to the base game — as part of Update 2.0 which will be available on the current-gen consoles and PC!

Take a closer look at the new features coming to #Cyberpunk2077 together with the #PhantomLiberty expansion! Some of these features will be also added to the base game — as part of Update 2.0 which will be available on the current-gen consoles and PC!

Cyberpunk 2077

1,997,033 views • 3 years ago

If you think GTA VI is going to be difficult to optimize for current gen consoles which all have 16GB of RAM respectively (except Xbox Series S) Then you can't imagine the pain Rockstar Games must have gone through when they had to optimize GTA V for PS3 and Xbox 360. Each one of these consoles had 512MB of total memory, PS3 had 256MB XDR DRAM and 256MB VRAM. For comparison, google chrome with 3 tabs open can take up to 2GB of RAM memory today. Yet, somehow Rockstar Games managed to have this game running on an internal resolution of 720p, targeting 30FPS which is an incredible feat in my opinion. For how massive GTA V was, they truly pushed these consoles to the absolute maximum

If you think GTA VI is going to be difficult to optimize for current gen consoles which all have 16GB of RAM respectively (except Xbox Series S) Then you can't imagine the pain Rockstar Games must have gone through when they had to optimize GTA V for PS3 and Xbox 360. Each one of these consoles had 512MB of total memory, PS3 had 256MB XDR DRAM and 256MB VRAM. For comparison, google chrome with 3 tabs open can take up to 2GB of RAM memory today. Yet, somehow Rockstar Games managed to have this game running on an internal resolution of 720p, targeting 30FPS which is an incredible feat in my opinion. For how massive GTA V was, they truly pushed these consoles to the absolute maximum

NikTek

1,291,265 views • 4 months ago

Take a closer look at the new features coming to #Cyberpunk2077 together with the #Phantom Liberty expansion! Some of these features will be also added to the base game — as part of Update 2.0 which will be available on the current-gen consoles and PC!

Take a closer look at the new features coming to #Cyberpunk2077 together with the #Phantom Liberty expansion! Some of these features will be also added to the base game — as part of Update 2.0 which will be available on the current-gen consoles and PC!

CD PROJEKT RED

611,198 views • 3 years ago

** 3D Engine Update 6 ** New Buffer System. 15-20% Faster !! Using Planes A and B interleaved horizontally we can get the best of all worlds with a near continuous frame buffer for fast Triangle & Quad writes and still use DMA for fast sending of the RAM frame buffer to VRAM. Normally you get one benefit without the other leading to tradeoffs. There is a caveat ofcourse, I lose a spare plane , however thats not a large concern for what im trying to achieve, i still have the full sprite load if i need to simulate ground etc - we can also multiplex sprites if needed. So each plane is full of half fill tiles ( 4 pixels set in each tile row ) , which are then slightly offset to make the full image. Also changed to H40 mode for faster DMA albiet centered view , plenty of Hud options for side panels etc. What is quite cool is the layout in VRAM is very similar to what you see on screen just with some striping . VRAM layout to the right. Theres no more VRAM used verse a normal double buffer , the trick is here to send the buffer in one frame always as theres no room for a double buffer in vram. I have to implement partial ofscreen drawing yet hence the snapping from front to back you see still . #SGDK #SegaGenesis #SegaMegadrive

3D Engine Update 6 New Buffer System. 15-20% Faster !! Using Planes A and B interleaved horizontally we can get the best of all worlds with a near continuous frame buffer for fast Triangle & Quad writes and still use DMA for fast sending of the RAM frame buffer to VRAM. Normally you get one benefit without the other leading to tradeoffs. There is a caveat ofcourse, I lose a spare plane , however thats not a large concern for what im trying to achieve, i still have the full sprite load if i need to simulate ground etc - we can also multiplex sprites if needed. So each plane is full of half fill tiles ( 4 pixels set in each tile row ) , which are then slightly offset to make the full image. Also changed to H40 mode for faster DMA albiet centered view , plenty of Hud options for side panels etc. What is quite cool is the layout in VRAM is very similar to what you see on screen just with some striping . VRAM layout to the right. Theres no more VRAM used verse a normal double buffer , the trick is here to send the buffer in one frame always as theres no room for a double buffer in vram. I have to implement partial ofscreen drawing yet hence the snapping from front to back you see still . #SGDK #SegaGenesis #SegaMegadrive

Shannon Birt

12,052 views • 3 months ago

$I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.$

I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.

Alok

36,617 views • 18 days ago

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own

NO1ennn

40,079 views • 1 month ago

While Siege X (beta) feels pretty stable so far, the impact on performance is huge. On the current Live build I get about ~300-320FPS avg. and on Siege X I get about ~150-165FPS on average. Also my GPU Latency is 2 to 2.5x higher then on Live Servers. (I play on Low - Medium Settings on 1080p | specs in comments) With that said, it's literally only been out for a couple hours yet I am a bit worried. Hoping they optimize it soon because losing 50% of my FPS is rough. Also please let me know what FPS you guys are getting and what your specs are.

While Siege X (beta) feels pretty stable so far, the impact on performance is huge. On the current Live build I get about ~300-320FPS avg. and on Siege X I get about ~150-165FPS on average. Also my GPU Latency is 2 to 2.5x higher then on Live Servers. (I play on Low - Medium Settings on 1080p | specs in comments) With that said, it's literally only been out for a couple hours yet I am a bit worried. Hoping they optimize it soon because losing 50% of my FPS is rough. Also please let me know what FPS you guys are getting and what your specs are.

Marc 'Reaxys' Seidl 🇦🇹

515,481 views • 1 year ago

The pre-release version of Crimson Desert reviewed by Digital Foundry on PS5 Pro had some significant dips in framerate in optimized mode which targets 60FPS. This happened during a combat sequence where you have tons of enemies and NPC around you. As Digital Foundry suggested this is a sign of a CPU-bottleneck from the Zen 2 architecture on the PS5 Pro. When they tested out Quality mode, it had a locked 30FPS with little to no dips.

The pre-release version of Crimson Desert reviewed by Digital Foundry on PS5 Pro had some significant dips in framerate in optimized mode which targets 60FPS. This happened during a combat sequence where you have tons of enemies and NPC around you. As Digital Foundry suggested this is a sign of a CPU-bottleneck from the Zen 2 architecture on the PS5 Pro. When they tested out Quality mode, it had a locked 30FPS with little to no dips.

NikTek

146,202 views • 5 months ago