Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

We've partnered to bring more Gemma 3 quantized models to you! 🚀 We worked with Georgi Gerganov llama.cpp, LM Studio, MLX, ollama to make sure you can run it using your favorite tool! Gemma models optimized with QAT, reduce memory requirements while keeping quality! All models checkpoints are available... show more

Philipp Schmid

41,946 subscribers

15,996 views • 1 year ago •via X (Twitter)

Health & Wellness Education Science & Technology

Anya Rossi• Live Now

Private livecam show

11 Comments

Philipp Schmid1 year ago

Read more about it, explore quantized Gemma and start building with your favorite tool: Blog: Llama cpp: LMStudio: Ollama: MLX: Google AI Edge (For Phones):

UNPLUGGED PERFORMANCE2 years ago

Upgrade your Tesla with UP-03 Forged Wheels from Unplugged Performance! Unmatched strength, lightweight design, and track-proven durability—perfect for Model S, 3, X, and Y. Ready to ship with a lifetime warranty. #Tesla #UP03 #UnpluggedPerformance

Elaina1 year ago

@ggerganov @lmstudio @ollama Community enthusiasts have further reduced the GGUF size of the Gemma-3-27b-it-qat model to just 15.6GB. 24GB VRAM X 16GB VRAM ✓

Ivan Baldo1 year ago

@ggerganov @lmstudio @ollama Any comparison between the regular Q4_K_M ones and these QAT Q4_0?

Christian Stelter1 year ago

May be you should also partner with @NeuralMagic to get handy quants out of their llm-compressor for vLLM junkies like me. ;-)

moskstraumen1 year ago

@ggerganov @lmstudio @ollama @awnihannun Maybe it's time for MLX to have it's own X account, so it can be tagged? 🙂

Idan Buzaglo 🎗️1 year ago

@ggerganov @lmstudio @ollama Do they truly keep quality? Because when I tested them, they didn’t work the same at Hebrew language.

D.1 year ago

@ggerganov @lmstudio @ollama 1) How does your 27B QAT int4 version compare to traditional Q6_K in precision/accuracy relative to the regular model? 2) Will Vision capabilities work on ollama? There have been issues with some Gemma versions there Great work! Now, we need Gemma 3 Thinking 👀

Yousef Amirghofran1 year ago

@ggerganov @lmstudio @ollama Fantastic!

LuisDataGuru1 year ago

@ggerganov @lmstudio @ollama ¡Innovación tangible! Gemma 3 con QAT acerca los LLMs a más desarrolladores. ¿Cómo impactará en chatbots o análisis en-edge? 🚀 La optimización para RTX 4060/3090 es un logro notable.

AI Wave1 year ago

@ggerganov @lmstudio @ollama Big models. Small GPUs. Still fast, still good. This isn’t just about tooling anymore llama.cpp, MLX, and Ollama are laying the groundwork for something bigger. Training or inference, local is back.

Related Videos

a new 8GB VRAM GPU dense Local LLM leader was born yesterday runs on: RTX 4060 / RTX 3070 / RTX 2080. any 8GB card Qwen 3.5 9B (dense) was the go to for 6-8GB VRAM builds. Gemma 4 12B QAT (dense) just changed that. same llama.cpp + cuda 13.2. i7 12700H. 16GB RAM. same -ngl 99 flags. same 48k context. unsloth gemma-4-12b-it-Q4_K_M.gguf → 15 tok/sec @ 48k ctx unsloth gemma-4-12B-it-qat-UD-Q4_K_XL.gguf → 32 tok/sec @ 48k ctx → 26 tok/sec @ 64k ctx 64k context is a big deal. Hermes 3 agent requires 64k minimum to run. you're now getting full hermes compatible context on a budget consumer GPU at 26 tok/sec locally. 2.1x faster on identical hardware. and here's the part that breaks your brain: the QAT-UD-Q4_K_XL is actually SMALLER than the Q4_K_M "XL" why? QAT = Quantization Aware Training Google didn't train the model first and compress it later they trained it to be quantized from day one the weights already know how to survive low precision that's why you get more quality per byte llamacpp flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 48000 -v fits in 8GB VRAM clean. no API. no cloud. no subscription. and this isn't even the MTP variant yet Gemma-4-E2B QAT runs on 3GB RAM, E4B on 5GB, 12B on 7GB, 26-A4B on 15GB and 31B on 18GB. I have benchmarked the 26b and 31b qat as well on a single RTX 4090, checkout the comments for details. If you have a 6GB or 8GB VRAM GPU, post your numbers. more benchmarks and configs coming soon

a new 8GB VRAM GPU dense Local LLM leader was born yesterday runs on: RTX 4060 / RTX 3070 / RTX 2080. any 8GB card Qwen 3.5 9B (dense) was the go to for 6-8GB VRAM builds. Gemma 4 12B QAT (dense) just changed that. same llama.cpp + cuda 13.2. i7 12700H. 16GB RAM. same -ngl 99 flags. same 48k context. unsloth gemma-4-12b-it-Q4_K_M.gguf → 15 tok/sec @ 48k ctx unsloth gemma-4-12B-it-qat-UD-Q4_K_XL.gguf → 32 tok/sec @ 48k ctx → 26 tok/sec @ 64k ctx 64k context is a big deal. Hermes 3 agent requires 64k minimum to run. you're now getting full hermes compatible context on a budget consumer GPU at 26 tok/sec locally. 2.1x faster on identical hardware. and here's the part that breaks your brain: the QAT-UD-Q4_K_XL is actually SMALLER than the Q4_K_M "XL" why? QAT = Quantization Aware Training Google didn't train the model first and compress it later they trained it to be quantized from day one the weights already know how to survive low precision that's why you get more quality per byte llamacpp flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 48000 -v fits in 8GB VRAM clean. no API. no cloud. no subscription. and this isn't even the MTP variant yet Gemma-4-E2B QAT runs on 3GB RAM, E4B on 5GB, 12B on 7GB, 26-A4B on 15GB and 31B on 18GB. I have benchmarked the 26b and 31b qat as well on a single RTX 4090, checkout the comments for details. If you have a 6GB or 8GB VRAM GPU, post your numbers. more benchmarks and configs coming soon

Alok

259,993 views • 2 months ago

Laguna S 2.1 is built to run locally. We heard the ask for a serious open model in the 80–120B range. Laguna S 2.1 is our answer, and we want it to redefine what this size class can do. Weights ship in BF16, FP8, INT4 and NVFP4, with official GGUF and MLX conversions, plus official DFlash draft models for faster inference. Day-one support across llama.cpp, ollama, Valerie Lee Make Up, SGLang, NVIDIA TRT-LLM, Hugging Face transformers and ZML. Watch it run on an NVIDIA AI DGX Spark.

Laguna S 2.1 is built to run locally. We heard the ask for a serious open model in the 80–120B range. Laguna S 2.1 is our answer, and we want it to redefine what this size class can do. Weights ship in BF16, FP8, INT4 and NVFP4, with official GGUF and MLX conversions, plus official DFlash draft models for faster inference. Day-one support across llama.cpp, ollama, Valerie Lee Make Up, SGLang, NVIDIA TRT-LLM, Hugging Face transformers and ZML. Watch it run on an NVIDIA AI DGX Spark.

Poolside

31,242 views • 14 days ago

🗓️ Release Week Recap Big week. mlx-audio and mlx-vlm are now among some of the fastest-growing OSS projects. Here’s what we shipped last week. Gemma 4 on Apple Silicon Two awesome releases by our partner Google DeepMind & Google Gemma : > Gemma 4 12B — their new dense, unified multimodal model. It uses an encoder free audio path and simplified vision encoder. > Gemma 4 QAT — quantization-aware training checkpoints, optimized to run locally on consumer GPUs and edge devices, compressing the model while preserving the quality you expect from Gemma 4. On the audio 🎧 side we added support for 15+ new TTS, ASR & VAD models, faster long-form transcription, and an expanded OpenAI-compatible audio server. All local on Apple Silicon. Huge thanks to every contributor and my co-maintainer Lucas Newman. 🙏🏽

🗓️ Release Week Recap Big week. mlx-audio and mlx-vlm are now among some of the fastest-growing OSS projects. Here’s what we shipped last week. Gemma 4 on Apple Silicon Two awesome releases by our partner Google DeepMind & Google Gemma : > Gemma 4 12B — their new dense, unified multimodal model. It uses an encoder free audio path and simplified vision encoder. > Gemma 4 QAT — quantization-aware training checkpoints, optimized to run locally on consumer GPUs and edge devices, compressing the model while preserving the quality you expect from Gemma 4. On the audio 🎧 side we added support for 15+ new TTS, ASR & VAD models, faster long-form transcription, and an expanded OpenAI-compatible audio server. All local on Apple Silicon. Huge thanks to every contributor and my co-maintainer Lucas Newman. 🙏🏽

Prince Canuma

11,731 views • 1 month ago

$I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.$

I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.

Alok

36,617 views • 12 days ago

Today, we’re launching Gemma 4, our most intelligent open models to date. Built with the same breakthrough technology as Gemini 3, Gemma 4 brings advanced reasoning to your personal hardware and devices. Here’s what Gemma 4 unlocks for developers: — Intelligence-per-parameter: Our 31B (Dense) and 26B (MoE) models deliver state-of-the-art performance for their size, outcompeting models 20x their size on Arena.ai — Commercial flexibility: Released under a permissive Apache 2.0 license for complete developer flexibility and digital sovereignty — Agentic workflows: Native support for function-calling and structured JSON output allows you to build reliable, autonomous agents — Multimodal edge AI: The E2B and E4B models bring native vision, audio, and low latency to mobile and IoT devices — Long-context reasoning: Up to 256K context windows allow you to process entire repositories or large documents in a single prompt Whether you're building global applications in 140+ languages or local-first AI code assistants, Gemma 4 is built to be your foundation. Explore in Google AI Studio or download the weights on Hugging Face, Kaggle, and ollama.

Today, we’re launching Gemma 4, our most intelligent open models to date. Built with the same breakthrough technology as Gemini 3, Gemma 4 brings advanced reasoning to your personal hardware and devices. Here’s what Gemma 4 unlocks for developers: — Intelligence-per-parameter: Our 31B (Dense) and 26B (MoE) models deliver state-of-the-art performance for their size, outcompeting models 20x their size on Arena.ai — Commercial flexibility: Released under a permissive Apache 2.0 license for complete developer flexibility and digital sovereignty — Agentic workflows: Native support for function-calling and structured JSON output allows you to build reliable, autonomous agents — Multimodal edge AI: The E2B and E4B models bring native vision, audio, and low latency to mobile and IoT devices — Long-context reasoning: Up to 256K context windows allow you to process entire repositories or large documents in a single prompt Whether you're building global applications in 140+ languages or local-first AI code assistants, Gemma 4 is built to be your foundation. Explore in Google AI Studio or download the weights on Hugging Face, Kaggle, and ollama.

Google AI

1,669,728 views • 4 months ago

Google's Gemma 4 26B A4B QAT hits 25+ tokens/sec and 320+ tokens/sec prefill on 8 GB VRAM (RTX 4060) + 16 GB RAM using TurboQuant Prefill just went from 200 → 320+ tok/s on the same 8GB card. 1.6x, no new hardware, no new quant, just a KV cache trick stacked on top of the Gemma 4 26B MoE setup from a few days ago. A few days ago I posted Gemma 4 26B A4B hitting 28 tok/s decode on 8GB VRAM using native MTP. prefill was stuck around 200 tok/s. fair callout by the community. So today I tested something I'd already been meaning to try: TheTom/llama-cpp-turboquant, the TurboQuant KV cache fork by Tom Turney (Tom Turney). (github link in the comments) thanks to him, the fork just got resynced to mainline, so MTP + TurboQuant now run together cleanly (I didnt see any meaningful gains by using MTP with this setup though but you can try). The flags (No MTP): -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -c 64000 --cache-type-k q8_0 --cache-type-v turbo3 Results on the same RTX 4060 8GB, tested with a 27k token prompt at 64k context loaded: Prefill: 200 tok/s → 320+ tok/s Decode: stayed above 25 tok/s (without MTP) Why it works: TurboQuant uses walsh hadamard rotation + polar quantization on the KV cache. keys are sensitive to compression, values aren't much, so it splits the difference: K stays at q8_0, V drops to turbo3 (~3 bits). bonus from the memory savings: same 8GB card can now stretch to 100-120k context with minimal decode penalty. It should now be snappier with any agent harness such as hermes agent without compromise on intelligence. If you're already running Gemma 4 on a small card, this stacks on top for free. Try --cache-type-k q8_0 --cache-type-v turbo3 on your setup and report back what your prefill/decode split looks like. unsloth model gguf and llama.cpp turboquant fork links in the comments. what's your prefill number before vs after?

Google's Gemma 4 26B A4B QAT hits 25+ tokens/sec and 320+ tokens/sec prefill on 8 GB VRAM (RTX 4060) + 16 GB RAM using TurboQuant Prefill just went from 200 → 320+ tok/s on the same 8GB card. 1.6x, no new hardware, no new quant, just a KV cache trick stacked on top of the Gemma 4 26B MoE setup from a few days ago. A few days ago I posted Gemma 4 26B A4B hitting 28 tok/s decode on 8GB VRAM using native MTP. prefill was stuck around 200 tok/s. fair callout by the community. So today I tested something I'd already been meaning to try: TheTom/llama-cpp-turboquant, the TurboQuant KV cache fork by Tom Turney (Tom Turney). (github link in the comments) thanks to him, the fork just got resynced to mainline, so MTP + TurboQuant now run together cleanly (I didnt see any meaningful gains by using MTP with this setup though but you can try). The flags (No MTP): -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -c 64000 --cache-type-k q8_0 --cache-type-v turbo3 Results on the same RTX 4060 8GB, tested with a 27k token prompt at 64k context loaded: Prefill: 200 tok/s → 320+ tok/s Decode: stayed above 25 tok/s (without MTP) Why it works: TurboQuant uses walsh hadamard rotation + polar quantization on the KV cache. keys are sensitive to compression, values aren't much, so it splits the difference: K stays at q8_0, V drops to turbo3 (~3 bits). bonus from the memory savings: same 8GB card can now stretch to 100-120k context with minimal decode penalty. It should now be snappier with any agent harness such as hermes agent without compromise on intelligence. If you're already running Gemma 4 on a small card, this stacks on top for free. Try --cache-type-k q8_0 --cache-type-v turbo3 on your setup and report back what your prefill/decode split looks like. unsloth model gguf and llama.cpp turboquant fork links in the comments. what's your prefill number before vs after?

Alok

119,821 views • 1 month ago

Anyone with 8GB or 12GB VRAM setups needs to understand that "-ncmoe" is the key flag to boost performance on llama.cpp Here are my results for Qwen3.6 35B A3B, with 64k q8_0 context on a 8GB RTX 3070Ti: ⚪️ no flag → 8.7 tok/s RAM: 13.6GB & VRAM: 7.8GB 🔴 -ncmoe 35 → 27.5 tok/s RAM: 12.1GB & VRAM: 4.3GB 🟢 -ncmoe 30 → 32.5 tok/s RAM: 12GB & VRAM: 5.6GB 🔵 -ncmoe 25 → 40.9 tok/s RAM: 12GB & VRAM: 6.9GB Please note the ram and vram usage you see are total usage of a windows pc, with the model running. My friend's setup: 8GB VRAM and 16GB RAM. You can boost performance by switching to Linux, just something to keep in mind. Basically, this flag keeps the MoE experts in the first X layers on your CPU + RAM, instead of eating all your VRAM straight away. This is a smart hybrid offload way that lets you run bigger models without OOM while keeping the rest on your GPU for speed. As we can see on the data, there's a sweet spot. When we lower it from 35 to 25, speed bumps +50% because there are more layers on your GPU (look at the VRAM usage). The key here is to play around with the number and fit as much as possible on your VRAM, goal is to have 1GB/800MB headroom to avoid stress. ↓ server flags below

Anyone with 8GB or 12GB VRAM setups needs to understand that "-ncmoe" is the key flag to boost performance on llama.cpp Here are my results for Qwen3.6 35B A3B, with 64k q8_0 context on a 8GB RTX 3070Ti: ⚪️ no flag → 8.7 tok/s RAM: 13.6GB & VRAM: 7.8GB 🔴 -ncmoe 35 → 27.5 tok/s RAM: 12.1GB & VRAM: 4.3GB 🟢 -ncmoe 30 → 32.5 tok/s RAM: 12GB & VRAM: 5.6GB 🔵 -ncmoe 25 → 40.9 tok/s RAM: 12GB & VRAM: 6.9GB Please note the ram and vram usage you see are total usage of a windows pc, with the model running. My friend's setup: 8GB VRAM and 16GB RAM. You can boost performance by switching to Linux, just something to keep in mind. Basically, this flag keeps the MoE experts in the first X layers on your CPU + RAM, instead of eating all your VRAM straight away. This is a smart hybrid offload way that lets you run bigger models without OOM while keeping the rest on your GPU for speed. As we can see on the data, there's a sweet spot. When we lower it from 35 to 25, speed bumps +50% because there are more layers on your GPU (look at the VRAM usage). The key here is to play around with the number and fit as much as possible on your VRAM, goal is to have 1GB/800MB headroom to avoid stress. ↓ server flags below

left curve dev

166,368 views • 2 months ago

𝗥𝘂𝗻 𝗚𝗼𝗼𝗴𝗹𝗲'𝘀 𝗚𝗲𝗺𝗺𝗮 𝟰 + 𝗢𝗽𝗲𝗻𝗖𝗹𝗮𝘄 𝗮𝘀 𝗮 𝗳𝗿𝗲𝗲 𝗽𝗿𝗶𝘃𝗮𝘁𝗲 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁 𝗼𝗻 𝘆𝗼𝘂𝗿 𝗼𝘄𝗻 𝗺𝗮𝗰𝗵𝗶𝗻𝗲 𝗶𝗻 𝟯 𝘀𝘁𝗲𝗽𝘀. No API bills. No usage limits. No subscription. Nothing leaves your computer. Here's the full setup: → Step 1: Go to Download and install. Update to version 0.2.2.0 or higher. → Step 2: Open terminal. Type: ollama pull gemma4. Downloads the model. Done. → Step 3: Install OpenClaw. Select Ollama as your provider. Point it at port 11434. Pick Gemma 4. That's it. Your AI agent is now running locally. Message it through Telegram, Slack, Discord, or WhatsApp like a coworker. Read files. Write code. Remember context across every conversation. All on your own hardware. Gemma 4 ranked number 3 on the global open model leaderboard on launch day. It beat models with 20 times more parameters. The 26B version activates only 4B parameters at a time so you get near large-model quality at small-model speed. Every AI subscription you're paying for right now could be replaced with this.

𝗥𝘂𝗻 𝗚𝗼𝗼𝗴𝗹𝗲'𝘀 𝗚𝗲𝗺𝗺𝗮 𝟰 + 𝗢𝗽𝗲𝗻𝗖𝗹𝗮𝘄 𝗮𝘀 𝗮 𝗳𝗿𝗲𝗲 𝗽𝗿𝗶𝘃𝗮𝘁𝗲 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁 𝗼𝗻 𝘆𝗼𝘂𝗿 𝗼𝘄𝗻 𝗺𝗮𝗰𝗵𝗶𝗻𝗲 𝗶𝗻 𝟯 𝘀𝘁𝗲𝗽𝘀. No API bills. No usage limits. No subscription. Nothing leaves your computer. Here's the full setup: → Step 1: Go to Download and install. Update to version 0.2.2.0 or higher. → Step 2: Open terminal. Type: ollama pull gemma4. Downloads the model. Done. → Step 3: Install OpenClaw. Select Ollama as your provider. Point it at port 11434. Pick Gemma 4. That's it. Your AI agent is now running locally. Message it through Telegram, Slack, Discord, or WhatsApp like a coworker. Read files. Write code. Remember context across every conversation. All on your own hardware. Gemma 4 ranked number 3 on the global open model leaderboard on launch day. It beat models with 20 times more parameters. The 26B version activates only 4B parameters at a time so you get near large-model quality at small-model speed. Every AI subscription you're paying for right now could be replaced with this.

Julian Goldie SEO

36,898 views • 4 months ago

$Auto regressive LLMs are officially on notice. run Gemma 4 26B diffusion gguf with llama.cpp Google just dropped DiffusionGemma-26B, and it completely flips how we generate text. instead of predicting words one by one, it generates 256 tokens in parallel using bi-directional attention. its like stable diffusion, but for language. the model starts with random text "noise" and iteratively refines and self-corrects the entire block in real-time to fix formatting and reasoning errors on the fly. since it’s a Mixture of Experts (MoE) that only activates 3.8B parameters during inference, it fits perfectly on consumer hardware. You can run the Q4_K_M quant with an 18GB VRAM budget on a single RTX 3090 or RTX 4090 with exceptional throughput. Tested on Ubuntu 22 with CUDA 13.1 using the cutting edge experimental llama.cpp branch. Here is how to compile and run it with the live terminal denoising visualizer: # 1. Clone & check out the experimental PR (#24423) - 1) git clone && cd llama.cpp -git fetch origin 2) pull/24423/head:diffusiongemma && --git checkout diffusiongemma # 2. Build with CUDA support 1) cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native 2) cmake --build build -j $(nproc) --config Release --target llama-diffusion-cli # 3. Run with live visual denoising (llama.cpp flags) ./build/bin/llama-diffusion-cli \ -m /path/to/diffusiongemma-26B-A4B-it-Q4_K_M.gguf \ -ngl 99 -cnv -n 2048 --diffusion-visual Watch the video below to see the live --diffusion-visual canvas iteratively de noising the prompt output in real time. guide and unsloth's hugging face GGUF model links are in the comments below! Is auto regressive generation officially legacy tech? Let me know what you think.$

Auto regressive LLMs are officially on notice. run Gemma 4 26B diffusion gguf with llama.cpp Google just dropped DiffusionGemma-26B, and it completely flips how we generate text. instead of predicting words one by one, it generates 256 tokens in parallel using bi-directional attention. its like stable diffusion, but for language. the model starts with random text "noise" and iteratively refines and self-corrects the entire block in real-time to fix formatting and reasoning errors on the fly. since it’s a Mixture of Experts (MoE) that only activates 3.8B parameters during inference, it fits perfectly on consumer hardware. You can run the Q4_K_M quant with an 18GB VRAM budget on a single RTX 3090 or RTX 4090 with exceptional throughput. Tested on Ubuntu 22 with CUDA 13.1 using the cutting edge experimental llama.cpp branch. Here is how to compile and run it with the live terminal denoising visualizer: # 1. Clone & check out the experimental PR (#24423) - 1) git clone && cd llama.cpp -git fetch origin 2) pull/24423/head:diffusiongemma && --git checkout diffusiongemma # 2. Build with CUDA support 1) cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native 2) cmake --build build -j $(nproc) --config Release --target llama-diffusion-cli # 3. Run with live visual denoising (llama.cpp flags) ./build/bin/llama-diffusion-cli \ -m /path/to/diffusiongemma-26B-A4B-it-Q4_K_M.gguf \ -ngl 99 -cnv -n 2048 --diffusion-visual Watch the video below to see the live --diffusion-visual canvas iteratively de noising the prompt output in real time. guide and unsloth's hugging face GGUF model links are in the comments below! Is auto regressive generation officially legacy tech? Let me know what you think.

Alok

52,656 views • 1 month ago

HERMES CAN NOW RUN FOR FREE FOREVER A new Gemma 4 update just removed the biggest problem with local AI agents. The Upgrade: → Gemma 4 runs around 90% faster on Apple Silicon using Ollama + MLX → Plug it into Hermes with a few terminal commands → Run private agent workflows without subscriptions, token limits, or API bills What It Can Actually Do: ✓ Build and preview apps locally ✓ Use /learn to turn tutorials into permanent skills ✓ Read files, analyse landing pages, summarise emails, and track AI news ✓ Run loops until a task is complete instead of waiting for another prompt ✓ Launch multiple Gemma 4 subagents to work on separate tasks in parallel No Apple Silicon? ✔ OpenRouter offers the 31B model through a free API The real win is not faster chat. It is having Hermes build, retry, check its work, and operate in the background without charging you for every loop.

HERMES CAN NOW RUN FOR FREE FOREVER A new Gemma 4 update just removed the biggest problem with local AI agents. The Upgrade: → Gemma 4 runs around 90% faster on Apple Silicon using Ollama + MLX → Plug it into Hermes with a few terminal commands → Run private agent workflows without subscriptions, token limits, or API bills What It Can Actually Do: ✓ Build and preview apps locally ✓ Use /learn to turn tutorials into permanent skills ✓ Read files, analyse landing pages, summarise emails, and track AI news ✓ Run loops until a task is complete instead of waiting for another prompt ✓ Launch multiple Gemma 4 subagents to work on separate tasks in parallel No Apple Silicon? ✔ OpenRouter offers the 31B model through a free API The real win is not faster chat. It is having Hermes build, retry, check its work, and operate in the background without charging you for every loop.

Julian Goldie SEO

11,234 views • 1 month ago

Unsloth AI and NVIDIA are Revolutionizing Local LLM Fine-Tuning: From RTX Desktops to DGX Spark Fine-tune popular AI models faster with Unsloth on NVIDIA RTX AI PCs such as GeForce RTX desktops and laptops to RTX PRO workstations and the new DGX Spark to build personalized assistants for coding, creative work, and complex agentic workflows. The landscape of modern AI is shifting. We are moving away from a total reliance on massive, generalized cloud models and entering the era of local, agentic AI. Whether it is tuning a chatbot to handle hyper-specific product support or building a personal assistant that manages intricate schedules, the potential for generative AI on local hardware is boundless. However, developers face a persistent bottleneck: How do you get a Small Language Model (SLM) to punch above its weight class and respond with high accuracy for specialized tasks? The answer is Fine-Tuning, and the tool of choice is Unsloth. Unsloth provides an easy and high-speed method to customize models. Optimized for efficient, low-memory training on NVIDIA GPUs, Unsloth scales effortlessly from GeForce RTX desktops and laptop all the way to the DGX Spark, the world’s smallest AI supercomputer...... Full analysis: NVIDIA NVIDIA AI NVIDIA AIDev NVIDIAnewsroom Unsloth AI Unsloth

Unsloth AI and NVIDIA are Revolutionizing Local LLM Fine-Tuning: From RTX Desktops to DGX Spark Fine-tune popular AI models faster with Unsloth on NVIDIA RTX AI PCs such as GeForce RTX desktops and laptops to RTX PRO workstations and the new DGX Spark to build personalized assistants for coding, creative work, and complex agentic workflows. The landscape of modern AI is shifting. We are moving away from a total reliance on massive, generalized cloud models and entering the era of local, agentic AI. Whether it is tuning a chatbot to handle hyper-specific product support or building a personal assistant that manages intricate schedules, the potential for generative AI on local hardware is boundless. However, developers face a persistent bottleneck: How do you get a Small Language Model (SLM) to punch above its weight class and respond with high accuracy for specialized tasks? The answer is Fine-Tuning, and the tool of choice is Unsloth. Unsloth provides an easy and high-speed method to customize models. Optimized for efficient, low-memory training on NVIDIA GPUs, Unsloth scales effortlessly from GeForce RTX desktops and laptop all the way to the DGX Spark, the world’s smallest AI supercomputer...... Full analysis: NVIDIA NVIDIA AI NVIDIA AIDev NVIDIAnewsroom Unsloth AI Unsloth

Marktechpost AI Dev News ⚡

31,551 views • 7 months ago

Google just shipped a free dictation app for Mac and iPhone called AI Edge Eloquent, and the model running it is Gemma 4 12B, entirely on your device. What it actually does: you speak, it transcribes, removes filler words, and polishes the result before dropping clean text into whatever app you are using. You can also give it voice commands to reformat, like "translate to Hindi" or "make this a bullet list." Audio and video file transcription is built in too. The interesting part is the privacy angle. Most AI writing tools send your audio to a server. This one never leaves your machine, which matters if you are dictating anything sensitive. The tradeoff is hardware: you need a Mac with at least 16GB of RAM to run a 12B parameter model comfortably.

Google just shipped a free dictation app for Mac and iPhone called AI Edge Eloquent, and the model running it is Gemma 4 12B, entirely on your device. What it actually does: you speak, it transcribes, removes filler words, and polishes the result before dropping clean text into whatever app you are using. You can also give it voice commands to reformat, like "translate to Hindi" or "make this a bullet list." Audio and video file transcription is built in too. The interesting part is the privacy angle. Most AI writing tools send your audio to a server. This one never leaves your machine, which matters if you are dictating anything sensitive. The tradeoff is hardware: you need a Mac with at least 16GB of RAM to run a 12B parameter model comfortably.

Aditya ⚡Rao

84,988 views • 1 month ago

$I nearly 2x'd the speed while only using +1GB VRAM with the new MTP update in llama.cpp 🤯 You need to add these flags to start using it: --spec-type draft-mtp \ --spec-draft-p-min 0.75 \ --spec-draft-n-max 2 My results with Qwen3.6 27B on a single RTX 5080 ↓ ⚪️ no flag (without mtp) → 54.3 tok/s with 13.26GB VRAM 🔵 --spec-draft-n-max 2 → 90.7 tok/s with 14.29GB VRAM 🔴 --spec-draft-n-max 2 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.30GB VRAM 🟢 --spec-draft-n-max 6 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.87GB VRAM Increasing to 6 draft tokens didn't help my setup for some reason. I made sure to test with a low context length to have enough headroom and eliminate risk of vram stress. From my understanding: 1) The speed gains are very task-dependent. You need to test across a wide range of tasks to get a realistic idea of the benefits 2) We’re already running heavily quantized GGUF models (Q3, Q4, Q6, etc.), so we already benefit from strong speed/performance thanks to the reduced size. That’s why some people are seeing little to no improvement compared to MLX or other quantized versions The progress over the past few days has been insane to say the least. However, MTP now consumes significantly more VRAM. Personally 16GB just isn't enough to use MTP and run it with a good context size. Time to upgrade lads, 24GB+ users are eating GOOD today 🔥 Full setup below ↓$

I nearly 2x'd the speed while only using +1GB VRAM with the new MTP update in llama.cpp 🤯 You need to add these flags to start using it: --spec-type draft-mtp \ --spec-draft-p-min 0.75 \ --spec-draft-n-max 2 My results with Qwen3.6 27B on a single RTX 5080 ↓ ⚪️ no flag (without mtp) → 54.3 tok/s with 13.26GB VRAM 🔵 --spec-draft-n-max 2 → 90.7 tok/s with 14.29GB VRAM 🔴 --spec-draft-n-max 2 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.30GB VRAM 🟢 --spec-draft-n-max 6 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.87GB VRAM Increasing to 6 draft tokens didn't help my setup for some reason. I made sure to test with a low context length to have enough headroom and eliminate risk of vram stress. From my understanding: 1) The speed gains are very task-dependent. You need to test across a wide range of tasks to get a realistic idea of the benefits 2) We’re already running heavily quantized GGUF models (Q3, Q4, Q6, etc.), so we already benefit from strong speed/performance thanks to the reduced size. That’s why some people are seeing little to no improvement compared to MLX or other quantized versions The progress over the past few days has been insane to say the least. However, MTP now consumes significantly more VRAM. Personally 16GB just isn't enough to use MTP and run it with a good context size. Time to upgrade lads, 24GB+ users are eating GOOD today 🔥 Full setup below ↓

left curve dev

29,946 views • 2 months ago

🥊 Time for a new fight Qwen3.5 397B A17B vs Qwen3.6 27B 🌸 "Cherry Blossom" (↓ prompt below) Using OpenRouter for 397B Running 27B locally on a single RTX 5080 Wow! 🤯 I'm sure this one is going to divide opinions. 397B feels cinematic with the flare and shadows it added, while 27B is just beautiful; it has a completely different color palette and the leaves are falling from the branches as asked. It's really tough to decide which model is the best. I was about to call it a tie, but then I remembered that 397B needs 180GB VRAM to run locally 😅 Don't get me wrong the model is amazing, but right now the VRAM it requires doesn't feel justified. It's simply not 10× better than 27B. We'll have to dig deeper with more prompts to be sure, let's see what each one has in store for us. What do you think?

🥊 Time for a new fight Qwen3.5 397B A17B vs Qwen3.6 27B 🌸 "Cherry Blossom" (↓ prompt below) Using OpenRouter for 397B Running 27B locally on a single RTX 5080 Wow! 🤯 I'm sure this one is going to divide opinions. 397B feels cinematic with the flare and shadows it added, while 27B is just beautiful; it has a completely different color palette and the leaves are falling from the branches as asked. It's really tough to decide which model is the best. I was about to call it a tie, but then I remembered that 397B needs 180GB VRAM to run locally 😅 Don't get me wrong the model is amazing, but right now the VRAM it requires doesn't feel justified. It's simply not 10× better than 27B. We'll have to dig deeper with more prompts to be sure, let's see what each one has in store for us. What do you think?

left curve dev

21,224 views • 3 months ago

Real-time Fire and Liquid Simulation 🤯 Achieved using NVIDIA RTX PRO 6000 with 96 GB of VRAM 🤯🤯 Only Nerds like us get why this awesome!! Anybody who worked with 3D softwares knows how hard is it to learn and work with particle simulations in Real-time. Thanks to Dell and JangaFX Software we saw this Demo using the amazing softwares of EmberGen and LiquiGen where newcomers can quickly learn and execute complex simulations in real-time. Best part is you can export these simulations and bring it to your 3D software of choice such as Unreal Engine or Blender 🔶 😎😎

Real-time Fire and Liquid Simulation 🤯 Achieved using NVIDIA RTX PRO 6000 with 96 GB of VRAM 🤯🤯 Only Nerds like us get why this awesome!! Anybody who worked with 3D softwares knows how hard is it to learn and work with particle simulations in Real-time. Thanks to Dell and JangaFX Software we saw this Demo using the amazing softwares of EmberGen and LiquiGen where newcomers can quickly learn and execute complex simulations in real-time. Best part is you can export these simulations and bring it to your 3D software of choice such as Unreal Engine or Blender 🔶 😎😎

Bad Decisions Studio

39,942 views • 11 months ago

Parameter scaling just broke in front of me. I gave 90 math problems as images to 10 local vision models, 3 runs each, only consistent answers across all 3 runs counted. Two findings: > Gemma 4 was the most consistent family, 31B took the crown at 89.6%. > But Qwen 3.5 4B came 2 answers behind it. A 4B. Ranked 2nd out of 10. Vision-math isn't one skill, it's two: reading the image, then solving it. The real lesson for anyone running local: small ≠ weak. If you're building agentic workflows, finding the right model for each task matters more than reaching for the biggest one. In this test, the 4B ran far faster because of its size, scored higher, and left VRAM free for the rest of your stack. Full results: 🥇 Gemma 4 31B — 242/270 (89.6%) 🥈 Qwen 3.5 4B — 240/270 (88.9%) 🥉 Gemma 4 E4B — 222/270 (82.2%) 🥉 Qwen 3.6 27B — 222/270 (82.2%) 5. Gemma 4 26B A4B — 216/270 (80.0%) 6. Qwen 3.5 2B — 201/270 (74.4%) 7. Gemma 4 E2B — 192/270 (71.1%) 8. Qwen 3.6 35B A3B — 192/270 (71.1%) 9. Qwen 3.5 9B — 168/270 (62.2%) 10. Qwen 3.5 0.8B — 45/270 (16.7%) All GGUF + mmproj, Unsloth's Q6_K_XL quant.

Parameter scaling just broke in front of me. I gave 90 math problems as images to 10 local vision models, 3 runs each, only consistent answers across all 3 runs counted. Two findings: > Gemma 4 was the most consistent family, 31B took the crown at 89.6%. > But Qwen 3.5 4B came 2 answers behind it. A 4B. Ranked 2nd out of 10. Vision-math isn't one skill, it's two: reading the image, then solving it. The real lesson for anyone running local: small ≠ weak. If you're building agentic workflows, finding the right model for each task matters more than reaching for the biggest one. In this test, the 4B ran far faster because of its size, scored higher, and left VRAM free for the rest of your stack. Full results: 🥇 Gemma 4 31B — 242/270 (89.6%) 🥈 Qwen 3.5 4B — 240/270 (88.9%) 🥉 Gemma 4 E4B — 222/270 (82.2%) 🥉 Qwen 3.6 27B — 222/270 (82.2%) 5. Gemma 4 26B A4B — 216/270 (80.0%) 6. Qwen 3.5 2B — 201/270 (74.4%) 7. Gemma 4 E2B — 192/270 (71.1%) 8. Qwen 3.6 35B A3B — 192/270 (71.1%) 9. Qwen 3.5 9B — 168/270 (62.2%) 10. Qwen 3.5 0.8B — 45/270 (16.7%) All GGUF + mmproj, Unsloth's Q6_K_XL quant.

stevibe

22,882 views • 2 months ago

LLMs can take gigabytes of memory to store, which limits what can be run on consumer hardware. But quantization can dramatically compress models, making a wider selection of models available to developers. You can often reduce model size by 4x or more while maintaining reasonable performance. In our new short course Quantization Fundamentals taught by Hugging Face's @younesbelkada and Marc Sun, you'll: - Learn how to quantize nearly any open source model - Use int8 and bfloat16 (Brain float 16) data types to load and run LLMs using PyTorch and the Hugging Face Transformers library - Dive into the technical details of linear quantization to map 32-bit floats to 8-bit integers As models get bigger and bigger, quantization becomes more important for making models practical and accessible. Please check out the course here:

LLMs can take gigabytes of memory to store, which limits what can be run on consumer hardware. But quantization can dramatically compress models, making a wider selection of models available to developers. You can often reduce model size by 4x or more while maintaining reasonable performance. In our new short course Quantization Fundamentals taught by Hugging Face's @younesbelkada and Marc Sun, you'll: - Learn how to quantize nearly any open source model - Use int8 and bfloat16 (Brain float 16) data types to load and run LLMs using PyTorch and the Hugging Face Transformers library - Dive into the technical details of linear quantization to map 32-bit floats to 8-bit integers As models get bigger and bigger, quantization becomes more important for making models practical and accessible. Please check out the course here:

Andrew Ng

288,304 views • 2 years ago

I freaked out when my WiFi router suddenly died. then realized my autonomous Hermes agent is running fully local, nothing stopped. Hermes Agent + Gemma 4 26B A4B QAT MoE, 100% local on my laptop, building my side projects while I scroll my phone zero API calls. zero cost. 100% private. fully offline. This might be the most satisfying thing I’ve watched in a while. last post: showed Hermes + local Gemma 4 26B pull off backtest a trading strategy. this time I asked it to develop something i'd use myself everyday: # A full unpacked extension with: - React side panel UI - Local llama.cpp backend (offline AI) - Live tab sync + status tracking - Auto context extraction via Readability.js Vision on Demand → captures viewport screenshots as compressed JPEGs Deterministic action system -> model outputs tokens -> directly controls page scrolling It planned everything first. Then started executing step by step. all i did was say 'ok'. only once. # What’s wild: - It reports back after every phase - Auto compresses context when nearing limits - Actualy, stays on track llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 --cache-type-k q8_0 --cache-type-v q8_0 --port 8080 # Performance on a single NVIDIA RTX 4060 (8GB VRAM) + 16 GB DDR4 RAM Gaming Laptop: - 300 tokens/sec prefill - 25+ tokens/sec decode More than usable for real dev workflows. This isn’t AI demo territory anymore. This is autonomous local software actually building things.

I freaked out when my WiFi router suddenly died. then realized my autonomous Hermes agent is running fully local, nothing stopped. Hermes Agent + Gemma 4 26B A4B QAT MoE, 100% local on my laptop, building my side projects while I scroll my phone zero API calls. zero cost. 100% private. fully offline. This might be the most satisfying thing I’ve watched in a while. last post: showed Hermes + local Gemma 4 26B pull off backtest a trading strategy. this time I asked it to develop something i'd use myself everyday: # A full unpacked extension with: - React side panel UI - Local llama.cpp backend (offline AI) - Live tab sync + status tracking - Auto context extraction via Readability.js Vision on Demand → captures viewport screenshots as compressed JPEGs Deterministic action system -> model outputs tokens -> directly controls page scrolling It planned everything first. Then started executing step by step. all i did was say 'ok'. only once. # What’s wild: - It reports back after every phase - Auto compresses context when nearing limits - Actualy, stays on track llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 --cache-type-k q8_0 --cache-type-v q8_0 --port 8080 # Performance on a single NVIDIA RTX 4060 (8GB VRAM) + 16 GB DDR4 RAM Gaming Laptop: - 300 tokens/sec prefill - 25+ tokens/sec decode More than usable for real dev workflows. This isn’t AI demo territory anymore. This is autonomous local software actually building things.

Alok

56,039 views • 1 month ago

Sriram Krishnan reveals why it's easier to defend American code with a Chinese model than an American one right now: "I don't think it is great that the leading open weight models or open source models are not American. There's some great innovation happening with Moonshot and with DeepSeek and with Qwen, but I would much rather prefer that the leading models are American." "Linus Torvalds of Linux fame had a law that given enough eyes, all bugs are shallow. Open weight models are inherently secure because when you download a model off Hugging Face, the entire world can take it apart, inspect it, fine-tune it, look at it in ways that you absolutely cannot if they are closed." "There was an incident with Hugging Face that got reported on earlier today. Hugging Face seems to have found that somebody was using an AI LLM agent to hammer them at multiple places and to find ways to break in." "The way to counter that is to make sure American or Western or allies defenders have access to the best models to make our software just more secure." "It may be harder to look at your own code for exploits with Fable than it is when you use a Chinese model. That's just like a weird spot to be, and I think we should fix that." Sriram Krishnan

Sriram Krishnan reveals why it's easier to defend American code with a Chinese model than an American one right now: "I don't think it is great that the leading open weight models or open source models are not American. There's some great innovation happening with Moonshot and with DeepSeek and with Qwen, but I would much rather prefer that the leading models are American." "Linus Torvalds of Linux fame had a law that given enough eyes, all bugs are shallow. Open weight models are inherently secure because when you download a model off Hugging Face, the entire world can take it apart, inspect it, fine-tune it, look at it in ways that you absolutely cannot if they are closed." "There was an incident with Hugging Face that got reported on earlier today. Hugging Face seems to have found that somebody was using an AI LLM agent to hammer them at multiple places and to find ways to break in." "The way to counter that is to make sure American or Western or allies defenders have access to the best models to make our software just more secure." "It may be harder to look at your own code for exploits with Fable than it is when you use a Chinese model. That's just like a weird spot to be, and I think we should fix that." Sriram Krishnan

MTS

61,227 views • 15 days ago