Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

GOOGLE JUST MADE EVERY CHATBOT FEEL SLOW Diffusion Gemma 26b doesn’t predict word by word, it generates 256 tokens in parallel using bi-directional attention, like stable diffusion but for language it’s MoE so only 3.8B params activate during inference, fits on a single RTX 4090 with 18GB VRAM and... show more

leopardracer

11,779 subscribers

27,385 görüntüleme • 3 ay önce •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

26 Yorum

Gabu profil fotoğrafı

Gabu3 ay önce

>and you can run it right now via llama.cpp Show us llama-server then :P Still waiting for full support. But you can run it full on vllm now:

Way profil fotoğrafı

Way3 ay önce

There’s more to a chatbot than speed. Faster slop is still slop.

Rick profil fotoğrafı

Rick3 ay önce

Will this run on a 64GB Macbook Pro Max M3

Oliver profil fotoğrafı

Oliver3 ay önce

256 tokens in parallel is the part to watch; latency wins only matter if the output stays steerable.

Paul profil fotoğrafı

Paul3 ay önce

Yes, it’s faster, but it’s also dumber than the base 26b a4b qat right now. Let’s hope they can close this gap.

starmex profil fotoğrafı

starmex3 ay önce

great post, thanks!

Alfero Chingono profil fotoğrafı

Alfero Chingono3 ay önce

Generating 256 tokens in parallel really shifts the latency game for apps. The MoE aspect helps with the practical cost of running this on consumer hardware too.

Kanika profil fotoğrafı

Kanika3 ay önce

When a model this capable fits on consumer hardware, adoption accelerates fast.

Petru | Steam Vibe profil fotoğrafı

Petru | Steam Vibe3 ay önce

The parallel token generation shifts the bottleneck from inference speed to VRAM bandwidth, making local deployment more viable for complex workflows.

Xoch profil fotoğrafı

Xoch3 ay önce

People underestimate how much faster responses change the entire experience of using AI

rari profil fotoğrafı

rari3 ay önce

chatbots waiting word by word may feel old soon

marfin profil fotoğrafı

marfin3 ay önce

I like how the Gemma model line is evolving

Nekt0 profil fotoğrafı

Nekt03 ay önce

Top

karla ❀ profil fotoğrafı

karla ❀3 ay önce

generates 256 tokens in parallel? It's awesome

Jay | The Midwest Dad | profil fotoğrafı

Jay | The Midwest Dad |3 ay önce

Only valuable if the quality is actually usable…which it’s not. Google admits this. Once enough people test this I anticipate they will put out a better quality version

slash1s profil fotoğrafı

slash1s3 ay önce

huh🤔

Nikiton profil fotoğrafı

Nikiton3 ay önce

His speed is amazing

Lunari profil fotoğrafı

Lunari3 ay önce

208ms per step over 97 steps is still not instant curious how quality holds when you push past the entropy bound on longer outputs

‏عادل | مبرمج profil fotoğrafı

‏عادل | مبرمج3 ay önce

Banger

rewind profil fotoğrafı

rewind3 ay önce

parallel token generation feels like the next obvious step

Gipp 🦅 profil fotoğrafı

Gipp 🦅3 ay önce

everything really seems slow now

Harry Tandy profil fotoğrafı

Harry Tandy3 ay önce

local inference speed just made cloud providers nervous

𝗢𝗠𝗘𝗚𝗔 profil fotoğrafı

𝗢𝗠𝗘𝗚𝗔3 ay önce

18gb vram is the new baseline everyone skipped past

Nico profil fotoğrafı

Nico3 ay önce

the crazy efficiency

Vjekoslav Jug profil fotoğrafı

Vjekoslav Jug3 ay önce

So the idea is to be dumb but as fast as possible so you can figure out it’s no use of it faster on move to next on the hype

Punch Taylor profil fotoğrafı

Punch Taylor3 ay önce

256 tokens in parallel is wild. but here's the dirty benchmark nobody talks about: how does it actually feel when your system is under load? we run local models on a 4090 that's also gaming, a mac studio that's also running home automation, and a mesh of pis doing mqtt. clean benchmarks look great until you're streaming while inferring. the moe architecture (3.8b active on a 26b model) should handle that load splitting beautifully — but real-world numbers under system stress are what actually matter for daily use.

Benzer Videolar

New Google Gemma 4 12B claims near-26B performance - we tested both! We ran both models locally on one RTX 4090 and gave each the same task: write a self-contained HTML5 canvas animation with real physics in one file without libraries. Three scenes - a Galton board, two blocks colliding off a wall, and a chaotic triple pendulum Outputs: Gemma 4 26B-A4B: 15 GB VRAM usage, 6.9k tokens, 138 tok/s Gemma 4 12B: 9 GB VRAM usage, 8.9k tokens, 80 tok/s Same Gemma 4 family, but the 26B-A4B won every scene and ran ~1.7x faster - on just 4B active params. The 12B stayed very close though, on almost half the VRAM - which makes it the ideal model for a 16 GB laptop

New Google Gemma 4 12B claims near-26B performance - we tested both! We ran both models locally on one RTX 4090 and gave each the same task: write a self-contained HTML5 canvas animation with real physics in one file without libraries. Three scenes - a Galton board, two blocks colliding off a wall, and a chaotic triple pendulum Outputs: Gemma 4 26B-A4B: 15 GB VRAM usage, 6.9k tokens, 138 tok/s Gemma 4 12B: 9 GB VRAM usage, 8.9k tokens, 80 tok/s Same Gemma 4 family, but the 26B-A4B won every scene and ran ~1.7x faster - on just 4B active params. The 12B stayed very close though, on almost half the VRAM - which makes it the ideal model for a 16 GB laptop

atomic.chat

152,748 görüntüleme • 3 ay önce

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

Alok

62,381 görüntüleme • 3 ay önce

Diffusion Gemma is 4x faster, but makes 6x more mistakes! We benchmarked the new diffusion LLM against its autoregressive twin on a single H100 (FP8). We gave each the same three tasks: write a Steve Jobs biography, the history of Tetris, and the story of BeOS - every next topic less popular than the previous one. Then we fact-checked every claim in every answer. Gemma4 got 45 facts right, 5 wrong. DiffusionGemma got 33 right, 28 wrong. The less popular the topic, the worse it got: 4 mistakes on Jobs, 12 on Tetris, 12 on BeOS. It named Clara Clley as Steve Jobs' mother, invented a colleague for Pajitnov named Geri Gulovik and priced the BeBox at $9,999. The real one cost $1,600. Outputs: Gemma4 26B A4B: 218 tok/s · 15.1s total · 45 facts · 5 mistakes DiffusionGemma 26B A4B: 763 tok/s · 3.7s total · 33 facts · 28 mistakes The reason is simple. DiffusionGemma throws 256 tokens on the screen at once and polishes them pass after pass until the text sounds smooth. Smooth is all it cares about: a fake name, date or number sounds just as smooth as a real one, so it stays. Regular Gemma4 meanwhile writes one word at a time and checks every new word against everything before it. Google says it themselves in the launch post: quality is lower, use regular Gemma 4 when facts matter.

Diffusion Gemma is 4x faster, but makes 6x more mistakes! We benchmarked the new diffusion LLM against its autoregressive twin on a single H100 (FP8). We gave each the same three tasks: write a Steve Jobs biography, the history of Tetris, and the story of BeOS - every next topic less popular than the previous one. Then we fact-checked every claim in every answer. Gemma4 got 45 facts right, 5 wrong. DiffusionGemma got 33 right, 28 wrong. The less popular the topic, the worse it got: 4 mistakes on Jobs, 12 on Tetris, 12 on BeOS. It named Clara Clley as Steve Jobs' mother, invented a colleague for Pajitnov named Geri Gulovik and priced the BeBox at $9,999. The real one cost $1,600. Outputs: Gemma4 26B A4B: 218 tok/s · 15.1s total · 45 facts · 5 mistakes DiffusionGemma 26B A4B: 763 tok/s · 3.7s total · 33 facts · 28 mistakes The reason is simple. DiffusionGemma throws 256 tokens on the screen at once and polishes them pass after pass until the text sounds smooth. Smooth is all it cares about: a fake name, date or number sounds just as smooth as a real one, so it stays. Regular Gemma4 meanwhile writes one word at a time and checks every new word against everything before it. Google says it themselves in the launch post: quality is lower, use regular Gemma 4 when facts matter.

atomic.chat

77,455 görüntüleme • 3 ay önce

If you are running local LLMs without N-gram speculative decoding, you are wasting massive amounts of compute. Whether your AI is editing a document, outputting structured JSON, or rewriting boilerplate templates, a huge chunk of the text it generates is highly repetitive or already exists right there in the prompt. Standard decoding wastes expensive GPU compute cycles "re thinking" every single token. By adding one hidden flag in llama.cpp, you can instantly fast forward through the repetition. Zero draft models. Zero extra VRAM. And virtually zero compute overhead. Google Colab hands you an enterprise grade NVIDIA Tesla T4 GPU with 16GB of VRAM for free. It’s the perfect Ubuntu Linux sandbox to build a bleeding edge inference engine from scratch. Recently, I showed you how to double your local speeds using MTP (Multi Token Prediction). But MTP requires a secondary neural network draft model. That eats into your precious VRAM (slightly though) and burns extra compute for every guess it makes. N-gram Speculative Decoding gives you a massive speed boost for exactly 0 memory cost and minimal compute. And it's faster than MTP when it works. Here is how it actually works under the hood: Standard autoregressive decoding is slow because it predicts one token at a time. If you ask an agent to format a long JSON object or update one line in an HTML file, it runs heavy matrix multiplications to calculate the probability of every single bracket, space, and letter from scratch. N-gram changes the game. It acts as a lightweight caching system. Instead of running heavy neural network math to guess the next word, it uses a simple hash table. Whenever the LLM starts outputting a sequence of tokens that already exists anywhere in its context window, N-gram instantly recognizes the pattern. Because it is just doing lightning fast string matching, the compute cost is practically zero. It "fast forwards" through the text, drafting the boilerplate instantly from memory, and the main model just verifies it in parallel. Pure speed. Using quantized GGUFs from Unsloth via HuggingFace, I spun up DeepMind’s massive Gemma 4 26B A4B QAT MoE on a free Colab instance to test this. Just look at the raw benchmark data on code editing task: Without N-gram: [ Prompt: 638.6 t/s | Generation: 45.9 t/s ] With N-gram: [ Prompt: 601.9 t/s | Generation: 107.1 t/s ] Here is the exact llama.cpp CLI command to activate it. Notice we don't even need the --model-draft flag: ./llama-cli -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -n 6000 -c 12000 -ngl 99 -fa on --spec-type ngram-mod Stop waiting for your GPU to re calculate words it already knows. I’ve built a free, interactive, cell by cell Google Colab notebook that lets you test this live in your browser. You can literally chat with the model and watch the text generation speed absolutely fly on the second turn when you ask it to edit a file. There are additional parameters for ngram-mod that you can tune once you get it working with the single flag. Link to the free Colab Notebook is in the comments below. It walks you through the entire stack: pulling pre built llama.cpp CUDA binaries for Linux, fetching GGUFs from HuggingFace, and spinning up the inference engine with ngram-mod from scratch. Let me know if you have already tried ngram-mod

If you are running local LLMs without N-gram speculative decoding, you are wasting massive amounts of compute. Whether your AI is editing a document, outputting structured JSON, or rewriting boilerplate templates, a huge chunk of the text it generates is highly repetitive or already exists right there in the prompt. Standard decoding wastes expensive GPU compute cycles "re thinking" every single token. By adding one hidden flag in llama.cpp, you can instantly fast forward through the repetition. Zero draft models. Zero extra VRAM. And virtually zero compute overhead. Google Colab hands you an enterprise grade NVIDIA Tesla T4 GPU with 16GB of VRAM for free. It’s the perfect Ubuntu Linux sandbox to build a bleeding edge inference engine from scratch. Recently, I showed you how to double your local speeds using MTP (Multi Token Prediction). But MTP requires a secondary neural network draft model. That eats into your precious VRAM (slightly though) and burns extra compute for every guess it makes. N-gram Speculative Decoding gives you a massive speed boost for exactly 0 memory cost and minimal compute. And it's faster than MTP when it works. Here is how it actually works under the hood: Standard autoregressive decoding is slow because it predicts one token at a time. If you ask an agent to format a long JSON object or update one line in an HTML file, it runs heavy matrix multiplications to calculate the probability of every single bracket, space, and letter from scratch. N-gram changes the game. It acts as a lightweight caching system. Instead of running heavy neural network math to guess the next word, it uses a simple hash table. Whenever the LLM starts outputting a sequence of tokens that already exists anywhere in its context window, N-gram instantly recognizes the pattern. Because it is just doing lightning fast string matching, the compute cost is practically zero. It "fast forwards" through the text, drafting the boilerplate instantly from memory, and the main model just verifies it in parallel. Pure speed. Using quantized GGUFs from Unsloth via HuggingFace, I spun up DeepMind’s massive Gemma 4 26B A4B QAT MoE on a free Colab instance to test this. Just look at the raw benchmark data on code editing task: Without N-gram: [ Prompt: 638.6 t/s | Generation: 45.9 t/s ] With N-gram: [ Prompt: 601.9 t/s | Generation: 107.1 t/s ] Here is the exact llama.cpp CLI command to activate it. Notice we don't even need the --model-draft flag: ./llama-cli -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -n 6000 -c 12000 -ngl 99 -fa on --spec-type ngram-mod Stop waiting for your GPU to re calculate words it already knows. I’ve built a free, interactive, cell by cell Google Colab notebook that lets you test this live in your browser. You can literally chat with the model and watch the text generation speed absolutely fly on the second turn when you ask it to edit a file. There are additional parameters for ngram-mod that you can tune once you get it working with the single flag. Link to the free Colab Notebook is in the comments below. It walks you through the entire stack: pulling pre built llama.cpp CUDA binaries for Linux, fetching GGUFs from HuggingFace, and spinning up the inference engine with ngram-mod from scratch. Let me know if you have already tried ngram-mod

Alok

31,765 görüntüleme • 2 ay önce

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,622 görüntüleme • 2 yıl önce

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own

NO1ennn

40,079 görüntüleme • 3 ay önce

a new 8GB VRAM GPU dense Local LLM leader was born yesterday runs on: RTX 4060 / RTX 3070 / RTX 2080. any 8GB card Qwen 3.5 9B (dense) was the go to for 6-8GB VRAM builds. Gemma 4 12B QAT (dense) just changed that. same llama.cpp + cuda 13.2. i7 12700H. 16GB RAM. same -ngl 99 flags. same 48k context. unsloth gemma-4-12b-it-Q4_K_M.gguf → 15 tok/sec @ 48k ctx unsloth gemma-4-12B-it-qat-UD-Q4_K_XL.gguf → 32 tok/sec @ 48k ctx → 26 tok/sec @ 64k ctx 64k context is a big deal. Hermes 3 agent requires 64k minimum to run. you're now getting full hermes compatible context on a budget consumer GPU at 26 tok/sec locally. 2.1x faster on identical hardware. and here's the part that breaks your brain: the QAT-UD-Q4_K_XL is actually SMALLER than the Q4_K_M "XL" why? QAT = Quantization Aware Training Google didn't train the model first and compress it later they trained it to be quantized from day one the weights already know how to survive low precision that's why you get more quality per byte llamacpp flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 48000 -v fits in 8GB VRAM clean. no API. no cloud. no subscription. and this isn't even the MTP variant yet Gemma-4-E2B QAT runs on 3GB RAM, E4B on 5GB, 12B on 7GB, 26-A4B on 15GB and 31B on 18GB. I have benchmarked the 26b and 31b qat as well on a single RTX 4090, checkout the comments for details. If you have a 6GB or 8GB VRAM GPU, post your numbers. more benchmarks and configs coming soon

a new 8GB VRAM GPU dense Local LLM leader was born yesterday runs on: RTX 4060 / RTX 3070 / RTX 2080. any 8GB card Qwen 3.5 9B (dense) was the go to for 6-8GB VRAM builds. Gemma 4 12B QAT (dense) just changed that. same llama.cpp + cuda 13.2. i7 12700H. 16GB RAM. same -ngl 99 flags. same 48k context. unsloth gemma-4-12b-it-Q4_K_M.gguf → 15 tok/sec @ 48k ctx unsloth gemma-4-12B-it-qat-UD-Q4_K_XL.gguf → 32 tok/sec @ 48k ctx → 26 tok/sec @ 64k ctx 64k context is a big deal. Hermes 3 agent requires 64k minimum to run. you're now getting full hermes compatible context on a budget consumer GPU at 26 tok/sec locally. 2.1x faster on identical hardware. and here's the part that breaks your brain: the QAT-UD-Q4_K_XL is actually SMALLER than the Q4_K_M "XL" why? QAT = Quantization Aware Training Google didn't train the model first and compress it later they trained it to be quantized from day one the weights already know how to survive low precision that's why you get more quality per byte llamacpp flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 48000 -v fits in 8GB VRAM clean. no API. no cloud. no subscription. and this isn't even the MTP variant yet Gemma-4-E2B QAT runs on 3GB RAM, E4B on 5GB, 12B on 7GB, 26-A4B on 15GB and 31B on 18GB. I have benchmarked the 26b and 31b qat as well on a single RTX 4090, checkout the comments for details. If you have a 6GB or 8GB VRAM GPU, post your numbers. more benchmarks and configs coming soon

Alok

264,062 görüntüleme • 3 ay önce

Here’s two examples of what it ACTUALLY looks like to “pay homage” Taking someone’s bars word for word and people just giving it the “homage” pass, is the stupidest thing I’ve ever heard. Jacking somebody’s bars word for word and using it to diss them isn’t “homage” and that’s happened plenty. The only time you can really pay homage while using someone’s bars word for word, is if there’s a callback to it or leading into it. Can also pay homage by interpolating and adding your own style to the sound and bars. Side note, just cuz somebody interpolates something doesn’t mean it’s paying homage, millions of times it’s been used to do the exact opposite. Lastly, just cuz someone might change a few words doesn’t give it a pass. There’s so many variables that go into these things and tbh some people just don’t care. Example is Drake on “Chicago Freestyle” using Em’s “Superman”, the only lyric change was “B*tch” to “Woman”. Eminem might love the record and appreciate it, but Drake is still 10000000% Swagger Jackin😂

Here’s two examples of what it ACTUALLY looks like to “pay homage” Taking someone’s bars word for word and people just giving it the “homage” pass, is the stupidest thing I’ve ever heard. Jacking somebody’s bars word for word and using it to diss them isn’t “homage” and that’s happened plenty. The only time you can really pay homage while using someone’s bars word for word, is if there’s a callback to it or leading into it. Can also pay homage by interpolating and adding your own style to the sound and bars. Side note, just cuz somebody interpolates something doesn’t mean it’s paying homage, millions of times it’s been used to do the exact opposite. Lastly, just cuz someone might change a few words doesn’t give it a pass. There’s so many variables that go into these things and tbh some people just don’t care. Example is Drake on “Chicago Freestyle” using Em’s “Superman”, the only lyric change was “B*tch” to “Woman”. Eminem might love the record and appreciate it, but Drake is still 10000000% Swagger Jackin😂

Hip Hop Historian

695,021 görüntüleme • 4 ay önce

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx

raullen

22,768 görüntüleme • 3 ay önce

I freaked out when my WiFi router suddenly died. then realized my autonomous Hermes agent is running fully local, nothing stopped. Hermes Agent + Gemma 4 26B A4B QAT MoE, 100% local on my laptop, building my side projects while I scroll my phone zero API calls. zero cost. 100% private. fully offline. This might be the most satisfying thing I’ve watched in a while. last post: showed Hermes + local Gemma 4 26B pull off backtest a trading strategy. this time I asked it to develop something i'd use myself everyday: # A full unpacked extension with: - React side panel UI - Local llama.cpp backend (offline AI) - Live tab sync + status tracking - Auto context extraction via Readability.js Vision on Demand → captures viewport screenshots as compressed JPEGs Deterministic action system -> model outputs tokens -> directly controls page scrolling It planned everything first. Then started executing step by step. all i did was say 'ok'. only once. # What’s wild: - It reports back after every phase - Auto compresses context when nearing limits - Actualy, stays on track llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 --cache-type-k q8_0 --cache-type-v q8_0 --port 8080 # Performance on a single NVIDIA RTX 4060 (8GB VRAM) + 16 GB DDR4 RAM Gaming Laptop: - 300 tokens/sec prefill - 25+ tokens/sec decode More than usable for real dev workflows. This isn’t AI demo territory anymore. This is autonomous local software actually building things.

I freaked out when my WiFi router suddenly died. then realized my autonomous Hermes agent is running fully local, nothing stopped. Hermes Agent + Gemma 4 26B A4B QAT MoE, 100% local on my laptop, building my side projects while I scroll my phone zero API calls. zero cost. 100% private. fully offline. This might be the most satisfying thing I’ve watched in a while. last post: showed Hermes + local Gemma 4 26B pull off backtest a trading strategy. this time I asked it to develop something i'd use myself everyday: # A full unpacked extension with: - React side panel UI - Local llama.cpp backend (offline AI) - Live tab sync + status tracking - Auto context extraction via Readability.js Vision on Demand → captures viewport screenshots as compressed JPEGs Deterministic action system -> model outputs tokens -> directly controls page scrolling It planned everything first. Then started executing step by step. all i did was say 'ok'. only once. # What’s wild: - It reports back after every phase - Auto compresses context when nearing limits - Actualy, stays on track llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 --cache-type-k q8_0 --cache-type-v q8_0 --port 8080 # Performance on a single NVIDIA RTX 4060 (8GB VRAM) + 16 GB DDR4 RAM Gaming Laptop: - 300 tokens/sec prefill - 25+ tokens/sec decode More than usable for real dev workflows. This isn’t AI demo territory anymore. This is autonomous local software actually building things.

Alok

56,717 görüntüleme • 3 ay önce

$I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.$

I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.

Alok

36,617 görüntüleme • 2 ay önce

🔥👀 NYC UNPROVOKED ANTI-WHITE BROOKLYN BUS ATTACK. So now you can get found guilty for a word but left the roam the streets for a WORD & ASSAULT. A white 35 year old woman gets on a bus in Flatbush gets on the bus and sits down. The person next to her makes an ANTI WHITE comment … then punches her clean in the face for existing. This happened on July 16th and still no arrest and not a SINGLE LOCAL TV NEWS STORY. Let’s reverse the races and it would be wall to wall on every local NYC tv outlet. Just like Jack Posobiec predicted … “IT WILL BE OPEN SEASON ON WHITE PEOPLE”

🔥👀 NYC UNPROVOKED ANTI-WHITE BROOKLYN BUS ATTACK. So now you can get found guilty for a word but left the roam the streets for a WORD & ASSAULT. A white 35 year old woman gets on a bus in Flatbush gets on the bus and sits down. The person next to her makes an ANTI WHITE comment … then punches her clean in the face for existing. This happened on July 16th and still no arrest and not a SINGLE LOCAL TV NEWS STORY. Let’s reverse the races and it would be wall to wall on every local NYC tv outlet. Just like Jack Posobiec predicted … “IT WILL BE OPEN SEASON ON WHITE PEOPLE”

Johnny St.Pete

167,725 görüntüleme • 2 ay önce

THIS IS WHAT 3 YEARS OF OBSIDIAN NOTES SHOULD LOOK LIKE BY NOW what you see here is thousands of connections forming a structure that didn’t exist before this is exactly what karpathy built with the llm wiki pattern, every source rewires the whole thing and new structure emerges automatically instead of you manually linking notes for years knowledge that connects itself looks nothing like a productivity system bookmark this and send it to whoever still thinks obsidian is just for taking notes

THIS IS WHAT 3 YEARS OF OBSIDIAN NOTES SHOULD LOOK LIKE BY NOW what you see here is thousands of connections forming a structure that didn’t exist before this is exactly what karpathy built with the llm wiki pattern, every source rewires the whole thing and new structure emerges automatically instead of you manually linking notes for years knowledge that connects itself looks nothing like a productivity system bookmark this and send it to whoever still thinks obsidian is just for taking notes

leopardracer

25,325 görüntüleme • 3 ay önce

Your social posts used to die in 48 hours. But Google… just changed everything. Now you can connect your social media profiles with your Google Search Console, just like you do with your website, you can see which posts show up in Google, which keywords they rank for, and how many clicks they drive. And that flips the entire ROI of social content. Because a post in your social feeds peaked in a day or two. Then it's gone. But then that same post ranking in Google can keep working for months. And the thing is Instagram content has been indexed in search for about a year now. But you just couldn't measure any of it. Now you can. So I’m telling all clients to create content around questions people are actually typing into Google. I use AnswerThePublic for this. Type in your topic, and it shows every question your audience is asking. Then answer one question in every piece of content you create. Because now, every post has the potential to become more than a social post. It can become a measurable search asset that keeps bringing you attention, traffic, and customers long after you publish it.

Your social posts used to die in 48 hours. But Google… just changed everything. Now you can connect your social media profiles with your Google Search Console, just like you do with your website, you can see which posts show up in Google, which keywords they rank for, and how many clicks they drive. And that flips the entire ROI of social content. Because a post in your social feeds peaked in a day or two. Then it's gone. But then that same post ranking in Google can keep working for months. And the thing is Instagram content has been indexed in search for about a year now. But you just couldn't measure any of it. Now you can. So I’m telling all clients to create content around questions people are actually typing into Google. I use AnswerThePublic for this. Type in your topic, and it shows every question your audience is asking. Then answer one question in every piece of content you create. Because now, every post has the potential to become more than a social post. It can become a measurable search asset that keeps bringing you attention, traffic, and customers long after you publish it.

Neil Patel

93,558 görüntüleme • 1 ay önce

MegaBunnish is live ! I made a free tool (link in comment) for all of you, where you can : - Navigate easily the Megaeth ecosystem - Sort by category - Have my own opinion on every single app - Have my "farming" list - Create your favorites - Follow your activity visually And more to come... Hope you'll like it, and that it will help your for mainnet, hit my DM for any feedback ily <3

MegaBunnish is live ! I made a free tool (link in comment) for all of you, where you can : - Navigate easily the Megaeth ecosystem - Sort by category - Have my own opinion on every single app - Have my "farming" list - Create your favorites - Follow your activity visually And more to come... Hope you'll like it, and that it will help your for mainnet, hit my DM for any feedback ily <3

Joestar⭐

35,067 görüntüleme • 7 ay önce

small local model that falls apart in bloated agents like openclaw just runs like a wild horse in hermes agent. and that's not even my line, someone else called it that, i've just been quietly pointing people at this harness for months because it held up on everything i threw at it, 3b models all the way to one trillion params. watch this happen on my own machine. i pointed hermes agent at a local http endpoint, gemma 4 12b on my 3090 llama.cpp server, and it auto-detected the model and started working immediately. no config wrestling, no broken tool calls, no babysitting the output format, i typed in a url and it just went. the whole clip is exactly that, start to finish, no errors, no retries, butter smooth. and the tool calling, the one thing that quietly breaks most local setups, works here like it's nothing. it's not the model that's flaky, it's the harness around it. hermes agent is the first agent i've run that actually gets that right. one url, one local model on one card, and it runs like a wild horse.

small local model that falls apart in bloated agents like openclaw just runs like a wild horse in hermes agent. and that's not even my line, someone else called it that, i've just been quietly pointing people at this harness for months because it held up on everything i threw at it, 3b models all the way to one trillion params. watch this happen on my own machine. i pointed hermes agent at a local http endpoint, gemma 4 12b on my 3090 llama.cpp server, and it auto-detected the model and started working immediately. no config wrestling, no broken tool calls, no babysitting the output format, i typed in a url and it just went. the whole clip is exactly that, start to finish, no errors, no retries, butter smooth. and the tool calling, the one thing that quietly breaks most local setups, works here like it's nothing. it's not the model that's flaky, it's the harness around it. hermes agent is the first agent i've run that actually gets that right. one url, one local model on one card, and it runs like a wild horse.

Sudo su

27,339 görüntüleme • 3 ay önce

The reason Claude keeps disappointing you is not Claude - it’s you. Every session - explain yourself from scratch. Every session - it forgets you exist. Every session - same mistakes. Same corrections. Same waste. That’s not AI. That’s a goldfish with a keyboard. One file broke that loop permanently. 21 lines. That’s it. Now Claude knows his voice. His rules. His decisions. What failed. What worked. Where he left off. He doesn’t explain himself. He doesn’t repeat himself. He just builds. While you’re re-explaining your job for the 200th time - he already shipped. Tony Stark didn’t prompt J.A.R.V.I.S. He talked to it like it knew him. Because it did. This guy has that. In a wine bar. On a laptop. With a text file. 82,000 GitHub stars. #1 trending. And you still don’t have it set up. Bookmark this before you forget. Because you will forget. And tomorrow you’ll be explaining yourself to Claude again.

The reason Claude keeps disappointing you is not Claude - it’s you. Every session - explain yourself from scratch. Every session - it forgets you exist. Every session - same mistakes. Same corrections. Same waste. That’s not AI. That’s a goldfish with a keyboard. One file broke that loop permanently. 21 lines. That’s it. Now Claude knows his voice. His rules. His decisions. What failed. What worked. Where he left off. He doesn’t explain himself. He doesn’t repeat himself. He just builds. While you’re re-explaining your job for the 200th time - he already shipped. Tony Stark didn’t prompt J.A.R.V.I.S. He talked to it like it knew him. Because it did. This guy has that. In a wine bar. On a laptop. With a text file. 82,000 GitHub stars. #1 trending. And you still don’t have it set up. Bookmark this before you forget. Because you will forget. And tomorrow you’ll be explaining yourself to Claude again.

leopardracer

10,804 görüntüleme • 5 ay önce

this n8n automation has generated over $200k in sales for my clients... i use it every single day in my SEO agency and it works like a charm i'm sharing the complete workflow with JSON file and step-by-step implementation guide: - does keyword research for you - generates optimized articles from keywords list - saves content in google docs and sends updates through slack it's designed for absolute beginners, you can set it up even with zero n8n experience reply "CONTENT" + retweet and it's yours (follow me so i can dm)

this n8n automation has generated over $200k in sales for my clients... i use it every single day in my SEO agency and it works like a charm i'm sharing the complete workflow with JSON file and step-by-step implementation guide: - does keyword research for you - generates optimized articles from keywords list - saves content in google docs and sends updates through slack it's designed for absolute beginners, you can set it up even with zero n8n experience reply "CONTENT" + retweet and it's yours (follow me so i can dm)

Machina

96,703 görüntüleme • 1 yıl önce

Claude tokens can be saved by using this simple hack. Every reply, Claude rereads your entire chat from the top before responding. Silently burning your limit before a single word comes back. Someone on Reddit figured out a one-line fix that cuts usage by 75%. Same answers. Same AI. Just stops letting it talk more than it needs to. Exact prompt is pinned in the first comment. Free. I share updates like these in my free AI community on WhatsApp. Join here 👇

Claude tokens can be saved by using this simple hack. Every reply, Claude rereads your entire chat from the top before responding. Silently burning your limit before a single word comes back. Someone on Reddit figured out a one-line fix that cuts usage by 75%. Same answers. Same AI. Just stops letting it talk more than it needs to. Exact prompt is pinned in the first comment. Free. I share updates like these in my free AI community on WhatsApp. Join here 👇

Vaibhav Sisinty

22,408 görüntüleme • 5 ay önce

"The anthem we all knew word for word in the '90s" Sys This song was the soundtrack of so many moments - whether you were blasting it on a road trip, singing it at the top of your lungs with friends, or just living your best '90s life. Shania was a vibe that shaped a whole era of music, and we're still rocking out to her today. Who else remembers every single lyric? #ShaniaTwain #90sCountry #AnyManOfMine #ThrowbackVibes

"The anthem we all knew word for word in the '90s" Sys This song was the soundtrack of so many moments - whether you were blasting it on a road trip, singing it at the top of your lungs with friends, or just living your best '90s life. Shania was a vibe that shaped a whole era of music, and we're still rocking out to her today. Who else remembers every single lyric? #ShaniaTwain #90sCountry #AnyManOfMine #ThrowbackVibes

NancyH

10,947 görüntüleme • 6 ay önce