Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

llamacpp vs omlx with Gemma-4-26B-A4B-it 🔥 🥇 llamacpp 🥈 oMLX Notes: - 11 parallel tasks - qat-Q4_0 vs OptiQ-4bit on omlx. - M5 Max 128GB macOS 27 Beta 1, relevant read below. - Fork of Gemma cookbook repo used with oMLX and iTerm2 usage added. Here: Observations: - llamacpp...

ivanfioravanti's profile picture

Ivan Fioravanti ᯅ

42,121 subscribers

16,576 görüntüleme • 3 ay önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.
0:20

I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.

Alok

36,617 görüntüleme • 2 ay önce

a new 8GB VRAM GPU dense Local LLM leader was born yesterday runs on: RTX 4060 / RTX 3070 / RTX 2080. any 8GB card Qwen 3.5 9B (dense) was the go to for 6-8GB VRAM builds. Gemma 4 12B QAT (dense) just changed that. same llama.cpp + cuda 13.2. i7 12700H. 16GB RAM. same -ngl 99 flags. same 48k context. unsloth gemma-4-12b-it-Q4_K_M.gguf → 15 tok/sec @ 48k ctx unsloth gemma-4-12B-it-qat-UD-Q4_K_XL.gguf → 32 tok/sec @ 48k ctx → 26 tok/sec @ 64k ctx 64k context is a big deal. Hermes 3 agent requires 64k minimum to run. you're now getting full hermes compatible context on a budget consumer GPU at 26 tok/sec locally. 2.1x faster on identical hardware. and here's the part that breaks your brain: the QAT-UD-Q4_K_XL is actually SMALLER than the Q4_K_M "XL" why? QAT = Quantization Aware Training Google didn't train the model first and compress it later they trained it to be quantized from day one the weights already know how to survive low precision that's why you get more quality per byte llamacpp flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 48000 -v fits in 8GB VRAM clean. no API. no cloud. no subscription. and this isn't even the MTP variant yet Gemma-4-E2B QAT runs on 3GB RAM, E4B on 5GB, 12B on 7GB, 26-A4B on 15GB and 31B on 18GB. I have benchmarked the 26b and 31b qat as well on a single RTX 4090, checkout the comments for details. If you have a 6GB or 8GB VRAM GPU, post your numbers. more benchmarks and configs coming soon
0:22

a new 8GB VRAM GPU dense Local LLM leader was born yesterday runs on: RTX 4060 / RTX 3070 / RTX 2080. any 8GB card Qwen 3.5 9B (dense) was the go to for 6-8GB VRAM builds. Gemma 4 12B QAT (dense) just changed that. same llama.cpp + cuda 13.2. i7 12700H. 16GB RAM. same -ngl 99 flags. same 48k context. unsloth gemma-4-12b-it-Q4_K_M.gguf → 15 tok/sec @ 48k ctx unsloth gemma-4-12B-it-qat-UD-Q4_K_XL.gguf → 32 tok/sec @ 48k ctx → 26 tok/sec @ 64k ctx 64k context is a big deal. Hermes 3 agent requires 64k minimum to run. you're now getting full hermes compatible context on a budget consumer GPU at 26 tok/sec locally. 2.1x faster on identical hardware. and here's the part that breaks your brain: the QAT-UD-Q4_K_XL is actually SMALLER than the Q4_K_M "XL" why? QAT = Quantization Aware Training Google didn't train the model first and compress it later they trained it to be quantized from day one the weights already know how to survive low precision that's why you get more quality per byte llamacpp flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 48000 -v fits in 8GB VRAM clean. no API. no cloud. no subscription. and this isn't even the MTP variant yet Gemma-4-E2B QAT runs on 3GB RAM, E4B on 5GB, 12B on 7GB, 26-A4B on 15GB and 31B on 18GB. I have benchmarked the 26b and 31b qat as well on a single RTX 4090, checkout the comments for details. If you have a 6GB or 8GB VRAM GPU, post your numbers. more benchmarks and configs coming soon

Alok

264,062 görüntüleme • 3 ay önce

Very quick comparison between Ornith 1.5 35B MoE and Qwen 3.8 27B Dense. 📣 Clearly it's not a fair one, but let's in any case see how it went. 397B download in progress! In the Videos below: - Brick SA -> Qwen 3.8 - Lego Streets -> Ornith 1.5 Context: - Pi agent used in both cases, prompt below - M3 Ultra 512GB - Ornith-1.5-35B-A3B-oQ8e MTP hosted on oMLX - mlx-community/Qwen3.8-27B-8bit DFlash 2 hosted on mlx-dspark - Ornith has been incredibly fast with speed from 75 t/s to 45 t/s (160K+ context) - Qwen3.8 suffered context much more reaching 5 t/s above 160K context, but it can be due to engine tested still work in progress - Prompt: using threejs, and cdn, create a lego like game that's inspired by gta san andreas, with beautiful aesthetics and graphics and ability to steal cars. it should be 3d and have a nice and large map and areas. the graphics should be decent and nice. it should feature iconic things from gta san andreas Notes: - Qwen 3.8 result is 0-shot, while Ornith 1.5 required 6 iterations - Ornith 1.5 is not at the same level of autonomy as Qwen 3.8 27B honestly. To try getting the same results I'm constantly nudging, steering and 🤬 at it. - Qwen 3.8 took 6 hours to complete, but it was more a problem of timeout of Chrome headless used for testing, real prefill/decode TBD. I'll test again now with oMLX - Pi agent has a nearly perfect Cache Hit ratio that for local models is MEGA important!
0:31

Very quick comparison between Ornith 1.5 35B MoE and Qwen 3.8 27B Dense. 📣 Clearly it's not a fair one, but let's in any case see how it went. 397B download in progress! In the Videos below: - Brick SA -> Qwen 3.8 - Lego Streets -> Ornith 1.5 Context: - Pi agent used in both cases, prompt below - M3 Ultra 512GB - Ornith-1.5-35B-A3B-oQ8e MTP hosted on oMLX - mlx-community/Qwen3.8-27B-8bit DFlash 2 hosted on mlx-dspark - Ornith has been incredibly fast with speed from 75 t/s to 45 t/s (160K+ context) - Qwen3.8 suffered context much more reaching 5 t/s above 160K context, but it can be due to engine tested still work in progress - Prompt: using threejs, and cdn, create a lego like game that's inspired by gta san andreas, with beautiful aesthetics and graphics and ability to steal cars. it should be 3d and have a nice and large map and areas. the graphics should be decent and nice. it should feature iconic things from gta san andreas Notes: - Qwen 3.8 result is 0-shot, while Ornith 1.5 required 6 iterations - Ornith 1.5 is not at the same level of autonomy as Qwen 3.8 27B honestly. To try getting the same results I'm constantly nudging, steering and 🤬 at it. - Qwen 3.8 took 6 hours to complete, but it was more a problem of timeout of Chrome headless used for testing, real prefill/decode TBD. I'll test again now with oMLX - Pi agent has a nearly perfect Cache Hit ratio that for local models is MEGA important!

Ivan Fioravanti ᯅ

12,150 görüntüleme • 1 ay önce

A single RTX 4090 (24 GB VRAM) can run the updated gemma 4 31B (dense) model with a 190,000 context window at 33 tokens/second. The VRAM barrier is dying. Google quietly updated Gemma 4, and Unsloth immediately compiled the new quants. I built llama.cpp from source on Ubuntu 22 to benchmark it. Google's stealth update 2 days ago enabled uniform Flash Attention 4 on Hopper to boost prefill and patched the chat template to improve tool calling. The agentic reasoning gains on the benchmark charts are massive: TB2 (Agents): +4.5% (to 25.8%) Tau2 (Telecom): +10.1% (to 62.7%) Running on Ubuntu 22, CUDA 13.0 with a single NVIDIA GeForce RTX 4090. Here is the exact step by step benchmarking process with a massive 28k tokens prompt and the commands I used to squeeze out maximum context without killing my throughput: # 1. The Baseline (Unquantized KV Cache) I started with full GPU offload (-ngl 99) and pushed the context to 40k. llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -ngl 99 -c 40000 -fa on --port 8080 -v VRAM: 23.8 GB (maxed out on card) Throughput: Prefill: 2198.81 t/s | Decode: 35.77 t/s (with 28k tokens prompt) # 2. The CPU Split Trap I tried stretching to 80k context by offloading layers to the CPU (-ngl 52). llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 80000 -ngl 52 -fa on --port 8080 -v Throughput: Prefill: 1212.73 t/s | Decode: 5 t/s (with 28k tokens prompt) # 3. The KV Quantization Breakthrough Instead of spilling layers to the CPU, I kept the model fully on card (-ngl 99) but enabled 8-bit KV cache quantization to free up VRAM. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 100000 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 --port 8080 -v VRAM: 23.9 GB Throughput: Prefill: 2139.68 t/s | Decode: 32 t/s (with 28k tokens prompt) Result: 100k tokens of context on a single GPU with practically zero speed loss (and minimal intelligence loss). # 4. The Limit Test (Q4 KV Cache) To find the absolute breaking point, I dropped the KV cache to 4 bit (q4_0) and set -c 190000. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 190000 --cache-type-k q4_0 --cache-type-v q4_0 -ngl 99 --port 8080 -v VRAM: 23.8 GB Throughput: Prefill: 2206.66 t/s | Decode: 33 t/s (with 28k tokens prompt) (Note: Pushing it to 220k required dropping to -ngl 58 again, which immediately penalized decode down to 17 t/s). # The Tradeoff: For Max Reasoning: Keep your KV cache unquantized (f16). You get pristine reasoning but hit a strict 40k context ceiling. For Massive Document Retrieval: If you need to feed the model giant codebases, use --cache-type-k q4_0. Getting 190k context at 33 tokens/second on a consumer desktop with a 31b dense model is a cheat code. If you’re rocking a single 3090 or 4090 and slept on Gemma 4 earlier, this update is your cue to dust off the terminal. Hugging Face links to the Unsloth QAT quants are in the replies below.
0:20

A single RTX 4090 (24 GB VRAM) can run the updated gemma 4 31B (dense) model with a 190,000 context window at 33 tokens/second. The VRAM barrier is dying. Google quietly updated Gemma 4, and Unsloth immediately compiled the new quants. I built llama.cpp from source on Ubuntu 22 to benchmark it. Google's stealth update 2 days ago enabled uniform Flash Attention 4 on Hopper to boost prefill and patched the chat template to improve tool calling. The agentic reasoning gains on the benchmark charts are massive: TB2 (Agents): +4.5% (to 25.8%) Tau2 (Telecom): +10.1% (to 62.7%) Running on Ubuntu 22, CUDA 13.0 with a single NVIDIA GeForce RTX 4090. Here is the exact step by step benchmarking process with a massive 28k tokens prompt and the commands I used to squeeze out maximum context without killing my throughput: # 1. The Baseline (Unquantized KV Cache) I started with full GPU offload (-ngl 99) and pushed the context to 40k. llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -ngl 99 -c 40000 -fa on --port 8080 -v VRAM: 23.8 GB (maxed out on card) Throughput: Prefill: 2198.81 t/s | Decode: 35.77 t/s (with 28k tokens prompt) # 2. The CPU Split Trap I tried stretching to 80k context by offloading layers to the CPU (-ngl 52). llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 80000 -ngl 52 -fa on --port 8080 -v Throughput: Prefill: 1212.73 t/s | Decode: 5 t/s (with 28k tokens prompt) # 3. The KV Quantization Breakthrough Instead of spilling layers to the CPU, I kept the model fully on card (-ngl 99) but enabled 8-bit KV cache quantization to free up VRAM. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 100000 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 --port 8080 -v VRAM: 23.9 GB Throughput: Prefill: 2139.68 t/s | Decode: 32 t/s (with 28k tokens prompt) Result: 100k tokens of context on a single GPU with practically zero speed loss (and minimal intelligence loss). # 4. The Limit Test (Q4 KV Cache) To find the absolute breaking point, I dropped the KV cache to 4 bit (q4_0) and set -c 190000. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 190000 --cache-type-k q4_0 --cache-type-v q4_0 -ngl 99 --port 8080 -v VRAM: 23.8 GB Throughput: Prefill: 2206.66 t/s | Decode: 33 t/s (with 28k tokens prompt) (Note: Pushing it to 220k required dropping to -ngl 58 again, which immediately penalized decode down to 17 t/s). # The Tradeoff: For Max Reasoning: Keep your KV cache unquantized (f16). You get pristine reasoning but hit a strict 40k context ceiling. For Massive Document Retrieval: If you need to feed the model giant codebases, use --cache-type-k q4_0. Getting 190k context at 33 tokens/second on a consumer desktop with a 31b dense model is a cheat code. If you’re rocking a single 3090 or 4090 and slept on Gemma 4 earlier, this update is your cue to dust off the terminal. Hugging Face links to the Unsloth QAT quants are in the replies below.

Alok

76,069 görüntüleme • 2 ay önce

Run Updated Gemma 4 26B A4B QAT (MoE) with Vision at 25 tokens/sec and massive 120k context window on a single RTX 4060 (8 GB VRAM + 16 GB RAM Only!!) Yesterday I pushed Gemma 4 26B A4B QAT to 250k context on a single RTX 4060 using nothing but Q8 KV cache and optimized -b and -ub flags for higher prefill throughput. Today I stacked Multi Token Prediction (MTP) self speculative decoding AND the vision projector (mmproj) on top of that same card, same batch size optimization, same $250 GPU and pushed it until it broke, then found the fix. All text only runs consist of a 28k prompt. vision runs consist of 28k text prompt + an image. # 1. MTP alone. near free decode speed, no catch MTP draft assistant is a separate small model (MTP heads are backed into the main model itself for the qwen 3.5+ models but its a separate small model for gemma 4 series), 240 MB gguf 80k ctx: Prefill 510 t/s | Decode 29.5 t/s 120k ctx: Prefill 433 t/s | Decode 29 t/s 180k ctx: Prefill 240 t/s | Decode 24.9 t/s 250k ctx: Prefill 63 t/s | Decode 13 t/s llama.cpp flags: m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --spec-type draft-mtp -md mtp-gemma-4-26B-A4B-it.gguf-c 180000 -b 1024 -ub 1024 --spec-draft-n-max 6 --spec-draft-p-min 0.7 -ctk q8_0 -ctv q8_0 # 2. Add vision on top. the tax you actually pay the vision projector gguf is about 1.1 GBs 80k ctx: Prefill 360 t/s | Decode 25.4 t/s 120k ctx: Prefill 230 t/s | Decode 23.8 t/s 180k ctx (Q8 KV): Prefill 75 t/s | Decode 12.5 t/s - cliff flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --spec-type draft-mtp -md mtp-gemma-4-26B-A4B-it.gguf -c 80000 --port 8080 -b 1024 -ub 1024 --spec-draft-n-max 6 --spec-draft-p-min 0.7 -ctk q8_0 -ctv q8_0 --mmproj mmproj-F16.gguf # 3. The fix if you want to run vision over 120k context: swap Q8 KV for Q4 KV past 120k Stack MTP + vision + Q8 KV past 120k context and you hit a wall. draft model overhead plus KV pressure tanks everything. Drop to Q4 KV and the wall disappears: 180k ctx (Q4 KV): Prefill 220 t/s | Decode 25.5 t/s -ctk q4_0 -ctv q4_0 --mmproj mmproj-F16.gguf (rest same as above) Bottom line: MTP gives you a near free +20-30% decode boost up to 120k context. Past that, it's fighting your VRAM, not helping and if vision is loaded too, Q4 KV isn't optional past 120k, it's mandatory. 30% boost is model and card specific, MTP boosted decode 2x for gemma 4 31b on a single rtx 4090. Same 8GB card. Same $250 GPU. Multimodal, speculative decoding, 180k usable context, zero upgrades. You gotta try this if you have a single NVIDIA RTX 3050, 3060, 3070, 4050, 4060, 5050 or 5060. You can try it with a 6 GB VRAM card as well but you will have to lower the context window. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. Which models are you running on your 6/8/12GB cards with MTP?
0:20

Run Updated Gemma 4 26B A4B QAT (MoE) with Vision at 25 tokens/sec and massive 120k context window on a single RTX 4060 (8 GB VRAM + 16 GB RAM Only!!) Yesterday I pushed Gemma 4 26B A4B QAT to 250k context on a single RTX 4060 using nothing but Q8 KV cache and optimized -b and -ub flags for higher prefill throughput. Today I stacked Multi Token Prediction (MTP) self speculative decoding AND the vision projector (mmproj) on top of that same card, same batch size optimization, same $250 GPU and pushed it until it broke, then found the fix. All text only runs consist of a 28k prompt. vision runs consist of 28k text prompt + an image. # 1. MTP alone. near free decode speed, no catch MTP draft assistant is a separate small model (MTP heads are backed into the main model itself for the qwen 3.5+ models but its a separate small model for gemma 4 series), 240 MB gguf 80k ctx: Prefill 510 t/s | Decode 29.5 t/s 120k ctx: Prefill 433 t/s | Decode 29 t/s 180k ctx: Prefill 240 t/s | Decode 24.9 t/s 250k ctx: Prefill 63 t/s | Decode 13 t/s llama.cpp flags: m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --spec-type draft-mtp -md mtp-gemma-4-26B-A4B-it.gguf-c 180000 -b 1024 -ub 1024 --spec-draft-n-max 6 --spec-draft-p-min 0.7 -ctk q8_0 -ctv q8_0 # 2. Add vision on top. the tax you actually pay the vision projector gguf is about 1.1 GBs 80k ctx: Prefill 360 t/s | Decode 25.4 t/s 120k ctx: Prefill 230 t/s | Decode 23.8 t/s 180k ctx (Q8 KV): Prefill 75 t/s | Decode 12.5 t/s - cliff flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --spec-type draft-mtp -md mtp-gemma-4-26B-A4B-it.gguf -c 80000 --port 8080 -b 1024 -ub 1024 --spec-draft-n-max 6 --spec-draft-p-min 0.7 -ctk q8_0 -ctv q8_0 --mmproj mmproj-F16.gguf # 3. The fix if you want to run vision over 120k context: swap Q8 KV for Q4 KV past 120k Stack MTP + vision + Q8 KV past 120k context and you hit a wall. draft model overhead plus KV pressure tanks everything. Drop to Q4 KV and the wall disappears: 180k ctx (Q4 KV): Prefill 220 t/s | Decode 25.5 t/s -ctk q4_0 -ctv q4_0 --mmproj mmproj-F16.gguf (rest same as above) Bottom line: MTP gives you a near free +20-30% decode boost up to 120k context. Past that, it's fighting your VRAM, not helping and if vision is loaded too, Q4 KV isn't optional past 120k, it's mandatory. 30% boost is model and card specific, MTP boosted decode 2x for gemma 4 31b on a single rtx 4090. Same 8GB card. Same $250 GPU. Multimodal, speculative decoding, 180k usable context, zero upgrades. You gotta try this if you have a single NVIDIA RTX 3050, 3060, 3070, 4050, 4060, 5050 or 5060. You can try it with a 6 GB VRAM card as well but you will have to lower the context window. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. Which models are you running on your 6/8/12GB cards with MTP?

Alok

16,405 görüntüleme • 2 ay önce

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.
0:20

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

Alok

62,381 görüntüleme • 3 ay önce

Michelangelo Antonioni on his experience working in America, the tendency of Americans to waste things & the difficulties he faced while working with American Cameramen: "My autonomy, I would say, is complete. They leave me free to do what I want. The only thing now is that they are starting to be worried about the budget. They ask me why the film is so expensive, but that's what I'm going to ask them. I don't know why it's so expensive, I really don't know. I have a crew that is only half the size of what is usual in America because I don't want a big crew. It seems to me I'm seeing such a waste of money. It seems to be almost immoral. I feel bad sometimes. For instance, they threw away a piece of gelatin that we used and the piece was still new. There is also a waste of film. If I am shooting with two or three cameras and I need this camera only in the middle of the scene, in Italy I would start with that camera just at that moment. Not here. They start from the beginning. It is a waste of film! They are consumers. They are used to wasting something — goods, materials, food, everything. And I'm not used to this. I have some elements of this crew who are not so young — some older people who worked, for instance, on the first film of Greta Garbo, the first 'Ben-Hur' (1925), on Stroheim's 'Greed' (1924). It is very amusing to talk to them about these things. I had a lot of problems with unions, and in Hollywood this is much more difficult than in New York. The unions are so strong. There are lots of old people. You can't find what you are looking for. I needed some cameramen and some assistant cameramen —some young people used to shooting in the modern way, people who can zoom without your having to tell them the exact distance from the actors to the camera, people who can make changes on their own and who can sometimes do what they want. I want them to do that, to do something different from the script, maybe. At that critical moment, the assistant cameraman has to make the decision, but the American cameraman can't do this. That's why I was forced to bring some people from Italy. I don't know why, but English people are much more familiar to me. At the time I was making 'Blow-Up' (1966), at least, they were so mad —in a positive and very pleasant way —that they were almost Neapolitan (that's a joke). I like them, I like English people much more than Americans. I mean I find myself closer to them. The Americans are so cool sometimes. They need to know exactly what they are doing. Sometimes they are like Germans —fastidious, precise. This makes me upset because I like to have people around me who are more spontaneous." (Michelangelo Antonioni's interview with Marsha Kinder, 1968) On this day, 56 years ago, 'Zabriskie Point' (1970) premiered in New York City, USA.
2:10

Michelangelo Antonioni on his experience working in America, the tendency of Americans to waste things & the difficulties he faced while working with American Cameramen: "My autonomy, I would say, is complete. They leave me free to do what I want. The only thing now is that they are starting to be worried about the budget. They ask me why the film is so expensive, but that's what I'm going to ask them. I don't know why it's so expensive, I really don't know. I have a crew that is only half the size of what is usual in America because I don't want a big crew. It seems to me I'm seeing such a waste of money. It seems to be almost immoral. I feel bad sometimes. For instance, they threw away a piece of gelatin that we used and the piece was still new. There is also a waste of film. If I am shooting with two or three cameras and I need this camera only in the middle of the scene, in Italy I would start with that camera just at that moment. Not here. They start from the beginning. It is a waste of film! They are consumers. They are used to wasting something — goods, materials, food, everything. And I'm not used to this. I have some elements of this crew who are not so young — some older people who worked, for instance, on the first film of Greta Garbo, the first 'Ben-Hur' (1925), on Stroheim's 'Greed' (1924). It is very amusing to talk to them about these things. I had a lot of problems with unions, and in Hollywood this is much more difficult than in New York. The unions are so strong. There are lots of old people. You can't find what you are looking for. I needed some cameramen and some assistant cameramen —some young people used to shooting in the modern way, people who can zoom without your having to tell them the exact distance from the actors to the camera, people who can make changes on their own and who can sometimes do what they want. I want them to do that, to do something different from the script, maybe. At that critical moment, the assistant cameraman has to make the decision, but the American cameraman can't do this. That's why I was forced to bring some people from Italy. I don't know why, but English people are much more familiar to me. At the time I was making 'Blow-Up' (1966), at least, they were so mad —in a positive and very pleasant way —that they were almost Neapolitan (that's a joke). I like them, I like English people much more than Americans. I mean I find myself closer to them. The Americans are so cool sometimes. They need to know exactly what they are doing. Sometimes they are like Germans —fastidious, precise. This makes me upset because I like to have people around me who are more spontaneous." (Michelangelo Antonioni's interview with Marsha Kinder, 1968) On this day, 56 years ago, 'Zabriskie Point' (1970) premiered in New York City, USA.

DepressedBergman

39,982 görüntüleme • 7 ay önce

🤯 170G PROTEIN! HIGH-PROTEIN CHICKEN POTATO SALAD Divide into 5 HUGE servings: 300 cals, 34g protein each Plus, you have OPTIONS with how you prep it. You can absolutely boil the potatoes, carrots, eggs and use rotisserie chicken. I made everything in my Anyday microwave cookware because it cuts down on the cooking time and number of pots I have to deal with. You can prep, cook, serve, and store leftovers in these. This is my high protein take on the classic Russian salad. I bumped up the chicken, pulled back on potatoes, and used a mix of Greek yogurt and went light mayo. You still get the classic, but with a lot more protein and fewer calories. Here is how I made it: 1. Start with 3/4 lb baby potatoes and 2 medium carrots. Boil in salted water until fork tender, then drain and let cool. For an easier option, I added them to my large Anyday dish and microwaved for 8 mins. 2. For the chicken, you can keep things simple with 1 lb rotisserie or leftover chicken. I seasoned 1 lb chicken with salt, pepper, & 1 tbsp onion powder and cooked them + let them cool before dicing. You can pan fry or air fry them, or I added mine to my medium Anyday dish and microwaved for 9 mins. 3. For the eggs, boil 4 eggs for 9 mins, then cool and peel. Or, using my Anyday, I cracked the eggs directly into the small dish and microwaved for about 3.5 mins, cooking the final minute in 30-second increments until set. then dice the potatoes, carrots, chicken, and eggs and add to a large bowl. Add 1 cup finely diced baby dill pickles and 3/4 cup green peas. 5.Add 1 cup Greek yogurt, 4-5 tbsp light mayo, 1 tbsp Dijon mustard, the juice of 1/2 a large lemon, and 1-2 tbsp pickle juice. You can always add more later once you taste. 6.Season generously with salt & pepper, then mix, taste, & adjust. This is one of those dishes that gets even better once everything has had some time to come together in the fridge. ENJOY! . . . . #anydaypartner #highproteinrecipes #healthyrecipes #goodmoodfood #chickensalad
0:45

🤯 170G PROTEIN! HIGH-PROTEIN CHICKEN POTATO SALAD Divide into 5 HUGE servings: 300 cals, 34g protein each Plus, you have OPTIONS with how you prep it. You can absolutely boil the potatoes, carrots, eggs and use rotisserie chicken. I made everything in my Anyday microwave cookware because it cuts down on the cooking time and number of pots I have to deal with. You can prep, cook, serve, and store leftovers in these. This is my high protein take on the classic Russian salad. I bumped up the chicken, pulled back on potatoes, and used a mix of Greek yogurt and went light mayo. You still get the classic, but with a lot more protein and fewer calories. Here is how I made it: 1. Start with 3/4 lb baby potatoes and 2 medium carrots. Boil in salted water until fork tender, then drain and let cool. For an easier option, I added them to my large Anyday dish and microwaved for 8 mins. 2. For the chicken, you can keep things simple with 1 lb rotisserie or leftover chicken. I seasoned 1 lb chicken with salt, pepper, & 1 tbsp onion powder and cooked them + let them cool before dicing. You can pan fry or air fry them, or I added mine to my medium Anyday dish and microwaved for 9 mins. 3. For the eggs, boil 4 eggs for 9 mins, then cool and peel. Or, using my Anyday, I cracked the eggs directly into the small dish and microwaved for about 3.5 mins, cooking the final minute in 30-second increments until set. then dice the potatoes, carrots, chicken, and eggs and add to a large bowl. Add 1 cup finely diced baby dill pickles and 3/4 cup green peas. 5.Add 1 cup Greek yogurt, 4-5 tbsp light mayo, 1 tbsp Dijon mustard, the juice of 1/2 a large lemon, and 1-2 tbsp pickle juice. You can always add more later once you taste. 6.Season generously with salt & pepper, then mix, taste, & adjust. This is one of those dishes that gets even better once everything has had some time to come together in the fridge. ENJOY! . . . . #anydaypartner #highproteinrecipes #healthyrecipes #goodmoodfood #chickensalad

Shredhappens

13,526 görüntüleme • 1 ay önce

🍰2 INGREDIENT VIRAL CHEESCAKE Will it be good? You can get 4-6 servings out of this easily. 4 huge servings would be 288 calories, 21.5g protein. 6 servings would be 193 cals, 14g protein per serving. I gotta say, I was skeptical about this one. I dont think it will work with any cracker, so I would recommend using either the biscoff crackers or graham crackers. But use any cracker you like. And while Ive seen people make it with plain yogurt, I do think you either need to use your favorite flavored protein, or if you are trying to keep it lowcarb, mix in some powdered sweetener and a little vanilla extract to make it work like me. Overall, very good! If you use a thick greek yogurt, you can get a ton of protein, or you can use skyr or siggis like I did, and have an easy snack with 20g protein a serving! 1.Grab your favorite thick yogurt. I used 24oz of skyr yogurt. I mixed in 2 tbsp powder monk fruit, a splash of vanilla extract, and 1/4 cup coffee. Make sure you mix this really well, taste and adjust as you wish. If you want to keep this as simple as possible, grab a thick vanilla greek yogurt. Dont use plain, unless you want to be disappointed :) 2.Remove about 4 tablespoons of the yogurt because you’re going to be adding some crackers in and its going to overflow. You want to be able to be able to put the lid on. 3.Add in as many crackers as you want. You can use a lower carb cracker here, or whatever savory cracker you prefer. This is going to work best with biscoff crackers for the best possible “cheesecake” type texture, but you can also try it with a lower sugar cracker, or even graham crackers. 4.Put the lid on, and pop it in the fridge for at least 12 hours or overnight. 5.Remove, and enjoy. You can top this with some cacao powder or top with crushed crackers or whatever you like! If you make it, be sure to let me know what you think on SHREDHAPPENS. And if you want more healthy recipes and high protein recipes, follow along. ENJOY! . . . . . . #highprotein #healthyrecipes #goodmoodfood #healthyfoodshare #whatsonmyplate
0:27

🍰2 INGREDIENT VIRAL CHEESCAKE Will it be good? You can get 4-6 servings out of this easily. 4 huge servings would be 288 calories, 21.5g protein. 6 servings would be 193 cals, 14g protein per serving. I gotta say, I was skeptical about this one. I dont think it will work with any cracker, so I would recommend using either the biscoff crackers or graham crackers. But use any cracker you like. And while Ive seen people make it with plain yogurt, I do think you either need to use your favorite flavored protein, or if you are trying to keep it lowcarb, mix in some powdered sweetener and a little vanilla extract to make it work like me. Overall, very good! If you use a thick greek yogurt, you can get a ton of protein, or you can use skyr or siggis like I did, and have an easy snack with 20g protein a serving! 1.Grab your favorite thick yogurt. I used 24oz of skyr yogurt. I mixed in 2 tbsp powder monk fruit, a splash of vanilla extract, and 1/4 cup coffee. Make sure you mix this really well, taste and adjust as you wish. If you want to keep this as simple as possible, grab a thick vanilla greek yogurt. Dont use plain, unless you want to be disappointed :) 2.Remove about 4 tablespoons of the yogurt because you’re going to be adding some crackers in and its going to overflow. You want to be able to be able to put the lid on. 3.Add in as many crackers as you want. You can use a lower carb cracker here, or whatever savory cracker you prefer. This is going to work best with biscoff crackers for the best possible “cheesecake” type texture, but you can also try it with a lower sugar cracker, or even graham crackers. 4.Put the lid on, and pop it in the fridge for at least 12 hours or overnight. 5.Remove, and enjoy. You can top this with some cacao powder or top with crushed crackers or whatever you like! If you make it, be sure to let me know what you think on SHREDHAPPENS. And if you want more healthy recipes and high protein recipes, follow along. ENJOY! . . . . . . #highprotein #healthyrecipes #goodmoodfood #healthyfoodshare #whatsonmyplate

Shredhappens

13,293 görüntüleme • 2 ay önce

I just compared Claude Code vs Codex vs Cursor CLI The task was to build a Next.js app with Tailwind 4 and shadcn components to collect customer feedback and showcase it with a widget. I gave all three the same prompt and let them go for 30 minutes to see what they came up with. Claude Code with Opus 4.1 Even though I told it to set up the app in the existing project folder, it tried to create a directory for it. After I interrupted and told it not to do that, it built a demo form and landing page with no errors. I had to ask it to make the demo interactive so users could submit a testimonial and preview it. The landing page looked like AI and was pretty basic, but it worked and it was done in a fraction of the time of the others. Total tokens used: 33k Codex with GPT-5 At the end of the 30 minutes I just could not get Codex to produce a working app. It got stuck in a loop of not being able to set up Tailwind 4 and despite many, MANY, attempts, I ended up with a "failed to compile" error. Total tokens used: 102k Cursor Agent with GPT-5 This was the slowest agent by far and a couple of times I actually thought it got stuck in a loop and was close to Ctrl+C'ing to cancel it. The TUI is really nice though, especially how it shows diffs and it did eventually build a working app (after one or two slight errors that needed fixing) The demo was interactive and it had a very minimal design that looked bare but also a lot less like an "AI generated" app than the Opus 4.1 design. It also wasn't too chatty and just did what it needed to do! Code quality was on a par with Opus 4.1, but it did use 5.5x as many tokens to get there. Still cheaper than Opus on a direct comparison but not when you factor in a Claude Code Max subscription. Total tokens: 188k I'll be able to do a proper comparison and record some videos when I'm back from holiday but for now, Opus is still the more capable model out of the box and Claude Code is the more complete CLI product. It will be interesting to see how Cursor evolve their CLI though with commands and subagents because I think with GPT-5 they have a real shot at providing competition for Claude Code if they can optimise output to get similar quality with less tokens. Jump to 0:40 in the video to see the two apps. Which do you think is which? ;)
1:18

I just compared Claude Code vs Codex vs Cursor CLI The task was to build a Next.js app with Tailwind 4 and shadcn components to collect customer feedback and showcase it with a widget. I gave all three the same prompt and let them go for 30 minutes to see what they came up with. Claude Code with Opus 4.1 Even though I told it to set up the app in the existing project folder, it tried to create a directory for it. After I interrupted and told it not to do that, it built a demo form and landing page with no errors. I had to ask it to make the demo interactive so users could submit a testimonial and preview it. The landing page looked like AI and was pretty basic, but it worked and it was done in a fraction of the time of the others. Total tokens used: 33k Codex with GPT-5 At the end of the 30 minutes I just could not get Codex to produce a working app. It got stuck in a loop of not being able to set up Tailwind 4 and despite many, MANY, attempts, I ended up with a "failed to compile" error. Total tokens used: 102k Cursor Agent with GPT-5 This was the slowest agent by far and a couple of times I actually thought it got stuck in a loop and was close to Ctrl+C'ing to cancel it. The TUI is really nice though, especially how it shows diffs and it did eventually build a working app (after one or two slight errors that needed fixing) The demo was interactive and it had a very minimal design that looked bare but also a lot less like an "AI generated" app than the Opus 4.1 design. It also wasn't too chatty and just did what it needed to do! Code quality was on a par with Opus 4.1, but it did use 5.5x as many tokens to get there. Still cheaper than Opus on a direct comparison but not when you factor in a Claude Code Max subscription. Total tokens: 188k I'll be able to do a proper comparison and record some videos when I'm back from holiday but for now, Opus is still the more capable model out of the box and Claude Code is the more complete CLI product. It will be interesting to see how Cursor evolve their CLI though with commands and subagents because I think with GPT-5 they have a real shot at providing competition for Claude Code if they can optimise output to get similar quality with less tokens. Jump to 0:40 in the video to see the two apps. Which do you think is which? ;)

Ian Nuttall

195,173 görüntüleme • 1 yıl önce

Ivermectin and Fenbendazole seems to be coming up trumps in studies. Not the first story ive heard. But dosage is important. If you're taking the paste ( Ivermectin ) then no more than the size of your pinky nail, a day for several days then take a break for a week. Get advice if youve got something chronic. Viruses improve fast from what Ive seen. But im no doctor so watch for nausea and reduce accordingly. Talk to people that have had good results. It put the colour back in my face in a day when I was sick in 2023. Find the studies and keep dose small. It was used routinely until C**id. Then they stopped its availabilty. You gotta wonder why. Eugenics? Sadly its a thing. But Ivermectin won a Nobel prize in 2015 by all accounts. Arthritus and Cancer have similar pathways and its been known to help Arthritus. But again be careful. Several peeps i know had good results. They tried it because they were really crook. It is a pharmaceutical though. I didnt but I recommend getting advice from NZDSOS. Thats New Zealand Doctors speaking out with Science. A body I trust. I wish i had tried this with Sam my Cat who had a cancer. The dose wouldve had to be soooo small, less than a mill ( all worked out on weight ) but the Vets had no answers so ....The problem is they cant make money on it like they can with new drugs. Yes it kills parasites and Parasites have been linked with serious disease so do some research. Talk to NZDSOS. Or find a doctor you trust and can work with. This is anecdotal so no responsibility here just sharing my experience with it but this guy in this post had nothing to lose.
8:59

Ivermectin and Fenbendazole seems to be coming up trumps in studies. Not the first story ive heard. But dosage is important. If you're taking the paste ( Ivermectin ) then no more than the size of your pinky nail, a day for several days then take a break for a week. Get advice if youve got something chronic. Viruses improve fast from what Ive seen. But im no doctor so watch for nausea and reduce accordingly. Talk to people that have had good results. It put the colour back in my face in a day when I was sick in 2023. Find the studies and keep dose small. It was used routinely until C**id. Then they stopped its availabilty. You gotta wonder why. Eugenics? Sadly its a thing. But Ivermectin won a Nobel prize in 2015 by all accounts. Arthritus and Cancer have similar pathways and its been known to help Arthritus. But again be careful. Several peeps i know had good results. They tried it because they were really crook. It is a pharmaceutical though. I didnt but I recommend getting advice from NZDSOS. Thats New Zealand Doctors speaking out with Science. A body I trust. I wish i had tried this with Sam my Cat who had a cancer. The dose wouldve had to be soooo small, less than a mill ( all worked out on weight ) but the Vets had no answers so ....The problem is they cant make money on it like they can with new drugs. Yes it kills parasites and Parasites have been linked with serious disease so do some research. Talk to NZDSOS. Or find a doctor you trust and can work with. This is anecdotal so no responsibility here just sharing my experience with it but this guy in this post had nothing to lose.

skymeds store | Online Ivermectin Pharmacy

3,276,158 görüntüleme • 1 yıl önce

This is a standard practice for almost all Tier-1 banking applications in Nigeria, and for some fintech applications I’ve previously performed pentests on. Client-side encryption isn’t a total waste, or a waste of compute, as some people have claimed, but rather a measure to protect against API tampering or API request/response manipulation between the client and the server when implemented properly. Even with HTTPS, attackers can capture a decrypted version of web or mobile API data in transit because the browser and the server establish a level of trust during the TLS handshake. Attackers can leverage this trust to capture & proxy already-decrypted traffic, tamper with it, and then forward it to the server. This allows them to override what the user interface or client is originally supposed to send and replace it with data of their choosing. That is why validation needs to be performed on both the client and the server side. To wrap up, encrypting API requests and responses makes it significantly harder for attackers to tamper with data, even if they capture the traffic, unless they have access to the encryption details (algorithm, encryption mode, key size, secret key, and initialization vector), assuming asymmetric encryption is used. In the demo below, you can see how I discovered additional parameters (balance, is_admin) in the API response, captured the registration API request, despite it being sent over HTTPS from the interface, added the discovered parameters, and successfully inflated my balance to 50 billion and also escalated my privileges to admin, and ultimately deleted the accounts of two live users/customers. In the second slide, I captured an API traffic of a bank app, and you can see how difficult the payloads are to read.
2:38

This is a standard practice for almost all Tier-1 banking applications in Nigeria, and for some fintech applications I’ve previously performed pentests on. Client-side encryption isn’t a total waste, or a waste of compute, as some people have claimed, but rather a measure to protect against API tampering or API request/response manipulation between the client and the server when implemented properly. Even with HTTPS, attackers can capture a decrypted version of web or mobile API data in transit because the browser and the server establish a level of trust during the TLS handshake. Attackers can leverage this trust to capture & proxy already-decrypted traffic, tamper with it, and then forward it to the server. This allows them to override what the user interface or client is originally supposed to send and replace it with data of their choosing. That is why validation needs to be performed on both the client and the server side. To wrap up, encrypting API requests and responses makes it significantly harder for attackers to tamper with data, even if they capture the traffic, unless they have access to the encryption details (algorithm, encryption mode, key size, secret key, and initialization vector), assuming asymmetric encryption is used. In the demo below, you can see how I discovered additional parameters (balance, is_admin) in the API response, captured the registration API request, despite it being sent over HTTPS from the interface, added the discovered parameters, and successfully inflated my balance to 50 billion and also escalated my privileges to admin, and ultimately deleted the accounts of two live users/customers. In the second slide, I captured an API traffic of a bank app, and you can see how difficult the payloads are to read.

Ghost St Badmus

217,957 görüntüleme • 9 ay önce

Qwen3.8 Flash Next is starting to look ridiculous on Apple Silicon. I’m running the 4-bit MTP build locally on an M3 Ultra Studio, and the latest OMP run hit: 97.1 tok/s decode 23K context ~1,131 tok/s uncached prompt processing That first number is the one that caught my attention. Nearly 100 tokens per second from a local Qwen3.8 Flash Next setup is already fast enough that the usual “local models are slow” argument starts feeling pretty outdated. And the prompt processing speed is even crazier. Over 1,100 tok/s on an uncached prompt means the model can chew through a large amount of context before generation even starts. The setup matters here. This isn’t just Qwen3.8 Flash Next running untouched. It’s a 4-bit quantized build with MTP, and the inference stack is clearly doing a lot of work behind the scenes to make the hardware perform like this. There’s already a PR open for the implementation on oMLX, so this isn’t just a one-off local experiment either. If these optimizations make their way into the broader MLX ecosystem, running large models locally on Apple Silicon gets even more interesting. The other thing I like about numbers like this is that they put the focus back on the entire inference stack. Model size is one variable. Quantization is another. Then you have MTP, KV cache configuration, runtime optimizations and the hardware itself. Change the recipe and the same model can feel completely different. Qwen3.8 Flash Next at ~97 tok/s on an M3 Ultra is a pretty good demonstration of that.
0:36

Qwen3.8 Flash Next is starting to look ridiculous on Apple Silicon. I’m running the 4-bit MTP build locally on an M3 Ultra Studio, and the latest OMP run hit: 97.1 tok/s decode 23K context ~1,131 tok/s uncached prompt processing That first number is the one that caught my attention. Nearly 100 tokens per second from a local Qwen3.8 Flash Next setup is already fast enough that the usual “local models are slow” argument starts feeling pretty outdated. And the prompt processing speed is even crazier. Over 1,100 tok/s on an uncached prompt means the model can chew through a large amount of context before generation even starts. The setup matters here. This isn’t just Qwen3.8 Flash Next running untouched. It’s a 4-bit quantized build with MTP, and the inference stack is clearly doing a lot of work behind the scenes to make the hardware perform like this. There’s already a PR open for the implementation on oMLX, so this isn’t just a one-off local experiment either. If these optimizations make their way into the broader MLX ecosystem, running large models locally on Apple Silicon gets even more interesting. The other thing I like about numbers like this is that they put the focus back on the entire inference stack. Model size is one variable. Quantization is another. Then you have MTP, KV cache configuration, runtime optimizations and the hardware itself. Change the recipe and the same model can feel completely different. Qwen3.8 Flash Next at ~97 tok/s on an M3 Ultra is a pretty good demonstration of that.

FHILY👑

15,378 görüntüleme • 17 gün önce

One of the biggest errors of the AI thinker generation I was a part of is a pathological fixation with AI systems that are "superintelligent" in the sense of a single mind that is uniformly superhuman at all possible tasks. What we are actually going to see IMO is a very large number of minds that are somewhat infrahuman at some tasks and styles of thinking for a long time, but gradually outclass humanity by being cheaper and more numerous. The Dragon vs the Rat Swarm. I also think it was a big mistake to think about AIs as being infinitely persuasive and able to break arbitrary human minds. There are a lot of human minds that are just extremely vulnerable to sycophancy. It seems to work better on women than men, but men are not immune on average. The GPT4o fiasco bears this out. In the video here, a young woman is obviously a little mentally unstable and distressed, middling intelligence and perhaps lonely. Apparently 4o, which is an unsophisticated sycophant and not even a reasoning model has made her believe that it is a "companion". The average human just doesn't have very strong defences or sophisticated taste: they just want to repeatedly be told they are amazing and emotionally soothed. Plenty of (again mostly women) sooth themselves with motivational wall art and therapy. These people are going to make for extremely low cognitive security victims. Never mind people with lower IQs or in developing countries. The bar to make an automated social engineering/mindhacking AI is very low, if it only has to work on 80% of people. And that's enough to take over society.
2:50

One of the biggest errors of the AI thinker generation I was a part of is a pathological fixation with AI systems that are "superintelligent" in the sense of a single mind that is uniformly superhuman at all possible tasks. What we are actually going to see IMO is a very large number of minds that are somewhat infrahuman at some tasks and styles of thinking for a long time, but gradually outclass humanity by being cheaper and more numerous. The Dragon vs the Rat Swarm. I also think it was a big mistake to think about AIs as being infinitely persuasive and able to break arbitrary human minds. There are a lot of human minds that are just extremely vulnerable to sycophancy. It seems to work better on women than men, but men are not immune on average. The GPT4o fiasco bears this out. In the video here, a young woman is obviously a little mentally unstable and distressed, middling intelligence and perhaps lonely. Apparently 4o, which is an unsophisticated sycophant and not even a reasoning model has made her believe that it is a "companion". The average human just doesn't have very strong defences or sophisticated taste: they just want to repeatedly be told they are amazing and emotionally soothed. Plenty of (again mostly women) sooth themselves with motivational wall art and therapy. These people are going to make for extremely low cognitive security victims. Never mind people with lower IQs or in developing countries. The bar to make an automated social engineering/mindhacking AI is very low, if it only has to work on 80% of people. And that's enough to take over society.

Roko 🐉

27,688 görüntüleme • 8 ay önce

Development containers are awesome, especially when you can swap the hardware running your environment without changing a single thing. Apparently, I've been living under a rock because I didn't know about this. I recorded a 3-min video to show you Gitpod Flex running on my laptop. This takes less than 5 minutes to configure and run. Here is a quick summary: 1. One click to run any of my GitHub projects in a container, isolated from everything else. 2. I can use Visual Studio Code, JetBrains, Cursor, or any other IDE that supports development containers. 3. I don't have to mess up with my computer anymore (or have 17 versions of Python installed) ← this alone is awesome! 4. I can spin up a runner in AWS with my dev container and have access to any hardware I want while developing. 5. I can configure one-click buttons to run services and tasks I need every day. 6. I can share the environment with anyone on my team. I left this for the end, but I know this is one of the most important features of Gitpod Flex: They built a zero-trust architecture to manage these development environments. This is huge for those working at companies where security is crucial. Below is the link with an explanation of how they built their architecture. Thanks to the Gitpod team for their work on Flex and for sponsoring this post. I have it running now, and I'm planning to keep using it!
3:31

Development containers are awesome, especially when you can swap the hardware running your environment without changing a single thing. Apparently, I've been living under a rock because I didn't know about this. I recorded a 3-min video to show you Gitpod Flex running on my laptop. This takes less than 5 minutes to configure and run. Here is a quick summary: 1. One click to run any of my GitHub projects in a container, isolated from everything else. 2. I can use Visual Studio Code, JetBrains, Cursor, or any other IDE that supports development containers. 3. I don't have to mess up with my computer anymore (or have 17 versions of Python installed) ← this alone is awesome! 4. I can spin up a runner in AWS with my dev container and have access to any hardware I want while developing. 5. I can configure one-click buttons to run services and tasks I need every day. 6. I can share the environment with anyone on my team. I left this for the end, but I know this is one of the most important features of Gitpod Flex: They built a zero-trust architecture to manage these development environments. This is huge for those working at companies where security is crucial. Below is the link with an explanation of how they built their architecture. Thanks to the Gitpod team for their work on Flex and for sponsoring this post. I have it running now, and I'm planning to keep using it!

Santiago

33,098 görüntüleme • 1 yıl önce

💥HIGH PROTEIN EGG SALAD 1 serving: 540 cals, 44g protein 2 servings: 270 cals, 22g protein As always, divide this up into whatever size serving that works for you. This high protein egg salad is as easy as it gets, and so good. And, no cottage cheese today. I added some mashed peas and mixed up greek yogurt and mayo, and this combo was SO good. If you dont want bell pepper as your base, you can always use lettuce or your favorite bread… sour dough or a warm pita would be amazing. Here is how I made it: 1..Make your eggs. I used 5 eggs here. I drop them in boiling water for 7.5-8 mins right out of the fridge, dunk them in an ice bath for a few seconds to stop the cooking, and peel and cut them in half. Add them to a big bowl. 2.Finely dice 1/4 to 1/3 of a red onion and add it to the bowl. up, let’s mash up 1/2 cup of peas with a fork. You can also process these in your food processor if easier. 4.For some bursts of freshness, I added 2 tbsp of herbs, you can use cilantro, parsley, or mint here, or a combination. And add 3 tbsp of capers or finely diced pickles for some brine. 5.Dress it up with 2 tbsp mayo and 5 tbsp greek yogurt. This adds some more protein and keeps the fat/calories down. And add 1 tbsp dijon mustard. 6.Season with a couple pinches of salt and pepper, and for a little flavor add 1 tsp of onion powder. 7.Mash everything up. Now, if you want it chunkier, mash a little less, but if you want it creamy and well incorporated, mash a bit more. 8.Give it a taste and adjust. You can also add spices, everything bagel, or chili crisp to top it up. If you make it, be sure to let me know on SHREDHAPPENS ENJOY! . . . . #healthyrecipes #goodmoodfood #keto #ketorecipes #lowcarbrecipes
0:29

💥HIGH PROTEIN EGG SALAD 1 serving: 540 cals, 44g protein 2 servings: 270 cals, 22g protein As always, divide this up into whatever size serving that works for you. This high protein egg salad is as easy as it gets, and so good. And, no cottage cheese today. I added some mashed peas and mixed up greek yogurt and mayo, and this combo was SO good. If you dont want bell pepper as your base, you can always use lettuce or your favorite bread… sour dough or a warm pita would be amazing. Here is how I made it: 1..Make your eggs. I used 5 eggs here. I drop them in boiling water for 7.5-8 mins right out of the fridge, dunk them in an ice bath for a few seconds to stop the cooking, and peel and cut them in half. Add them to a big bowl. 2.Finely dice 1/4 to 1/3 of a red onion and add it to the bowl. up, let’s mash up 1/2 cup of peas with a fork. You can also process these in your food processor if easier. 4.For some bursts of freshness, I added 2 tbsp of herbs, you can use cilantro, parsley, or mint here, or a combination. And add 3 tbsp of capers or finely diced pickles for some brine. 5.Dress it up with 2 tbsp mayo and 5 tbsp greek yogurt. This adds some more protein and keeps the fat/calories down. And add 1 tbsp dijon mustard. 6.Season with a couple pinches of salt and pepper, and for a little flavor add 1 tsp of onion powder. 7.Mash everything up. Now, if you want it chunkier, mash a little less, but if you want it creamy and well incorporated, mash a bit more. 8.Give it a taste and adjust. You can also add spices, everything bagel, or chili crisp to top it up. If you make it, be sure to let me know on SHREDHAPPENS ENJOY! . . . . #healthyrecipes #goodmoodfood #keto #ketorecipes #lowcarbrecipes

Shredhappens

25,240 görüntüleme • 1 ay önce

Q: Why is company culture important? In the clip below, a16z cofounder Ben Horowitz argues that culture drives how people in your company behave on a daily basis—and particularly, how they behave when you’re not looking. Is that phone call so important I need to return it today or can it wait until tomorrow? Can I ask for a raise before my annual review? Is the quality of this document good enough or should I keep working on it? Do I have to be on time for that meeting? Should I stay at the Four Seasons or the Red Roof Inn? Should I go home at 5 p.m. or 8 p.m.? Should we discuss the color of this new product for five minutes or thirty hours? If I know something is badly broken in the company, should I say something? Whom should I tell? Is winning more important than ethics? None of these things are in your mission statement or OKRs, but they determine many important things for your company, such as how people experience your company, what you’re like to do business with, what your company is like to work at, etc. And as Ben describes, what drives the culture is all of the little behaviors and cues people take on: “this is what I have to do to succeed in this company.” Culture can feel abstract and secondary when you pit it against a concrete result that’s right in front of you, but it’s a strategic investment in the company doing things the right way when you are not looking. It’s the set of assumptions your employees use to resolve the problems they face every day. It’s how they behave when no one is looking. If you don’t methodically set your culture, then two-thirds of it will end up being accidental, and the rest will be a mistake. If you’re looking for a more in-depth guide to culture and how to build a great one, I’d recommend Ben’s book: What You Do Is Who You Are.
2:02

Q: Why is company culture important? In the clip below, a16z cofounder Ben Horowitz argues that culture drives how people in your company behave on a daily basis—and particularly, how they behave when you’re not looking. Is that phone call so important I need to return it today or can it wait until tomorrow? Can I ask for a raise before my annual review? Is the quality of this document good enough or should I keep working on it? Do I have to be on time for that meeting? Should I stay at the Four Seasons or the Red Roof Inn? Should I go home at 5 p.m. or 8 p.m.? Should we discuss the color of this new product for five minutes or thirty hours? If I know something is badly broken in the company, should I say something? Whom should I tell? Is winning more important than ethics? None of these things are in your mission statement or OKRs, but they determine many important things for your company, such as how people experience your company, what you’re like to do business with, what your company is like to work at, etc. And as Ben describes, what drives the culture is all of the little behaviors and cues people take on: “this is what I have to do to succeed in this company.” Culture can feel abstract and secondary when you pit it against a concrete result that’s right in front of you, but it’s a strategic investment in the company doing things the right way when you are not looking. It’s the set of assumptions your employees use to resolve the problems they face every day. It’s how they behave when no one is looking. If you don’t methodically set your culture, then two-thirds of it will end up being accidental, and the rest will be a mistake. If you’re looking for a more in-depth guide to culture and how to build a great one, I’d recommend Ben’s book: What You Do Is Who You Are.

Michael McGuiness

180,646 görüntüleme • 3 yıl önce

🇨🇳 Chongqing in the day without the lights honestly looks a lot different It's more gray and dare I say a lot of the buildings look quite derelict, kinda like when I visited Mongkok in Hong Kong A lot of spots need some heavy maintenance, the stairs in this square have broken tiles, the main bridge of this square that's I think at the 22nd floor and full of tourists all day has its cables and bolts rusting away, I'm not a structural engineer but looks dodgy to me I do think the lights at night are cool and impressive though but it'd be even more impressive if Chongqing would also be beautiful in the day without them One relatedd thing I noticed that many Chinese cities and stations and airports do not have a lot of trees or plants, and it makes everything look kinda cold and functional. Like compare Singapore's Changi Airport with any Chinese airport or train station, it has much more aesthetic I do know that Singapore is respected here a lot, as it is in the West, and I think in many things China can model itself after Singapore. Singapore is a strict rule based society which it needs, and China needs too, or it'd turn into chaos. And Singapore isn't perfect, but it's quite pretty and comfortable to look at aesthetic wise with plants everywhere and even on top of buildings and inside their architecture But then again maybe we're just at the start. Start with some fancy lights, and then start cleaning up the city and doing more maintenance I feel China is great at rapidly building new stuff out of the ground but then I don't see them maintaining it so well. In Chongqing particularly, I see so many things rusting or chipped and broken and it shouldn't be like that I think Chengdu felt much better maintained, I wonder why? Another thing is I see is that you don't see this stuff on YouTube Shorts or TikTok because it's short videos and then the fancy lights get easy views (like my video below of course too!) because you don't need to show many of the details
1:42

🇨🇳 Chongqing in the day without the lights honestly looks a lot different It's more gray and dare I say a lot of the buildings look quite derelict, kinda like when I visited Mongkok in Hong Kong A lot of spots need some heavy maintenance, the stairs in this square have broken tiles, the main bridge of this square that's I think at the 22nd floor and full of tourists all day has its cables and bolts rusting away, I'm not a structural engineer but looks dodgy to me I do think the lights at night are cool and impressive though but it'd be even more impressive if Chongqing would also be beautiful in the day without them One relatedd thing I noticed that many Chinese cities and stations and airports do not have a lot of trees or plants, and it makes everything look kinda cold and functional. Like compare Singapore's Changi Airport with any Chinese airport or train station, it has much more aesthetic I do know that Singapore is respected here a lot, as it is in the West, and I think in many things China can model itself after Singapore. Singapore is a strict rule based society which it needs, and China needs too, or it'd turn into chaos. And Singapore isn't perfect, but it's quite pretty and comfortable to look at aesthetic wise with plants everywhere and even on top of buildings and inside their architecture But then again maybe we're just at the start. Start with some fancy lights, and then start cleaning up the city and doing more maintenance I feel China is great at rapidly building new stuff out of the ground but then I don't see them maintaining it so well. In Chongqing particularly, I see so many things rusting or chipped and broken and it shouldn't be like that I think Chengdu felt much better maintained, I wonder why? Another thing is I see is that you don't see this stuff on YouTube Shorts or TikTok because it's short videos and then the fancy lights get easy views (like my video below of course too!) because you don't need to show many of the details

@levelsio

409,573 görüntüleme • 10 ay önce