Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

MiniMax H3 ⚡️ Spectrum acceleration in ComfyUI — 34% lower Euler sampling time, 30% lower RES time Tested at approximately 0.5 MP, 8 seconds and 20 steps on an RTX PRO 6000 under WSL, using High VRAM mode 👇

SD_Tutorial's profile picture

Stable Diffusion Tutorials

8,383 subscribers

29,096 Aufrufe • vor 27 Tagen •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

⚡️ We just added support for Nvidia G4 (RTX Pro 6000 Blackwell Server Edition) GPUs in Colaboratory! 🔥 With a peak rating of 960 BF16 TFLOPs (~50% more than the A100-80G) and 96 GB of VRAM (20% more than the A100-80G), it’s the most efficient high-performance GPU we’ve ever released. ⏱️Time to make GPUs go brrrr 🏎️
0:14

⚡️ We just added support for Nvidia G4 (RTX Pro 6000 Blackwell Server Edition) GPUs in Colaboratory! 🔥 With a peak rating of 960 BF16 TFLOPs (~50% more than the A100-80G) and 96 GB of VRAM (20% more than the A100-80G), it’s the most efficient high-performance GPU we’ve ever released. ⏱️Time to make GPUs go brrrr 🏎️

Colaboratory

62,470 Aufrufe • vor 5 Monaten

Open source AI is actually moving at an unhinged pace right now. I literally hadn't even finished typing up my last Gemma 4 12b benchmark notes before Google went ahead and dropped the official Quantization Aware Training (QAT) checkpoints on Hugging Face. If you missed the news, QAT basically bakes the compression directly into the training process. Instead of standard post training quantization degrading the model's reasoning capabilities, QAT trains the model with compression in mind. Unsloth is reporting near original performance at 4-bit with ~72% lower memory footprint. Details in the comments. Naturally, had to instantly pull the new GGUFs to see what a single RTX 4090 card (24 GB VRAM, Cuda 12.8, ubuntu 22) could do. i fired up llama.cpp engine again Look at these numbers: 1. Unsloth Gemma 4 26B-A4B IT (QAT Q4_K_XL) flags: ./build/bin/llama-cli -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 250000 -fa on -v VRAM Used: 19.5 GB context: 250,000 tokens decode throughput: 193 tps 2. Unsloth Gemma 4 31B IT (QAT Q4_K_XL) flags: Command: ./build/bin/llama-cli -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 60000 -fa on -v - VRAM Used: 23 GB (Tight, but zero system RAM spillover) - context: 60,000 tokens - decode throughput: 47 tps We are essentially watching hardware bottlenecks evaporate in real time. An update literally drops before you can finish benchmarking the previous one. What a time to be running local hardware. If you have a single rtx 3090, rtx 4090, these are the latest gemma models to try this week.
0:14

Open source AI is actually moving at an unhinged pace right now. I literally hadn't even finished typing up my last Gemma 4 12b benchmark notes before Google went ahead and dropped the official Quantization Aware Training (QAT) checkpoints on Hugging Face. If you missed the news, QAT basically bakes the compression directly into the training process. Instead of standard post training quantization degrading the model's reasoning capabilities, QAT trains the model with compression in mind. Unsloth is reporting near original performance at 4-bit with ~72% lower memory footprint. Details in the comments. Naturally, had to instantly pull the new GGUFs to see what a single RTX 4090 card (24 GB VRAM, Cuda 12.8, ubuntu 22) could do. i fired up llama.cpp engine again Look at these numbers: 1. Unsloth Gemma 4 26B-A4B IT (QAT Q4_K_XL) flags: ./build/bin/llama-cli -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 250000 -fa on -v VRAM Used: 19.5 GB context: 250,000 tokens decode throughput: 193 tps 2. Unsloth Gemma 4 31B IT (QAT Q4_K_XL) flags: Command: ./build/bin/llama-cli -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -cnv -ngl 99 -c 60000 -fa on -v - VRAM Used: 23 GB (Tight, but zero system RAM spillover) - context: 60,000 tokens - decode throughput: 47 tps We are essentially watching hardware bottlenecks evaporate in real time. An update literally drops before you can finish benchmarking the previous one. What a time to be running local hardware. If you have a single rtx 3090, rtx 4090, these are the latest gemma models to try this week.

Alok

26,841 Aufrufe • vor 2 Monaten

you're paying $20/mo for something your $500 GPU can already do. Gemma 4 26B A4B QAT MoE + Hermes Agent running on a single RTX 4060 (8GB VRAM). Built a vision capable, 100% free, 100% local, private AI assistant that lives in my Chrome browser. No API keys. No cloud. No subscriptions. 100% vibe coded. 0% handholding. It has full context of whatever's on my screen can answer questions, summarize pages, extract data, and see images. Same local model handles everything, no external calls, ever. keep reading for the model and hermes agent tips i learnt while building this locally. Here's the exact setup for anyone running local LLMs on 6-8 GB VRAM: llama.cpp server flags (on my NVIDIA RTX 4060 8gb VRAM): -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --cache-type-k q8_0 --cache-type-v q8_0 -c 150000 --port 8080 Throughput with quantization: Prefill: 200-250 tokens/sec Decode: 20-25 tokens/sec reduce context if oom on 6 gb vram card. Key learnings: - Quantize KV cache to q8 for faster prefill/decode. Prefill goes from 100-150 (unquantized) to 200-250 tok/s (q8). - But watch out, once actual context grows past ~50k tokens on high entropy workloads, q8 KV quantization can cause hallucinations. Low entropy workloads are mostly unaffected. If you see it happening, drop the quantization. This is common across all local models. - In Hermes Agent settings -> Memory & Context, bump compression threshold from default 0.5 to 0.7. Default triggers way too frequent context compression and eats time. Up next: add persistent memory, web search, tool calling, streaming output and whatever you suggest. Running a 26B MoE with vision + 150k context window on 8GB VRAM would've sounded impossible 6 months ago. Works the same on the NVIDIA RTX 3060 Ti, 3070, 4060 Ti, 5060, 2080, or any 8GB card. VRAM is the only requirement. Local AI agents are closer than people think. You just need to know where the knobs are. Model's Unsloth quant hugging face link in the comments. Have you tried Hermes agent by Nous Research yet? What are you building with local LLMs? Drop it below, let's see what this community is shipping.
0:19

you're paying $20/mo for something your $500 GPU can already do. Gemma 4 26B A4B QAT MoE + Hermes Agent running on a single RTX 4060 (8GB VRAM). Built a vision capable, 100% free, 100% local, private AI assistant that lives in my Chrome browser. No API keys. No cloud. No subscriptions. 100% vibe coded. 0% handholding. It has full context of whatever's on my screen can answer questions, summarize pages, extract data, and see images. Same local model handles everything, no external calls, ever. keep reading for the model and hermes agent tips i learnt while building this locally. Here's the exact setup for anyone running local LLMs on 6-8 GB VRAM: llama.cpp server flags (on my NVIDIA RTX 4060 8gb VRAM): -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --cache-type-k q8_0 --cache-type-v q8_0 -c 150000 --port 8080 Throughput with quantization: Prefill: 200-250 tokens/sec Decode: 20-25 tokens/sec reduce context if oom on 6 gb vram card. Key learnings: - Quantize KV cache to q8 for faster prefill/decode. Prefill goes from 100-150 (unquantized) to 200-250 tok/s (q8). - But watch out, once actual context grows past ~50k tokens on high entropy workloads, q8 KV quantization can cause hallucinations. Low entropy workloads are mostly unaffected. If you see it happening, drop the quantization. This is common across all local models. - In Hermes Agent settings -> Memory & Context, bump compression threshold from default 0.5 to 0.7. Default triggers way too frequent context compression and eats time. Up next: add persistent memory, web search, tool calling, streaming output and whatever you suggest. Running a 26B MoE with vision + 150k context window on 8GB VRAM would've sounded impossible 6 months ago. Works the same on the NVIDIA RTX 3060 Ti, 3070, 4060 Ti, 5060, 2080, or any 8GB card. VRAM is the only requirement. Local AI agents are closer than people think. You just need to know where the knobs are. Model's Unsloth quant hugging face link in the comments. Have you tried Hermes agent by Nous Research yet? What are you building with local LLMs? Drop it below, let's see what this community is shipping.

Alok

36,691 Aufrufe • vor 1 Monat

50% more context unlocked for Qwen 3.8 27b Q4_K_XL dflash 2 on a single RTX 4090 (24 GB VRAM) I found a hidden VRAM tax in llama.cpp. By combining my custom 2 bit DFlash 2 drafter with one overlooked server flag, I just unlocked another +80,000 tokens of context. Qwen3.8-27B is now running a massive 250,000 context at 75 tokens/s on a single RTX 4090. Here is the secret: By default, `llama-server` reserves massive chunks of your VRAM to handle multiple concurrent users (batching). If you are running a single user session, you are bleeding memory for features you aren't using. By passing the `--parallel 1` flag, you force the engine to dedicate 100% of your 24GB VRAM buffer to a single user. When we combine the VRAM saved by our Q2_K 2-bit drafter with the VRAM saved by `--parallel 1`, the context ceilings absolutely explode: Note: all benchmarks carried out with a massive 28k prompt. Ubuntu 22. ### THE NEW 24GB PHYSICAL LIMITS (Single RTX 4090): # 1. The "Repo Swallower" (Q4 KV Cache): - Context: 250,000 tokens (Up from 170k!) - Speed: 73.66 t/s decode | 1,608 t/s prefill - Peak VRAM: 23.8 GB # 2. The "High-Precision SWE" (Q8 KV Cache): - Context: 150,000 tokens (Up from 100k!) - Speed: 75.01 t/s decode | 1,667 t/s prefill - Peak VRAM: 23.9 GB # 3. The "Pristine Attention" (Unquantized FP16 KV): - Context: 90,000 tokens - Speed: 80.58 t/s decode | 1,699 t/s prefill - Peak VRAM: 23.92 GB ### HOW TO RUN THE 250K GOD STACK TODAY: (Requires PR #27342 + my Q2_K Hugging Face drafter) llama.cpp flags: ./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -md Qwen3.8-27B-DFlash2-Q2_K.gguf --spec-type draft-dflash --spec-draft-n-max 3 -c 250000 -ngl 99 --parallel 1 --port 8080 -ctv q4_0 -ctk q4_0 We are pushing a quarter million tokens of context with speculative DFlash 2 decoding at 73 tokens/second on a single consumer gaming GPU. I dropped my custom 2 bit Hugging Face GGUF links, visual performance graphs, and the PR #27342 build instructions in the replies below. If you own a single RTX 3090 or 4090, it is officially time to cancel your API subscriptions and let local silicon eat the cloud. how much monthly API spend does an optimized 4090 rig like this actually replace for you?
0:14

50% more context unlocked for Qwen 3.8 27b Q4_K_XL dflash 2 on a single RTX 4090 (24 GB VRAM) I found a hidden VRAM tax in llama.cpp. By combining my custom 2 bit DFlash 2 drafter with one overlooked server flag, I just unlocked another +80,000 tokens of context. Qwen3.8-27B is now running a massive 250,000 context at 75 tokens/s on a single RTX 4090. Here is the secret: By default, `llama-server` reserves massive chunks of your VRAM to handle multiple concurrent users (batching). If you are running a single user session, you are bleeding memory for features you aren't using. By passing the `--parallel 1` flag, you force the engine to dedicate 100% of your 24GB VRAM buffer to a single user. When we combine the VRAM saved by our Q2_K 2-bit drafter with the VRAM saved by `--parallel 1`, the context ceilings absolutely explode: Note: all benchmarks carried out with a massive 28k prompt. Ubuntu 22. ### THE NEW 24GB PHYSICAL LIMITS (Single RTX 4090): # 1. The "Repo Swallower" (Q4 KV Cache): - Context: 250,000 tokens (Up from 170k!) - Speed: 73.66 t/s decode | 1,608 t/s prefill - Peak VRAM: 23.8 GB # 2. The "High-Precision SWE" (Q8 KV Cache): - Context: 150,000 tokens (Up from 100k!) - Speed: 75.01 t/s decode | 1,667 t/s prefill - Peak VRAM: 23.9 GB # 3. The "Pristine Attention" (Unquantized FP16 KV): - Context: 90,000 tokens - Speed: 80.58 t/s decode | 1,699 t/s prefill - Peak VRAM: 23.92 GB ### HOW TO RUN THE 250K GOD STACK TODAY: (Requires PR #27342 + my Q2_K Hugging Face drafter) llama.cpp flags: ./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -md Qwen3.8-27B-DFlash2-Q2_K.gguf --spec-type draft-dflash --spec-draft-n-max 3 -c 250000 -ngl 99 --parallel 1 --port 8080 -ctv q4_0 -ctk q4_0 We are pushing a quarter million tokens of context with speculative DFlash 2 decoding at 73 tokens/second on a single consumer gaming GPU. I dropped my custom 2 bit Hugging Face GGUF links, visual performance graphs, and the PR #27342 build instructions in the replies below. If you own a single RTX 3090 or 4090, it is officially time to cancel your API subscriptions and let local silicon eat the cloud. how much monthly API spend does an optimized 4090 rig like this actually replace for you?

Alok

39,189 Aufrufe • vor 10 Tagen

Run Gemma 4 26B MoE on 8GB VRAM with 250k context at 20+ tokens/sec If you own any 8GB VRAM graphics card, stop what you are doing. Local AI just had its absolute "Holy Shit" moment for budget hardware. Yesterday, I benchmarked Unsloth Gemma 4 12B Q4_K_XL on an 8GB card. The community went wild but immediately demanded more: "Can we run a 25B+ model on budget GPUs?" Today, I’m delivering exactly that. I am running a massive 26B parameter Mixture of Experts (MoE) model locally on a standard 8GB VRAM setup with 250k full native context!. If you own an RTX 3060, 3070, 4060, or any budget GPU with 8GB of VRAM, the local AI paradigm has completely changed. The performance metrics are astonishing: - 20 tokens/sec flat decode throughput. - Stable, flat decode speed even with massive prompts. - I threw a 60k token prompt at it, and it still clocked in at 20 TPS without dropping a single frame. # What about prefill? Yes, Time To First Token (TTFT) is slightly high when swallowing massive contexts. But with a solid 200 tokens/sec prefill speed, the wait is barely noticeable and highly usable. And this is running completely without Multi Token Prediction (MTP) active. How is this possible? It’s the magic of Google's new QAT (Quantization Aware Training) quants for Gemma 4. The model weight file (unsloth gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf) is only 13.2 GB, making it the ultimate local powerhouse. # The Test Setup: CPU: Intel Core i7 RAM: 16GB System RAM GPU: NVIDIA GeForce RTX 4060 Laptop GPU (8GB VRAM) # The Secret Sauce (The -cmoe Flag) To make this work properly on any 8GB card, you must use the -cmoe (CPU MoE) flag in llama.cpp. This flag isolates the heavy MoE expert weights directly to system memory (CPU/RAM) while letting your GPU focus strictly on the Attention layers and the KV Cache. It prevents VRAM spillage and holds the throughput rock solid. # The flags: -m "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf" -cmoe -c 248000 -v Once running, just open the UI on localhost and toggle the new reasoning lightbulb icon in the text input box to watch the model perform multi step thinking. Are you still running smaller models, or are you ready to scale up your budget local setups? Let's discuss in the replies
0:19

Run Gemma 4 26B MoE on 8GB VRAM with 250k context at 20+ tokens/sec If you own any 8GB VRAM graphics card, stop what you are doing. Local AI just had its absolute "Holy Shit" moment for budget hardware. Yesterday, I benchmarked Unsloth Gemma 4 12B Q4_K_XL on an 8GB card. The community went wild but immediately demanded more: "Can we run a 25B+ model on budget GPUs?" Today, I’m delivering exactly that. I am running a massive 26B parameter Mixture of Experts (MoE) model locally on a standard 8GB VRAM setup with 250k full native context!. If you own an RTX 3060, 3070, 4060, or any budget GPU with 8GB of VRAM, the local AI paradigm has completely changed. The performance metrics are astonishing: - 20 tokens/sec flat decode throughput. - Stable, flat decode speed even with massive prompts. - I threw a 60k token prompt at it, and it still clocked in at 20 TPS without dropping a single frame. # What about prefill? Yes, Time To First Token (TTFT) is slightly high when swallowing massive contexts. But with a solid 200 tokens/sec prefill speed, the wait is barely noticeable and highly usable. And this is running completely without Multi Token Prediction (MTP) active. How is this possible? It’s the magic of Google's new QAT (Quantization Aware Training) quants for Gemma 4. The model weight file (unsloth gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf) is only 13.2 GB, making it the ultimate local powerhouse. # The Test Setup: CPU: Intel Core i7 RAM: 16GB System RAM GPU: NVIDIA GeForce RTX 4060 Laptop GPU (8GB VRAM) # The Secret Sauce (The -cmoe Flag) To make this work properly on any 8GB card, you must use the -cmoe (CPU MoE) flag in llama.cpp. This flag isolates the heavy MoE expert weights directly to system memory (CPU/RAM) while letting your GPU focus strictly on the Attention layers and the KV Cache. It prevents VRAM spillage and holds the throughput rock solid. # The flags: -m "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf" -cmoe -c 248000 -v Once running, just open the UI on localhost and toggle the new reasoning lightbulb icon in the text input box to watch the model perform multi step thinking. Are you still running smaller models, or are you ready to scale up your budget local setups? Let's discuss in the replies

Alok

292,770 Aufrufe • vor 2 Monaten

Gemma 4 26B A4B MoE - 500+ t/s decode - Single RTX 4090 (24 GB VRAM) - Llama.cpp concurrency 24 - q8 kv cache How many API users can you simultaneously host on a single RTX 4090 (24 GB VRAM) before it crashes? Yesterday, I proved you can host 14 active users using unquantized memory. Today, I used 8 bit KV Cache Quantization to hack the VRAM footprint. I successfully scaled to 24 concurrent users without a single dropped connection. A 71% server capacity boost for free. By adding the -ctk q8_0 -ctv q8_0 flags to llama.cpp, you compress the KV cache context memory from 16 bit to 8 bit. This unlocks massive concurrency limits on Gemma 4 26B (MoE) on a single 24GB consumer GPU. Here is the exact telemetry from pushing 8 bit quantization to its absolute physical edge: # TEST 1: The 24 User Concurrency Max Server Config: 24 slots (np 24) | 4,096 context per slot | 98,304 Total Context Client Load: 24 simultaneous requests (2,000 token prompt per user) Unquantized KV cache for this load requires 28GB+ VRAM (Instant OOM). Quantized to Q8, it allocated safely at 23.35 GB. The C++ engine crunched the entire batch in 28.5 seconds. Decode Speed: 21 t/s (Per User) | 500 t/s (Agg) # TEST 2: The 48 User Queue Overload What happens to a compressed cache during a traffic spike? Server Config: 24 slots (np 24) | 4,096 context per slot | 98,304 Total Context Client Load: 48 simultaneous requests (2k token prompt per user) Zero queue drops. The scheduler flushed and hot swapped the 8 bit memory flawlessly on the fly, completing all 48 users in 66.0 seconds (a perfect 2.3x queue scaling multiplier). Decode Speed: 18 t/s (Per User) | 430 t/s (Agg) # TEST 3: The 8 User RAG Slam Server Config: 8 slots (np 8) | 60,000 context per slot | 480,000 Total Context Client Load: 8 simultaneous requests (30k token prompt per user) It allocated 23.83 GB VRAM and chewed through ~240,000 prefill tokens in 46 seconds under massive memory pressure. Prefill Speed: 6,200 t/s (Agg) Decode Speed: 22 t/s (Per User) | 175 t/s (Agg) # The Engineering Alpha (The Quantization Tradeoff): You gain a massive 71% increase in server capacity, but what do you lose? Compute latency. Because the cache is stored in 8 bit, the GPU's cores have to dequantize the memory back to 16 bit on the fly during every single prefill step. In my unquantized tests yesterday, single slot prefill was hitting ~1,500+ t/s. Today, under the heavy 48-user Q8 load, prefill dropped as low as ~750 t/s. You trade a few seconds of initial prefill latency to essentially double your API hosting capacity. For production high volume SaaS, this is the ultimate unit economics cheat code. Here is the exact command to run a 24 user Q8 continuous batching server on your own single 4090, single 3090 or any 24gb vram rig: ./build/bin/llama-server -m gemma-4-26B-A4B-it.gguf -c 98304 -np 24 -b 2048 -ub 2048 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 --port 8080 (Note: -c 98304 allocates exactly 4,096 tokens of context per user across 24 slots). Hugging Face links to the Unsloth Gemma 4 26B QAT quants along with performance graphs available in the replies. Would you trade 3 seconds of Time To First Token latency to double your active user capacity?
0:18

Gemma 4 26B A4B MoE - 500+ t/s decode - Single RTX 4090 (24 GB VRAM) - Llama.cpp concurrency 24 - q8 kv cache How many API users can you simultaneously host on a single RTX 4090 (24 GB VRAM) before it crashes? Yesterday, I proved you can host 14 active users using unquantized memory. Today, I used 8 bit KV Cache Quantization to hack the VRAM footprint. I successfully scaled to 24 concurrent users without a single dropped connection. A 71% server capacity boost for free. By adding the -ctk q8_0 -ctv q8_0 flags to llama.cpp, you compress the KV cache context memory from 16 bit to 8 bit. This unlocks massive concurrency limits on Gemma 4 26B (MoE) on a single 24GB consumer GPU. Here is the exact telemetry from pushing 8 bit quantization to its absolute physical edge: # TEST 1: The 24 User Concurrency Max Server Config: 24 slots (np 24) | 4,096 context per slot | 98,304 Total Context Client Load: 24 simultaneous requests (2,000 token prompt per user) Unquantized KV cache for this load requires 28GB+ VRAM (Instant OOM). Quantized to Q8, it allocated safely at 23.35 GB. The C++ engine crunched the entire batch in 28.5 seconds. Decode Speed: 21 t/s (Per User) | 500 t/s (Agg) # TEST 2: The 48 User Queue Overload What happens to a compressed cache during a traffic spike? Server Config: 24 slots (np 24) | 4,096 context per slot | 98,304 Total Context Client Load: 48 simultaneous requests (2k token prompt per user) Zero queue drops. The scheduler flushed and hot swapped the 8 bit memory flawlessly on the fly, completing all 48 users in 66.0 seconds (a perfect 2.3x queue scaling multiplier). Decode Speed: 18 t/s (Per User) | 430 t/s (Agg) # TEST 3: The 8 User RAG Slam Server Config: 8 slots (np 8) | 60,000 context per slot | 480,000 Total Context Client Load: 8 simultaneous requests (30k token prompt per user) It allocated 23.83 GB VRAM and chewed through ~240,000 prefill tokens in 46 seconds under massive memory pressure. Prefill Speed: 6,200 t/s (Agg) Decode Speed: 22 t/s (Per User) | 175 t/s (Agg) # The Engineering Alpha (The Quantization Tradeoff): You gain a massive 71% increase in server capacity, but what do you lose? Compute latency. Because the cache is stored in 8 bit, the GPU's cores have to dequantize the memory back to 16 bit on the fly during every single prefill step. In my unquantized tests yesterday, single slot prefill was hitting ~1,500+ t/s. Today, under the heavy 48-user Q8 load, prefill dropped as low as ~750 t/s. You trade a few seconds of initial prefill latency to essentially double your API hosting capacity. For production high volume SaaS, this is the ultimate unit economics cheat code. Here is the exact command to run a 24 user Q8 continuous batching server on your own single 4090, single 3090 or any 24gb vram rig: ./build/bin/llama-server -m gemma-4-26B-A4B-it.gguf -c 98304 -np 24 -b 2048 -ub 2048 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 --port 8080 (Note: -c 98304 allocates exactly 4,096 tokens of context per user across 24 slots). Hugging Face links to the Unsloth Gemma 4 26B QAT quants along with performance graphs available in the replies. Would you trade 3 seconds of Time To First Token latency to double your active user capacity?

Alok

17,465 Aufrufe • vor 1 Monat

The (Processing) Level Required to Be Super We’ve had clients ask us whether our 3D/4DGS capture and processing pipeline holds up in high-end VFX cinematic shots. Last year we were fortunate to work on the Superman production with Framestore (capturing Superman’s parents for a hologram), with our work shown on IMAX screens ~30 m high—so we’re confident the quality delivers. For this R&D preview, we ran test captures using an affordable off-the-shelf temporal plugin to reduce flicker and improve stability over time. We recorded @ 48fps from our 196x machine vision cameras and trained the 4DGS using Postshot at Ultra quality ~20 million splat resolution. What you’re seeing are stop-motion 3DGS frames. Each frame is an individual Gaussian Splat—processed to ~20 million splats for the full scene, then cropped and cleaned to ~6 million splats per frame. Our system—and the 3DGS process—picks up fine details like stray fine hairs, skin pores and woven cloth threads. Thanks for watching & stay super! Learn more @ #3DGS #4DGS #VFX #GaussianSplatting #IMAX #VolumetricCapture #Superman 4K Version here -
0:19

The (Processing) Level Required to Be Super We’ve had clients ask us whether our 3D/4DGS capture and processing pipeline holds up in high-end VFX cinematic shots. Last year we were fortunate to work on the Superman production with Framestore (capturing Superman’s parents for a hologram), with our work shown on IMAX screens ~30 m high—so we’re confident the quality delivers. For this R&D preview, we ran test captures using an affordable off-the-shelf temporal plugin to reduce flicker and improve stability over time. We recorded @ 48fps from our 196x machine vision cameras and trained the 4DGS using Postshot at Ultra quality ~20 million splat resolution. What you’re seeing are stop-motion 3DGS frames. Each frame is an individual Gaussian Splat—processed to ~20 million splats for the full scene, then cropped and cleaned to ~6 million splats per frame. Our system—and the 3DGS process—picks up fine details like stray fine hairs, skin pores and woven cloth threads. Thanks for watching & stay super! Learn more @ #3DGS #4DGS #VFX #GaussianSplatting #IMAX #VolumetricCapture #Superman 4K Version here -

Infinite-Realities

44,109 Aufrufe • vor 11 Monaten

🔄 Ever wondered how a substation “understands” electricity? Currents flow. Voltages fluctuate. Temperatures rise. But protection relays don’t read analog waves — they read numbers. Before a breaker trips or an alarm appears on SCADA, an invisible process converts raw electrical signals into digital intelligence. That silent hero? Analog-to-Digital Conversion (ADC). Let’s break it down step by step 👇 📡➡💻 From Analog Signal to Digital Decision 1️⃣ Low-Pass Filtering (Anti-Aliasing) ⚙️ Purpose: Remove high-frequency noise before sampling. Why? If unwanted frequencies enter the ADC, they cause aliasing — distortion that corrupts measurements. 💡 Clean input = accurate digital output. 2️⃣ Sampling 📉 The continuous analog waveform is captured at fixed time intervals. Sampling rate matters. Too slow → information loss. Too fast → unnecessary data overload. In power systems, sampling is often synchronized with system frequency (50/60 Hz). 3️⃣ Quantization 🪜 Each sampled value is approximated to the nearest discrete level. Resolution depends on ADC bits: • 8-bit → 256 levels • 12-bit → 4096 levels • 16-bit → 65,536 levels Higher resolution = finer accuracy in current and voltage measurement. 4️⃣ Encoding 🔢 Quantized values are converted into binary form. Example: Analog value → Quantized level → 110010 Now the signal is no longer a waveform — it’s data. 5️⃣ Digital Output 📲 The binary stream is transmitted to: ✔ Protection relays ✔ IEDs ✔ RTUs ✔ SCADA systems From here, algorithms detect faults, measure harmonics, calculate power, and trigger protective actions. 🧠 Without accurate ADC: ❌ Protection misoperations increase ❌ Fault detection becomes unreliable ❌ Automation decisions degrade With precise ADC: ✔ Real-time protection ✔ Advanced fault diagnostics ✔ Accurate metering ✔ Seamless IEC 61850 integration Every digital substation depends on clean, time-synchronized data. ⚡ From a simple CT secondary current to a breaker trip command — it all begins with precise sampling and conversion. Next time you analyze disturbance records or SCADA logs, remember: Behind every number lies an ADC working in microseconds.
0:12

🔄 Ever wondered how a substation “understands” electricity? Currents flow. Voltages fluctuate. Temperatures rise. But protection relays don’t read analog waves — they read numbers. Before a breaker trips or an alarm appears on SCADA, an invisible process converts raw electrical signals into digital intelligence. That silent hero? Analog-to-Digital Conversion (ADC). Let’s break it down step by step 👇 📡➡💻 From Analog Signal to Digital Decision 1️⃣ Low-Pass Filtering (Anti-Aliasing) ⚙️ Purpose: Remove high-frequency noise before sampling. Why? If unwanted frequencies enter the ADC, they cause aliasing — distortion that corrupts measurements. 💡 Clean input = accurate digital output. 2️⃣ Sampling 📉 The continuous analog waveform is captured at fixed time intervals. Sampling rate matters. Too slow → information loss. Too fast → unnecessary data overload. In power systems, sampling is often synchronized with system frequency (50/60 Hz). 3️⃣ Quantization 🪜 Each sampled value is approximated to the nearest discrete level. Resolution depends on ADC bits: • 8-bit → 256 levels • 12-bit → 4096 levels • 16-bit → 65,536 levels Higher resolution = finer accuracy in current and voltage measurement. 4️⃣ Encoding 🔢 Quantized values are converted into binary form. Example: Analog value → Quantized level → 110010 Now the signal is no longer a waveform — it’s data. 5️⃣ Digital Output 📲 The binary stream is transmitted to: ✔ Protection relays ✔ IEDs ✔ RTUs ✔ SCADA systems From here, algorithms detect faults, measure harmonics, calculate power, and trigger protective actions. 🧠 Without accurate ADC: ❌ Protection misoperations increase ❌ Fault detection becomes unreliable ❌ Automation decisions degrade With precise ADC: ✔ Real-time protection ✔ Advanced fault diagnostics ✔ Accurate metering ✔ Seamless IEC 61850 integration Every digital substation depends on clean, time-synchronized data. ⚡ From a simple CT secondary current to a breaker trip command — it all begins with precise sampling and conversion. Next time you analyze disturbance records or SCADA logs, remember: Behind every number lies an ADC working in microseconds.

Taha Alasemi | طه العاصمي

24,450 Aufrufe • vor 6 Monaten

First day with Gardner-Webb Baseball means a massive focus on teaching the skill of Acceleration and building an Upper Body to handle the demands of throwing. When athletes struggle to accelerate there is usually a correlation between the amount of inhibitions they show at depth, so dosing the Deep Tier (Matt) appropriately in the warm up increases their amount of time spent at depth, thus breaking that inhibition. Put them on the sled and get some practice at putting more force into the ground at various velocities. Follow up with some Med Ball Throws at max intent with varying amounts of lower body contribution, static or moving start, rotational or linear, and then blast some Jake Tuura style Hypertrophy Clusters for a great day! Warm Up: 3x through Deep Split Exchange Leaps x10 yards Light Sled Push x10 yards Deep Leaps x10 yards Heavy Sled Push x10 yards Timed Sprint: 2x through 10 yard Acceleration w/ 1 yard Run-In Intensive MB Throws: 3x through EMOM MB Rotational Scoop Throw w/ Step x1 each Static Start Prone MB Throw x2 Static Start Roller MB Overhead Throw x2 Upper HC Superset: 10x through DB Bench Press x4 @ 70% Chest-Supported DB Row x8 @ RPE 7-8
0:13

First day with Gardner-Webb Baseball means a massive focus on teaching the skill of Acceleration and building an Upper Body to handle the demands of throwing. When athletes struggle to accelerate there is usually a correlation between the amount of inhibitions they show at depth, so dosing the Deep Tier (Matt) appropriately in the warm up increases their amount of time spent at depth, thus breaking that inhibition. Put them on the sled and get some practice at putting more force into the ground at various velocities. Follow up with some Med Ball Throws at max intent with varying amounts of lower body contribution, static or moving start, rotational or linear, and then blast some Jake Tuura style Hypertrophy Clusters for a great day! Warm Up: 3x through Deep Split Exchange Leaps x10 yards Light Sled Push x10 yards Deep Leaps x10 yards Heavy Sled Push x10 yards Timed Sprint: 2x through 10 yard Acceleration w/ 1 yard Run-In Intensive MB Throws: 3x through EMOM MB Rotational Scoop Throw w/ Step x1 each Static Start Prone MB Throw x2 Static Start Roller MB Overhead Throw x2 Upper HC Superset: 10x through DB Bench Press x4 @ 70% Chest-Supported DB Row x8 @ RPE 7-8

Nat Sauberan

16,369 Aufrufe • vor 2 Jahren

TESLA SEMI: THE FUTURE OF TRUCKING IS ELECTRIC, EFFICIENT & AUTONOMOUS The Tesla Semi isn’t just another truck—it’s a complete redefinition of long-haul trucking. With production ramping up and deliveries accelerating, it’s already proving why electric Class 8 semis will dominate the industry. Why the Tesla Semi is critically important: •Unmatched efficiency & cost savings ~1.7 kWh/mile energy consumption (real-world data from PepsiCo & early fleets) — far lower than diesel equivalents. Operating costs drop dramatically with electricity vs. diesel, plus massive reductions in maintenance (no oil changes, fewer brake replacements thanks to regenerative braking). •Insane performance 0–60 mph in ~20 seconds fully loaded (500-mile range version), 0–60 in under 10 seconds in lighter configs. Instant torque makes hills and merging effortless—safer and faster than diesel trucks. •500+ mile range on a single charge Real-world highway range exceeds 500 miles even with heavy loads, eliminating most range anxiety for long-haul routes. Megachargers add hundreds of miles in ~30 minutes. •Autonomy-ready platform Built on Tesla’s Full Self-Driving hardware and software stack. Future unsupervised FSD will eliminate driver fatigue, reduce accidents (94% of which involve human error), and enable 24/7 operation—transforming driver economics and safety. •Environmental & regulatory edge Zero tailpipe emissions, helping fleets meet tightening emissions standards (California, EU, etc.). Lower noise pollution for urban deliveries and night operations. •Fleet economics revolution Lower TCO (total cost of ownership) than diesel trucks over the vehicle life. When combined with solar + Megapack charging depots, operating costs approach near-zero marginal energy cost. The Tesla Semi isn’t competing in trucking—it’s redefining the entire category. From PepsiCo’s early fleet to upcoming high-volume production, it’s the truck that makes electric long-haul not just possible, but inevitable 🚛⚡
0:17

TESLA SEMI: THE FUTURE OF TRUCKING IS ELECTRIC, EFFICIENT & AUTONOMOUS The Tesla Semi isn’t just another truck—it’s a complete redefinition of long-haul trucking. With production ramping up and deliveries accelerating, it’s already proving why electric Class 8 semis will dominate the industry. Why the Tesla Semi is critically important: •Unmatched efficiency & cost savings ~1.7 kWh/mile energy consumption (real-world data from PepsiCo & early fleets) — far lower than diesel equivalents. Operating costs drop dramatically with electricity vs. diesel, plus massive reductions in maintenance (no oil changes, fewer brake replacements thanks to regenerative braking). •Insane performance 0–60 mph in ~20 seconds fully loaded (500-mile range version), 0–60 in under 10 seconds in lighter configs. Instant torque makes hills and merging effortless—safer and faster than diesel trucks. •500+ mile range on a single charge Real-world highway range exceeds 500 miles even with heavy loads, eliminating most range anxiety for long-haul routes. Megachargers add hundreds of miles in ~30 minutes. •Autonomy-ready platform Built on Tesla’s Full Self-Driving hardware and software stack. Future unsupervised FSD will eliminate driver fatigue, reduce accidents (94% of which involve human error), and enable 24/7 operation—transforming driver economics and safety. •Environmental & regulatory edge Zero tailpipe emissions, helping fleets meet tightening emissions standards (California, EU, etc.). Lower noise pollution for urban deliveries and night operations. •Fleet economics revolution Lower TCO (total cost of ownership) than diesel trucks over the vehicle life. When combined with solar + Megapack charging depots, operating costs approach near-zero marginal energy cost. The Tesla Semi isn’t competing in trucking—it’s redefining the entire category. From PepsiCo’s early fleet to upcoming high-volume production, it’s the truck that makes electric long-haul not just possible, but inevitable 🚛⚡

Tesla Owners Silicon Valley

15,161 Aufrufe • vor 6 Monaten

A molecular beam epitaxy system is not a machine in the traditional sense. Even though it looks like a Time Travel Machine to the untrained eye. It is an ultra high vacuum atomic assembly chamber operating near 10⁻¹⁰ torr, lower pressure than low Earth orbit, where even a single stray molecule can become a crystal defect. Inside, solid materials are heated in effusion cells until they form directed atomic beams. These atoms travel in free molecular flow and condense on a substrate one atomic layer at a time, with thickness control at the angstrom scale. Growth rates are often 0.1 to 1 monolayer per second. A single nanometer of material can take several minutes to form, because precision is the entire constraint. A full research grade system typically costs between $800,000 and $2 million, while advanced multi chamber platforms exceed $5 million and complete cluster installations can reach $10-15 million. Because every subsystem is a precision instrument controlling matter at atomic scale under conditions where randomness itself becomes a defect. What looks like a vacuum chamber is actually one of the most controlled environments ever built for matter manipulation on the planet, this is the furthest frontier of Modern day Science. 🎥armanj3189
0:17

A molecular beam epitaxy system is not a machine in the traditional sense. Even though it looks like a Time Travel Machine to the untrained eye. It is an ultra high vacuum atomic assembly chamber operating near 10⁻¹⁰ torr, lower pressure than low Earth orbit, where even a single stray molecule can become a crystal defect. Inside, solid materials are heated in effusion cells until they form directed atomic beams. These atoms travel in free molecular flow and condense on a substrate one atomic layer at a time, with thickness control at the angstrom scale. Growth rates are often 0.1 to 1 monolayer per second. A single nanometer of material can take several minutes to form, because precision is the entire constraint. A full research grade system typically costs between $800,000 and $2 million, while advanced multi chamber platforms exceed $5 million and complete cluster installations can reach $10-15 million. Because every subsystem is a precision instrument controlling matter at atomic scale under conditions where randomness itself becomes a defect. What looks like a vacuum chamber is actually one of the most controlled environments ever built for matter manipulation on the planet, this is the furthest frontier of Modern day Science. 🎥armanj3189

Ammanichanda

23,949 Aufrufe • vor 1 Monat

List of accused arrested in connection with the attack on Jahangir Mohalla Town Outpost (West Bengal), where an Islamic mob vandalised the police outpost, attacked police personnel, pelted stones, and attempted to disrupt law and order in the area: 1. Md Chand (22) 2. Md Sahabuddin 3. Md Mohsin Nazir (22) 4. Md Salman (22) 5. Md Saddam (26) 6. Md Suraj (29) 7. Md Jawed Ansari (30) 8. Md Israfil (28) 9. Md Irfan (20) 10. Md Naushad (20) 11. Md Salim (20) 12. Ahmed Raza (26) 13. Md Samsad (23) 14. Md Wasim (23) 15. Md Parwez (23) 16. Md Hasnain (24) 17. Md Abir (26) 18. Md Hasan (22) 19. Aqib Khan (25) 20. Md Ismail Khan (22) 21. Md Irsad (18) 22. Md Aslam (35) 23. Sahanaj Hossain (20) 24. Md Jabid Sha (21) 25. Md Sonu (35) 26. Md Sarfaraz (19) All are residents under Asansol North Police Station, Dist- Paschim Bardhaman... Do you think these many arrests of Islamists were possible if there was no BJP? It will take time but they'll learn to live peacefully... Also, look at their age, most are in the 20-30 age group....
0:15

List of accused arrested in connection with the attack on Jahangir Mohalla Town Outpost (West Bengal), where an Islamic mob vandalised the police outpost, attacked police personnel, pelted stones, and attempted to disrupt law and order in the area: 1. Md Chand (22) 2. Md Sahabuddin 3. Md Mohsin Nazir (22) 4. Md Salman (22) 5. Md Saddam (26) 6. Md Suraj (29) 7. Md Jawed Ansari (30) 8. Md Israfil (28) 9. Md Irfan (20) 10. Md Naushad (20) 11. Md Salim (20) 12. Ahmed Raza (26) 13. Md Samsad (23) 14. Md Wasim (23) 15. Md Parwez (23) 16. Md Hasnain (24) 17. Md Abir (26) 18. Md Hasan (22) 19. Aqib Khan (25) 20. Md Ismail Khan (22) 21. Md Irsad (18) 22. Md Aslam (35) 23. Sahanaj Hossain (20) 24. Md Jabid Sha (21) 25. Md Sonu (35) 26. Md Sarfaraz (19) All are residents under Asansol North Police Station, Dist- Paschim Bardhaman... Do you think these many arrests of Islamists were possible if there was no BJP? It will take time but they'll learn to live peacefully... Also, look at their age, most are in the 20-30 age group....

Mr Sinha

426,296 Aufrufe • vor 3 Monaten

I just ran Gemma 4 31B on @CerebrasSystems at 1,800+ tokens/sec and it's multimodal. For context: that's 35x faster than a typical GPU endpoint, and the first token (reasoning included) lands in 1.5 seconds. This isn't a benchmark slide, I recorded the inference live. Prompt I used: "Create a simulation of an iPhone. Include at least one working dummy note taking app, a functional notification pulldown, high quality graphics, single HTML file, any libs via CDN." - Generation time: 3 seconds. - Notes app worked. - Notification panel worked. - Rendered first try. This is what wafer-scale inference unlocks, not just "faster," but a different category of product. When generation is this fast, you stop waiting and start iterating in real time. Why this matters: Gemma 4 31B is Google DeepMind's flagship open weight model, Apache 2.0 licensed, dense (not MoE), and built for efficiency over raw parameter count. It scores close to Claude Haiku 4.5 on the Artificial Analysis Intelligence Index (30 vs 29) but runs ~18x faster on Cerebras. It's also the first multimodal model on Cerebras's platform, meaning you can now feed it screenshots, documents, charts, and UI states at wafer scale speed. # Applications I'm most excited about: - Screenshot → Insight: Drop in a dashboard or document screenshot, get structured findings back instantly. no waiting, no batching. - Live UI generation: Full interactive interfaces (like my iPhone sim) generated and rendered in under 2 seconds. - Screenshot -> Patch: Feed it a broken UI + console error, get a minimal code fix and verification steps back. - Computer use & agentic loops: See -> reason -> act - verify, fast enough to keep a human in the loop instead of waiting on the model. - Long context summarization: Full research reports condensed into decision ready summaries you can read and requery in one sitting. The bigger unlock isn't the speed number itself, it's that agentic and multimodal loops (see -> reason -> output -> tool call -> verify -> retry) finally run in real time instead of feeling sluggish. As Logan Kilpatrick (Logan Kilpatrick) put it: "If every model was doing 2,000 tokens per second, you wouldn't build the same product and just have it be faster, you'd build different products." Gemma 4 31B is live now on Cerebras Inference Cloud in public preview. If you're building multimodal, agentic, or real time apps, this is worth testing today. What would you build with such insane inference throughput?
0:12

I just ran Gemma 4 31B on @CerebrasSystems at 1,800+ tokens/sec and it's multimodal. For context: that's 35x faster than a typical GPU endpoint, and the first token (reasoning included) lands in 1.5 seconds. This isn't a benchmark slide, I recorded the inference live. Prompt I used: "Create a simulation of an iPhone. Include at least one working dummy note taking app, a functional notification pulldown, high quality graphics, single HTML file, any libs via CDN." - Generation time: 3 seconds. - Notes app worked. - Notification panel worked. - Rendered first try. This is what wafer-scale inference unlocks, not just "faster," but a different category of product. When generation is this fast, you stop waiting and start iterating in real time. Why this matters: Gemma 4 31B is Google DeepMind's flagship open weight model, Apache 2.0 licensed, dense (not MoE), and built for efficiency over raw parameter count. It scores close to Claude Haiku 4.5 on the Artificial Analysis Intelligence Index (30 vs 29) but runs ~18x faster on Cerebras. It's also the first multimodal model on Cerebras's platform, meaning you can now feed it screenshots, documents, charts, and UI states at wafer scale speed. # Applications I'm most excited about: - Screenshot → Insight: Drop in a dashboard or document screenshot, get structured findings back instantly. no waiting, no batching. - Live UI generation: Full interactive interfaces (like my iPhone sim) generated and rendered in under 2 seconds. - Screenshot -> Patch: Feed it a broken UI + console error, get a minimal code fix and verification steps back. - Computer use & agentic loops: See -> reason -> act - verify, fast enough to keep a human in the loop instead of waiting on the model. - Long context summarization: Full research reports condensed into decision ready summaries you can read and requery in one sitting. The bigger unlock isn't the speed number itself, it's that agentic and multimodal loops (see -> reason -> output -> tool call -> verify -> retry) finally run in real time instead of feeling sluggish. As Logan Kilpatrick (Logan Kilpatrick) put it: "If every model was doing 2,000 tokens per second, you wouldn't build the same product and just have it be faster, you'd build different products." Gemma 4 31B is live now on Cerebras Inference Cloud in public preview. If you're building multimodal, agentic, or real time apps, this is worth testing today. What would you build with such insane inference throughput?

Alok

12,962 Aufrufe • vor 2 Monaten

You don't need a GPU for fast studio grade voice cloning anymore. Qwen3 TTS (1.7B Q4_K_M) + mainline llama.cpp is officially the fastest way to generate zero shot voice clones using 100% pure CPU execution. Following up on my last post where we ran the Q8 model on a GPU, we just took local C++ voice synthesis a massive step further. The open source community quantized Alibaba's SOTA Qwen3 TTS model down to Q4_K_M GGUF, completely freeing local audio pipelines from dedicated graphics hardware. Here is the real world benchmark and hardware breakdown of running SOTA voice cloning on CPU: # Architecture & Model Setup Using Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf paired with the 8 bit multimodal projector (mmproj-Q8_0.gguf), llama.cpp executes the entire pipeline in pure C++. No PyTorch, no CUDA dependencies, and no VRAM bottlenecks. # Real-World Memory Footprint - Baseline RAM: 1.6 GB system idle. - Peak Generation RAM: 8 GB RAM during active voice synthesis. - Requirement: Any basic machine with at least 8 GB of system RAM can run this easily. # Real World CPU Benchmarks - Google Colab Free Tier (Throttled 2 Core CPU): Synthesizes a 5 sec studio quality audio clip (~8 words) in 45 seconds. - Modern Consumer CPU (Intel i5/i7 13th/14th Gen or AMD Ryzen 7000/9000): generation should drop to 5 to 20 seconds (nearly 1:1 real-time generation speed!). # Zero Shot Voice Cloning Quality Pass any 5 to 20 second .wav audio sample to the C++ engine using the --tts-speaker-file flag. It yields clean, natural sounding cloned speech with virtually zero quality loss compared to unquantized FP16 weights. To make testing seamless, I built an updated zero config Google Colab notebook. It pulls the official pre built llama.cpp CPU binaries (zero compilation time!) launches a live Gradio web app right in your browser. Record a 5 second clip from your mic (or drop a .mp3, .wav file), type text, and generate cloned audio on CPU. Native C++ audio models are making edge based, offline AI voice agents a reality. Links to the free Q4 CPU Colab notebook and the Q4_K_M GGUF HuggingFace repository are in the replies below! Which models have you been running on your CPUs? What CPU hardware are you using for local inference?
0:16

You don't need a GPU for fast studio grade voice cloning anymore. Qwen3 TTS (1.7B Q4_K_M) + mainline llama.cpp is officially the fastest way to generate zero shot voice clones using 100% pure CPU execution. Following up on my last post where we ran the Q8 model on a GPU, we just took local C++ voice synthesis a massive step further. The open source community quantized Alibaba's SOTA Qwen3 TTS model down to Q4_K_M GGUF, completely freeing local audio pipelines from dedicated graphics hardware. Here is the real world benchmark and hardware breakdown of running SOTA voice cloning on CPU: # Architecture & Model Setup Using Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf paired with the 8 bit multimodal projector (mmproj-Q8_0.gguf), llama.cpp executes the entire pipeline in pure C++. No PyTorch, no CUDA dependencies, and no VRAM bottlenecks. # Real-World Memory Footprint - Baseline RAM: 1.6 GB system idle. - Peak Generation RAM: 8 GB RAM during active voice synthesis. - Requirement: Any basic machine with at least 8 GB of system RAM can run this easily. # Real World CPU Benchmarks - Google Colab Free Tier (Throttled 2 Core CPU): Synthesizes a 5 sec studio quality audio clip (~8 words) in 45 seconds. - Modern Consumer CPU (Intel i5/i7 13th/14th Gen or AMD Ryzen 7000/9000): generation should drop to 5 to 20 seconds (nearly 1:1 real-time generation speed!). # Zero Shot Voice Cloning Quality Pass any 5 to 20 second .wav audio sample to the C++ engine using the --tts-speaker-file flag. It yields clean, natural sounding cloned speech with virtually zero quality loss compared to unquantized FP16 weights. To make testing seamless, I built an updated zero config Google Colab notebook. It pulls the official pre built llama.cpp CPU binaries (zero compilation time!) launches a live Gradio web app right in your browser. Record a 5 second clip from your mic (or drop a .mp3, .wav file), type text, and generate cloned audio on CPU. Native C++ audio models are making edge based, offline AI voice agents a reality. Links to the free Q4 CPU Colab notebook and the Q4_K_M GGUF HuggingFace repository are in the replies below! Which models have you been running on your CPUs? What CPU hardware are you using for local inference?

Alok

60,514 Aufrufe • vor 23 Tagen

my 8 GB VRAM gaming laptop is absolutely going to hate me for this. but I still did it. ran a 31b dense model (Gemma 4 31b Q4) with only 8 GB VRAM last week I ran Gemma 4 26B A4B a mixture of experts model on my RTX 4060 and hit 25–28 tokens/sec using llama.cpp's new MTP support. smooth. snappy. but MoE has a secret: it only activates 4B parameters per token despite having 26B total. that's why it flies. so the real question started haunting me. what if I throw a full, no tricks, every parameter fires on every token, 31B DENSE model at the same machine? # Hardware: GPU: NVIDIA RTX 4060, 8 GB VRAM RAM: 16 GB CPU: Intel Core i7 H Laptop. Gaming. Modest. The model: gemma-4-31B-it-qat-UD-Q4_K_XL.gguf (model's unsloth huggingface link in the comments) This is Google DeepMind's flagship dense model in the Gemma 4 family that can run on single consumer GPU. It packs a hybrid attention architecture, supports up to 256K context natively, and is QAT (Quantization Aware Training) optimized, meaning it retains far more quality than standard post training quants at the same bit depth. This is NOT the MoE. This is 31 BILLION dense parameters, every single one of them loaded. # the flags I used: -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -cnv --spec-type draft-mtp --spec-draft-model mtp-gemma-4-31B-it.gguf --spec-draft-n-max 8 --spec-draft-p-min 0.6 -c 6000 -v Multi Token Prediction (MTP) is still active here. Separate draft GGUF required, same as the 26B setup. # Results: → Decode: ~3 tokens/sec → Prefill: ~2 tokens/sec → Context: 6000 tokens → Hardware crying quietly in the corner: yes so is 3 tps actually usable? For real time back and forth chat? Not ideal. You're not having a fluid conversation at 3 tps. but slow ≠ useless. And this is where it gets genuinely interesting. think about how senior devs actually work in a real team. But when something is architectural, deeply complex, or needs serious reasoning? they walk down the hall and escalate to the senior. That's exactly the local AI agent architecture this unlocks: → Fast orchestrator model (Gemma 4 26B MoE at 25+ tps) handles routing, simple queries, tool calls, memory. The junior dev. → Gemma 4 31B dense is the senior, called only when the fast model genuinely hits a wall. Hard multi step reasoning. Complex code generation. Deep architectural decisions. The agentic loop stays fast. Only the hard hops touch the 31B. That's a legitimate production grade local AI architecture on a budget hardware. (requires 2 8gb gpus) other workflows where 3 tps is completely fine: - overnight batch jobs. summarize documents, extract structured data, review code. Fire it off. Sleep. wake up to results. - One shot deep reasoning - Silent code audit loops, you write and test, the 31B reviews diffs and flags issues in the background between your sprints - Any workflow where output quality > output speed A few weeks ago, nobody was running a 30B+ dense model on a single consumer GPU with 8 GB VRAM. At all. Now we're doing it on an Intel i7-H gaming laptop with a NVIDIA RTX 4060, thanks to llama.cpp + QAT quants + MTP speculative drafting. Google DeepMind said the Gemma 4 31B targets "consumer GPUs and workstations." They were not exaggerating. The hardware bar to run serious frontier class models locally keeps dropping. the tools are here. the models are here. you just have to be willing to abuse your laptop a little. what workflows would you actually run on a local 3 tps 31B dense model? genuinely curious. drop it below.
0:18

my 8 GB VRAM gaming laptop is absolutely going to hate me for this. but I still did it. ran a 31b dense model (Gemma 4 31b Q4) with only 8 GB VRAM last week I ran Gemma 4 26B A4B a mixture of experts model on my RTX 4060 and hit 25–28 tokens/sec using llama.cpp's new MTP support. smooth. snappy. but MoE has a secret: it only activates 4B parameters per token despite having 26B total. that's why it flies. so the real question started haunting me. what if I throw a full, no tricks, every parameter fires on every token, 31B DENSE model at the same machine? # Hardware: GPU: NVIDIA RTX 4060, 8 GB VRAM RAM: 16 GB CPU: Intel Core i7 H Laptop. Gaming. Modest. The model: gemma-4-31B-it-qat-UD-Q4_K_XL.gguf (model's unsloth huggingface link in the comments) This is Google DeepMind's flagship dense model in the Gemma 4 family that can run on single consumer GPU. It packs a hybrid attention architecture, supports up to 256K context natively, and is QAT (Quantization Aware Training) optimized, meaning it retains far more quality than standard post training quants at the same bit depth. This is NOT the MoE. This is 31 BILLION dense parameters, every single one of them loaded. # the flags I used: -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -cnv --spec-type draft-mtp --spec-draft-model mtp-gemma-4-31B-it.gguf --spec-draft-n-max 8 --spec-draft-p-min 0.6 -c 6000 -v Multi Token Prediction (MTP) is still active here. Separate draft GGUF required, same as the 26B setup. # Results: → Decode: ~3 tokens/sec → Prefill: ~2 tokens/sec → Context: 6000 tokens → Hardware crying quietly in the corner: yes so is 3 tps actually usable? For real time back and forth chat? Not ideal. You're not having a fluid conversation at 3 tps. but slow ≠ useless. And this is where it gets genuinely interesting. think about how senior devs actually work in a real team. But when something is architectural, deeply complex, or needs serious reasoning? they walk down the hall and escalate to the senior. That's exactly the local AI agent architecture this unlocks: → Fast orchestrator model (Gemma 4 26B MoE at 25+ tps) handles routing, simple queries, tool calls, memory. The junior dev. → Gemma 4 31B dense is the senior, called only when the fast model genuinely hits a wall. Hard multi step reasoning. Complex code generation. Deep architectural decisions. The agentic loop stays fast. Only the hard hops touch the 31B. That's a legitimate production grade local AI architecture on a budget hardware. (requires 2 8gb gpus) other workflows where 3 tps is completely fine: - overnight batch jobs. summarize documents, extract structured data, review code. Fire it off. Sleep. wake up to results. - One shot deep reasoning - Silent code audit loops, you write and test, the 31B reviews diffs and flags issues in the background between your sprints - Any workflow where output quality > output speed A few weeks ago, nobody was running a 30B+ dense model on a single consumer GPU with 8 GB VRAM. At all. Now we're doing it on an Intel i7-H gaming laptop with a NVIDIA RTX 4060, thanks to llama.cpp + QAT quants + MTP speculative drafting. Google DeepMind said the Gemma 4 31B targets "consumer GPUs and workstations." They were not exaggerating. The hardware bar to run serious frontier class models locally keeps dropping. the tools are here. the models are here. you just have to be willing to abuse your laptop a little. what workflows would you actually run on a local 3 tps 31B dense model? genuinely curious. drop it below.

Alok

63,689 Aufrufe • vor 2 Monaten

Today marks General Availability of AgentCore, a set of infrastructure building blocks for developers and companies to build secure, scalable agents. When we first started AWS, the vast majority of developers were spending most of their time on the undifferentiated heavy lifting of infrastructure instead of what differentiated their feature. So, we solved that problem by building primitive building blocks like compute and storage and database that would allow teammates and customers to quickly build and deploy new experiences without having to reinvent the wheel each time. We realized the same thing was happening with AI agents. It's too difficult and it's slowing customers down. That's why we created AgentCore, a set of services to build, deploy, and operate highly capable agents using any framework or model, with enterprise-grade security and scalability. These building blocks (like serverless secure runtime, memory, observability, a gateway that does MCP translation, etc) help customers tackle some of the biggest challenges of going from prototype to production, much more quickly, securely, and scalably. AgentCore has been in preview for several weeks, and customers have been quite excited about it. The AgentCore SDK has already been downloaded over a million times and we're seeing transformative results, such as Cohere Health expecting to reduce medical review times by 30-40% in highly regulated healthcare, and teams at Cox Automotive and Experian are embracing its flexibility to deploy and operate agents at scale. Inside Amazon, our Amazon Devices Operations & Supply Chain team is using AgentCore to develop an agentic manufacturing approach where AI agents work together to automate manual processes – turning what used to be days of engineering time into processes that take under an hour with high precision. Just like AWS changed how companies build and scale applications, we believe AgentCore will do the same for AI agents, enabling the next generation of innovation.
0:15

Today marks General Availability of AgentCore, a set of infrastructure building blocks for developers and companies to build secure, scalable agents. When we first started AWS, the vast majority of developers were spending most of their time on the undifferentiated heavy lifting of infrastructure instead of what differentiated their feature. So, we solved that problem by building primitive building blocks like compute and storage and database that would allow teammates and customers to quickly build and deploy new experiences without having to reinvent the wheel each time. We realized the same thing was happening with AI agents. It's too difficult and it's slowing customers down. That's why we created AgentCore, a set of services to build, deploy, and operate highly capable agents using any framework or model, with enterprise-grade security and scalability. These building blocks (like serverless secure runtime, memory, observability, a gateway that does MCP translation, etc) help customers tackle some of the biggest challenges of going from prototype to production, much more quickly, securely, and scalably. AgentCore has been in preview for several weeks, and customers have been quite excited about it. The AgentCore SDK has already been downloaded over a million times and we're seeing transformative results, such as Cohere Health expecting to reduce medical review times by 30-40% in highly regulated healthcare, and teams at Cox Automotive and Experian are embracing its flexibility to deploy and operate agents at scale. Inside Amazon, our Amazon Devices Operations & Supply Chain team is using AgentCore to develop an agentic manufacturing approach where AI agents work together to automate manual processes – turning what used to be days of engineering time into processes that take under an hour with high precision. Just like AWS changed how companies build and scale applications, we believe AgentCore will do the same for AI agents, enabling the next generation of innovation.

Andy Jassy

24,990 Aufrufe • vor 10 Monaten

50 things to do instead of wasting another year: 1. wake up at the same time every day 2. track every expense for 30 days 3. cut one useless monthly subscription 4. build an emergency fund (first 1000) 5. learn one high-income skill 6. read 10 non-fiction books 7. delete tiktok and instagram for 30 days 8. train your body 3x per week 9. fix your sleep schedule 10. learn basic personal finance 11. start journaling daily 12. build a simple website or portfolio 13. learn how taxes work in your country 14. stop buying things to impress others 15. build one income stream outside your job 16. learn sales fundamentals 17. learn copywriting basics 18. improve your communication skills 19. track your time for one full week 20. cut processed food 21. learn to cook 5 basic meals 22. walk 8000–10000 steps daily 23. set clear 1-year goals 24. set clear 5-year goals 25. build discipline before motivation 26. network intentionally 27. learn how algorithms work (X, youtube) 28. build a morning routine 29. build an evening shutdown routine 30. stop complaining 31. stop blaming circumstances 32. learn to sit alone without stimulation 33. audit who you spend time with 34. learn basic investing principles 35. save before you spend 36. learn to say no 37. learn negotiation basics 38. improve posture and body language 39. learn deep focus (no multitasking) 40. build something small and finish it 41. document progress instead of flexing results 42. stop chasing trends 43. build consistency over intensity 44. learn stress management 45. learn delayed gratification 46. take responsibility for outcomes 47. reduce digital noise 48. build a simple life system 49. measure progress weekly 50. repeat for 365 days
0:14

50 things to do instead of wasting another year: 1. wake up at the same time every day 2. track every expense for 30 days 3. cut one useless monthly subscription 4. build an emergency fund (first 1000) 5. learn one high-income skill 6. read 10 non-fiction books 7. delete tiktok and instagram for 30 days 8. train your body 3x per week 9. fix your sleep schedule 10. learn basic personal finance 11. start journaling daily 12. build a simple website or portfolio 13. learn how taxes work in your country 14. stop buying things to impress others 15. build one income stream outside your job 16. learn sales fundamentals 17. learn copywriting basics 18. improve your communication skills 19. track your time for one full week 20. cut processed food 21. learn to cook 5 basic meals 22. walk 8000–10000 steps daily 23. set clear 1-year goals 24. set clear 5-year goals 25. build discipline before motivation 26. network intentionally 27. learn how algorithms work (X, youtube) 28. build a morning routine 29. build an evening shutdown routine 30. stop complaining 31. stop blaming circumstances 32. learn to sit alone without stimulation 33. audit who you spend time with 34. learn basic investing principles 35. save before you spend 36. learn to say no 37. learn negotiation basics 38. improve posture and body language 39. learn deep focus (no multitasking) 40. build something small and finish it 41. document progress instead of flexing results 42. stop chasing trends 43. build consistency over intensity 44. learn stress management 45. learn delayed gratification 46. take responsibility for outcomes 47. reduce digital noise 48. build a simple life system 49. measure progress weekly 50. repeat for 365 days

alexei

17,479 Aufrufe • vor 1 Monat

Longtime New Yorkers Wrongly Hit With Massive “Pied-à-Terre” Tax Bills — Maybe It’s Time to Leave New Yorkers who have lived in the city for decades — many in their only home — are receiving five-figure tax notices under the new pied-à-terre surcharge. The tax was sold as a levy on wealthy owners of luxury second homes. Instead, longtime residents say they are being forced to prove they actually live in the places they have called home for 30 years or more. Karen Young, who has lived in New York since 1972, received a $43,000 bill on her West 95th Street brownstone. “Whoever’s behind this, who I can only assume is the mayor, didn’t take the 30 seconds to research. Is this a witch hunt? As a faithful New Yorker since 1972, I find it not only insulting but painful. Is anybody paying any attention, or any thinking, or two minutes of research? Just look me up!” (Video: AI) An 81-year-old woman got a $55,000 notice addressed to her late husband. “I have lived here for 30 years; it’s my only residence. The City of New York didn’t get that right.” The appeal process is described as cumbersome, with tight deadlines and technical hurdles even for those who pay taxes, utilities, and jury duty from the same address. What was framed as “taxing the rich” is landing hard on people who never left. New York City is already seeing an exodus of residents to lower-tax states like Florida and Texas. It will be interesting to watch whether this pied-à-terre tax accelerates that trend. Once the high earners leave, who’s left for Mayor Mamdani to tax?
0:15

Longtime New Yorkers Wrongly Hit With Massive “Pied-à-Terre” Tax Bills — Maybe It’s Time to Leave New Yorkers who have lived in the city for decades — many in their only home — are receiving five-figure tax notices under the new pied-à-terre surcharge. The tax was sold as a levy on wealthy owners of luxury second homes. Instead, longtime residents say they are being forced to prove they actually live in the places they have called home for 30 years or more. Karen Young, who has lived in New York since 1972, received a $43,000 bill on her West 95th Street brownstone. “Whoever’s behind this, who I can only assume is the mayor, didn’t take the 30 seconds to research. Is this a witch hunt? As a faithful New Yorker since 1972, I find it not only insulting but painful. Is anybody paying any attention, or any thinking, or two minutes of research? Just look me up!” (Video: AI) An 81-year-old woman got a $55,000 notice addressed to her late husband. “I have lived here for 30 years; it’s my only residence. The City of New York didn’t get that right.” The appeal process is described as cumbersome, with tight deadlines and technical hurdles even for those who pay taxes, utilities, and jury duty from the same address. What was framed as “taxing the rich” is landing hard on people who never left. New York City is already seeing an exodus of residents to lower-tax states like Florida and Texas. It will be interesting to watch whether this pied-à-terre tax accelerates that trend. Once the high earners leave, who’s left for Mayor Mamdani to tax?

Paul A. Szypula 🇺🇸

235,278 Aufrufe • vor 1 Monat

IF I WAS FORCED to build a $20K/month AI creative agency using nothing but Photoshop, starting from 0, here's exactly what I would do in steps: The production setup (Days 1–3) 1. Download the Higgsfield plugin inside Photoshop — takes 5 minutes 2. You now have: sketch-to-image, layer decomposer, mockup studio, relight, upscale, face swap, character swap, background removal, AI stylist — all in 1 tool 3. Old creative agency workflow: designer + photographer + editor + 3–5 day turnaround 4. New workflow: 1 person, Photoshop, 30 minutes per deliverable The offer (Days 3–7) 5. Pick 1 niche — ecom brands, real estate agents, or course creators all need visuals constantly 6. Build a simple offer: "10 ad creatives delivered in 24 hours — $500" 7. Old agencies charge $2,000–$5,000/month for the same output 8. Your cost to deliver: $0 beyond the plugin. Pure margin. 9. Create 3 sample mockups using the tool — drop a product image in, generate 9 variations, pick the best 3 10. That's your portfolio. Built in under 1 hour. Cost: $0. The client machine (Days 7–20) 11. Go on X and search "[niche] + need a designer" or "[niche] + creatives" 12. DM 50 people per day — "I'll make you 3 free ad creatives in 24 hours, no catch" 13. Deliver them in 30 minutes using the plugin 14. 50 DMs/day × 14 days = 700 outreach messages 15. Conservative 3% conversion = 21 people see the free work 16. Close 5 of them at $500 = $2,500 in week 3 The scale (Days 20–30) 17. Upsell every client to a $1,500/month retainer — 10 creatives/week, unlimited revisions 18. 1 client per day in Photoshop takes 45 minutes max 19. 10 retainer clients × $1,500 = $15,000/month 20. Add 3 one-off clients at $500/month = $1,500 21. Add a $997 "AI creative system" course teaching other people this exact workflow = $3,000+/month from 3 sales The math: 50 DMs/day × 30 days = 1,500 outreach messages 3% book a call = 45 calls 40% close at $1,500/month retainer = 18 clients 18 × $1,500 = $27,000/month recurring Time per client per day: 45 minutes Total daily work: 4–5 hours Every mockup — AI. Every restyle — AI. Every layer rebuild — AI. Every variation — AI. No photographer. No designer and no reshoot. Start it here. 👇
0:19

IF I WAS FORCED to build a $20K/month AI creative agency using nothing but Photoshop, starting from 0, here's exactly what I would do in steps: The production setup (Days 1–3) 1. Download the Higgsfield plugin inside Photoshop — takes 5 minutes 2. You now have: sketch-to-image, layer decomposer, mockup studio, relight, upscale, face swap, character swap, background removal, AI stylist — all in 1 tool 3. Old creative agency workflow: designer + photographer + editor + 3–5 day turnaround 4. New workflow: 1 person, Photoshop, 30 minutes per deliverable The offer (Days 3–7) 5. Pick 1 niche — ecom brands, real estate agents, or course creators all need visuals constantly 6. Build a simple offer: "10 ad creatives delivered in 24 hours — $500" 7. Old agencies charge $2,000–$5,000/month for the same output 8. Your cost to deliver: $0 beyond the plugin. Pure margin. 9. Create 3 sample mockups using the tool — drop a product image in, generate 9 variations, pick the best 3 10. That's your portfolio. Built in under 1 hour. Cost: $0. The client machine (Days 7–20) 11. Go on X and search "[niche] + need a designer" or "[niche] + creatives" 12. DM 50 people per day — "I'll make you 3 free ad creatives in 24 hours, no catch" 13. Deliver them in 30 minutes using the plugin 14. 50 DMs/day × 14 days = 700 outreach messages 15. Conservative 3% conversion = 21 people see the free work 16. Close 5 of them at $500 = $2,500 in week 3 The scale (Days 20–30) 17. Upsell every client to a $1,500/month retainer — 10 creatives/week, unlimited revisions 18. 1 client per day in Photoshop takes 45 minutes max 19. 10 retainer clients × $1,500 = $15,000/month 20. Add 3 one-off clients at $500/month = $1,500 21. Add a $997 "AI creative system" course teaching other people this exact workflow = $3,000+/month from 3 sales The math: 50 DMs/day × 30 days = 1,500 outreach messages 3% book a call = 45 calls 40% close at $1,500/month retainer = 18 clients 18 × $1,500 = $27,000/month recurring Time per client per day: 45 minutes Total daily work: 4–5 hours Every mockup — AI. Every restyle — AI. Every layer rebuild — AI. Every variation — AI. No photographer. No designer and no reshoot. Start it here. 👇

ALEX SUZUKI

20,557 Aufrufe • vor 2 Monaten