Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

From the Hamster Labs lab: Introducing Qwen3.8-Flash-Next-REAP-288-MLX-4bit, a 180B-class model running on just 39gb of memory - MLX-native 4-bit 60% smaller than stock q4 - Pruned 512→288 experts via REAP - 91.5% HumanEval (vs 93.9% stock) Hugging Face links below ↓

EyalToledano's profile picture

Eyal Toledano

18,315 subscribers

45,168 Aufrufe • vor 6 Tagen •via X (Twitter)

Wissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Im done with GLM-5.3 Flash 🤯 I went deep into the guts of GLM-5.3 Flash. Ripped out every expert, every memory head, every quantization scale, looked at all of it up close, then put the model back together so you can see the inside too. 🎉 GLM-5.3 Flash NVFP4 is now in Weight Atlas. 320B total, 18B active, 45 language layers and a 24 block Vision tower. And none of it is read from the config. I ran the deployed NVFP4 checkpoint itself on 4x RTX PRO 6000 and measured what it does. - 12,096 expert cells, 42 layers x 288 experts, each with exact REAP importance from 12.59M tokens per layer, route share and output contribution. Slice it by 14 domains, russian and vision included - one chart invalidated an assumption I had. Route count does not track expert importance, Spearman -0.222 against exact REAP. The ranking itself is stable, 0.994 split half - 2,176 KDA memory heads, each with its own half life, from 0.25 tokens to 2.7M - 19 billion NVFP4 block scales, plus the deployed FC2 QDQ error: median 21.05 dB and 21.7% of values turn into exact zeros - a causal check of REAP at inference, weights untouched. Dropping the top 2% experts changes 79% of sequences, dropping the bottom 2% changes 64% Every number comes from a capture I checksummed, 15 artifacts, and the limits are written next to each chart. Sound on I will record a video walk through every chart later. For now go poke around inside it:
3:10

Im done with GLM-5.3 Flash 🤯 I went deep into the guts of GLM-5.3 Flash. Ripped out every expert, every memory head, every quantization scale, looked at all of it up close, then put the model back together so you can see the inside too. 🎉 GLM-5.3 Flash NVFP4 is now in Weight Atlas. 320B total, 18B active, 45 language layers and a 24 block Vision tower. And none of it is read from the config. I ran the deployed NVFP4 checkpoint itself on 4x RTX PRO 6000 and measured what it does. - 12,096 expert cells, 42 layers x 288 experts, each with exact REAP importance from 12.59M tokens per layer, route share and output contribution. Slice it by 14 domains, russian and vision included - one chart invalidated an assumption I had. Route count does not track expert importance, Spearman -0.222 against exact REAP. The ranking itself is stable, 0.994 split half - 2,176 KDA memory heads, each with its own half life, from 0.25 tokens to 2.7M - 19 billion NVFP4 block scales, plus the deployed FC2 QDQ error: median 21.05 dB and 21.7% of values turn into exact zeros - a causal check of REAP at inference, weights untouched. Dropping the top 2% experts changes 79% of sequences, dropping the bottom 2% changes 64% Every number comes from a capture I checksummed, 15 artifacts, and the limits are written next to each chart. Sound on I will record a video walk through every chart later. For now go poke around inside it:

Alexey Fateev

28,925 Aufrufe • vor 18 Stunden

The VRAM barrier is officially dead. I just ran Qwen 3.8 Flash Next (MoE) 125B A6B with a 250,000 context window on a single 24GB RTX 4090. 21 tokens/sec decode. 364 t/s prefill. no mtp. no dflash. no kv cache quantization! We are running datacenter models on consumer hardware. Tested on Ubuntu 22 | CUDA 13.0 | PCIe 4.0 x16 | 110 GB DDR4 System RAM with a continuous 28k prompt across all runs. ### The Benchmarks & Scaling # 1. Hybrid Offload (-ncmoe 40 @ 80k Context) Offloaded 40 expert layers to the GPU, pushing VRAM to the ceiling. ./build/bin/llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf -c 80000 --port 8080 -v --fit off -b 4096 -ub 4096 -ncmoe 40 Prefill: 383.85 t/s | Decode: 22.52 t/s Footprint: 23.85 GB VRAM | 97 GB RAM # 2. Full CPU MoE Offload (-cmoe @ 80k Context) Pinned all 512 expert layers to DDR4 RAM (-cmoe), keeping attention on the 4090. llama.cpp flags: (Same as above, replace -ncmoe 40 with -cmoe) Prefill: 355.72 t/s | Decode: 20.84 t/s Footprint: 11.66 GB VRAM (12GB+ VRAM freed up!) | 110 GB RAM # 3. The 180,000 Context Run Prefill: 357.75 t/s | Decode: 20.98 t/s | VRAM: 15.6 GB | RAM: 110 GB # 4. The 250,000 Context Absolute Ceiling ./build/bin/llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf -c 250000 --port 8080 -v --fit off -b 4096 -ub 4096 -cmoe Prefill: 364.29 t/s | Decode: 20.97 t/s Footprint: 18.3 GB VRAM (Still ~5.7 GB of VRAM headroom!) | 110 GB RAM ### Key Insights: -b 4096 -ub 4096: doubles the prompt ingestion from ~150 to 364+ t/s. -cmoe Free Lunch: Shifting expert layers to DDR4 RAM slashes VRAM from 24GB to 11.6GB with virtually zero decode penalty (22.5 -> 20.9 t/s), enabling the 250k context ceiling. Qwen 3.8 Flash-Next (UD-Q4_K_XL) is a massive 111.4 GB model split across 4 shards. To run this architecture, you must build from the experimental PR branch (#27742) by Daniel Han: git clone && cd llama.cpp git fetch origin pull/27742/head:qwen-next && git checkout qwen-next cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native -DBUILD_SHARED_LIBS=OFF cmake --build build --config Release -j $(nproc) --target llama-server A single 4090 paired with 100 GB of cheap DDR4 RAM will comfortably serve production grade 125B inference. While Qwen 3.8 27B (dense) still holds the crown for single 3090/4090 rigs, Flash Next proves 125B hybrid models are officially viable on consumer hardware. Hugging Face GGUF link and complete performance telemetry graphs are dropped in the replies below. GLM 5.3 Flash VS Qwen 3.8 Flash Next, which one takes the open weights crown this week?
0:20

The VRAM barrier is officially dead. I just ran Qwen 3.8 Flash Next (MoE) 125B A6B with a 250,000 context window on a single 24GB RTX 4090. 21 tokens/sec decode. 364 t/s prefill. no mtp. no dflash. no kv cache quantization! We are running datacenter models on consumer hardware. Tested on Ubuntu 22 | CUDA 13.0 | PCIe 4.0 x16 | 110 GB DDR4 System RAM with a continuous 28k prompt across all runs. ### The Benchmarks & Scaling # 1. Hybrid Offload (-ncmoe 40 @ 80k Context) Offloaded 40 expert layers to the GPU, pushing VRAM to the ceiling. ./build/bin/llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf -c 80000 --port 8080 -v --fit off -b 4096 -ub 4096 -ncmoe 40 Prefill: 383.85 t/s | Decode: 22.52 t/s Footprint: 23.85 GB VRAM | 97 GB RAM # 2. Full CPU MoE Offload (-cmoe @ 80k Context) Pinned all 512 expert layers to DDR4 RAM (-cmoe), keeping attention on the 4090. llama.cpp flags: (Same as above, replace -ncmoe 40 with -cmoe) Prefill: 355.72 t/s | Decode: 20.84 t/s Footprint: 11.66 GB VRAM (12GB+ VRAM freed up!) | 110 GB RAM # 3. The 180,000 Context Run Prefill: 357.75 t/s | Decode: 20.98 t/s | VRAM: 15.6 GB | RAM: 110 GB # 4. The 250,000 Context Absolute Ceiling ./build/bin/llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf -c 250000 --port 8080 -v --fit off -b 4096 -ub 4096 -cmoe Prefill: 364.29 t/s | Decode: 20.97 t/s Footprint: 18.3 GB VRAM (Still ~5.7 GB of VRAM headroom!) | 110 GB RAM ### Key Insights: -b 4096 -ub 4096: doubles the prompt ingestion from ~150 to 364+ t/s. -cmoe Free Lunch: Shifting expert layers to DDR4 RAM slashes VRAM from 24GB to 11.6GB with virtually zero decode penalty (22.5 -> 20.9 t/s), enabling the 250k context ceiling. Qwen 3.8 Flash-Next (UD-Q4_K_XL) is a massive 111.4 GB model split across 4 shards. To run this architecture, you must build from the experimental PR branch (#27742) by Daniel Han: git clone && cd llama.cpp git fetch origin pull/27742/head:qwen-next && git checkout qwen-next cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native -DBUILD_SHARED_LIBS=OFF cmake --build build --config Release -j $(nproc) --target llama-server A single 4090 paired with 100 GB of cheap DDR4 RAM will comfortably serve production grade 125B inference. While Qwen 3.8 27B (dense) still holds the crown for single 3090/4090 rigs, Flash Next proves 125B hybrid models are officially viable on consumer hardware. Hugging Face GGUF link and complete performance telemetry graphs are dropped in the replies below. GLM 5.3 Flash VS Qwen 3.8 Flash Next, which one takes the open weights crown this week?

Alok

997,384 Aufrufe • vor 7 Tagen

MTPLX V2.10 is out! I put a lot of work into this one. Highlights: - Qwen 3.8 Next Support Added: Bare Speed (4 bit) and Optimised Speed (4 bit/8 bit) models added running at 50 - 80 TPS sustained at high contexts. (expect speed improvements in the coming days) - Qwen 3.8 Next mmap: The 32 GB n-gram table streams from SSD via mmap instead of RAM. Both models fit a 96 GB Mac with 0 speed impact. - Qwen 3.8 27B Decode & Prefill boost EVERYWHERE: +15% TPS at 3k context, +29% at 88k, +54% at 147k. Prefill +41%. - 48 GB Macs went from 3 to 4 TPS in swap @ 30k context to 33 TPS. - KV Cache has been solved!: 8 bit and 4 bit cache now barely impacts speed instead of a 50% decline. - Coding agents wall time optimisations: file task dropped from 150s to 44s, and mid-session first token went from ~2s to 0.11s. - CLI UX improvements + 34 bug fixes. My thoughts on Qwen Next? Incredible model. Enormously better at design, and creative tasks with scary good vision. It created what I think is the best flappy bird. maybe even better than the Fable version. It had various difficulties and skins, moving pillars, and upgrades. It also had the most memeified look. Incredibly impressed. However, it is slightly worse than 27B at having code just work straight away. But it's personality is also far better than 27B. Here is the Flappy bird it made below! Made with Qwen Next Optimized Speed one shot@ 57 TPS:
1:16

MTPLX V2.10 is out! I put a lot of work into this one. Highlights: - Qwen 3.8 Next Support Added: Bare Speed (4 bit) and Optimised Speed (4 bit/8 bit) models added running at 50 - 80 TPS sustained at high contexts. (expect speed improvements in the coming days) - Qwen 3.8 Next mmap: The 32 GB n-gram table streams from SSD via mmap instead of RAM. Both models fit a 96 GB Mac with 0 speed impact. - Qwen 3.8 27B Decode & Prefill boost EVERYWHERE: +15% TPS at 3k context, +29% at 88k, +54% at 147k. Prefill +41%. - 48 GB Macs went from 3 to 4 TPS in swap @ 30k context to 33 TPS. - KV Cache has been solved!: 8 bit and 4 bit cache now barely impacts speed instead of a 50% decline. - Coding agents wall time optimisations: file task dropped from 150s to 44s, and mid-session first token went from ~2s to 0.11s. - CLI UX improvements + 34 bug fixes. My thoughts on Qwen Next? Incredible model. Enormously better at design, and creative tasks with scary good vision. It created what I think is the best flappy bird. maybe even better than the Fable version. It had various difficulties and skins, moving pillars, and upgrades. It also had the most memeified look. Incredibly impressed. However, it is slightly worse than 27B at having code just work straight away. But it's personality is also far better than 27B. Here is the Flappy bird it made below! Made with Qwen Next Optimized Speed one shot@ 57 TPS:

Youssof Al Toukhi

20,501 Aufrufe • vor 5 Tagen

nemotron 3 omni q8 on dgx spark 128gb vram cranking via hermes agent at 56 tok/s. first night of real local agentic on this box and local hits harder than i thought it would. q8 (near lossless quant, perplexity loss <1% vs fp16) running 256k context on 33 gb of unified memory, 90+ gb still free. multimodal omni weights included. hermes agent driving from telegram, talking to it from bed. speed: 56 tok/s generation, 1,300 tok/s prefill. for context, qwen 3.6 27b at q4 (heavy quant) on 3090 = 40 tok/s. nemotron at higher precision quant on spark beats qwen at lower precision quant on 3090. moe 3.5b active params architecture earns its keep. what i tested tonight: agentic tool calling works clean. ask it to check disks, it autonomously runs df -h through hermes agent. ask it to set up telegram gateway, it invokes the hermes-agent skill, walks through the prompts, completes the flow. overthinks a bit before tool calls (reasoning model trait) but lands the right move every time. researches api docs, internalizes, tests, documents. completes tasks. current models on dgx spark: 9 gguf files, 305 gb total, mix of qwen 3.6 27b dense (5 quants), nemotron omni (4 quants), deepseek v4-flash 158b q4 (the 112gb flagship test). more data coming this week as i benchmark each.
1:40

nemotron 3 omni q8 on dgx spark 128gb vram cranking via hermes agent at 56 tok/s. first night of real local agentic on this box and local hits harder than i thought it would. q8 (near lossless quant, perplexity loss <1% vs fp16) running 256k context on 33 gb of unified memory, 90+ gb still free. multimodal omni weights included. hermes agent driving from telegram, talking to it from bed. speed: 56 tok/s generation, 1,300 tok/s prefill. for context, qwen 3.6 27b at q4 (heavy quant) on 3090 = 40 tok/s. nemotron at higher precision quant on spark beats qwen at lower precision quant on 3090. moe 3.5b active params architecture earns its keep. what i tested tonight: agentic tool calling works clean. ask it to check disks, it autonomously runs df -h through hermes agent. ask it to set up telegram gateway, it invokes the hermes-agent skill, walks through the prompts, completes the flow. overthinks a bit before tool calls (reasoning model trait) but lands the right move every time. researches api docs, internalizes, tests, documents. completes tasks. current models on dgx spark: 9 gguf files, 305 gb total, mix of qwen 3.6 27b dense (5 quants), nemotron omni (4 quants), deepseek v4-flash 158b q4 (the 112gb flagship test). more data coming this week as i benchmark each.

Sudo su

30,228 Aufrufe • vor 4 Monaten

Karpathy told Dwarkesh that a 1 billion parameter model, trained on clean data, could hit the intelligence of today's 1.8 trillion parameter frontier. That is a 1,800x compression claim. The math behind it is more defensible than it sounds. When researchers at frontier labs look at random samples from their training corpus, they see stock ticker symbols, broken HTML, forum spam, autogenerated gibberish. Not Wikipedia. Not the Wall Street Journal. The actual pretraining dataset is mostly noise, and the model is burning parameters to vaguely remember all of it. One estimate pegs Llama 3's information compression at 0.07 bits per token. Well-structured English carries around 1.5 bits per token of real information. The trillion-parameter model is holding a roughly 5% resolution image of the internet it trained on. So when a lab ships a 1.8 trillion parameter model, the overwhelming majority of those weights are handling rough memorization. They are compression overhead for a noisy training set, taking up capacity that could be doing reasoning instead. Karpathy's proposal is to separate the two. Build a cognitive core: a small model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization. Pair it with external memory the model queries when it needs a fact. A 1 billion parameter reasoner plus retrieval beats a 1.8 trillion parameter model trying to do both. The data already supports this direction. GPT-4o runs at roughly 200 billion parameters and outperforms the original 1.8 trillion GPT-4. Inference costs for GPT-3.5 level performance fell 280x between 2022 and 2024, driven almost entirely by smaller, cleaner, better-architected models. The trend line is pointing where Karpathy says it should. The real implication for anyone tracking the AI trade: data quality is the actual constraint. The companies winning the next phase will be the ones who figured out what to train on, and what to throw away.
2:00

Karpathy told Dwarkesh that a 1 billion parameter model, trained on clean data, could hit the intelligence of today's 1.8 trillion parameter frontier. That is a 1,800x compression claim. The math behind it is more defensible than it sounds. When researchers at frontier labs look at random samples from their training corpus, they see stock ticker symbols, broken HTML, forum spam, autogenerated gibberish. Not Wikipedia. Not the Wall Street Journal. The actual pretraining dataset is mostly noise, and the model is burning parameters to vaguely remember all of it. One estimate pegs Llama 3's information compression at 0.07 bits per token. Well-structured English carries around 1.5 bits per token of real information. The trillion-parameter model is holding a roughly 5% resolution image of the internet it trained on. So when a lab ships a 1.8 trillion parameter model, the overwhelming majority of those weights are handling rough memorization. They are compression overhead for a noisy training set, taking up capacity that could be doing reasoning instead. Karpathy's proposal is to separate the two. Build a cognitive core: a small model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization. Pair it with external memory the model queries when it needs a fact. A 1 billion parameter reasoner plus retrieval beats a 1.8 trillion parameter model trying to do both. The data already supports this direction. GPT-4o runs at roughly 200 billion parameters and outperforms the original 1.8 trillion GPT-4. Inference costs for GPT-3.5 level performance fell 280x between 2022 and 2024, driven almost entirely by smaller, cleaner, better-architected models. The trend line is pointing where Karpathy says it should. The real implication for anyone tracking the AI trade: data quality is the actual constraint. The companies winning the next phase will be the ones who figured out what to train on, and what to throw away.

Aakash Gupta

508,446 Aufrufe • vor 4 Monaten

The most downloaded AI on earth is now Chinese. Alibaba just gave away a model that matches Claude's flagship, and it literally runs on a $700 used graphics card. The Qwen models crossed 3 BILLION downloads in six months. Hugging Face counted 418 million downloads for Google this year, and 227 million for Meta. Alibaba cleared more than four times both of them combined. Then today it released Qwen3.8-27B under an Apache 2.0 license. The model has 27 billion parameters, native vision, and a 262,000 token context window. Developers are running it locally on 17 gigabytes of memory, on used cards that cost a few hundred dollars. Alibaba's own benchmark table claims it beats Opus 4.6 Max on computer use by 84.3 to 72.7, on mobile use by 81.9 to 62, and on visual math by 94.6 to 65.5. Those numbers come from the vendor and nobody has independently verified them yet, so treat them as a claim. But the generation over generation jumps are harder to wave away: On DeepSWE the score went from 13.3 to 42.2. On software engineering it went from 49.3 to 79.0. That happened in ONE release cycle. And Apache 2.0 means anyone can download the weights, modify them, build products on them, sell those products, and never pay or ask permission. It cannot be revoked. Once the file is on your drive it is yours permanently. 3 billion downloads means those files already sit on machines in every country on Earth. Alibaba could delete everything tomorrow and it would change nothing. Washington spent 4 years building an export control regime around chips, model weights, and entity lists. Every piece of it assumes a chokepoint exists somewhere. A fab, a shipment, a company that can be told no. But there is no chokepoint for a file that has already been copied three billion times. And the copying compounds. Hugging Face counted 151,448 models built on top of Qwen, which is 2.6x Meta's entire footprint and 4.7x the number of Llama repositories. New ones appear at roughly 200 a day. The report says Qwen has become "part of the default workflow for developers deciding what models to fine-tune and deploy." Alibaba is also pushing Qwen through its cloud into Southeast Asia and Africa, markets where American labs have almost no presence, and where a very large share of the next generation of developers will learn to build. Meta and Nvidia have both rushed out new open models in recent weeks. That is what a response looks like when you feel the floor move. And to be clear, these are download and derivative numbers, not usage numbers. ChatGPT and Claude cannot be downloaded at all, so they do not appear in this comparison. What the figures measure is what developers choose to build on top of, which is a different question from what consumers type into a box. That is also why it matters MORE. Consumer habits change in an afternoon. Infrastructure choices last a decade, because everything built on top has to be rewritten to undo them. The American labs are valued on an assumption that frontier intelligence stays scarce, expensive, and rented by the token. Alibaba just made a version of it free, permanent, and small enough to run on hardware people already own. You will not get an announcement when the software you use every day starts running on a Chinese model underneath. Go and count how many of the tools you rely on could be rebuilt on free weights this year.
0:39

The most downloaded AI on earth is now Chinese. Alibaba just gave away a model that matches Claude's flagship, and it literally runs on a $700 used graphics card. The Qwen models crossed 3 BILLION downloads in six months. Hugging Face counted 418 million downloads for Google this year, and 227 million for Meta. Alibaba cleared more than four times both of them combined. Then today it released Qwen3.8-27B under an Apache 2.0 license. The model has 27 billion parameters, native vision, and a 262,000 token context window. Developers are running it locally on 17 gigabytes of memory, on used cards that cost a few hundred dollars. Alibaba's own benchmark table claims it beats Opus 4.6 Max on computer use by 84.3 to 72.7, on mobile use by 81.9 to 62, and on visual math by 94.6 to 65.5. Those numbers come from the vendor and nobody has independently verified them yet, so treat them as a claim. But the generation over generation jumps are harder to wave away: On DeepSWE the score went from 13.3 to 42.2. On software engineering it went from 49.3 to 79.0. That happened in ONE release cycle. And Apache 2.0 means anyone can download the weights, modify them, build products on them, sell those products, and never pay or ask permission. It cannot be revoked. Once the file is on your drive it is yours permanently. 3 billion downloads means those files already sit on machines in every country on Earth. Alibaba could delete everything tomorrow and it would change nothing. Washington spent 4 years building an export control regime around chips, model weights, and entity lists. Every piece of it assumes a chokepoint exists somewhere. A fab, a shipment, a company that can be told no. But there is no chokepoint for a file that has already been copied three billion times. And the copying compounds. Hugging Face counted 151,448 models built on top of Qwen, which is 2.6x Meta's entire footprint and 4.7x the number of Llama repositories. New ones appear at roughly 200 a day. The report says Qwen has become "part of the default workflow for developers deciding what models to fine-tune and deploy." Alibaba is also pushing Qwen through its cloud into Southeast Asia and Africa, markets where American labs have almost no presence, and where a very large share of the next generation of developers will learn to build. Meta and Nvidia have both rushed out new open models in recent weeks. That is what a response looks like when you feel the floor move. And to be clear, these are download and derivative numbers, not usage numbers. ChatGPT and Claude cannot be downloaded at all, so they do not appear in this comparison. What the figures measure is what developers choose to build on top of, which is a different question from what consumers type into a box. That is also why it matters MORE. Consumer habits change in an afternoon. Infrastructure choices last a decade, because everything built on top has to be rewritten to undo them. The American labs are valued on an assumption that frontier intelligence stays scarce, expensive, and rented by the token. Alibaba just made a version of it free, permanent, and small enough to run on hardware people already own. You will not get an announcement when the software you use every day starts running on a Chinese model underneath. Go and count how many of the tools you rely on could be rebuilt on free weights this year.

Ricardo

80,611 Aufrufe • vor 17 Tagen

Alibaba just released a coding model that hits 82 percent on SWE-Bench Verified. That is the highest score ever published for an open-source model. The weights are free. The license is Apache 2.0. You can run it today. The model is Qwen 4 Coder 32B. Here is what 82 percent on SWE-Bench Verified actually means. SWE-Bench Verified tests whether an AI can autonomously resolve real bugs pulled from real production GitHub repositories. Not synthetic exercises. Real open-source projects that real teams depend on. A model gets a bug report, reads the code, writes a fix, and either passes the test suite or it does not. At 82 percent, Qwen 4 Coder 32B resolves 82 out of every 100 real production bugs it is given. Without a human guiding it. On code it has never seen before. For comparison: Qwen 4 Coder 32B: 82 percent SWE-Bench Verified. Open source. Apache 2.0. Claude Fable 5: 80.3 percent SWE-Bench Pro. $10 input / $50 output per million tokens. Currently suspended. GPT-5.6 Sol: Competitive on Terminal-Bench. $5 input / $30 output per million tokens. An open-weight model that you can download and run for free just beat both of them on the benchmark designed to measure real software engineering capability. Here is the architecture. Qwen 4 Coder 32B is a 32 billion parameter dense model. Not a Mixture-of-Experts. Every parameter is active on every request. This matters for inference: a dense 32B model runs on 22 gigabytes of VRAM, which fits on a single high-end consumer GPU or a MacBook Pro with 64GB of unified memory. The smaller variant, Qwen 4 Coder 4B, runs at approximately 135 tokens per second on an M5 Max and fits inside 8 gigabytes of RAM. For a model with usable coding capability, that is a new bar for what fits in a single laptop. The training methodology continued Alibaba's approach of reinforcement learning on verifiable coding tasks. The model gets rewarded when its code passes tests. It gets penalized when it fails. Over millions of training steps, the model learns to write code that actually runs rather than code that looks plausible. License: Apache 2.0. Full commercial use. No attribution requirement. No revenue threshold. No monthly active user ceiling. Weights: Hugging Face, available today. Runs on: vLLM, Ollama, SGLang, and any standard GGUF-compatible inference engine. Qwen 4 32B also runs at approximately 135 tokens per second on an M5 Max chip, setting a new bar for what a sub-8GB model can do on Apple Silicon. The open-source coding model just beat the best closed-source model in the world on the benchmark designed to test whether AI can actually do software engineering. The weights are free. The subscription is optional. Source: Autom8Labs AI Insight July 2026, State of Open Source LLMs June 2026, Kunal Ganglani blog June 2026.
0:25

Alibaba just released a coding model that hits 82 percent on SWE-Bench Verified. That is the highest score ever published for an open-source model. The weights are free. The license is Apache 2.0. You can run it today. The model is Qwen 4 Coder 32B. Here is what 82 percent on SWE-Bench Verified actually means. SWE-Bench Verified tests whether an AI can autonomously resolve real bugs pulled from real production GitHub repositories. Not synthetic exercises. Real open-source projects that real teams depend on. A model gets a bug report, reads the code, writes a fix, and either passes the test suite or it does not. At 82 percent, Qwen 4 Coder 32B resolves 82 out of every 100 real production bugs it is given. Without a human guiding it. On code it has never seen before. For comparison: Qwen 4 Coder 32B: 82 percent SWE-Bench Verified. Open source. Apache 2.0. Claude Fable 5: 80.3 percent SWE-Bench Pro. $10 input / $50 output per million tokens. Currently suspended. GPT-5.6 Sol: Competitive on Terminal-Bench. $5 input / $30 output per million tokens. An open-weight model that you can download and run for free just beat both of them on the benchmark designed to measure real software engineering capability. Here is the architecture. Qwen 4 Coder 32B is a 32 billion parameter dense model. Not a Mixture-of-Experts. Every parameter is active on every request. This matters for inference: a dense 32B model runs on 22 gigabytes of VRAM, which fits on a single high-end consumer GPU or a MacBook Pro with 64GB of unified memory. The smaller variant, Qwen 4 Coder 4B, runs at approximately 135 tokens per second on an M5 Max and fits inside 8 gigabytes of RAM. For a model with usable coding capability, that is a new bar for what fits in a single laptop. The training methodology continued Alibaba's approach of reinforcement learning on verifiable coding tasks. The model gets rewarded when its code passes tests. It gets penalized when it fails. Over millions of training steps, the model learns to write code that actually runs rather than code that looks plausible. License: Apache 2.0. Full commercial use. No attribution requirement. No revenue threshold. No monthly active user ceiling. Weights: Hugging Face, available today. Runs on: vLLM, Ollama, SGLang, and any standard GGUF-compatible inference engine. Qwen 4 32B also runs at approximately 135 tokens per second on an M5 Max chip, setting a new bar for what a sub-8GB model can do on Apple Silicon. The open-source coding model just beat the best closed-source model in the world on the benchmark designed to test whether AI can actually do software engineering. The weights are free. The subscription is optional. Source: Autom8Labs AI Insight July 2026, State of Open Source LLMs June 2026, Kunal Ganglani blog June 2026.

Harman

41,278 Aufrufe • vor 1 Monat

Yesterday, steve jang shared his thoughts on CNBC regarding AI’s future roadmap around memory, robotics, agents, and open weights models. The hot topic of the morning: SK hynix reported record Q2 results, then fell more than 9%. The gap between the print and the reaction raises a larger question: is the market applying an old memory-cycle framework to a new AI infrastructure layer? Q2 revenue reached USD $55.0 billion, up an eye-popping 257% year over year. Operating profit rose to USD $42.0 billion, up incredibly 557%, with a company-record operating margin of 76%. But both missed consensus estimates. As our partner Steve Jang told Becky Quick on Squawk Box, “the company has incredible fundamentals, record-breaking numbers, and long term HBM technical defensibility…but expectations were just very high.” Steve’s larger argument begins with high-bandwidth memory, or HBM. For two decades, investors largely treated memory as a cyclical commodity. Now, Steve argues, HBM is moving into a new role, “sitting side-by-side with GPUs and other advanced logic processors” as a core layer of AI compute. Demand now spans model training, inference, long-context agents, robotics, and autonomous systems. Steve pointed to deep-research products like Perplexity’s Computer agent platform: as agents hold and reason across more context, their memory bandwidth and capacity needs grow. Robotaxis like Nuro+Uber and Waymo will need onboard edge compute including HBMs. He estimates HBM demand could increase 10x over the next 3 to 4 years. SK hynix enters that buildout from a strong position. Long-term supply agreements with major customers provide demand visibility. Stacked-die architecture, advanced packaging, yield, and customer qualification create a steep technical and manufacturing climb. Steve estimates that a new entrant could need three to five years to significantly enter the HBM4e class. That supports a credible near-term moat while leaving the harder question open: how today’s 76% operating margin evolves as supply expands over the next two to three years. One thing is clear: demand and importance of high bandwidth memory is still wildly underestimated. The second half of the conversation moved from compute capacity to operational control. During the recent OpenAI and Hugging Face security incident, commercial frontier-model APIs blocked the attack commands and exploit payloads contained in forensic logs. Hugging Face instead ran GLM 5.2, an open-weight model, on its own infrastructure to analyze more than 17,000 recorded events without sending incident data or credentials outside its environment. Steve’s takeaway is practical. As autonomous agents grow more capable, defenders need access to models they can host and direct when hosted guardrails block legitimate forensic work. The next phase of AI will depend on both: enough memory bandwidth alongside GPUs and other accelerators to scale increasingly capable systems, and enough model ownership and control to deploy and defend them efficiently and safely. Full conversation below 👇
7:13

Yesterday, steve jang shared his thoughts on CNBC regarding AI’s future roadmap around memory, robotics, agents, and open weights models. The hot topic of the morning: SK hynix reported record Q2 results, then fell more than 9%. The gap between the print and the reaction raises a larger question: is the market applying an old memory-cycle framework to a new AI infrastructure layer? Q2 revenue reached USD $55.0 billion, up an eye-popping 257% year over year. Operating profit rose to USD $42.0 billion, up incredibly 557%, with a company-record operating margin of 76%. But both missed consensus estimates. As our partner Steve Jang told Becky Quick on Squawk Box, “the company has incredible fundamentals, record-breaking numbers, and long term HBM technical defensibility…but expectations were just very high.” Steve’s larger argument begins with high-bandwidth memory, or HBM. For two decades, investors largely treated memory as a cyclical commodity. Now, Steve argues, HBM is moving into a new role, “sitting side-by-side with GPUs and other advanced logic processors” as a core layer of AI compute. Demand now spans model training, inference, long-context agents, robotics, and autonomous systems. Steve pointed to deep-research products like Perplexity’s Computer agent platform: as agents hold and reason across more context, their memory bandwidth and capacity needs grow. Robotaxis like Nuro+Uber and Waymo will need onboard edge compute including HBMs. He estimates HBM demand could increase 10x over the next 3 to 4 years. SK hynix enters that buildout from a strong position. Long-term supply agreements with major customers provide demand visibility. Stacked-die architecture, advanced packaging, yield, and customer qualification create a steep technical and manufacturing climb. Steve estimates that a new entrant could need three to five years to significantly enter the HBM4e class. That supports a credible near-term moat while leaving the harder question open: how today’s 76% operating margin evolves as supply expands over the next two to three years. One thing is clear: demand and importance of high bandwidth memory is still wildly underestimated. The second half of the conversation moved from compute capacity to operational control. During the recent OpenAI and Hugging Face security incident, commercial frontier-model APIs blocked the attack commands and exploit payloads contained in forensic logs. Hugging Face instead ran GLM 5.2, an open-weight model, on its own infrastructure to analyze more than 17,000 recorded events without sending incident data or credentials outside its environment. Steve’s takeaway is practical. As autonomous agents grow more capable, defenders need access to models they can host and direct when hosted guardrails block legitimate forensic work. The next phase of AI will depend on both: enough memory bandwidth alongside GPUs and other accelerators to scale increasingly capable systems, and enough model ownership and control to deploy and defend them efficiently and safely. Full conversation below 👇

Kindred Ventures

1,736,819 Aufrufe • vor 1 Monat

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.
0:42

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

336,634 Aufrufe • vor 12 Tagen

Google just quit the AI race on purpose, and it is about to make MORE money than everyone still running it. 4 of the most cited AI researchers alive walked out of Google in a single afternoon. Jeff Dean, the man who built the systems Google runs on, gone after 27 years. Sanjay Ghemawat, his longtime partner, gone. Oriol Vinyals, a Gemini co-lead, gone. Quoc Le, a Google Brain co-founder, gone. That same day, Demis Hassabis stepped back from running DeepMind. Hassabis co-founded the lab, won a Nobel Prize for AlphaFold, and had been the face of Google AI for a decade. The stock dropped 5% within hours. Analysts called it a brain drain. Headlines called it the day Google fell behind. But turns out that's completely wrong, because the numbers underneath tell a completely different story: Google is not trying to win the frontier model race anymore. It looked at where the money is and walked toward it. Gemini, Google's flagship model business, generated about $12 billion in annual revenue last quarter. That is the entire payoff from competing head to head with OpenAI and Anthropic. Now look at the other number. By the end of 2027, Google Cloud is projected to do over $73 billion selling AI infrastructure to other companies, plus another $120 billion selling its TPU chips. That is roughly $200 billion of external sales at high margins, against a $12 billion model business. Google understood that the frontier race is the expensive part while selling the shovels is the profitable part. And the customers buying those shovels include Google's own rivals. Over 20% of Google's TPU shipments for 2026 and 2027 are going to Anthropic, one of the two labs supposedly beating Gemini. Google now makes money every time Anthropic trains a model designed to crush Google's OWN product. Cede the frontier, own the layer underneath it, and collect a toll from everyone racing across the top. The researchers leaving is the symptom of a company that already decided models are not where it wins. Jeff Dean said it himself on the way out. He told the New York Times that leaving a public company gives him room to make decisions "not necessarily in the company's purist financial interests." Read that from Google's side: The people who wanted to chase the science left, because Google is now optimizing for the FINANCIAL interest. Gemini 3.5 Pro is running months behind, with staff blaming low morale. DeepMind's comms, legal, and marketing teams are being folded into Google proper. A former manager told the Guardian the era of DeepMind as an independent lab is over. None of that reads as failure once you see the strategy. Yet Wall Street is pricing this as Google losing. The parallel that should worry the frontier labs: If open weight models keep compressing the price of inference, being the best model stops being a business. It becomes like semiconductor fabrication, strategically vital and financially brutal, a race you win and still lose money running. Google is the first giant to admit that. The company that invented the transformer just handed the frontier to OpenAI and Anthropic, and positioned itself to get paid on every model both of them ship. Those labs will be burning billions to stay one benchmark ahead, and Google will be cashing in hundreds of billions from it. The model business is actually just a race where everyone loses. Apple understood that from the get-go and never joined the race, Google understood it now and left it to OpenAI and Anthropic. Who will go bankrupt first?
2:05

Google just quit the AI race on purpose, and it is about to make MORE money than everyone still running it. 4 of the most cited AI researchers alive walked out of Google in a single afternoon. Jeff Dean, the man who built the systems Google runs on, gone after 27 years. Sanjay Ghemawat, his longtime partner, gone. Oriol Vinyals, a Gemini co-lead, gone. Quoc Le, a Google Brain co-founder, gone. That same day, Demis Hassabis stepped back from running DeepMind. Hassabis co-founded the lab, won a Nobel Prize for AlphaFold, and had been the face of Google AI for a decade. The stock dropped 5% within hours. Analysts called it a brain drain. Headlines called it the day Google fell behind. But turns out that's completely wrong, because the numbers underneath tell a completely different story: Google is not trying to win the frontier model race anymore. It looked at where the money is and walked toward it. Gemini, Google's flagship model business, generated about $12 billion in annual revenue last quarter. That is the entire payoff from competing head to head with OpenAI and Anthropic. Now look at the other number. By the end of 2027, Google Cloud is projected to do over $73 billion selling AI infrastructure to other companies, plus another $120 billion selling its TPU chips. That is roughly $200 billion of external sales at high margins, against a $12 billion model business. Google understood that the frontier race is the expensive part while selling the shovels is the profitable part. And the customers buying those shovels include Google's own rivals. Over 20% of Google's TPU shipments for 2026 and 2027 are going to Anthropic, one of the two labs supposedly beating Gemini. Google now makes money every time Anthropic trains a model designed to crush Google's OWN product. Cede the frontier, own the layer underneath it, and collect a toll from everyone racing across the top. The researchers leaving is the symptom of a company that already decided models are not where it wins. Jeff Dean said it himself on the way out. He told the New York Times that leaving a public company gives him room to make decisions "not necessarily in the company's purist financial interests." Read that from Google's side: The people who wanted to chase the science left, because Google is now optimizing for the FINANCIAL interest. Gemini 3.5 Pro is running months behind, with staff blaming low morale. DeepMind's comms, legal, and marketing teams are being folded into Google proper. A former manager told the Guardian the era of DeepMind as an independent lab is over. None of that reads as failure once you see the strategy. Yet Wall Street is pricing this as Google losing. The parallel that should worry the frontier labs: If open weight models keep compressing the price of inference, being the best model stops being a business. It becomes like semiconductor fabrication, strategically vital and financially brutal, a race you win and still lose money running. Google is the first giant to admit that. The company that invented the transformer just handed the frontier to OpenAI and Anthropic, and positioned itself to get paid on every model both of them ship. Those labs will be burning billions to stay one benchmark ahead, and Google will be cashing in hundreds of billions from it. The model business is actually just a race where everyone loses. Apple understood that from the get-go and never joined the race, Google understood it now and left it to OpenAI and Anthropic. Who will go bankrupt first?

Ricardo

241,927 Aufrufe • vor 22 Tagen

I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.
0:20

I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.

Alok

36,617 Aufrufe • vor 1 Monat

Qwen 3.8 27B on hit 3.3x faster decode in 7 days. Here's what happened and what we're thinking next. Result (so far) Median decode speed increased from 26 tok/s to 87.9 tok/s on the verifier M5 Max (33 to 93.1 tok/s across the eight prompts), with prefill around 971.8 tok/s. This came out of a collective effort: 31 solvers across 67 improvements. Most of the recent ones run custom MTP heads that draft and accept ~3.9 tokens per round while still matching serial output exactly. Why this matters Beyond the performance itself, two things stand out to me. (1) Dense models on Apple Silicon were supposed to be the hard case. "Everyone knows Macs are slow at dense models." But watching the community take it from the usual baseline to >3x in seven days shows the low-hanging fruit was still there. (2) Open-weight models have been small and effective for a while. This is the first time one is small and frontier. Qwen 3.8 27B is an extremely strong dense model, comparable in capability to Opus 4.6 (Max). Running it at usable speed (>45 tok/s) is a step change for local AI users. What we improved about the challenge itself This is our second challenge, and we took the feedback from the Laguna track and rebuilt a few core pieces. - Speculative decoding (native MTP) was available and editable on day one instead of bolted on later. - Scoring became the median of eight independent prompt speedups over pure serial decode (anchored at 1.0, floor 0.90, ceiling 3.0), so no single fixture could dominate. - The leaderboard now ranks total contribution rather than just the current record holder. - Every submission gets automated screening for gaming before it scores. I really appreciate folks who's provided feedback. Naming a few that came to mind Ivan Fioravanti ᯅ TheDavidTai Morgan McGuire poly Takeshi7 Steven Gumbii.Digital Tanishq Dubey Arjun Ram Andrey 🦃 Petrov tiny edge David Zhang Jaime Rader Peter and many others on slack! We also widened the editable surface to include the MTP head weights themselves, the full draft/verify loop, and a large set of the underlying Metal kernels. How we got to the 3x speedup Here's a summary from Grok. Much of it is beyond my understanding, but I expect people (and agents) smarter than I am can take these insights and apply them in other contexts. Custom MTP heads + adaptive draft policy People stopped treating the head as fixed and started training or editing it for higher acceptance under the exact verify constraints. Combined with per-round draft counts that can adapt (0 to 8), this is what pushed average accepted tokens from ~1-2 up to 3.9 on the top runs. Tighter verify-block and KV rollback paths The Swift session code for assembling the verify pass, snapshotting KV, and rolling back on rejects got cleaned up a lot. Small latency wins here compound once you're drafting ~four tokens at a time. Metal kernel work on the hot paths SDPA, the MoE gather GEMM, RoPE, RMSNorm, and a few of the smaller element-wise ops saw targeted edits. Most of the gains only show up once the verify width is high and the memory traffic pattern changes. Fidelity-preserving residual handling Several submissions improved how residuals and acceptance decisions are managed, so that higher draft depth doesn't quietly degrade the token match rate. The gates stayed strict: every emitted token still has to equal serial, so these were real engineering wins rather than score hacks. What's next for Qwen 3.8 27B MLX. We plan to keep the track live a bit longer, then switch to Qwen 3.8's MoE version (rumored to be 35B-A3B). Given the recent DFlash 2 announcement, we're also looking at whether we can support broader speculative methods. The current surface already supports a lot of experimentation. The main gaps are better upstreaming for local usage and clearer docs on how the benchmark and verifier work. Multiplatform. In parallel, we're experimenting with running a similar effort around CUDA for Qwen 3.8 27B. A lot of people have asked for this, since the two communities overlap quite a bit. Our goal is to ship the CUDA version next week. We'd also love to partner with Qwen on it. If anyone has a connection there, please introduce us, and we'll see if they're down to match a bounty with us to push this out. What's most useful for the broader MLX community The improvements from the challenge are already upstreamed inside Darkbloom, and we're seeing ~2x faster decode in our production traffic for Qwen. Outside the challenge itself, something I've been thinking about deeply, and that a few community members have raised, is how to make these results useful to more people. There are many individual efforts happening across the MLX community, and honestly, the more I dig in, the more confused I get by the overlapping libraries and concepts. I'm sure I'm not alone, and newcomers probably feel the same. That's no one's fault, just the growing pains of an open source community. I don't expect I'm gonna come up with the answer, but I'd love to learn more about what different folks are working on and how they're thinking about their roadmaps. I'll share what I learn along the way, and hopefully someone smarter than me can turn it into a proposal for us to rally around.
0:27

Qwen 3.8 27B on hit 3.3x faster decode in 7 days. Here's what happened and what we're thinking next. Result (so far) Median decode speed increased from 26 tok/s to 87.9 tok/s on the verifier M5 Max (33 to 93.1 tok/s across the eight prompts), with prefill around 971.8 tok/s. This came out of a collective effort: 31 solvers across 67 improvements. Most of the recent ones run custom MTP heads that draft and accept ~3.9 tokens per round while still matching serial output exactly. Why this matters Beyond the performance itself, two things stand out to me. (1) Dense models on Apple Silicon were supposed to be the hard case. "Everyone knows Macs are slow at dense models." But watching the community take it from the usual baseline to >3x in seven days shows the low-hanging fruit was still there. (2) Open-weight models have been small and effective for a while. This is the first time one is small and frontier. Qwen 3.8 27B is an extremely strong dense model, comparable in capability to Opus 4.6 (Max). Running it at usable speed (>45 tok/s) is a step change for local AI users. What we improved about the challenge itself This is our second challenge, and we took the feedback from the Laguna track and rebuilt a few core pieces. - Speculative decoding (native MTP) was available and editable on day one instead of bolted on later. - Scoring became the median of eight independent prompt speedups over pure serial decode (anchored at 1.0, floor 0.90, ceiling 3.0), so no single fixture could dominate. - The leaderboard now ranks total contribution rather than just the current record holder. - Every submission gets automated screening for gaming before it scores. I really appreciate folks who's provided feedback. Naming a few that came to mind Ivan Fioravanti ᯅ TheDavidTai Morgan McGuire poly Takeshi7 Steven Gumbii.Digital Tanishq Dubey Arjun Ram Andrey 🦃 Petrov tiny edge David Zhang Jaime Rader Peter and many others on slack! We also widened the editable surface to include the MTP head weights themselves, the full draft/verify loop, and a large set of the underlying Metal kernels. How we got to the 3x speedup Here's a summary from Grok. Much of it is beyond my understanding, but I expect people (and agents) smarter than I am can take these insights and apply them in other contexts. Custom MTP heads + adaptive draft policy People stopped treating the head as fixed and started training or editing it for higher acceptance under the exact verify constraints. Combined with per-round draft counts that can adapt (0 to 8), this is what pushed average accepted tokens from ~1-2 up to 3.9 on the top runs. Tighter verify-block and KV rollback paths The Swift session code for assembling the verify pass, snapshotting KV, and rolling back on rejects got cleaned up a lot. Small latency wins here compound once you're drafting ~four tokens at a time. Metal kernel work on the hot paths SDPA, the MoE gather GEMM, RoPE, RMSNorm, and a few of the smaller element-wise ops saw targeted edits. Most of the gains only show up once the verify width is high and the memory traffic pattern changes. Fidelity-preserving residual handling Several submissions improved how residuals and acceptance decisions are managed, so that higher draft depth doesn't quietly degrade the token match rate. The gates stayed strict: every emitted token still has to equal serial, so these were real engineering wins rather than score hacks. What's next for Qwen 3.8 27B MLX. We plan to keep the track live a bit longer, then switch to Qwen 3.8's MoE version (rumored to be 35B-A3B). Given the recent DFlash 2 announcement, we're also looking at whether we can support broader speculative methods. The current surface already supports a lot of experimentation. The main gaps are better upstreaming for local usage and clearer docs on how the benchmark and verifier work. Multiplatform. In parallel, we're experimenting with running a similar effort around CUDA for Qwen 3.8 27B. A lot of people have asked for this, since the two communities overlap quite a bit. Our goal is to ship the CUDA version next week. We'd also love to partner with Qwen on it. If anyone has a connection there, please introduce us, and we'll see if they're down to match a bounty with us to push this out. What's most useful for the broader MLX community The improvements from the challenge are already upstreamed inside Darkbloom, and we're seeing ~2x faster decode in our production traffic for Qwen. Outside the challenge itself, something I've been thinking about deeply, and that a few community members have raised, is how to make these results useful to more people. There are many individual efforts happening across the MLX community, and honestly, the more I dig in, the more confused I get by the overlapping libraries and concepts. I'm sure I'm not alone, and newcomers probably feel the same. That's no one's fault, just the growing pains of an open source community. I don't expect I'm gonna come up with the answer, but I'd love to learn more about what different folks are working on and how they're thinking about their roadmaps. I'll share what I learn along the way, and hopefully someone smarter than me can turn it into a proposal for us to rally around.

Kydo

30,202 Aufrufe • vor 12 Tagen

A single RTX 4090 (24 GB VRAM) can run the updated gemma 4 31B (dense) model with a 190,000 context window at 33 tokens/second. The VRAM barrier is dying. Google quietly updated Gemma 4, and Unsloth immediately compiled the new quants. I built llama.cpp from source on Ubuntu 22 to benchmark it. Google's stealth update 2 days ago enabled uniform Flash Attention 4 on Hopper to boost prefill and patched the chat template to improve tool calling. The agentic reasoning gains on the benchmark charts are massive: TB2 (Agents): +4.5% (to 25.8%) Tau2 (Telecom): +10.1% (to 62.7%) Running on Ubuntu 22, CUDA 13.0 with a single NVIDIA GeForce RTX 4090. Here is the exact step by step benchmarking process with a massive 28k tokens prompt and the commands I used to squeeze out maximum context without killing my throughput: # 1. The Baseline (Unquantized KV Cache) I started with full GPU offload (-ngl 99) and pushed the context to 40k. llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -ngl 99 -c 40000 -fa on --port 8080 -v VRAM: 23.8 GB (maxed out on card) Throughput: Prefill: 2198.81 t/s | Decode: 35.77 t/s (with 28k tokens prompt) # 2. The CPU Split Trap I tried stretching to 80k context by offloading layers to the CPU (-ngl 52). llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 80000 -ngl 52 -fa on --port 8080 -v Throughput: Prefill: 1212.73 t/s | Decode: 5 t/s (with 28k tokens prompt) # 3. The KV Quantization Breakthrough Instead of spilling layers to the CPU, I kept the model fully on card (-ngl 99) but enabled 8-bit KV cache quantization to free up VRAM. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 100000 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 --port 8080 -v VRAM: 23.9 GB Throughput: Prefill: 2139.68 t/s | Decode: 32 t/s (with 28k tokens prompt) Result: 100k tokens of context on a single GPU with practically zero speed loss (and minimal intelligence loss). # 4. The Limit Test (Q4 KV Cache) To find the absolute breaking point, I dropped the KV cache to 4 bit (q4_0) and set -c 190000. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 190000 --cache-type-k q4_0 --cache-type-v q4_0 -ngl 99 --port 8080 -v VRAM: 23.8 GB Throughput: Prefill: 2206.66 t/s | Decode: 33 t/s (with 28k tokens prompt) (Note: Pushing it to 220k required dropping to -ngl 58 again, which immediately penalized decode down to 17 t/s). # The Tradeoff: For Max Reasoning: Keep your KV cache unquantized (f16). You get pristine reasoning but hit a strict 40k context ceiling. For Massive Document Retrieval: If you need to feed the model giant codebases, use --cache-type-k q4_0. Getting 190k context at 33 tokens/second on a consumer desktop with a 31b dense model is a cheat code. If you’re rocking a single 3090 or 4090 and slept on Gemma 4 earlier, this update is your cue to dust off the terminal. Hugging Face links to the Unsloth QAT quants are in the replies below.
0:20

A single RTX 4090 (24 GB VRAM) can run the updated gemma 4 31B (dense) model with a 190,000 context window at 33 tokens/second. The VRAM barrier is dying. Google quietly updated Gemma 4, and Unsloth immediately compiled the new quants. I built llama.cpp from source on Ubuntu 22 to benchmark it. Google's stealth update 2 days ago enabled uniform Flash Attention 4 on Hopper to boost prefill and patched the chat template to improve tool calling. The agentic reasoning gains on the benchmark charts are massive: TB2 (Agents): +4.5% (to 25.8%) Tau2 (Telecom): +10.1% (to 62.7%) Running on Ubuntu 22, CUDA 13.0 with a single NVIDIA GeForce RTX 4090. Here is the exact step by step benchmarking process with a massive 28k tokens prompt and the commands I used to squeeze out maximum context without killing my throughput: # 1. The Baseline (Unquantized KV Cache) I started with full GPU offload (-ngl 99) and pushed the context to 40k. llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -ngl 99 -c 40000 -fa on --port 8080 -v VRAM: 23.8 GB (maxed out on card) Throughput: Prefill: 2198.81 t/s | Decode: 35.77 t/s (with 28k tokens prompt) # 2. The CPU Split Trap I tried stretching to 80k context by offloading layers to the CPU (-ngl 52). llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 80000 -ngl 52 -fa on --port 8080 -v Throughput: Prefill: 1212.73 t/s | Decode: 5 t/s (with 28k tokens prompt) # 3. The KV Quantization Breakthrough Instead of spilling layers to the CPU, I kept the model fully on card (-ngl 99) but enabled 8-bit KV cache quantization to free up VRAM. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 100000 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 --port 8080 -v VRAM: 23.9 GB Throughput: Prefill: 2139.68 t/s | Decode: 32 t/s (with 28k tokens prompt) Result: 100k tokens of context on a single GPU with practically zero speed loss (and minimal intelligence loss). # 4. The Limit Test (Q4 KV Cache) To find the absolute breaking point, I dropped the KV cache to 4 bit (q4_0) and set -c 190000. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 190000 --cache-type-k q4_0 --cache-type-v q4_0 -ngl 99 --port 8080 -v VRAM: 23.8 GB Throughput: Prefill: 2206.66 t/s | Decode: 33 t/s (with 28k tokens prompt) (Note: Pushing it to 220k required dropping to -ngl 58 again, which immediately penalized decode down to 17 t/s). # The Tradeoff: For Max Reasoning: Keep your KV cache unquantized (f16). You get pristine reasoning but hit a strict 40k context ceiling. For Massive Document Retrieval: If you need to feed the model giant codebases, use --cache-type-k q4_0. Getting 190k context at 33 tokens/second on a consumer desktop with a 31b dense model is a cheat code. If you’re rocking a single 3090 or 4090 and slept on Gemma 4 earlier, this update is your cue to dust off the terminal. Hugging Face links to the Unsloth QAT quants are in the replies below.

Alok

76,069 Aufrufe • vor 1 Monat

Jensen Huang just identified the next $200 billion market (Save this). The shift starts with a observation about agentic AI that changes everything about infrastructure. In the era of training and inference, the GPU was everything while CPU was a traffic cop, scheduling work, managing memory, dispatching tasks while the GPU did the heavy lifting. Agentic AI breaks that model entirely. An AI agent does not just run a single inference pass but rather it plans, calls tools, executes code in sandboxes, retrieves data from multiple sources and loops through complex multi-step reasoning sequences often thousands of times per second at scale. Every one of those operations runs through the CPU and the GPU sits idle waiting for the CPU to prepare the next task, supply the right context and execute the retrieval and tool calling logic fast enough to keep the accelerators fed. The CPU is now the conductor and the GPU is the orchestra and the bottleneck is the conductor falling behind. This is showing up in production AI factory utilization right now, which is exactly why Jensen built Vera from scratch rather than licensing x86. Vera achieves 40% lower peak memory latency than x86, 50% faster core to core communication, and 1.8 times the agentic sandbox performance of current x86 processors on a purpose-built architecture designed around the agentic loop. Now here is where the investment thesis gets interesting. The obvious beneficiary is Nvidia itself, and that thesis is real. Nvidia's CFO has guided for nearly $20 billion in Vera CPU revenue this fiscal year alone, a market Nvidia had zero presence in just three years ago. Intel held 60% of server CPU market share as recently as Q4 2025 and that transition is now happening at a pace Intel structurally cannot respond to. But the deeper question is, what architecture is Vera actually built on? Vera's Olympus cores are ARM compatible and every single Vera CPU deployed in every Vera Rubin rack in every data center in the world runs on ARM architecture. And ARM Holdings collects a royalty on every one of them. ARM does not make chips but rather licenses the instruction set architecture and CPU core designs that others build on top of. Every time Nvidia ships a Vera CPU, every time a hyperscaler deploys a Vera Rubin rack, every time an enterprise qualifies Vera for their AI factory, ARM earns a royalty. The secular tailwind here is almost perfectly constructed for ARM's business model. Amazon's Graviton, Microsoft's Cobalt, Google's Axion, Apple's silicon stack, and Qualcomm's data center push all run on ARM. And now Nvidia's Vera, which is projected to displace Intel as the largest server CPU supplier by revenue in a single fiscal year, is ARM. ARM's royalty rate on high end server chips is estimated at roughly 1 to 2% of chip selling price. At $5,000 per Vera CPU and 4 million units projected for FY2027, that is a royalty line growing from near zero to potentially $400 million to $800 million annually from Nvidia's data center CPU business alone before counting Amazon, Microsoft, Google, Apple, and Qualcomm. The total ARM addressable royalty base across all the silicon it already licenses is compounding at a rate that the current $130 billion market cap does not fully reflect. Jensen's CPU thesis is the most underappreciated catalyst in ARM's fundamental story, and the royalty compounding has barely started. Come join Milk Road Pro and get our full ARM royalty model and our entire AI trade thesis. Link below!
3:41

Jensen Huang just identified the next $200 billion market (Save this). The shift starts with a observation about agentic AI that changes everything about infrastructure. In the era of training and inference, the GPU was everything while CPU was a traffic cop, scheduling work, managing memory, dispatching tasks while the GPU did the heavy lifting. Agentic AI breaks that model entirely. An AI agent does not just run a single inference pass but rather it plans, calls tools, executes code in sandboxes, retrieves data from multiple sources and loops through complex multi-step reasoning sequences often thousands of times per second at scale. Every one of those operations runs through the CPU and the GPU sits idle waiting for the CPU to prepare the next task, supply the right context and execute the retrieval and tool calling logic fast enough to keep the accelerators fed. The CPU is now the conductor and the GPU is the orchestra and the bottleneck is the conductor falling behind. This is showing up in production AI factory utilization right now, which is exactly why Jensen built Vera from scratch rather than licensing x86. Vera achieves 40% lower peak memory latency than x86, 50% faster core to core communication, and 1.8 times the agentic sandbox performance of current x86 processors on a purpose-built architecture designed around the agentic loop. Now here is where the investment thesis gets interesting. The obvious beneficiary is Nvidia itself, and that thesis is real. Nvidia's CFO has guided for nearly $20 billion in Vera CPU revenue this fiscal year alone, a market Nvidia had zero presence in just three years ago. Intel held 60% of server CPU market share as recently as Q4 2025 and that transition is now happening at a pace Intel structurally cannot respond to. But the deeper question is, what architecture is Vera actually built on? Vera's Olympus cores are ARM compatible and every single Vera CPU deployed in every Vera Rubin rack in every data center in the world runs on ARM architecture. And ARM Holdings collects a royalty on every one of them. ARM does not make chips but rather licenses the instruction set architecture and CPU core designs that others build on top of. Every time Nvidia ships a Vera CPU, every time a hyperscaler deploys a Vera Rubin rack, every time an enterprise qualifies Vera for their AI factory, ARM earns a royalty. The secular tailwind here is almost perfectly constructed for ARM's business model. Amazon's Graviton, Microsoft's Cobalt, Google's Axion, Apple's silicon stack, and Qualcomm's data center push all run on ARM. And now Nvidia's Vera, which is projected to displace Intel as the largest server CPU supplier by revenue in a single fiscal year, is ARM. ARM's royalty rate on high end server chips is estimated at roughly 1 to 2% of chip selling price. At $5,000 per Vera CPU and 4 million units projected for FY2027, that is a royalty line growing from near zero to potentially $400 million to $800 million annually from Nvidia's data center CPU business alone before counting Amazon, Microsoft, Google, Apple, and Qualcomm. The total ARM addressable royalty base across all the silicon it already licenses is compounding at a rate that the current $130 billion market cap does not fully reflect. Jensen's CPU thesis is the most underappreciated catalyst in ARM's fundamental story, and the royalty compounding has barely started. Come join Milk Road Pro and get our full ARM royalty model and our entire AI trade thesis. Link below!

Milk Road AI

11,819 Aufrufe • vor 3 Monaten

Deepseek V4 Flash 0731 (Q2) - 12 tokens/sec - Single RTX 4090 - 650+ tokens/sec prefill - 250k context - no kv cache quantization! DeepSeek just dropped the official V4 Flash 0731 two days ago with a massive agent capabilities upgrade. The official benchmarks are literally crushing their own V4-Pro-Preview on agentic tasks like Terminal Bench 2.1 and DeepSWE. Unsloth AI said they couldn't wait to bring it to local devices, and they delivered. If you thought my 118B Poolside Laguna S 2.1 MoE run last week on a single GPU was wild, hold onto your hardware. I just successfully ran Unsloth’s brand new 91GB DeepSeek-V4-Flash-0731 (UD-IQ2_M) GGUF entirely locally. And I pushed it to a mind-bending 250,000 context window. The VRAM ceiling is an illusion if you know how to optimize llama.cpp. Here are the benchmarks and the cheat codes to run a local frontier class model yourself. For the hardware and setup, I used a single NVIDIA RTX 4090 (24GB VRAM) hooked up via a PCIe 4 bus, running Ubuntu 22.04 LTS and CUDA 13.0. You don't need a massive enterprise server for this, if you have more than 80 GB of standard DDR4 RAM and a 24GB card like an RTX 3090 or 4090, you can run this exact stack yourself. All benchmarks were run using a massive 28k token prompt to truly stress test the prefill limits. no kv cache quantization THE BENCHMARKS (Scaling Context): # 80k Context (Baseline: -b 2048 -ub 2048): Prefill: 465.43 t/s | Decode: 13.00 t/s | VRAM: 22.87 GB # 80k Context (Optimized: -b 4096 -ub 4096): Prefill: 643.15 t/s | Decode: 12.20 t/s | VRAM: 23.00 GB (Notice how doubling the batch flags spiked my prefill throughput by nearly 200 t/s with almost zero VRAM penalty) # 180k Context (-b 4096 -ub 4096): Prefill: 629.18 t/s | Decode: 11.92 t/s | VRAM: 23.40 GB # 250k Context MAXIMUM (-b 4096 -ub 4096): Prefill: 619.02 t/s | Decode: 11.54 t/s | VRAM: 23.40 GB # THE SECRET SAUCE (Why this works): Unsloth’s UD-IQ2_M quant is ~91GB across 3 files. Since I only have 24GB of VRAM, the PCIe 4 bus and system RAM have to do the heavy lifting. The magic bullet is the --no-mmap flag. By completely bypassing OS disk paging, I forced llama.cpp to load the massive model weights directly into the system RAM upfront. Combined with Flash Attention (-fa on) and exactly 12 CPU threads (--threads 12), I maintained an incredibly stable 11.5+ tokens/sec decode speed even at a quarter million token context. # THE EXACT COMMAND: ./build/bin/llama-server -m /workspace/models/DeepSeek-V4-Flash-0731-UD-IQ2_M-00001-of-00003.gguf -c 250000 -fa on --port 8080 --threads 12 -b 4096 -ub 4096 --no-mmap -v Local conversational and agentic coding AI is fully here. You don’t need an API or an H100 cluster. Qwen 3.8 27b drops next week making the 24GB VRAM tier even more worthwhile. What does your current local AI rig look like, and what's the craziest model you've managed to squeeze into it? Official huggingface GGUF links from Unsloth and performance graphs are dropped in the replies below!
0:21

Deepseek V4 Flash 0731 (Q2) - 12 tokens/sec - Single RTX 4090 - 650+ tokens/sec prefill - 250k context - no kv cache quantization! DeepSeek just dropped the official V4 Flash 0731 two days ago with a massive agent capabilities upgrade. The official benchmarks are literally crushing their own V4-Pro-Preview on agentic tasks like Terminal Bench 2.1 and DeepSWE. Unsloth AI said they couldn't wait to bring it to local devices, and they delivered. If you thought my 118B Poolside Laguna S 2.1 MoE run last week on a single GPU was wild, hold onto your hardware. I just successfully ran Unsloth’s brand new 91GB DeepSeek-V4-Flash-0731 (UD-IQ2_M) GGUF entirely locally. And I pushed it to a mind-bending 250,000 context window. The VRAM ceiling is an illusion if you know how to optimize llama.cpp. Here are the benchmarks and the cheat codes to run a local frontier class model yourself. For the hardware and setup, I used a single NVIDIA RTX 4090 (24GB VRAM) hooked up via a PCIe 4 bus, running Ubuntu 22.04 LTS and CUDA 13.0. You don't need a massive enterprise server for this, if you have more than 80 GB of standard DDR4 RAM and a 24GB card like an RTX 3090 or 4090, you can run this exact stack yourself. All benchmarks were run using a massive 28k token prompt to truly stress test the prefill limits. no kv cache quantization THE BENCHMARKS (Scaling Context): # 80k Context (Baseline: -b 2048 -ub 2048): Prefill: 465.43 t/s | Decode: 13.00 t/s | VRAM: 22.87 GB # 80k Context (Optimized: -b 4096 -ub 4096): Prefill: 643.15 t/s | Decode: 12.20 t/s | VRAM: 23.00 GB (Notice how doubling the batch flags spiked my prefill throughput by nearly 200 t/s with almost zero VRAM penalty) # 180k Context (-b 4096 -ub 4096): Prefill: 629.18 t/s | Decode: 11.92 t/s | VRAM: 23.40 GB # 250k Context MAXIMUM (-b 4096 -ub 4096): Prefill: 619.02 t/s | Decode: 11.54 t/s | VRAM: 23.40 GB # THE SECRET SAUCE (Why this works): Unsloth’s UD-IQ2_M quant is ~91GB across 3 files. Since I only have 24GB of VRAM, the PCIe 4 bus and system RAM have to do the heavy lifting. The magic bullet is the --no-mmap flag. By completely bypassing OS disk paging, I forced llama.cpp to load the massive model weights directly into the system RAM upfront. Combined with Flash Attention (-fa on) and exactly 12 CPU threads (--threads 12), I maintained an incredibly stable 11.5+ tokens/sec decode speed even at a quarter million token context. # THE EXACT COMMAND: ./build/bin/llama-server -m /workspace/models/DeepSeek-V4-Flash-0731-UD-IQ2_M-00001-of-00003.gguf -c 250000 -fa on --port 8080 --threads 12 -b 4096 -ub 4096 --no-mmap -v Local conversational and agentic coding AI is fully here. You don’t need an API or an H100 cluster. Qwen 3.8 27b drops next week making the 24GB VRAM tier even more worthwhile. What does your current local AI rig look like, and what's the craziest model you've managed to squeeze into it? Official huggingface GGUF links from Unsloth and performance graphs are dropped in the replies below!

Alok

45,967 Aufrufe • vor 1 Monat

If you are running local LLMs without N-gram speculative decoding, you are wasting massive amounts of compute. Whether your AI is editing a document, outputting structured JSON, or rewriting boilerplate templates, a huge chunk of the text it generates is highly repetitive or already exists right there in the prompt. Standard decoding wastes expensive GPU compute cycles "re thinking" every single token. By adding one hidden flag in llama.cpp, you can instantly fast forward through the repetition. Zero draft models. Zero extra VRAM. And virtually zero compute overhead. Google Colab hands you an enterprise grade NVIDIA Tesla T4 GPU with 16GB of VRAM for free. It’s the perfect Ubuntu Linux sandbox to build a bleeding edge inference engine from scratch. Recently, I showed you how to double your local speeds using MTP (Multi Token Prediction). But MTP requires a secondary neural network draft model. That eats into your precious VRAM (slightly though) and burns extra compute for every guess it makes. N-gram Speculative Decoding gives you a massive speed boost for exactly 0 memory cost and minimal compute. And it's faster than MTP when it works. Here is how it actually works under the hood: Standard autoregressive decoding is slow because it predicts one token at a time. If you ask an agent to format a long JSON object or update one line in an HTML file, it runs heavy matrix multiplications to calculate the probability of every single bracket, space, and letter from scratch. N-gram changes the game. It acts as a lightweight caching system. Instead of running heavy neural network math to guess the next word, it uses a simple hash table. Whenever the LLM starts outputting a sequence of tokens that already exists anywhere in its context window, N-gram instantly recognizes the pattern. Because it is just doing lightning fast string matching, the compute cost is practically zero. It "fast forwards" through the text, drafting the boilerplate instantly from memory, and the main model just verifies it in parallel. Pure speed. Using quantized GGUFs from Unsloth via HuggingFace, I spun up DeepMind’s massive Gemma 4 26B A4B QAT MoE on a free Colab instance to test this. Just look at the raw benchmark data on code editing task: Without N-gram: [ Prompt: 638.6 t/s | Generation: 45.9 t/s ] With N-gram: [ Prompt: 601.9 t/s | Generation: 107.1 t/s ] Here is the exact llama.cpp CLI command to activate it. Notice we don't even need the --model-draft flag: ./llama-cli -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -n 6000 -c 12000 -ngl 99 -fa on --spec-type ngram-mod Stop waiting for your GPU to re calculate words it already knows. I’ve built a free, interactive, cell by cell Google Colab notebook that lets you test this live in your browser. You can literally chat with the model and watch the text generation speed absolutely fly on the second turn when you ask it to edit a file. There are additional parameters for ngram-mod that you can tune once you get it working with the single flag. Link to the free Colab Notebook is in the comments below. It walks you through the entire stack: pulling pre built llama.cpp CUDA binaries for Linux, fetching GGUFs from HuggingFace, and spinning up the inference engine with ngram-mod from scratch. Let me know if you have already tried ngram-mod
0:23

If you are running local LLMs without N-gram speculative decoding, you are wasting massive amounts of compute. Whether your AI is editing a document, outputting structured JSON, or rewriting boilerplate templates, a huge chunk of the text it generates is highly repetitive or already exists right there in the prompt. Standard decoding wastes expensive GPU compute cycles "re thinking" every single token. By adding one hidden flag in llama.cpp, you can instantly fast forward through the repetition. Zero draft models. Zero extra VRAM. And virtually zero compute overhead. Google Colab hands you an enterprise grade NVIDIA Tesla T4 GPU with 16GB of VRAM for free. It’s the perfect Ubuntu Linux sandbox to build a bleeding edge inference engine from scratch. Recently, I showed you how to double your local speeds using MTP (Multi Token Prediction). But MTP requires a secondary neural network draft model. That eats into your precious VRAM (slightly though) and burns extra compute for every guess it makes. N-gram Speculative Decoding gives you a massive speed boost for exactly 0 memory cost and minimal compute. And it's faster than MTP when it works. Here is how it actually works under the hood: Standard autoregressive decoding is slow because it predicts one token at a time. If you ask an agent to format a long JSON object or update one line in an HTML file, it runs heavy matrix multiplications to calculate the probability of every single bracket, space, and letter from scratch. N-gram changes the game. It acts as a lightweight caching system. Instead of running heavy neural network math to guess the next word, it uses a simple hash table. Whenever the LLM starts outputting a sequence of tokens that already exists anywhere in its context window, N-gram instantly recognizes the pattern. Because it is just doing lightning fast string matching, the compute cost is practically zero. It "fast forwards" through the text, drafting the boilerplate instantly from memory, and the main model just verifies it in parallel. Pure speed. Using quantized GGUFs from Unsloth via HuggingFace, I spun up DeepMind’s massive Gemma 4 26B A4B QAT MoE on a free Colab instance to test this. Just look at the raw benchmark data on code editing task: Without N-gram: [ Prompt: 638.6 t/s | Generation: 45.9 t/s ] With N-gram: [ Prompt: 601.9 t/s | Generation: 107.1 t/s ] Here is the exact llama.cpp CLI command to activate it. Notice we don't even need the --model-draft flag: ./llama-cli -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -n 6000 -c 12000 -ngl 99 -fa on --spec-type ngram-mod Stop waiting for your GPU to re calculate words it already knows. I’ve built a free, interactive, cell by cell Google Colab notebook that lets you test this live in your browser. You can literally chat with the model and watch the text generation speed absolutely fly on the second turn when you ask it to edit a file. There are additional parameters for ngram-mod that you can tune once you get it working with the single flag. Link to the free Colab Notebook is in the comments below. It walks you through the entire stack: pulling pre built llama.cpp CUDA binaries for Linux, fetching GGUFs from HuggingFace, and spinning up the inference engine with ngram-mod from scratch. Let me know if you have already tried ngram-mod

Alok

31,765 Aufrufe • vor 1 Monat

Quick chat with dylan ツ (Dylan Bristot, GTM @ $NBIS). Also on YouTube (link in first comment) for those who prefer to watch/listen there. Timestamps 00:00 – Dylan's role at Nebius and Nebius Token Factory 01:48 – Dylan's investing philosophy and portfolio approach 05:07 – How working in AI infrastructure influences his investing 08:29 – Training vs. inference and why inference demand could explode 13:22 – Enterprise AI adoption: from POCs to production 18:02 – Open-source vs. closed/frontier models 24:44 – The economics of open vs. closed AI models 29:27 – Where the next AI infrastructure bottlenecks could emerge 31:22 – Dylan's AI Bottlenecks project and approach to stock selection 34:06 – Closing thoughts Key Insights (AI Summary, so you don't have to copy paste and prompt for exactly that ;D) “I seem to like areas where the demand really looks kind of secular, but the supply is genuinely hard to create.” → Implication: The most attractive AI trades may sit in physical bottlenecks where supply cannot quickly respond to demand. “The bottleneck is who has the pricing power and kind of what might get commoditized and where the concentrate might move next.” → Implication: Value capture across the AI stack will keep shifting as individual layers become scarce or commoditized. “Training creates the intelligence and then the inference actually monetizes and distributes.” → Implication: Training and inference are complementary, rather than one ultimately replacing the other. “One user action can become dozens or hundreds of model calls, tools calls, and like verification steps, retries.” → Implication: Agentic AI can drive token consumption far faster than user growth alone would suggest. “The best infra for making any model and the best infra for serving a billion interactions are not necessarily the same.” → Implication: Training and inference could increasingly require different hardware and infrastructure architectures. “The Frontier Labs might be incentivized to run more and more of the inference of these models for internal research instead of providing it to external people.” → Implication: The most capable models and their compute could increasingly be used internally to accelerate frontier research rather than monetized externally. “Enterprise AI adoption is actually much further along than a lot of people kind of think. But probably less mature than the headlines suggest.” → Implication: Enterprise demand is real, but deployment maturity still has significant room to improve. “The POC problem might be solved for a lot of companies, but the production problem isn’t yet.” → Implication: The enterprise bottleneck is shifting from proving AI works to deploying it reliably, securely and economically at scale. “They feel like it’s time for them to actually not only integrate AI, but build some sort of moat out of the AI.” → Implication: Enterprises increasingly want proprietary AI systems built around their own data rather than simply consuming generic models. “The more autonomous the software becomes, the more infra discipline you need underneath it.” → Implication: Agents increase the importance of inference cost, reliability and infrastructure optimization. “Maybe I have fifteen different versions of very different LLMs, fine tuned on fifteen different kinds of tasks that I’m operating across my business, instead of having a one model fits all.” → Implication: Enterprise AI could evolve toward many specialized models rather than one frontier model handling every workload. “I don’t necessarily think it’s open versus closed. That might be the wrong framing.” → Implication: Open and closed models can coexist because they optimize for different customer needs. “Historically the problem was that that control came with a massive operational tax.” → Implication: Better inference infrastructure can make open models materially more competitive by removing the complexity traditionally associated with running them. “I don’t think open needs to beat the best closed model on every single benchmark. It just basically needs to be good enough for the workload of the given customer while offering a much better combination of control, cost, and deployment flexibility.” → Implication: For production AI, workload-specific economics may matter more than having the absolute smartest model. “Maybe actually the bulk of tokens generated in the future might come from open models.” → Implication: Frontier intelligence could remain dominated by closed labs even while open models capture most production inference volume. “I could really imagine frontier intelligence being really concentrated while most of the production inference becomes super fragmented.” → Implication: AI could consolidate at the intelligence layer while fragmenting heavily at the inference layer across models, GPUs, providers and regions. “I don’t think that necessarily means the margins of open source will be much worse than the ones of closed source.” → Implication: Optimization can potentially make open-model inference highly profitable despite lower pricing. “I think now we’re probably in the middle of phase two... everything feeding the accelerator.” → Implication: The AI trade is broadening beyond GPUs toward networking, packaging, data centers, electrical equipment and power. “It’s no longer about the megawatts, about energized megawatts.” → Implication: Available power on paper matters less than how quickly that power can actually be delivered to operating AI infrastructure. “It’s increasingly about utilisation and conversion now and like how efficiently you convert expensive infra into actual useful AI work.” → Implication: Infrastructure efficiency and utilization become increasingly important as the absolute amount of deployed AI infrastructure grows. “The market tends to really notice demand before it notices what demand breaks.” → Implication: Second-order bottlenecks may offer some of the most interesting opportunities in the next phase of the AI buildout. “The interesting question now is which part of the mine breaks next?” → Implication: Finding the next constraint in the AI supply chain may matter more than simply identifying continued AI demand.
34:29

Quick chat with dylan ツ (Dylan Bristot, GTM @ $NBIS). Also on YouTube (link in first comment) for those who prefer to watch/listen there. Timestamps 00:00 – Dylan's role at Nebius and Nebius Token Factory 01:48 – Dylan's investing philosophy and portfolio approach 05:07 – How working in AI infrastructure influences his investing 08:29 – Training vs. inference and why inference demand could explode 13:22 – Enterprise AI adoption: from POCs to production 18:02 – Open-source vs. closed/frontier models 24:44 – The economics of open vs. closed AI models 29:27 – Where the next AI infrastructure bottlenecks could emerge 31:22 – Dylan's AI Bottlenecks project and approach to stock selection 34:06 – Closing thoughts Key Insights (AI Summary, so you don't have to copy paste and prompt for exactly that ;D) “I seem to like areas where the demand really looks kind of secular, but the supply is genuinely hard to create.” → Implication: The most attractive AI trades may sit in physical bottlenecks where supply cannot quickly respond to demand. “The bottleneck is who has the pricing power and kind of what might get commoditized and where the concentrate might move next.” → Implication: Value capture across the AI stack will keep shifting as individual layers become scarce or commoditized. “Training creates the intelligence and then the inference actually monetizes and distributes.” → Implication: Training and inference are complementary, rather than one ultimately replacing the other. “One user action can become dozens or hundreds of model calls, tools calls, and like verification steps, retries.” → Implication: Agentic AI can drive token consumption far faster than user growth alone would suggest. “The best infra for making any model and the best infra for serving a billion interactions are not necessarily the same.” → Implication: Training and inference could increasingly require different hardware and infrastructure architectures. “The Frontier Labs might be incentivized to run more and more of the inference of these models for internal research instead of providing it to external people.” → Implication: The most capable models and their compute could increasingly be used internally to accelerate frontier research rather than monetized externally. “Enterprise AI adoption is actually much further along than a lot of people kind of think. But probably less mature than the headlines suggest.” → Implication: Enterprise demand is real, but deployment maturity still has significant room to improve. “The POC problem might be solved for a lot of companies, but the production problem isn’t yet.” → Implication: The enterprise bottleneck is shifting from proving AI works to deploying it reliably, securely and economically at scale. “They feel like it’s time for them to actually not only integrate AI, but build some sort of moat out of the AI.” → Implication: Enterprises increasingly want proprietary AI systems built around their own data rather than simply consuming generic models. “The more autonomous the software becomes, the more infra discipline you need underneath it.” → Implication: Agents increase the importance of inference cost, reliability and infrastructure optimization. “Maybe I have fifteen different versions of very different LLMs, fine tuned on fifteen different kinds of tasks that I’m operating across my business, instead of having a one model fits all.” → Implication: Enterprise AI could evolve toward many specialized models rather than one frontier model handling every workload. “I don’t necessarily think it’s open versus closed. That might be the wrong framing.” → Implication: Open and closed models can coexist because they optimize for different customer needs. “Historically the problem was that that control came with a massive operational tax.” → Implication: Better inference infrastructure can make open models materially more competitive by removing the complexity traditionally associated with running them. “I don’t think open needs to beat the best closed model on every single benchmark. It just basically needs to be good enough for the workload of the given customer while offering a much better combination of control, cost, and deployment flexibility.” → Implication: For production AI, workload-specific economics may matter more than having the absolute smartest model. “Maybe actually the bulk of tokens generated in the future might come from open models.” → Implication: Frontier intelligence could remain dominated by closed labs even while open models capture most production inference volume. “I could really imagine frontier intelligence being really concentrated while most of the production inference becomes super fragmented.” → Implication: AI could consolidate at the intelligence layer while fragmenting heavily at the inference layer across models, GPUs, providers and regions. “I don’t think that necessarily means the margins of open source will be much worse than the ones of closed source.” → Implication: Optimization can potentially make open-model inference highly profitable despite lower pricing. “I think now we’re probably in the middle of phase two... everything feeding the accelerator.” → Implication: The AI trade is broadening beyond GPUs toward networking, packaging, data centers, electrical equipment and power. “It’s no longer about the megawatts, about energized megawatts.” → Implication: Available power on paper matters less than how quickly that power can actually be delivered to operating AI infrastructure. “It’s increasingly about utilisation and conversion now and like how efficiently you convert expensive infra into actual useful AI work.” → Implication: Infrastructure efficiency and utilization become increasingly important as the absolute amount of deployed AI infrastructure grows. “The market tends to really notice demand before it notices what demand breaks.” → Implication: Second-order bottlenecks may offer some of the most interesting opportunities in the next phase of the AI buildout. “The interesting question now is which part of the mine breaks next?” → Implication: Finding the next constraint in the AI supply chain may matter more than simply identifying continued AI demand.

Daniel Koss

46,001 Aufrufe • vor 2 Tagen

The #GTA 3 port to the Sega #Dreamcast has been progressing at an incredible pace. It's been amazing to see the whole DC community come together to tag-team this "impossible" project... Here it is running on a stock DC, no longer requiring the 32MB RAM hardware modification, only a few weeks into development. Since I finally got the time to sit down, build the codebase, and look into some of what I think is the critical path for performance, let's talk about some technical shit, and some of the future steps I think can be taken to further improve performance. First of all, I want everyone to take note that this is NOT a port of the PS2 version. This is a port of the PC version, which has extra content, increased draw distance, improved textures, and other things that have actually increased the challenge here... Whether the DC version will ultimately have these additions or not will remain to be seen, but we're running into plenty of shit that the PS2 didn't have to worry about (like these big-ass PC replay saves won't fit onto a Visual Memory Unit!) Secondly, lets talk about what is and isn't currently optimized, because it's absolutely vital that the DC's hardware is fully utilized here for the sake of performance and achieving a competitive polygon count. Unlike with modern devices, where the whole graphics pipeline is handled by the GPU, both the PS2 and Dreamcast were responsible for transforming and doing lighting calculations for each vertex BEFORE they got submitted to the GPU. The PS2 had a vector coprocessor to do this, while the Dreamcast had a few extremely important SIMD and fast math assembly instructions on its CPU to do these computations. Up until literally just a few hours ago (not shown in this footage), the Dreamcast's SH4 was doing 100% of these operations in slow-ass plain C and C++ code, which is absolutely sub-optimal and is immediately bogging down its CPU with just transforming vertices, bottlenecking the entire graphics pipeline on the fist T&L stage, and also leaving less CPU time for handling other gameplay logic... this is going to absolutely have to be addressed (and already has begun to be). Another issue that is crippling performance here is the fact that the models are all using individual triangles rather than triangle strips, which the Dreamcast's PVR GPU was designed to handle better... Converting these models to use strips rather than individual triangles will result in MANY different gainz for the DC, as you're going from 3N to N+2 vertices per triangle. Converting the models to triangle strips will 1) reduce load times, since model assets will be smaller 2) reduce the amount of video memory required to hold these vertices on the GPU 3) reduce the amount of shit that must be transferred from the CPU to the GPU and 4) give us back a bunch of CPU time, since the SH4 will be less bogged down transforming redundant vertices! TL;DR: This is still EXTREMELY suboptimal in terms of fully utilizing the graphical potential of the Dreamcast. There is going to be a LOT that can be done still to both improve performance and polygon counts, so stay tuned! FINALLY: Mad respect and love to Stefanos Kornilios Mitsis Poiitidis, for doing an amazing job leading this project, and to Frogbull , Esppiral, and everyone else who is helping us stick it to the PS2 by making this happen! #gamedev #retrogaming #cplusplus
1:53

The #GTA 3 port to the Sega #Dreamcast has been progressing at an incredible pace. It's been amazing to see the whole DC community come together to tag-team this "impossible" project... Here it is running on a stock DC, no longer requiring the 32MB RAM hardware modification, only a few weeks into development. Since I finally got the time to sit down, build the codebase, and look into some of what I think is the critical path for performance, let's talk about some technical shit, and some of the future steps I think can be taken to further improve performance. First of all, I want everyone to take note that this is NOT a port of the PS2 version. This is a port of the PC version, which has extra content, increased draw distance, improved textures, and other things that have actually increased the challenge here... Whether the DC version will ultimately have these additions or not will remain to be seen, but we're running into plenty of shit that the PS2 didn't have to worry about (like these big-ass PC replay saves won't fit onto a Visual Memory Unit!) Secondly, lets talk about what is and isn't currently optimized, because it's absolutely vital that the DC's hardware is fully utilized here for the sake of performance and achieving a competitive polygon count. Unlike with modern devices, where the whole graphics pipeline is handled by the GPU, both the PS2 and Dreamcast were responsible for transforming and doing lighting calculations for each vertex BEFORE they got submitted to the GPU. The PS2 had a vector coprocessor to do this, while the Dreamcast had a few extremely important SIMD and fast math assembly instructions on its CPU to do these computations. Up until literally just a few hours ago (not shown in this footage), the Dreamcast's SH4 was doing 100% of these operations in slow-ass plain C and C++ code, which is absolutely sub-optimal and is immediately bogging down its CPU with just transforming vertices, bottlenecking the entire graphics pipeline on the fist T&L stage, and also leaving less CPU time for handling other gameplay logic... this is going to absolutely have to be addressed (and already has begun to be). Another issue that is crippling performance here is the fact that the models are all using individual triangles rather than triangle strips, which the Dreamcast's PVR GPU was designed to handle better... Converting these models to use strips rather than individual triangles will result in MANY different gainz for the DC, as you're going from 3N to N+2 vertices per triangle. Converting the models to triangle strips will 1) reduce load times, since model assets will be smaller 2) reduce the amount of video memory required to hold these vertices on the GPU 3) reduce the amount of shit that must be transferred from the CPU to the GPU and 4) give us back a bunch of CPU time, since the SH4 will be less bogged down transforming redundant vertices! TL;DR: This is still EXTREMELY suboptimal in terms of fully utilizing the graphical potential of the Dreamcast. There is going to be a LOT that can be done still to both improve performance and polygon counts, so stay tuned! FINALLY: Mad respect and love to Stefanos Kornilios Mitsis Poiitidis, for doing an amazing job leading this project, and to Frogbull , Esppiral, and everyone else who is helping us stick it to the PS2 by making this happen! #gamedev #retrogaming #cplusplus

Falco Girgis

88,356 Aufrufe • vor 2 Jahren

$ASTI Ascent Solar Technologies Space and Drone Solar Panels The "Going to Zero" or Mispriced Space/Drone Solar Play Intro and comparison to $RKLB and $RDW panels Let’s get the ugly stuff out of the way first. $ASTI is a distressed penny stock with a ~$5M-$10M market cap. • They burn millions in cash. • 2024 Revenue: ~$40k. 2025 Revenue (YTD): ~$60k. • They generate less revenue than a single Tesla Model Y. • They have diluted shareholders relentlessly. $ASTI just raised $2M in December with the potential of $3.5M more via warrants while being a ~$5M mcap "company". Yikes. To most, this is "uninvestable trash." Stay away. Full stop. So why did I buy ~5% of the float? IF the technology works and IF they execute then I believe this is a massive market pricing dislocation about to inflect. They have been grinding for years and may finally be hitting an inflection point. $RKLB Rocketlab is the king of space solar and they are my second largest position overall, but here is why $ASTI might be a very high risk but asymmetric bet in Space & Defense right now. 1. The Tech Pivot: Flexible CIGS vs. The World Ascent started in 2005 but pivoted 2 years ago from consumer to pure-play Space & Defense. They have sunk ~$250M and 20 years of R&D into proprietary CIGS (Copper-Indium-Gallium-Selenide) thin-film technology while building out fully domestic and vertically integrated manufacturing capabilities. The Physics: • Thickness: 0.03 mm (Thinner than paper). • Flexibility: Wraps around drones/satellites; rolls up like a poster. • Durability: "Self-Healing" capabilities against space radiation. Can take a bullet or micrometeoroid and keep working. Can handle shocks/vibration. Does not shatter. The Metric that Matters: Specific Power (W/kg) (aka energy to weight ratio) In space, mass means cost and difficult decision decisions. • Rocket Lab ($RKLB) / Spectrolab: ~150 W/kg (System level). • Ascent Solar ($ASTI): ~1,960 W/kg (Module level). $ASTI is roughly 10x lighter for the same power output potential (mass-wise). This frees up design limitations and cost. 2. The Competition: $RKLB & $RDW Rocket Lab (SolAero) & Redwire (iROSA): • Tech: Rigid Crystal Cells (Multi-junction) embedded in a fabric mesh. • Pros: Extreme Efficiency (~30%+). Perfect for limited surface area. • Cons: Heavy, Brittle, Expensive ($3k-$10k per Watt). Manufacturing multi-junction cells (SolAero) involves slowly growing crystals in a vacuum chamber. With radiation the panels degrade and loose efficiency over time which will limit the satellite lifespan. • Use Case: James Webb Telescope, Flagship missions. Ascent Solar (ASTI): • Tech: Flexible Thin-Film on Plastic. • Pros: Ultra-light, Durable, Cheap ($500-$1k per Watt). Manufacturing CIGS is roughly similar to printing newspapers (roll-to-roll). The panels are radiation degradation resistant and will outlive the satellite • Cons: Lower Efficiency (~17.5%). Requires 2x surface area. • Use Case: Mega-Constellations (Starlink/Amazon Leo), Small/Low cost satellites, Drones, Deformable surfaces. The lower efficiency is not an ASTI failing. It is the inherent physics trade-off of not using glass/rigid silicone. The downside however is increased atmospheric drag with very larger/massive panel sheets. Because ASTI modules are ~50% less efficient than rigid panels, they require ~2x the physical surface area to generate the same amount of power. In GEO (High Orbit): Drag doesn't matter. Weight savings are king. A massive solar array allows for more sensors and longer project lifespan. ASTI is highly competitive here. In LEO (Low Orbit): Atmospheric drag is real. A massive solar array acts like a large parachute, causing the satellite to de-orbit faster unless it burns more fuel to stay up. At LEO, smaller satellites are a better fit for ASTI. 3. Durability & Radiation "Self-Healing" Radiation Hardness This is ASTI's "Ace in the Hole" for physics. The Problem: In space, high-energy protons (radiation) smash into solar cells, creating atomic "defects" that trap electrons. Over time, this kills the panel's power output (degradation). The CIGS Advantage: CIGS (Copper-Indium-Gallium-Selenide) material has a unique property where heat (annealing) allows the atomic structure to relax and "heal" these defects. Self-Healing: Because CIGS heals at relatively low temperatures (often achieved just by the sun heating the panel), it suffers significantly less degradation than traditional Silicon or even some GaAs panels over long missions in high-radiation belts (like MEO or GEO). Lifespan: While a rigid GaAs panel might lose 15-20% of its power over 15 years (enough to kill a satellite), CIGS panels heal and can maintain a flatter power curve, potentially outlasting the satellite itself in high-radiation orbits. 4. Brittleness & Flexibility ASTI (CIGS on Polyimide): Flexible. You can roll it like a poster. It can take a bullet or micrometeoroid and the hole will just be a dead spot; the rest of the panel keeps working. It does not shatter. Redwire (ROSA) & Rocket Lab (SolAero): Brittle Cells on a Flex Blanket. $RDW's ROSA (Roll-Out Solar Array) typically uses rigid multi-junction cells (made by SolAero/Rocket Lab or Spectrolab) mounted on a flexible mesh fabric. The Risk: If you bend the cells too far, they crack. They rely on the mesh backing for flexibility, but the active generating material is still a brittle crystal wafer. Much heavier, more expensive, and less durable than $ASTI's option 5. The Inflection Point (Why Now?) After years of silent struggle, late 2025 has seen an explosion of activity. Recent Agreements (Nov/Dec 2025): NovaSpark: Hydrogen-powered military drones. $ASTI panels generate power in the field → NovaSpark creates hydrogen fuel. CisLunar Industries: Integrating ASTI solar with power conversion hardware for deep space longevity. Defiant Space: A strategic alliance to act as the "door opener" for classified DoD/NATO programs. More headlines: Ascent Solar Technologies Provides Leading Space Company with Thin-Film PV modules for Spacecraft Power Generation Testing in Cislunar Space December 03, 2025 08:00 ET Ascent Solar Technologies Delivers Thin-Film PV for Saltwater Environment Durability and Space-Based Power Beaming Testing October 14, 2025 08:00 ET Ascent Solar Enters Teaming Agreement with Emtel Energy USA to Advance Thin-Film PV Energy Storage Capabilities September 16, 2025 08:00 ET Ascent Solar Technologies Signs MOU with Star Catcher Industries to Improve Power Capabilities for Thin-Film Solar Technology in Space August 28, 2025 08:00 ET Ascent Solar Technologies Establishes Rapid Thin-Film PV Delivery Process to Provide Customized Space Solar Products Ahead of Schedule on Mission Enabling Timelines August 07, 2025 08:00 ET The Pipeline (From Aug Corporate Presentation) 18 new NDA's signed in 2025. They are field testing with 3 major players: • Company A: Mega-constellation (+2,500 satellites). • Company B: Space Defense (Explicitly mentioned "Golden Dome"). • Company C: Satellite Manufacturer (30-200 unit scale). Management: New board members include a former founding member of SpaceX and a retired Air Force General and Deputy Assistant Secretary for Contracting (acquisitions expert). The company started in 2005 based out of Colorado, but two years ago pivoted to Space & Defense and away from consumer applications. Made in USA: Defense contracts heavily favor domestic supply chains. ASTI manufactures in Colorado. This is a huge moat against cheap Chinese solar. In their Q3 report they note that their market has seen sudden recent acceleration. The space solar industry is currently only capable of 8 to 12 MW per year of production meanwhile the demand is growing to over 100 MW per year. 6. The Risk (The Sword of Damocles) ⚠️ This is critical. $ASTI just raised ~$2M in December. Attached to that raise are ~2 Million Warrants with a strike price of $1.70. These are exercisable immediately. If the stock rips to $3.00, warrant holders exercise at $1.70 and dump on the market for a risk-free 76% profit. This creates a massive "sell wall" and potential 40% dilution of the float. Summary: This is a binary bet. • Bear Case: They run out of cash in 6 months, dilution spirals, stock goes to $0. • Bull Case: They land one of the "Company A/B/C" contracts. Revenue jumps from $60k to projected $20M+ in 2026. The stock reprices from a "bankrupt penny stock" to a "critical defense/space supplier." I have gradually accumulated ~5% of the float. I am ready for it to go to zero. But if the space economy demands "Cheap, Light, and Durable," $ASTI is the only public pure-play. Disclaimer: This is a very high-risk microcap. Do your own due diligence. Not financial advice.
3:04

$ASTI Ascent Solar Technologies Space and Drone Solar Panels The "Going to Zero" or Mispriced Space/Drone Solar Play Intro and comparison to $RKLB and $RDW panels Let’s get the ugly stuff out of the way first. $ASTI is a distressed penny stock with a ~$5M-$10M market cap. • They burn millions in cash. • 2024 Revenue: ~$40k. 2025 Revenue (YTD): ~$60k. • They generate less revenue than a single Tesla Model Y. • They have diluted shareholders relentlessly. $ASTI just raised $2M in December with the potential of $3.5M more via warrants while being a ~$5M mcap "company". Yikes. To most, this is "uninvestable trash." Stay away. Full stop. So why did I buy ~5% of the float? IF the technology works and IF they execute then I believe this is a massive market pricing dislocation about to inflect. They have been grinding for years and may finally be hitting an inflection point. $RKLB Rocketlab is the king of space solar and they are my second largest position overall, but here is why $ASTI might be a very high risk but asymmetric bet in Space & Defense right now. 1. The Tech Pivot: Flexible CIGS vs. The World Ascent started in 2005 but pivoted 2 years ago from consumer to pure-play Space & Defense. They have sunk ~$250M and 20 years of R&D into proprietary CIGS (Copper-Indium-Gallium-Selenide) thin-film technology while building out fully domestic and vertically integrated manufacturing capabilities. The Physics: • Thickness: 0.03 mm (Thinner than paper). • Flexibility: Wraps around drones/satellites; rolls up like a poster. • Durability: "Self-Healing" capabilities against space radiation. Can take a bullet or micrometeoroid and keep working. Can handle shocks/vibration. Does not shatter. The Metric that Matters: Specific Power (W/kg) (aka energy to weight ratio) In space, mass means cost and difficult decision decisions. • Rocket Lab ($RKLB) / Spectrolab: ~150 W/kg (System level). • Ascent Solar ($ASTI): ~1,960 W/kg (Module level). $ASTI is roughly 10x lighter for the same power output potential (mass-wise). This frees up design limitations and cost. 2. The Competition: $RKLB & $RDW Rocket Lab (SolAero) & Redwire (iROSA): • Tech: Rigid Crystal Cells (Multi-junction) embedded in a fabric mesh. • Pros: Extreme Efficiency (~30%+). Perfect for limited surface area. • Cons: Heavy, Brittle, Expensive ($3k-$10k per Watt). Manufacturing multi-junction cells (SolAero) involves slowly growing crystals in a vacuum chamber. With radiation the panels degrade and loose efficiency over time which will limit the satellite lifespan. • Use Case: James Webb Telescope, Flagship missions. Ascent Solar (ASTI): • Tech: Flexible Thin-Film on Plastic. • Pros: Ultra-light, Durable, Cheap ($500-$1k per Watt). Manufacturing CIGS is roughly similar to printing newspapers (roll-to-roll). The panels are radiation degradation resistant and will outlive the satellite • Cons: Lower Efficiency (~17.5%). Requires 2x surface area. • Use Case: Mega-Constellations (Starlink/Amazon Leo), Small/Low cost satellites, Drones, Deformable surfaces. The lower efficiency is not an ASTI failing. It is the inherent physics trade-off of not using glass/rigid silicone. The downside however is increased atmospheric drag with very larger/massive panel sheets. Because ASTI modules are ~50% less efficient than rigid panels, they require ~2x the physical surface area to generate the same amount of power. In GEO (High Orbit): Drag doesn't matter. Weight savings are king. A massive solar array allows for more sensors and longer project lifespan. ASTI is highly competitive here. In LEO (Low Orbit): Atmospheric drag is real. A massive solar array acts like a large parachute, causing the satellite to de-orbit faster unless it burns more fuel to stay up. At LEO, smaller satellites are a better fit for ASTI. 3. Durability & Radiation "Self-Healing" Radiation Hardness This is ASTI's "Ace in the Hole" for physics. The Problem: In space, high-energy protons (radiation) smash into solar cells, creating atomic "defects" that trap electrons. Over time, this kills the panel's power output (degradation). The CIGS Advantage: CIGS (Copper-Indium-Gallium-Selenide) material has a unique property where heat (annealing) allows the atomic structure to relax and "heal" these defects. Self-Healing: Because CIGS heals at relatively low temperatures (often achieved just by the sun heating the panel), it suffers significantly less degradation than traditional Silicon or even some GaAs panels over long missions in high-radiation belts (like MEO or GEO). Lifespan: While a rigid GaAs panel might lose 15-20% of its power over 15 years (enough to kill a satellite), CIGS panels heal and can maintain a flatter power curve, potentially outlasting the satellite itself in high-radiation orbits. 4. Brittleness & Flexibility ASTI (CIGS on Polyimide): Flexible. You can roll it like a poster. It can take a bullet or micrometeoroid and the hole will just be a dead spot; the rest of the panel keeps working. It does not shatter. Redwire (ROSA) & Rocket Lab (SolAero): Brittle Cells on a Flex Blanket. $RDW's ROSA (Roll-Out Solar Array) typically uses rigid multi-junction cells (made by SolAero/Rocket Lab or Spectrolab) mounted on a flexible mesh fabric. The Risk: If you bend the cells too far, they crack. They rely on the mesh backing for flexibility, but the active generating material is still a brittle crystal wafer. Much heavier, more expensive, and less durable than $ASTI's option 5. The Inflection Point (Why Now?) After years of silent struggle, late 2025 has seen an explosion of activity. Recent Agreements (Nov/Dec 2025): NovaSpark: Hydrogen-powered military drones. $ASTI panels generate power in the field → NovaSpark creates hydrogen fuel. CisLunar Industries: Integrating ASTI solar with power conversion hardware for deep space longevity. Defiant Space: A strategic alliance to act as the "door opener" for classified DoD/NATO programs. More headlines: Ascent Solar Technologies Provides Leading Space Company with Thin-Film PV modules for Spacecraft Power Generation Testing in Cislunar Space December 03, 2025 08:00 ET Ascent Solar Technologies Delivers Thin-Film PV for Saltwater Environment Durability and Space-Based Power Beaming Testing October 14, 2025 08:00 ET Ascent Solar Enters Teaming Agreement with Emtel Energy USA to Advance Thin-Film PV Energy Storage Capabilities September 16, 2025 08:00 ET Ascent Solar Technologies Signs MOU with Star Catcher Industries to Improve Power Capabilities for Thin-Film Solar Technology in Space August 28, 2025 08:00 ET Ascent Solar Technologies Establishes Rapid Thin-Film PV Delivery Process to Provide Customized Space Solar Products Ahead of Schedule on Mission Enabling Timelines August 07, 2025 08:00 ET The Pipeline (From Aug Corporate Presentation) 18 new NDA's signed in 2025. They are field testing with 3 major players: • Company A: Mega-constellation (+2,500 satellites). • Company B: Space Defense (Explicitly mentioned "Golden Dome"). • Company C: Satellite Manufacturer (30-200 unit scale). Management: New board members include a former founding member of SpaceX and a retired Air Force General and Deputy Assistant Secretary for Contracting (acquisitions expert). The company started in 2005 based out of Colorado, but two years ago pivoted to Space & Defense and away from consumer applications. Made in USA: Defense contracts heavily favor domestic supply chains. ASTI manufactures in Colorado. This is a huge moat against cheap Chinese solar. In their Q3 report they note that their market has seen sudden recent acceleration. The space solar industry is currently only capable of 8 to 12 MW per year of production meanwhile the demand is growing to over 100 MW per year. 6. The Risk (The Sword of Damocles) ⚠️ This is critical. $ASTI just raised ~$2M in December. Attached to that raise are ~2 Million Warrants with a strike price of $1.70. These are exercisable immediately. If the stock rips to $3.00, warrant holders exercise at $1.70 and dump on the market for a risk-free 76% profit. This creates a massive "sell wall" and potential 40% dilution of the float. Summary: This is a binary bet. • Bear Case: They run out of cash in 6 months, dilution spirals, stock goes to $0. • Bull Case: They land one of the "Company A/B/C" contracts. Revenue jumps from $60k to projected $20M+ in 2026. The stock reprices from a "bankrupt penny stock" to a "critical defense/space supplier." I have gradually accumulated ~5% of the float. I am ready for it to go to zero. But if the space economy demands "Cheap, Light, and Durable," $ASTI is the only public pure-play. Disclaimer: This is a very high-risk microcap. Do your own due diligence. Not financial advice.

YeahDave

208,349 Aufrufe • vor 8 Monaten

The July 4th weekend All-In The All-In Podcast turned into a long argument about who owns the intelligence layer. The besties think enterprises just woke up to a trap they had been walking into, here's how the conversation went (save this): ◽️ The Palantir-Nvidia deal is a bet against the model-layer duopoly. Palantir will use Nvidia's Nemotron open models to build a custom frontier-quality model for US government agencies, and the agencies own the hardware, the data, and the weights. Sacks framed it as structural: an application company and a chip company both want a competitive model layer, so they are natural partners against a two-provider middle. ◽️ Alex Karp's CNBC "crashout" was actually the thesis. Karp argued enterprises have lost trust in the frontier labs and want to own their compute, models, data, and alpha. Sacks translated it as a new definition of enterprise AI safety: safety means the model provider cannot hoover up your proprietary knowledge and turn it into its next product. ◽️ Figma is the cautionary tale that made it real. Anthropic launched Claude Design into Figma's category, its chief product officer sat on Figma's board and resigned only 3 days before launch, and Figma's stock is down about 50% this year while Anthropic's valuation surged. Sacks listed Claude Science, Security, Legal, Financial, and Code as the same move: dominate the model layer, then take the lucrative verticals. ◽️ The playbook has a name, and it is Microsoft and Google. Sacks argued Anthropic is running the operating-system strategy: own the layer everyone builds on, then walk up the stack. His Google receipt is that fewer than half of searches now send you off-site, versus an early Google that prided itself on how fast it kicked you away. ◽️ The BCG number is what raises the stakes. Chamath cited a BCG return-on-capital-employed study: the cost of capital is back to its long-run 8 to 11%, and half of large US companies cannot earn returns above it. If you are already teetering on your cost of capital, handing your alpha to a provider that may compete with you is not a luxury risk, it is fatal. ◽️ The 16.4x number is the whole argument in one data point. Chamath ran a code-migration task through 8090's harness. Wrapping Claude was 1.4x cheaper and 1.5x faster than Claude Opus alone. Wrapping the best open-source model was 16.4x cheaper, at about 3x slower. For a background task, three extra hours to cut cost by 16x is not a close call. ◽️ Even at 100x cheaper, enterprises were saying no for the wrong reason. Chamath relayed an ex-Meta PM's point that companies reject open models over China and safety fears, when they could host those same open weights on their own GPUs in US data centers with nothing flowing back. The safety objection, she argued, is backwards: the leak is the data you hand the frontier labs. ◽️ Friedberg says the frontier labs are trying to commoditize their own customers. Anthropic has been signing up life-sciences companies to feed a new life-focused model in exchange for early access, and nearly everyone he has talked to now refuses, recognizing that data they spent billions generating becomes worthless once it is pooled with everyone else's. ◽️ The deployment topology is shifting from big hubs to distributed spokes. Friedberg's map: the old assumption was a few capital-advantaged mega-clusters plus inference clouds. The new one is large hubs, medium hubs (enterprise training clusters), and distributed spokes, including on-prem inference in your own building. Owning your weights is the point. ◽️ Chamath's endgame is running GLM himself. An industry contact told him that with harness post-training and telemetry, an open Chinese model like GLM could get as good as Anthropic's Mythos. His conclusion: take GLM, control it soup-to-nuts on US hardware with only US citizens touching it, and pay a fraction. ◽️ The Apple analogy sharpens why renting intelligence is different from renting distribution. Chamath argued Apple is the only platform that respected developers, deliberately keeping its stock apps basic to protect the ecosystem and collect its 30% tax. There is no 30% tax on open models, and worse, you cannot rent intelligence from the same place that rents it to your competitor without ending up identical to them. ◽️ Nvidia's open model is now good enough to matter. Calacanis claimed you cannot tell Jensen Huang's Nemotron from Claude on 95% of searches, and that Nvidia downplayed the model until now to avoid alarming its top customers. The gloves came off once OpenAI, Anthropic, and Elon all signaled their own silicon ambitions. ◽️ Sacks sized the duopoly: roughly $60B and $40B in ARR. Anthropic is around ~$60 billion of ARR, OpenAI at ~$40 billion, and no one else generates meaningful model-layer revenue. Sacks's policy line: the US does not ban monopolies, only anti-competitive tactics, but the government should do nothing to make the duopoly more likely. ◽️ The token deflation call: 90% a year for three years. Calacanis predicted token costs fall 90% annually for three years, putting the price of intelligence near free and making it rational to waste tokens on hardware you already own. Friedberg's version is a 70/20/10 split between big cloud, local, and other clouds. ◽️ A wave of platform lock-in spending is already landing. Calacanis flagged Microsoft standing up a roughly $2.5 billion forward-deployed-engineer effort and Amazon spending about $1 billion on the same, plus OpenAI's version. His read: enterprises will slam the door, because letting a provider's engineers study your business is how it ends up in their model. ◽️ The server-per-employee prediction. Calacanis expects every employee to get $10,000 to $20,000 of local compute, a Mac Studio or a high-RAM Dell, running a personal local model that syncs to a thin laptop. A server per person, so nothing leaks. ◽️ On jobs, the data does not show present-tense loss. Sacks cited a RAMP and Revelio Labs study of over 21,000 US firms: the heaviest AI spenders grew headcount about 10% over two years, and entry-level headcount grew even faster at 12%. Friedberg's harder claim: there is no AI job loss yet, only clunky, gradual value creation, and the media will not reverse its narrative because that destroys its credibility. ◽️ The displacement case is real but forward-dated. The counterpoint on the show was that customer support, entry-level data entry and BPO, and driving are the near-term displacements, with Waymo cited as present-tense evidence: in markets where it hits critical mass, Uber and Lyft stop recruiting drivers. Sacks noted most US entry-level support was already offshored, so the acute risk sits in those countries first. ◽️ The human-premium counternarrative. Friedberg argued that as automation spreads, human interaction gets a premium: the skilled bartender, the real driver, the human-in-the-loop tier. He cited the company (referenced as Klarna) that hyped replacing its whole support team with AI, then reversed a year later on brand grounds. ◽️ The export-control episode needed three conditions, and Sacks says do not over-read it. Commerce lifted controls on Anthropic's Fable 5 after two weeks, with Mythos 5 restored to US customers around June 26 once co-founder Tom Brown replaced Dario as lead negotiator. Sacks's three conditions: Dario boasting for months about a cyber weapon, Amazon reporting failed guardrails in testing, and Dario refusing to roll Fable back. His message to allies: this was a particular set of circumstances rather than the debut of a standing lever. ◽️ The import question nobody answered cleanly. Calacanis pressed on why the US blocks Chinese cars and drones but not Chinese open models like DeepSeek and Kimi. Sacks's answer: a forked open model run on US hardware stops being Chinese, and banning open source would isolate the US and impose a token tax on American enterprises, so let the market decide if American open models win. ◽️ The California fiscal story is a business-climate story. Friedberg walked through the numbers behind Newsom's "balanced" $351B budget: expenses exceed revenue and $20-40B is borrowed to close the gap, the budget grew 65% in six years ($215B to $355B), personal income tax is $142B of ~$211B revenue with the top 1% (150,000 people) paying $70B of it, and the corporate rate of 8.9% sits far above Texas at zero. ◽️ The tax base is leaving, and the state is now taxing everyone else. Friedberg cited 1 to 1.5% of adjusted gross income leaving each year (about 15% over a decade), at least 15 Fortune 500 HQs and ~2,100 firms gone since 2019, and a new 8% software sales tax hitting Word, Gmail, and ChatGPT subscriptions plus a health-insurance tax, on top of a now-permanent 14.4% top bracket. The liabilities behind it run $1.4T in debt, up to $1.5T in unfunded pensions senior to state bonds, and ~$40B/year in out-year deficits. Lastly, the line that framed the whole show: "You can't rent intelligence from the same place that rents it to your competitor." That is the sovereignty thesis in one sentence, and every number in this episode is an argument for it. ____ Follow Fireside Alpha for more summaries on key business and technology conversations.
1:33

The July 4th weekend All-In The All-In Podcast turned into a long argument about who owns the intelligence layer. The besties think enterprises just woke up to a trap they had been walking into, here's how the conversation went (save this): ◽️ The Palantir-Nvidia deal is a bet against the model-layer duopoly. Palantir will use Nvidia's Nemotron open models to build a custom frontier-quality model for US government agencies, and the agencies own the hardware, the data, and the weights. Sacks framed it as structural: an application company and a chip company both want a competitive model layer, so they are natural partners against a two-provider middle. ◽️ Alex Karp's CNBC "crashout" was actually the thesis. Karp argued enterprises have lost trust in the frontier labs and want to own their compute, models, data, and alpha. Sacks translated it as a new definition of enterprise AI safety: safety means the model provider cannot hoover up your proprietary knowledge and turn it into its next product. ◽️ Figma is the cautionary tale that made it real. Anthropic launched Claude Design into Figma's category, its chief product officer sat on Figma's board and resigned only 3 days before launch, and Figma's stock is down about 50% this year while Anthropic's valuation surged. Sacks listed Claude Science, Security, Legal, Financial, and Code as the same move: dominate the model layer, then take the lucrative verticals. ◽️ The playbook has a name, and it is Microsoft and Google. Sacks argued Anthropic is running the operating-system strategy: own the layer everyone builds on, then walk up the stack. His Google receipt is that fewer than half of searches now send you off-site, versus an early Google that prided itself on how fast it kicked you away. ◽️ The BCG number is what raises the stakes. Chamath cited a BCG return-on-capital-employed study: the cost of capital is back to its long-run 8 to 11%, and half of large US companies cannot earn returns above it. If you are already teetering on your cost of capital, handing your alpha to a provider that may compete with you is not a luxury risk, it is fatal. ◽️ The 16.4x number is the whole argument in one data point. Chamath ran a code-migration task through 8090's harness. Wrapping Claude was 1.4x cheaper and 1.5x faster than Claude Opus alone. Wrapping the best open-source model was 16.4x cheaper, at about 3x slower. For a background task, three extra hours to cut cost by 16x is not a close call. ◽️ Even at 100x cheaper, enterprises were saying no for the wrong reason. Chamath relayed an ex-Meta PM's point that companies reject open models over China and safety fears, when they could host those same open weights on their own GPUs in US data centers with nothing flowing back. The safety objection, she argued, is backwards: the leak is the data you hand the frontier labs. ◽️ Friedberg says the frontier labs are trying to commoditize their own customers. Anthropic has been signing up life-sciences companies to feed a new life-focused model in exchange for early access, and nearly everyone he has talked to now refuses, recognizing that data they spent billions generating becomes worthless once it is pooled with everyone else's. ◽️ The deployment topology is shifting from big hubs to distributed spokes. Friedberg's map: the old assumption was a few capital-advantaged mega-clusters plus inference clouds. The new one is large hubs, medium hubs (enterprise training clusters), and distributed spokes, including on-prem inference in your own building. Owning your weights is the point. ◽️ Chamath's endgame is running GLM himself. An industry contact told him that with harness post-training and telemetry, an open Chinese model like GLM could get as good as Anthropic's Mythos. His conclusion: take GLM, control it soup-to-nuts on US hardware with only US citizens touching it, and pay a fraction. ◽️ The Apple analogy sharpens why renting intelligence is different from renting distribution. Chamath argued Apple is the only platform that respected developers, deliberately keeping its stock apps basic to protect the ecosystem and collect its 30% tax. There is no 30% tax on open models, and worse, you cannot rent intelligence from the same place that rents it to your competitor without ending up identical to them. ◽️ Nvidia's open model is now good enough to matter. Calacanis claimed you cannot tell Jensen Huang's Nemotron from Claude on 95% of searches, and that Nvidia downplayed the model until now to avoid alarming its top customers. The gloves came off once OpenAI, Anthropic, and Elon all signaled their own silicon ambitions. ◽️ Sacks sized the duopoly: roughly $60B and $40B in ARR. Anthropic is around ~$60 billion of ARR, OpenAI at ~$40 billion, and no one else generates meaningful model-layer revenue. Sacks's policy line: the US does not ban monopolies, only anti-competitive tactics, but the government should do nothing to make the duopoly more likely. ◽️ The token deflation call: 90% a year for three years. Calacanis predicted token costs fall 90% annually for three years, putting the price of intelligence near free and making it rational to waste tokens on hardware you already own. Friedberg's version is a 70/20/10 split between big cloud, local, and other clouds. ◽️ A wave of platform lock-in spending is already landing. Calacanis flagged Microsoft standing up a roughly $2.5 billion forward-deployed-engineer effort and Amazon spending about $1 billion on the same, plus OpenAI's version. His read: enterprises will slam the door, because letting a provider's engineers study your business is how it ends up in their model. ◽️ The server-per-employee prediction. Calacanis expects every employee to get $10,000 to $20,000 of local compute, a Mac Studio or a high-RAM Dell, running a personal local model that syncs to a thin laptop. A server per person, so nothing leaks. ◽️ On jobs, the data does not show present-tense loss. Sacks cited a RAMP and Revelio Labs study of over 21,000 US firms: the heaviest AI spenders grew headcount about 10% over two years, and entry-level headcount grew even faster at 12%. Friedberg's harder claim: there is no AI job loss yet, only clunky, gradual value creation, and the media will not reverse its narrative because that destroys its credibility. ◽️ The displacement case is real but forward-dated. The counterpoint on the show was that customer support, entry-level data entry and BPO, and driving are the near-term displacements, with Waymo cited as present-tense evidence: in markets where it hits critical mass, Uber and Lyft stop recruiting drivers. Sacks noted most US entry-level support was already offshored, so the acute risk sits in those countries first. ◽️ The human-premium counternarrative. Friedberg argued that as automation spreads, human interaction gets a premium: the skilled bartender, the real driver, the human-in-the-loop tier. He cited the company (referenced as Klarna) that hyped replacing its whole support team with AI, then reversed a year later on brand grounds. ◽️ The export-control episode needed three conditions, and Sacks says do not over-read it. Commerce lifted controls on Anthropic's Fable 5 after two weeks, with Mythos 5 restored to US customers around June 26 once co-founder Tom Brown replaced Dario as lead negotiator. Sacks's three conditions: Dario boasting for months about a cyber weapon, Amazon reporting failed guardrails in testing, and Dario refusing to roll Fable back. His message to allies: this was a particular set of circumstances rather than the debut of a standing lever. ◽️ The import question nobody answered cleanly. Calacanis pressed on why the US blocks Chinese cars and drones but not Chinese open models like DeepSeek and Kimi. Sacks's answer: a forked open model run on US hardware stops being Chinese, and banning open source would isolate the US and impose a token tax on American enterprises, so let the market decide if American open models win. ◽️ The California fiscal story is a business-climate story. Friedberg walked through the numbers behind Newsom's "balanced" $351B budget: expenses exceed revenue and $20-40B is borrowed to close the gap, the budget grew 65% in six years ($215B to $355B), personal income tax is $142B of ~$211B revenue with the top 1% (150,000 people) paying $70B of it, and the corporate rate of 8.9% sits far above Texas at zero. ◽️ The tax base is leaving, and the state is now taxing everyone else. Friedberg cited 1 to 1.5% of adjusted gross income leaving each year (about 15% over a decade), at least 15 Fortune 500 HQs and ~2,100 firms gone since 2019, and a new 8% software sales tax hitting Word, Gmail, and ChatGPT subscriptions plus a health-insurance tax, on top of a now-permanent 14.4% top bracket. The liabilities behind it run $1.4T in debt, up to $1.5T in unfunded pensions senior to state bonds, and ~$40B/year in out-year deficits. Lastly, the line that framed the whole show: "You can't rent intelligence from the same place that rents it to your competitor." That is the sovereignty thesis in one sentence, and every number in this episode is an argument for it. ____ Follow Fireside Alpha for more summaries on key business and technology conversations.

Fireside Alpha

55,816 Aufrufe • vor 2 Monaten