正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

DUAL INTEL ARC PRO B70 GPUS AT $1,900 RUN QWEN 3 CODER 30B AT 268 TOKENS PER SECOND VIA VLLM ON 8 CONCURRENT REQUESTS, A NEW INTEL TIER UNDER YOUR MAP THAT BEATS RTX 3090 STACKS ON THROUGHPUT PER DOLLAR 02:22 the operator reads off his benchmark sheet, "vllm... was at 268 tokens per second which is just wild to me, getting 250+ tokens per second out of these overall kind of slow Intel cards" dual Intel Arc Pro B70 ships at $950 per card, 32GB GDDR6 each, 64GB combined VRAM for $1,900 total, the same budget gets you two used RTX 3090s at 48GB combined and worse multi-user throughput Qwen 3 coder 30B on dual B70 with vLLM hits 268 t/s at 8 concurrent users, llama.cpp tops out at 84 t/s on the same hardware, the gap is not 2x or 3x, vLLM with Intel's tweaks pulls 3.2x on parallel inference prefill on a 16,000 word context took 26 seconds on llama.cpp and 3 seconds on vLLM, agents spawning subagents or households sharing one box turn that 8x speedup into 250 t/s of usable throughput your map skips Intel because the Arc cards looked weak in 2025, the B70 plus the vLLM stack flipped that, $1,900 in dual Intel silicon now serves a 4 person team at the speed openai serves enterprise tiers for $400 a month per seat bookmark this and read the article belowshow more

starmex

9,602 subscribers

29,137 次观看 • 3 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

40 条评论

Myttle 的头像

Myttle3 个月前

Intel gets interesting the moment concurrency matters more than single-user speed

starmex 的头像

starmex3 个月前

ye concurrency is where the real game starts

kocer 的头像

kocer3 个月前

dual b70 for 1900 and beating 3090 stacks on concurrent stuff is wild. finally intel making sense for local runs

starmex 的头像

starmex3 个月前

ye intel finally pulling weight on concurrent runs

Insomnia 的头像

Insomnia3 个月前

intel also have good hardware

starmex 的头像

starmex3 个月前

ye intel hardware finally pulling its weight

Hussain Hashim | Building SundayBack 的头像

Hussain Hashim | Building SundayBack3 个月前

@starmexxx dude, that's wild. didn’t expect intel to push past 3090s like that. game changer for sure

Fokki 的头像

Fokki3 个月前

keep spreading this

starmex 的头像

starmex3 个月前

appreciate it bro, intel tier is sleeper alpha

Gipp 🦅 的头像

Gipp 🦅3 个月前

Intel really excels in all these features

starmex 的头像

starmex3 个月前

ye intel finally cooking, the timing is wild

Yarchi 的头像

Yarchi3 个月前

Intel quietly becoming viable

starmex 的头像

starmex3 个月前

ye intel quietly stacking wins under the radar

Scott Jordan 的头像

Scott Jordan3 个月前

So single stream is like 32 tok/s?

Mercer 的头像

Mercer3 个月前

this is whats gonna shift homelabs from 3090 stacks to mixed-vendor builds

Nicholas Hyperion 的头像

Nicholas Hyperion3 个月前

'Slow' Intel cards beating NVIDIA stacks on throughput because someone optimized the serving layer. ML remains not a serious science and the jankiest shit somehow works.

Dipanshu Kushwaha 的头像

Dipanshu Kushwaha3 个月前

Wow, that's pretty impressive performance for the price point. Intel is really stepping up.

FerTech 🇨🇭 的头像

FerTech 🇨🇭3 个月前

What do you think about this @grok ?

Dolfenspif 的头像

Dolfenspif2 个月前

now way in hell I would run LLM on a ǰèŵ.

Lorden 的头像

Lorden3 个月前

spending less for more is wild

starmex 的头像

starmex3 个月前

ye throughput per dollar finally flipped

Oracle Over Coffee 的头像

Oracle Over Coffee3 个月前

but what drivers support it, why not stay in the CUDA lane? and the VRAM what 3x slower than RTX?

rewind 的头像

rewind3 个月前

intel just got interesting again

@Mazarion 的头像

@Mazarion3 个月前

Intel finally making sense for local AI. Dual B70 for $1900 pulling 268 t/s is actually wild.

Harry Tandy 的头像

Harry Tandy2 个月前

local hardware setups are getting scary fast these days

Nick Venturi 的头像

Nick Venturi2 个月前

nvidia shaking in their overpriced boots 😂

monokern 的头像

monokern3 个月前

it's now more cost-effective to buy a graphics card than to pay for a monthly subscription

starmex 的头像

starmex3 个月前

exactly bro, the gpu pays itself off in months

cgomes 的头像

cgomes2 个月前

Hey @starmexxx who is the guy in this video? Does he have a youtube channel?

0xSlyth 的头像

0xSlyth3 个月前

intel just flipped the script hard

starmex 的头像

starmex3 个月前

ye intel coming in hot, throughput per dollar is wild

leopardracer 的头像

leopardracer3 个月前

bookmarked this!

starmex 的头像

starmex3 个月前

ty!

Alexey Fateev 的头像

Alexey Fateev3 个月前

Throughput per dollar maybe, but the layout matters more than the cards. On 4x3090 the trick is NOT tensor-parallel: 4 data-parallel copies of Gemma 12B do 3,425 tok/s vs 1,007 for TP4. Most 3090 stacks are just configured wrong:

That Guy 的头像

That Guy3 个月前

Tell me what OS version and kernel you are using please? My arc pro wedges between 60-90 seconds of agentic load. It’ll donthe benchmarks, but is unusable for real workloads.

Michał Piszczek 的头像

Michał Piszczek3 个月前

throughput per dollar is the right axis. the Intel tax shows up later in driver maturity and the hours you spend debugging the stack

Esteban L 的头像

Esteban L2 个月前

Hi @starmexxx grettings from Mexico, what CPU (Intel or AMD) and how much RAM do you have in your setup?? Also can you recommend me or suggest a good motherboard to support both Intel cards?

Faheem | FrontierMind AI 的头像

Faheem | FrontierMind AI2 个月前

Throughput per dollar is becoming a serious AI metric. The next deployment wave will care less about demos and more about what can run reliably at usable cost.

Kenzo🥷 的头像

Kenzo🥷3 个月前

268 t/s on Intel Arc via vLLM at $1,900-software beats hardware, again. I'm waiting for independent tests, but if this is confirmed, it changes the equation for small teams.

ChonkE 的头像

ChonkE3 个月前

Battlemage stronk af 🦾🦾🦾🦾🦾

相关视频

OPENAI PAYS NVIDIA $30K PER H100. INTEL SELLS THE SAME 32GB VRAM IN A $950 ARC PRO B70. HE STACKED 2 AND RUNS OPUS-5 LOCALLY FOR $3/MONTH buy 2 intel arc pro b70 for $1,900 -> plug them into any dual-x8 pcie motherboard -> boot ubuntu 26.04 -> gpus recognized instantly -> load a q4 moe model like qwen 35b-a3b -> run 64gb of local ai for $3/month in electricity that loop is why nvidia's 5090 monopoly on 32gb vram just quietly cracked and openai's inference costs are about to look ridiculous 2x intel arc pro b70 + asrock taichi lite + ryzen 5 9600 + 64gb ddr5 + ubuntu 26.04 + intel's ipex stack - that's the stack watch and save it, then order your first b70 this weekend

OPENAI PAYS NVIDIA $30K PER H100. INTEL SELLS THE SAME 32GB VRAM IN A $950 ARC PRO B70. HE STACKED 2 AND RUNS OPUS-5 LOCALLY FOR $3/MONTH buy 2 intel arc pro b70 for $1,900 -> plug them into any dual-x8 pcie motherboard -> boot ubuntu 26.04 -> gpus recognized instantly -> load a q4 moe model like qwen 35b-a3b -> run 64gb of local ai for $3/month in electricity that loop is why nvidia's 5090 monopoly on 32gb vram just quietly cracked and openai's inference costs are about to look ridiculous 2x intel arc pro b70 + asrock taichi lite + ryzen 5 9600 + 64gb ddr5 + ubuntu 26.04 + intel's ipex stack - that's the stack watch and save it, then order your first b70 this weekend

Antid

28,739 次观看 • 1 个月前

NVIDIA CHARGES $4,000 FOR AN RTX 5090 WITH 32GB. INTEL SELLS THE SAME 32GB FOR $950. HE STACKED TWO INTO 64GB AND KILLED $200/MO CLAUDE CODE AND $200/MO CHATGPT PRO FOR $3 IN ELECTRICITY 00:09 he's been at it since the middle of the night and admits it up front, "I've been testing these pretty much non stop since about 2am today" 2x intel arc pro b70 at $950 each -> 64gb pooled for less than half a 5090 -> asrock taichi lite, both in x8 slots -> ryzen 9600, 64gb ddr5 -> ubuntu 26.04 sees both cards instantly, zero driver hunting -> intel's own stack, not community packages -> mixture of experts models fly nvidia wants $4,000 for 32gb, intel put the same capacity in a card at 23% of that, so a 70b rig went from a $4,000 ticket to $1,900 for double the memory, and the $400 a month altman and amodei charge becomes $3 in electricity the honest part: q8 hits a hard bottleneck on battlemage, bizarre because intel's older alchemist cards were faster at q8 than q4, dense models are rough, and running both cards together has been ugly so far 2x intel arc pro b70 32gb + asrock taichi lite + ryzen 9600 + 64gb ddr5 + ubuntu 26.04 + intel's own inference stack watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below

NVIDIA CHARGES $4,000 FOR AN RTX 5090 WITH 32GB. INTEL SELLS THE SAME 32GB FOR $950. HE STACKED TWO INTO 64GB AND KILLED $200/MO CLAUDE CODE AND $200/MO CHATGPT PRO FOR $3 IN ELECTRICITY 00:09 he's been at it since the middle of the night and admits it up front, "I've been testing these pretty much non stop since about 2am today" 2x intel arc pro b70 at $950 each -> 64gb pooled for less than half a 5090 -> asrock taichi lite, both in x8 slots -> ryzen 9600, 64gb ddr5 -> ubuntu 26.04 sees both cards instantly, zero driver hunting -> intel's own stack, not community packages -> mixture of experts models fly nvidia wants $4,000 for 32gb, intel put the same capacity in a card at 23% of that, so a 70b rig went from a $4,000 ticket to $1,900 for double the memory, and the $400 a month altman and amodei charge becomes $3 in electricity the honest part: q8 hits a hard bottleneck on battlemage, bizarre because intel's older alchemist cards were faster at q8 than q4, dense models are rough, and running both cards together has been ugly so far 2x intel arc pro b70 32gb + asrock taichi lite + ryzen 9600 + 64gb ddr5 + ubuntu 26.04 + intel's own inference stack watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below

Antid

39,006 次观看 • 17 天前

INTEL JUST QUIETLY BUILT THE CHEAPEST PATH TO 128GB OF LOCAL AI VRAM, AND IT BEATS NVIDIA'S DGX SPARK BY 2X FOR THE SAME $4,000 00:11 four Arc Pro B70s racked in one chassis. 32GB of GDDR6 each, $949 MSRP, roughly $1,000 street per card stack four and you get 128GB of VRAM for about $4,000 total. a lab benchmarked the quad against a DGX Spark and a single RTX Pro 6000. the four B70s doubled the Spark or better they trail the Pro 6000, but that card is $10,000. so the real choice is now $4K for 128GB from Intel or $10K for 96GB from NVIDIA the article covers the DGX Spark at $3,999 as the single-desk local AI ceiling. this Intel quad matches the price, beats the memory throughput, and nobody is talking about it yet save this before Intel's VRAM-per-dollar advantage becomes the default homelab build ↓

INTEL JUST QUIETLY BUILT THE CHEAPEST PATH TO 128GB OF LOCAL AI VRAM, AND IT BEATS NVIDIA'S DGX SPARK BY 2X FOR THE SAME $4,000 00:11 four Arc Pro B70s racked in one chassis. 32GB of GDDR6 each, $949 MSRP, roughly $1,000 street per card stack four and you get 128GB of VRAM for about $4,000 total. a lab benchmarked the quad against a DGX Spark and a single RTX Pro 6000. the four B70s doubled the Spark or better they trail the Pro 6000, but that card is $10,000. so the real choice is now $4K for 128GB from Intel or $10K for 96GB from NVIDIA the article covers the DGX Spark at $3,999 as the single-desk local AI ceiling. this Intel quad matches the price, beats the memory throughput, and nobody is talking about it yet save this before Intel's VRAM-per-dollar advantage becomes the default homelab build ↓

NO1ennn

139,901 次观看 • 2 个月前

NVIDIA CEO JENSEN HUANG UNVEILED THE RTX 3090 AND THE USED VERSION ON EBAY NOW KILLS A $400/MONTH CLAUDE CODE AND OPENAI BILL the rtx 3090 has 24gb of vram, the same as a new rtx 4090 that costs $2,000. for local ai inference vram matters more than anything else and the 3090 sells for $650-800 used it runs qwen 3.6 27b comfortably, deepseek r1 14b at 60+ tokens per second and mistral 7b for fast automation. 16gb of qwen 3.6 fits with 8gb of headroom for long context alibaba's qwen 3.6 27b beats claude 4.5 opus on realworldqa vision 84.1 vs 77.0 and on ifbench instructions 76.5 vs 58.0. a free open weight model out-benchmarks anthropic's flagship a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and a used 3090 pays itself off in 3.5 months install ollama, pull qwen 3.6 27b, point claude code at localhost. same interface, nothing leaves the machine, nothing costs per request bookmark this and read the article below

NVIDIA CEO JENSEN HUANG UNVEILED THE RTX 3090 AND THE USED VERSION ON EBAY NOW KILLS A $400/MONTH CLAUDE CODE AND OPENAI BILL the rtx 3090 has 24gb of vram, the same as a new rtx 4090 that costs $2,000. for local ai inference vram matters more than anything else and the 3090 sells for $650-800 used it runs qwen 3.6 27b comfortably, deepseek r1 14b at 60+ tokens per second and mistral 7b for fast automation. 16gb of qwen 3.6 fits with 8gb of headroom for long context alibaba's qwen 3.6 27b beats claude 4.5 opus on realworldqa vision 84.1 vs 77.0 and on ifbench instructions 76.5 vs 58.0. a free open weight model out-benchmarks anthropic's flagship a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and a used 3090 pays itself off in 3.5 months install ollama, pull qwen 3.6 27b, point claude code at localhost. same interface, nothing leaves the machine, nothing costs per request bookmark this and read the article below

starmex

86,831 次观看 • 3 个月前

INTEL JUST SHIPPED A WORKSTATION CARD WITH TWO GPUs ON ONE PCB AND 48GB OF VRAM. FOUR CARDS GIVE A SINGLE MOTHERBOARD 192GB OF POOLED INFERENCE MEMORY FOR THE PRICE OF TWO RTX 5090s 00:14 he holds one card up to the camera, two GPU dies side by side under the cooler, each one running its own x8 PCIe lane back to the chipset the cluster sees eight discrete accelerators in software. intel's Battle Matrix stack shards a model across all eight, so a 235B parameter network loads in slices and answers requests in parallel what 192GB of VRAM unlocks: an entire 200B class model in memory without quantization. a vision agent reading 100 invoices at once. a research box that holds three frontier models loaded simultaneously, switching between them in under a second intel is the slow side of inference. nvidia is faster per token, that is the honest tradeoff. but the only other path to this much VRAM is a $40,000 nvidia rack or three networked Mac Studios four B60 cards plus the chassis lands under $5,000. power draw averages 800 watts, $55 a month in electricity. one engineer paying $400 a month for combined ChatGPT Pro and Claude Code Max pays the hardware off in less than a year

INTEL JUST SHIPPED A WORKSTATION CARD WITH TWO GPUs ON ONE PCB AND 48GB OF VRAM. FOUR CARDS GIVE A SINGLE MOTHERBOARD 192GB OF POOLED INFERENCE MEMORY FOR THE PRICE OF TWO RTX 5090s 00:14 he holds one card up to the camera, two GPU dies side by side under the cooler, each one running its own x8 PCIe lane back to the chipset the cluster sees eight discrete accelerators in software. intel's Battle Matrix stack shards a model across all eight, so a 235B parameter network loads in slices and answers requests in parallel what 192GB of VRAM unlocks: an entire 200B class model in memory without quantization. a vision agent reading 100 invoices at once. a research box that holds three frontier models loaded simultaneously, switching between them in under a second intel is the slow side of inference. nvidia is faster per token, that is the honest tradeoff. but the only other path to this much VRAM is a $40,000 nvidia rack or three networked Mac Studios four B60 cards plus the chassis lands under $5,000. power draw averages 800 watts, $55 a month in electricity. one engineer paying $400 a month for combined ChatGPT Pro and Claude Code Max pays the hardware off in less than a year

NO1ennn

48,247 次观看 • 2 个月前

AMD'S NEW MINI PC RUNS CLAUDE CODE OFFLINE FOR $0/MONTH AND REPLACES $200 OPENAI + $200 CLAUDE CODE SUBSCRIPTIONS amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on a single piece of silicon with 128gb of unified memory the gmktec evo-x2 runs qwen3 coder 30b at 40 to 50 tokens per second and llama 3.3 70b at 20 to 25 tokens per second locally. on linux you get 110gb of usable vram out of 128gb setup takes 90 minutes. wipe windows for linux, install ollama with one command, pull the model, point claude code at localhost. same interface, same agent loop, nothing leaves your network a heavy ai user pays $200 for claude code max, $200 for chatgpt pro and $20 for cursor. that's $5,040 a year and the box pays itself off in 5 months after break-even every month is $420 that used to leave your account. three year math is roughly $13,000 not paid to ai companies plus zero rate limits on overnight agents bookmark this and read the article below

AMD'S NEW MINI PC RUNS CLAUDE CODE OFFLINE FOR $0/MONTH AND REPLACES $200 OPENAI + $200 CLAUDE CODE SUBSCRIPTIONS amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on a single piece of silicon with 128gb of unified memory the gmktec evo-x2 runs qwen3 coder 30b at 40 to 50 tokens per second and llama 3.3 70b at 20 to 25 tokens per second locally. on linux you get 110gb of usable vram out of 128gb setup takes 90 minutes. wipe windows for linux, install ollama with one command, pull the model, point claude code at localhost. same interface, same agent loop, nothing leaves your network a heavy ai user pays $200 for claude code max, $200 for chatgpt pro and $20 for cursor. that's $5,040 a year and the box pays itself off in 5 months after break-even every month is $420 that used to leave your account. three year math is roughly $13,000 not paid to ai companies plus zero rate limits on overnight agents bookmark this and read the article below

starmex

33,516 次观看 • 3 个月前

Alibaba just released a coding model that hits 82 percent on SWE-Bench Verified. That is the highest score ever published for an open-source model. The weights are free. The license is Apache 2.0. You can run it today. The model is Qwen 4 Coder 32B. Here is what 82 percent on SWE-Bench Verified actually means. SWE-Bench Verified tests whether an AI can autonomously resolve real bugs pulled from real production GitHub repositories. Not synthetic exercises. Real open-source projects that real teams depend on. A model gets a bug report, reads the code, writes a fix, and either passes the test suite or it does not. At 82 percent, Qwen 4 Coder 32B resolves 82 out of every 100 real production bugs it is given. Without a human guiding it. On code it has never seen before. For comparison: Qwen 4 Coder 32B: 82 percent SWE-Bench Verified. Open source. Apache 2.0. Claude Fable 5: 80.3 percent SWE-Bench Pro. $10 input / $50 output per million tokens. Currently suspended. GPT-5.6 Sol: Competitive on Terminal-Bench. $5 input / $30 output per million tokens. An open-weight model that you can download and run for free just beat both of them on the benchmark designed to measure real software engineering capability. Here is the architecture. Qwen 4 Coder 32B is a 32 billion parameter dense model. Not a Mixture-of-Experts. Every parameter is active on every request. This matters for inference: a dense 32B model runs on 22 gigabytes of VRAM, which fits on a single high-end consumer GPU or a MacBook Pro with 64GB of unified memory. The smaller variant, Qwen 4 Coder 4B, runs at approximately 135 tokens per second on an M5 Max and fits inside 8 gigabytes of RAM. For a model with usable coding capability, that is a new bar for what fits in a single laptop. The training methodology continued Alibaba's approach of reinforcement learning on verifiable coding tasks. The model gets rewarded when its code passes tests. It gets penalized when it fails. Over millions of training steps, the model learns to write code that actually runs rather than code that looks plausible. License: Apache 2.0. Full commercial use. No attribution requirement. No revenue threshold. No monthly active user ceiling. Weights: Hugging Face, available today. Runs on: vLLM, Ollama, SGLang, and any standard GGUF-compatible inference engine. Qwen 4 32B also runs at approximately 135 tokens per second on an M5 Max chip, setting a new bar for what a sub-8GB model can do on Apple Silicon. The open-source coding model just beat the best closed-source model in the world on the benchmark designed to test whether AI can actually do software engineering. The weights are free. The subscription is optional. Source: Autom8Labs AI Insight July 2026, State of Open Source LLMs June 2026, Kunal Ganglani blog June 2026.

Alibaba just released a coding model that hits 82 percent on SWE-Bench Verified. That is the highest score ever published for an open-source model. The weights are free. The license is Apache 2.0. You can run it today. The model is Qwen 4 Coder 32B. Here is what 82 percent on SWE-Bench Verified actually means. SWE-Bench Verified tests whether an AI can autonomously resolve real bugs pulled from real production GitHub repositories. Not synthetic exercises. Real open-source projects that real teams depend on. A model gets a bug report, reads the code, writes a fix, and either passes the test suite or it does not. At 82 percent, Qwen 4 Coder 32B resolves 82 out of every 100 real production bugs it is given. Without a human guiding it. On code it has never seen before. For comparison: Qwen 4 Coder 32B: 82 percent SWE-Bench Verified. Open source. Apache 2.0. Claude Fable 5: 80.3 percent SWE-Bench Pro. $10 input / $50 output per million tokens. Currently suspended. GPT-5.6 Sol: Competitive on Terminal-Bench. $5 input / $30 output per million tokens. An open-weight model that you can download and run for free just beat both of them on the benchmark designed to measure real software engineering capability. Here is the architecture. Qwen 4 Coder 32B is a 32 billion parameter dense model. Not a Mixture-of-Experts. Every parameter is active on every request. This matters for inference: a dense 32B model runs on 22 gigabytes of VRAM, which fits on a single high-end consumer GPU or a MacBook Pro with 64GB of unified memory. The smaller variant, Qwen 4 Coder 4B, runs at approximately 135 tokens per second on an M5 Max and fits inside 8 gigabytes of RAM. For a model with usable coding capability, that is a new bar for what fits in a single laptop. The training methodology continued Alibaba's approach of reinforcement learning on verifiable coding tasks. The model gets rewarded when its code passes tests. It gets penalized when it fails. Over millions of training steps, the model learns to write code that actually runs rather than code that looks plausible. License: Apache 2.0. Full commercial use. No attribution requirement. No revenue threshold. No monthly active user ceiling. Weights: Hugging Face, available today. Runs on: vLLM, Ollama, SGLang, and any standard GGUF-compatible inference engine. Qwen 4 32B also runs at approximately 135 tokens per second on an M5 Max chip, setting a new bar for what a sub-8GB model can do on Apple Silicon. The open-source coding model just beat the best closed-source model in the world on the benchmark designed to test whether AI can actually do software engineering. The weights are free. The subscription is optional. Source: Autom8Labs AI Insight July 2026, State of Open Source LLMs June 2026, Kunal Ganglani blog June 2026.

Harman

41,278 次观看 • 2 个月前

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Avi Chawla

16,140 次观看 • 1 个月前

$I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.$

I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.

Alok

36,617 次观看 • 2 个月前

A single RTX 4090 (24 GB VRAM) can run the updated gemma 4 31B (dense) model with a 190,000 context window at 33 tokens/second. The VRAM barrier is dying. Google quietly updated Gemma 4, and Unsloth immediately compiled the new quants. I built llama.cpp from source on Ubuntu 22 to benchmark it. Google's stealth update 2 days ago enabled uniform Flash Attention 4 on Hopper to boost prefill and patched the chat template to improve tool calling. The agentic reasoning gains on the benchmark charts are massive: TB2 (Agents): +4.5% (to 25.8%) Tau2 (Telecom): +10.1% (to 62.7%) Running on Ubuntu 22, CUDA 13.0 with a single NVIDIA GeForce RTX 4090. Here is the exact step by step benchmarking process with a massive 28k tokens prompt and the commands I used to squeeze out maximum context without killing my throughput: # 1. The Baseline (Unquantized KV Cache) I started with full GPU offload (-ngl 99) and pushed the context to 40k. llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -ngl 99 -c 40000 -fa on --port 8080 -v VRAM: 23.8 GB (maxed out on card) Throughput: Prefill: 2198.81 t/s | Decode: 35.77 t/s (with 28k tokens prompt) # 2. The CPU Split Trap I tried stretching to 80k context by offloading layers to the CPU (-ngl 52). llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 80000 -ngl 52 -fa on --port 8080 -v Throughput: Prefill: 1212.73 t/s | Decode: 5 t/s (with 28k tokens prompt) # 3. The KV Quantization Breakthrough Instead of spilling layers to the CPU, I kept the model fully on card (-ngl 99) but enabled 8-bit KV cache quantization to free up VRAM. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 100000 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 --port 8080 -v VRAM: 23.9 GB Throughput: Prefill: 2139.68 t/s | Decode: 32 t/s (with 28k tokens prompt) Result: 100k tokens of context on a single GPU with practically zero speed loss (and minimal intelligence loss). # 4. The Limit Test (Q4 KV Cache) To find the absolute breaking point, I dropped the KV cache to 4 bit (q4_0) and set -c 190000. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 190000 --cache-type-k q4_0 --cache-type-v q4_0 -ngl 99 --port 8080 -v VRAM: 23.8 GB Throughput: Prefill: 2206.66 t/s | Decode: 33 t/s (with 28k tokens prompt) (Note: Pushing it to 220k required dropping to -ngl 58 again, which immediately penalized decode down to 17 t/s). # The Tradeoff: For Max Reasoning: Keep your KV cache unquantized (f16). You get pristine reasoning but hit a strict 40k context ceiling. For Massive Document Retrieval: If you need to feed the model giant codebases, use --cache-type-k q4_0. Getting 190k context at 33 tokens/second on a consumer desktop with a 31b dense model is a cheat code. If you’re rocking a single 3090 or 4090 and slept on Gemma 4 earlier, this update is your cue to dust off the terminal. Hugging Face links to the Unsloth QAT quants are in the replies below.

A single RTX 4090 (24 GB VRAM) can run the updated gemma 4 31B (dense) model with a 190,000 context window at 33 tokens/second. The VRAM barrier is dying. Google quietly updated Gemma 4, and Unsloth immediately compiled the new quants. I built llama.cpp from source on Ubuntu 22 to benchmark it. Google's stealth update 2 days ago enabled uniform Flash Attention 4 on Hopper to boost prefill and patched the chat template to improve tool calling. The agentic reasoning gains on the benchmark charts are massive: TB2 (Agents): +4.5% (to 25.8%) Tau2 (Telecom): +10.1% (to 62.7%) Running on Ubuntu 22, CUDA 13.0 with a single NVIDIA GeForce RTX 4090. Here is the exact step by step benchmarking process with a massive 28k tokens prompt and the commands I used to squeeze out maximum context without killing my throughput: # 1. The Baseline (Unquantized KV Cache) I started with full GPU offload (-ngl 99) and pushed the context to 40k. llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -ngl 99 -c 40000 -fa on --port 8080 -v VRAM: 23.8 GB (maxed out on card) Throughput: Prefill: 2198.81 t/s | Decode: 35.77 t/s (with 28k tokens prompt) # 2. The CPU Split Trap I tried stretching to 80k context by offloading layers to the CPU (-ngl 52). llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 80000 -ngl 52 -fa on --port 8080 -v Throughput: Prefill: 1212.73 t/s | Decode: 5 t/s (with 28k tokens prompt) # 3. The KV Quantization Breakthrough Instead of spilling layers to the CPU, I kept the model fully on card (-ngl 99) but enabled 8-bit KV cache quantization to free up VRAM. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 100000 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 --port 8080 -v VRAM: 23.9 GB Throughput: Prefill: 2139.68 t/s | Decode: 32 t/s (with 28k tokens prompt) Result: 100k tokens of context on a single GPU with practically zero speed loss (and minimal intelligence loss). # 4. The Limit Test (Q4 KV Cache) To find the absolute breaking point, I dropped the KV cache to 4 bit (q4_0) and set -c 190000. flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -c 190000 --cache-type-k q4_0 --cache-type-v q4_0 -ngl 99 --port 8080 -v VRAM: 23.8 GB Throughput: Prefill: 2206.66 t/s | Decode: 33 t/s (with 28k tokens prompt) (Note: Pushing it to 220k required dropping to -ngl 58 again, which immediately penalized decode down to 17 t/s). # The Tradeoff: For Max Reasoning: Keep your KV cache unquantized (f16). You get pristine reasoning but hit a strict 40k context ceiling. For Massive Document Retrieval: If you need to feed the model giant codebases, use --cache-type-k q4_0. Getting 190k context at 33 tokens/second on a consumer desktop with a 31b dense model is a cheat code. If you’re rocking a single 3090 or 4090 and slept on Gemma 4 earlier, this update is your cue to dust off the terminal. Hugging Face links to the Unsloth QAT quants are in the replies below.

Alok

76,069 次观看 • 2 个月前

THIS DEVELOPER CONNECTED 8 NVIDIA DGX SPARKS INTO ONE CLUSTER - AND RAN AN 800GB MODEL THAT MADE HIM 10X MORE PRODUCTIVE 21:47 he says it straight - "this is a terabyte of VRAM - we ran Quen 3.5, 800GB on disk, a model that doesn't even fit on a single Mac Studio - 24 tokens per second - I'd say that's a win" 8 Sparks connected through a $1,300 switch via RDMA over Ethernet - each node adding 128GB of memory into one unified pool of 1TB started with one Spark at 3 tokens per second - every added node doubled the speed - and eight together deliver 24 tokens on a model that physically cannot run anywhere else Kimi K2 at 600GB loaded in 15 minutes, 115GB per node, 13 tokens per second - a model that simply cannot run on anything smaller Claude helped configure the entire cluster - SSH mesh across all 8 machines, network config, jumbo frames, QSFP port speeds - all from one terminal most people rent cloud compute for models this size at $2,000+/month - he built the cluster once and now every token costs 20x less

THIS DEVELOPER CONNECTED 8 NVIDIA DGX SPARKS INTO ONE CLUSTER - AND RAN AN 800GB MODEL THAT MADE HIM 10X MORE PRODUCTIVE 21:47 he says it straight - "this is a terabyte of VRAM - we ran Quen 3.5, 800GB on disk, a model that doesn't even fit on a single Mac Studio - 24 tokens per second - I'd say that's a win" 8 Sparks connected through a $1,300 switch via RDMA over Ethernet - each node adding 128GB of memory into one unified pool of 1TB started with one Spark at 3 tokens per second - every added node doubled the speed - and eight together deliver 24 tokens on a model that physically cannot run anywhere else Kimi K2 at 600GB loaded in 15 minutes, 115GB per node, 13 tokens per second - a model that simply cannot run on anything smaller Claude helped configure the entire cluster - SSH mesh across all 8 machines, network config, jumbo frames, QSFP port speeds - all from one terminal most people rent cloud compute for models this size at $2,000+/month - he built the cluster once and now every token costs 20x less

Noisy

186,592 次观看 • 3 个月前

THE $4,000 DGX SPARK BENCHMARKS AT 342 T/S ON PROMPT PROCESSING - THEN DROPS TO 73 WHEN IT ACTUALLY GENERATES. that clip is a dgx spark running llama-bench on qwen3-coder 30b, side by side with a mac. two numbers decide everything here: pp512, prompt processing = 342 tokens/sec tg128, token generation = 73 tokens/sec that gap is the whole lesson. local ai isn't one speed, it's two. prompt processing is reading your input: your files, your long context. the spark is compute-heavy, so it eats that fast. token generation is writing the answer, and it's bound by memory bandwidth, where the spark is modest and unified-memory macs stay close. so "which box is faster" has no single winner. feeding huge context or running agents that read a lot? prompt processing wins, the spark shines. long chat replies, one token at a time? generation matters more, and a mac holds its own. no single benchmark number, no universal "fastest box", no upgrade that wins every workload. the uncomfortable part isn't that the spark is fast. it's that "tokens per second" was always two numbers, and the hype only ever quotes the flattering one. follow RetroChainer for the benchmark truth nobody screenshots, and bookmark this before you buy on one number.

THE $4,000 DGX SPARK BENCHMARKS AT 342 T/S ON PROMPT PROCESSING - THEN DROPS TO 73 WHEN IT ACTUALLY GENERATES. that clip is a dgx spark running llama-bench on qwen3-coder 30b, side by side with a mac. two numbers decide everything here: pp512, prompt processing = 342 tokens/sec tg128, token generation = 73 tokens/sec that gap is the whole lesson. local ai isn't one speed, it's two. prompt processing is reading your input: your files, your long context. the spark is compute-heavy, so it eats that fast. token generation is writing the answer, and it's bound by memory bandwidth, where the spark is modest and unified-memory macs stay close. so "which box is faster" has no single winner. feeding huge context or running agents that read a lot? prompt processing wins, the spark shines. long chat replies, one token at a time? generation matters more, and a mac holds its own. no single benchmark number, no universal "fastest box", no upgrade that wins every workload. the uncomfortable part isn't that the spark is fast. it's that "tokens per second" was always two numbers, and the hype only ever quotes the flattering one. follow RetroChainer for the benchmark truth nobody screenshots, and bookmark this before you buy on one number.

RetroChainer

42,451 次观看 • 1 个月前

THE CHINESE INSTALLED 6 AI AGENTS RUNNING 24/7 FOR $11 A MONTH. THE BOX UNDER ONE DESK REPLACED $1,200 IN CLOUD SUBSCRIPTIONS. The Minisforum MS-S1 Max ships with 128GB of unified memory shared between CPU and GPU. Qwen3-Coder 30B runs at 40 to 50 tokens per second. 6 agents run simultaneously without anyone watching the meter. An inbox sorter drafts replies every 15 minutes. A research monitor watches 30+ feeds and lands a digest by 7 AM, a job that costs $15 to $20 a day on cloud APIs. The same stack on cloud subscriptions ran $800 to $1,200 a month. The hardware costs $3,000 once. Break-even hits month 3 to 4. The real shift is behavior, not savings. A meeting prep agent never would have justified per-token cost. Running it free overnight made it obvious.

THE CHINESE INSTALLED 6 AI AGENTS RUNNING 24/7 FOR $11 A MONTH. THE BOX UNDER ONE DESK REPLACED $1,200 IN CLOUD SUBSCRIPTIONS. The Minisforum MS-S1 Max ships with 128GB of unified memory shared between CPU and GPU. Qwen3-Coder 30B runs at 40 to 50 tokens per second. 6 agents run simultaneously without anyone watching the meter. An inbox sorter drafts replies every 15 minutes. A research monitor watches 30+ feeds and lands a digest by 7 AM, a job that costs $15 to $20 a day on cloud APIs. The same stack on cloud subscriptions ran $800 to $1,200 a month. The hardware costs $3,000 once. Break-even hits month 3 to 4. The real shift is behavior, not savings. A meeting prep agent never would have justified per-token cost. Running it free overnight made it obvious.

Lummox

14,093 次观看 • 3 个月前

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

343,270 次观看 • 1 个月前

EVERYONE BUYING A $4,200 MAC STUDIO FOR LOCAL AI MISSED THAT TESLA V100s FROM DGX SERVERS ARE DUMPED ON EBAY FOR $250, AND ONE JUST RAN GPT-OSS 20B AT HOME a Tesla V100 SXM2 16GB shipped inside nvidia DGX-1 nodes at $10,000 per card in 2017, eight per chassis. those same cards now retire from cloud datacenters as H100s land, dumped on ebay at $200 to $300. drop a $50 SXM2 to PCIe adapter into a Z8 G4, slot the card with thermal pad, total bill $330. GPT-OSS 20B runs at 87°C and 180W the article calls the $700 used RTX 3090 the best memory per dollar. this V100 hack does the same job for one third the price with HBM2. the catch: HP Z620 fails on AVX2, install CUDA 11 not 12, flip above 4G decoding in BIOS the V100 pulls 250W and the blower screams. the mac mini from the article pulls 30W and runs silent. $300 versus $4,200 buys a lot of electricity and earplugs over four years

EVERYONE BUYING A $4,200 MAC STUDIO FOR LOCAL AI MISSED THAT TESLA V100s FROM DGX SERVERS ARE DUMPED ON EBAY FOR $250, AND ONE JUST RAN GPT-OSS 20B AT HOME a Tesla V100 SXM2 16GB shipped inside nvidia DGX-1 nodes at $10,000 per card in 2017, eight per chassis. those same cards now retire from cloud datacenters as H100s land, dumped on ebay at $200 to $300. drop a $50 SXM2 to PCIe adapter into a Z8 G4, slot the card with thermal pad, total bill $330. GPT-OSS 20B runs at 87°C and 180W the article calls the $700 used RTX 3090 the best memory per dollar. this V100 hack does the same job for one third the price with HBM2. the catch: HP Z620 fails on AVX2, install CUDA 11 not 12, flip above 4G decoding in BIOS the V100 pulls 250W and the blower screams. the mac mini from the article pulls 30W and runs silent. $300 versus $4,200 buys a lot of electricity and earplugs over four years

Shadow Nick

103,257 次观看 • 2 个月前

🤯 A localmaxxer hit ~381 tok/s on a SINGLE RTX 3090 with Qwen3.8-27B. This developer has turned a 24GB RTX 3090 into a monster Qwen inference box - w/some creativity. Four days ago 👉 ⚡ ~82 tok/s single-user Then 👉 ⚡ ~114 tok/s with optimized MTP ⚡ ~138 tok/s with DFlash2 + lookup drafting Now 👉 🔥 ~381 tok/s on ONE request How? The recipe combines ... 🧠 Qwen3.8-27B 🎮 1× RTX 3090 24GB @ 250W ⚙️ heavily optimized vLLM ⚡ DFlash2 speculative decoding 🔎 lookup-augmented drafting 📚 prefix caching 🧮 16-token verification blocks 💾 quantized KV / heads / activations DFlash2 normally proposes 7 tokens. The developer realized the verification block doesn't have to stop there. If Qwen is answering from a document already sitting in the prompt, the system can fill the remaining draft positions using tokens found directly in that context. 🎯 So the target model can verify 16 tokens at once. On a ~25K-token document reproduction task: Previous DFlash2 👉 ~260 tok/s Longer verification + context lookup 👉 🔥 ~382 tok/s Acceptance: 🤯 15 of 16 tokens per verification step That is where the crazy number comes from. ⚠️ On ordinary real-world chat prompts, the same setup is around ~133 tok/s Still extremely fast for a dense 27B model on an RTX 3090. The 381 tok/s mode shines when the answer largely comes from material already in context so these are best use cases 📚 RAG / document Q&A 💻 Coding assistants applying edits 📝 Quoting or rewriting documents 🔎 Extracting information from long prompts And another optimization 👉 With prefix caching, a second question against the same 25K-token document reportedly goes from: 🐌 22.4 sec TTFT → ⚡ 0.56 sec TTFT Because the model doesn't need to process the whole document again. 🎯 It's specifically a mode for RAG front ends and coding agents. Follow iamMess on Reddit or syv-ai on GitHub 🔗 Reddit: r/LocalLLaMA/comments/1vtup5s/ 🔗 GitHub: /syv-ai/qwen38-27b-rtx3090

🤯 A localmaxxer hit ~381 tok/s on a SINGLE RTX 3090 with Qwen3.8-27B. This developer has turned a 24GB RTX 3090 into a monster Qwen inference box - w/some creativity. Four days ago 👉 ⚡ ~82 tok/s single-user Then 👉 ⚡ ~114 tok/s with optimized MTP ⚡ ~138 tok/s with DFlash2 + lookup drafting Now 👉 🔥 ~381 tok/s on ONE request How? The recipe combines ... 🧠 Qwen3.8-27B 🎮 1× RTX 3090 24GB @ 250W ⚙️ heavily optimized vLLM ⚡ DFlash2 speculative decoding 🔎 lookup-augmented drafting 📚 prefix caching 🧮 16-token verification blocks 💾 quantized KV / heads / activations DFlash2 normally proposes 7 tokens. The developer realized the verification block doesn't have to stop there. If Qwen is answering from a document already sitting in the prompt, the system can fill the remaining draft positions using tokens found directly in that context. 🎯 So the target model can verify 16 tokens at once. On a ~25K-token document reproduction task: Previous DFlash2 👉 ~260 tok/s Longer verification + context lookup 👉 🔥 ~382 tok/s Acceptance: 🤯 15 of 16 tokens per verification step That is where the crazy number comes from. ⚠️ On ordinary real-world chat prompts, the same setup is around ~133 tok/s Still extremely fast for a dense 27B model on an RTX 3090. The 381 tok/s mode shines when the answer largely comes from material already in context so these are best use cases 📚 RAG / document Q&A 💻 Coding assistants applying edits 📝 Quoting or rewriting documents 🔎 Extracting information from long prompts And another optimization 👉 With prefix caching, a second question against the same 25K-token document reportedly goes from: 🐌 22.4 sec TTFT → ⚡ 0.56 sec TTFT Because the model doesn't need to process the whole document again. 🎯 It's specifically a mode for RAG front ends and coding agents. Follow iamMess on Reddit or syv-ai on GitHub 🔗 Reddit: r/LocalLLaMA/comments/1vtup5s/ 🔗 GitHub: /syv-ai/qwen38-27b-rtx3090

David Hendrickson

132,796 次观看 • 1 个月前

MiniMax M2.7 is 230B params. Can you actually run it at home? I tested Unsloth's UD-IQ3_XXS (80GB) on 4 different rigs: 🟠 4x RTX 4090 (96GB): 71.52 tok/s, TTFT 1045ms 🟢 4x RTX 5090 (128GB): 120.54 tok/s, TTFT 725ms 🟡 1x RTX PRO 6000 (96GB): 118.74 tok/s, TTFT 765ms 🟣 DGX Spark (128GB) — 24.41 tok/s, TTFT 741ms Backend: llama.cpp. Context: 32k. Max tokens: 4096. I went with IQ3_XXS because it's the biggest quant that fits in 96GB VRAM while still leaving safe headroom for 32k context. Same quant across all four rigs, fairest comparison I could run. Now look at rough peak GPU power draw: 🟠 4x4090 → 1,800W peak (450W × 4) 🟢 4x5090 → 2,300W peak (575W × 4) 🟡 RTX PRO 6000 → 600W peak 🟣 DGX Spark → 240W peak (whole system) The RTX PRO 6000 is the quiet winner. One card, 96GB, matching a 4x5090 rig at roughly a quarter of the power and zero multi-GPU headaches. Best tokens-per-watt by a wide margin. DGX Spark is slow on generation but pulls the least power of any rig here, around 240W for the whole system. Prefill-friendly, memory-rich, wall-socket-friendly. And yes, plenty of people cap their cards. Even then, 4x 4090 or 4x 5090 still pulls well over 1,200W from the GPUs alone.

MiniMax M2.7 is 230B params. Can you actually run it at home? I tested Unsloth's UD-IQ3_XXS (80GB) on 4 different rigs: 🟠 4x RTX 4090 (96GB): 71.52 tok/s, TTFT 1045ms 🟢 4x RTX 5090 (128GB): 120.54 tok/s, TTFT 725ms 🟡 1x RTX PRO 6000 (96GB): 118.74 tok/s, TTFT 765ms 🟣 DGX Spark (128GB) — 24.41 tok/s, TTFT 741ms Backend: llama.cpp. Context: 32k. Max tokens: 4096. I went with IQ3_XXS because it's the biggest quant that fits in 96GB VRAM while still leaving safe headroom for 32k context. Same quant across all four rigs, fairest comparison I could run. Now look at rough peak GPU power draw: 🟠 4x4090 → 1,800W peak (450W × 4) 🟢 4x5090 → 2,300W peak (575W × 4) 🟡 RTX PRO 6000 → 600W peak 🟣 DGX Spark → 240W peak (whole system) The RTX PRO 6000 is the quiet winner. One card, 96GB, matching a 4x5090 rig at roughly a quarter of the power and zero multi-GPU headaches. Best tokens-per-watt by a wide margin. DGX Spark is slow on generation but pulls the least power of any rig here, around 240W for the whole system. Prefill-friendly, memory-rich, wall-socket-friendly. And yes, plenty of people cap their cards. Even then, 4x 4090 or 4x 5090 still pulls well over 1,200W from the GPUs alone.

stevibe

193,127 次观看 • 5 个月前

ALIBABA GAVE QWEN AWAY FREE, ZUCKERBERG GAVE LLAMA AWAY FREE, AND JENSEN HUANG STILL CHARGES $300 FOR THE 12GB THAT RUNS THEM. THE $80 AMD BOARD HAS 16GB. HE IS TESTING BOTH THIS WEEK AGAINST $200/MO CLAUDE CODE MAX AND $200/MO CHATGPT PRO 00:19 he points at a dead gtx 680 he keeps on the shelf purely for decoration and says "i'm actually really gpu poor" $300 rtx 3060 with 12gb -> $80 bc250 with 16gb -> ubuntu server on both -> 8gb tier, 12 to 16 tier, 24gb ceiling on a four year old 3090 -> qwen against llama at every tier -> tokens per dollar published jensen shipped 12gb on the 3060 in 2021 and consumer vram has barely moved since, alibaba and meta gave the weights away for nothing, the models are free and the only thing left to buy is memory the honest part: he says it himself, this is apples and oranges, and there is not a single benchmark number in this video yet $80 amd bc250 with 16gb gddr6 + $300 rtx 3060 with 12gb + a four year old 3090 with 24gb + ubuntu server + ollama + qwen + llama watch and save it, then kill your $439 stack this weekend, kimi does the same work at $19 flat and the whole map is in the article below

ALIBABA GAVE QWEN AWAY FREE, ZUCKERBERG GAVE LLAMA AWAY FREE, AND JENSEN HUANG STILL CHARGES $300 FOR THE 12GB THAT RUNS THEM. THE $80 AMD BOARD HAS 16GB. HE IS TESTING BOTH THIS WEEK AGAINST $200/MO CLAUDE CODE MAX AND $200/MO CHATGPT PRO 00:19 he points at a dead gtx 680 he keeps on the shelf purely for decoration and says "i'm actually really gpu poor" $300 rtx 3060 with 12gb -> $80 bc250 with 16gb -> ubuntu server on both -> 8gb tier, 12 to 16 tier, 24gb ceiling on a four year old 3090 -> qwen against llama at every tier -> tokens per dollar published jensen shipped 12gb on the 3060 in 2021 and consumer vram has barely moved since, alibaba and meta gave the weights away for nothing, the models are free and the only thing left to buy is memory the honest part: he says it himself, this is apples and oranges, and there is not a single benchmark number in this video yet $80 amd bc250 with 16gb gddr6 + $300 rtx 3060 with 12gb + a four year old 3090 with 24gb + ubuntu server + ollama + qwen + llama watch and save it, then kill your $439 stack this weekend, kimi does the same work at $19 flat and the whole map is in the article below

Antid

13,974 次观看 • 1 个月前

STACKING 4 OLD NVIDIA GTX 1080s GIVES YOU LOCAL AI THAT BEATS $400/MO IN CLOUD SUBSCRIPTIONS people are pulling 4 used gtx 1080s off ebay for around $400 total and stacking them in one tower to build a local ai rig. that pools 32gb of vram across cards that originally retailed at $700 each old gaming cards became cheap when ethereum killed proof of work in 2022. miners dumped millions of them and developers picked them up to run open weight models locally ollama installs in one shell command and pulls a model in another. point claude code at localhost through one environment variable and the cli works identically without sending data to anthropic llama 3.2, mistral 7b, qwen 2.5 and phi 3 all run comfortably on a 1080 stack at 40 to 60 tokens per second. fast enough for real conversations, automation pipelines and document analysis a heavy ai user pays $200 chatgpt pro, $200 claude code and $20 cursor every month. that's $5,040 a year and a $400 rig pays itself off before month two bookmark this and read the article below

STACKING 4 OLD NVIDIA GTX 1080s GIVES YOU LOCAL AI THAT BEATS $400/MO IN CLOUD SUBSCRIPTIONS people are pulling 4 used gtx 1080s off ebay for around $400 total and stacking them in one tower to build a local ai rig. that pools 32gb of vram across cards that originally retailed at $700 each old gaming cards became cheap when ethereum killed proof of work in 2022. miners dumped millions of them and developers picked them up to run open weight models locally ollama installs in one shell command and pulls a model in another. point claude code at localhost through one environment variable and the cli works identically without sending data to anthropic llama 3.2, mistral 7b, qwen 2.5 and phi 3 all run comfortably on a 1080 stack at 40 to 60 tokens per second. fast enough for real conversations, automation pipelines and document analysis a heavy ai user pays $200 chatgpt pro, $200 claude code and $20 cursor every month. that's $5,040 a year and a $400 rig pays itself off before month two bookmark this and read the article below

starmex

371,235 次观看 • 3 个月前

A FARM OF 90 RTX 3090s BUILT FOR ETHEREUM MINING IN 2021 IS NOW SERVING LOCAL LLM INFERENCE FROM THE SAME GARAGE. THE PART THAT MATTERS ISN'T THE SIZE, IT'S THAT THE OWNER NEVER UNPLUGGED THE RIG, HE JUST CHANGED THE WORKLOAD the camera walks through the room. three custom wood frames stacked floor to ceiling. ninety 3090s breathing rainbow LEDs in unison. a yellow industrial duct ripped through the wall to pull the heat out. each card pulling 250W under load, the whole room drawing 22kW continuous. this is not a hobby setup. it is a small datacenter dressed up as a bedroom do the math. 90 cards times 24GB VRAM is 2,160GB of pooled memory, enough to load every frontier open weight model at once. at $700 used per card, each 3090 pays for itself against a Claude Code Max subscription in about three months. the same RTX 3090 i wrote about last week as the best value per dollar for a local AI rig, stacked 90 times in a garage this is not a crypto relic. it is the new datacenter, built from cards that gamers retired in 2022 and crypto miners gave up on in 2023, now quietly answering prompts that OpenAI charges you $200 a month for

A FARM OF 90 RTX 3090s BUILT FOR ETHEREUM MINING IN 2021 IS NOW SERVING LOCAL LLM INFERENCE FROM THE SAME GARAGE. THE PART THAT MATTERS ISN'T THE SIZE, IT'S THAT THE OWNER NEVER UNPLUGGED THE RIG, HE JUST CHANGED THE WORKLOAD the camera walks through the room. three custom wood frames stacked floor to ceiling. ninety 3090s breathing rainbow LEDs in unison. a yellow industrial duct ripped through the wall to pull the heat out. each card pulling 250W under load, the whole room drawing 22kW continuous. this is not a hobby setup. it is a small datacenter dressed up as a bedroom do the math. 90 cards times 24GB VRAM is 2,160GB of pooled memory, enough to load every frontier open weight model at once. at $700 used per card, each 3090 pays for itself against a Claude Code Max subscription in about three months. the same RTX 3090 i wrote about last week as the best value per dollar for a local AI rig, stacked 90 times in a garage this is not a crypto relic. it is the new datacenter, built from cards that gamers retired in 2022 and crypto miners gave up on in 2023, now quietly answering prompts that OpenAI charges you $200 a month for

Ridark

64,499 次观看 • 2 个月前