Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

THIS DEVELOPER CONNECTED 8 NVIDIA DGX SPARKS INTO ONE CLUSTER - AND RAN AN 800GB MODEL THAT MADE HIM 10X MORE PRODUCTIVE 21:47 he says it straight - "this is a terabyte of VRAM - we ran Quen 3.5, 800GB on disk, a model that doesn't even fit on...

noisyb0y1's profile picture

Noisy

24,078 subscribers

185,632 Aufrufe • vor 2 Monaten •via X (Twitter)

Wissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

THIS DEVELOPER JUST RAN A TRILLION PARAMETER MODEL ON 4 MAC STUDIOS - 10X FASTER AND 5X CHEAPER THAN CLOUD CODE 19:00 he says it out loud. "we just ran a trillion parameter model. 30 something tokens per second. wow." RDMA over Thunderbolt made the cluster 10x faster than before. tensor parallelism splits the model between machines in parallel instead of sequentially DeepSeek V3 full size without quantization. 183GB per machine. still 25 tokens per second. faster than most people can read Kimi K2 with trillion parameters and 256,000 context window. loaded, responded and adapted memory usage dynamically based on prompt size 4 machines consuming 66 watts total. cloud GPU for the same workload costs $1,900/month a cluster that used to cost millions now assembled from Mac Studios in a few hours
22:35

THIS DEVELOPER JUST RAN A TRILLION PARAMETER MODEL ON 4 MAC STUDIOS - 10X FASTER AND 5X CHEAPER THAN CLOUD CODE 19:00 he says it out loud. "we just ran a trillion parameter model. 30 something tokens per second. wow." RDMA over Thunderbolt made the cluster 10x faster than before. tensor parallelism splits the model between machines in parallel instead of sequentially DeepSeek V3 full size without quantization. 183GB per machine. still 25 tokens per second. faster than most people can read Kimi K2 with trillion parameters and 256,000 context window. loaded, responded and adapted memory usage dynamically based on prompt size 4 machines consuming 66 watts total. cloud GPU for the same workload costs $1,900/month a cluster that used to cost millions now assembled from Mac Studios in a few hours

Noisy

79,665 Aufrufe • vor 2 Monaten

THIS CLUSTER OF 4 MAC STUDIOS USES LESS POWER THAN ONE NVIDIA CHIP - AND COSTS 5X LESS 17:30 the number hits hard - 600 watts for all four Mac Studios - less than one H200 that costs $30,000 on its own instead of one $30,000 GPU you get 4 machines with 1.5TB of memory, 3x more performance and models that simply cannot run on a single chip Apple quietly added RDMA over Thunderbolt in a beta update - latency dropped 99% and the same cluster became 3x faster without buying any new hardware before the update: 5 tokens per second. after: 15.5 - a free upgrade through a software update 8 Nvidia DGX Sparks for the same memory would cost $32,000 - this cluster costs a fraction and gets faster with every update 10x more productive at half the price - and the only thing you need is knowing where to look
20:03

THIS CLUSTER OF 4 MAC STUDIOS USES LESS POWER THAN ONE NVIDIA CHIP - AND COSTS 5X LESS 17:30 the number hits hard - 600 watts for all four Mac Studios - less than one H200 that costs $30,000 on its own instead of one $30,000 GPU you get 4 machines with 1.5TB of memory, 3x more performance and models that simply cannot run on a single chip Apple quietly added RDMA over Thunderbolt in a beta update - latency dropped 99% and the same cluster became 3x faster without buying any new hardware before the update: 5 tokens per second. after: 15.5 - a free upgrade through a software update 8 Nvidia DGX Sparks for the same memory would cost $32,000 - this cluster costs a fraction and gets faster with every update 10x more productive at half the price - and the only thing you need is knowing where to look

Noisy

105,595 Aufrufe • vor 2 Monaten

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓
1:21

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

slash1s

101,317 Aufrufe • vor 2 Monaten

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables
21:59

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables

Noisy

48,615 Aufrufe • vor 2 Monaten

THIS DEVELOPER RAN THE LARGEST AI MODEL IN THE WORLD ON 5 MAC STUDIOS - AND IT COST 100X LESS THAN WHAT OPENAI USES 27:47 he says it after hours of setup - Llama 3.1 405B running locally on five Mac Studios - a model that normally requires 42 GPUs at $1,600 each or an entire data center 320GB unified memory combined - every Mac Studio adds its 64GB into one shared pool - and a model that needs 1TB of VRAM just distributes across all five it launched, it responded and not a single token went to OpenAI or Google - everything local, everything private, everything yours and 100x cheaper 5 Mac Studios, 46 watts for all of them in idle - less than one light bulb per machine what OpenAI keeps behind a subscription he ran at home with a one-time purchase in a few clicks
34:56

THIS DEVELOPER RAN THE LARGEST AI MODEL IN THE WORLD ON 5 MAC STUDIOS - AND IT COST 100X LESS THAN WHAT OPENAI USES 27:47 he says it after hours of setup - Llama 3.1 405B running locally on five Mac Studios - a model that normally requires 42 GPUs at $1,600 each or an entire data center 320GB unified memory combined - every Mac Studio adds its 64GB into one shared pool - and a model that needs 1TB of VRAM just distributes across all five it launched, it responded and not a single token went to OpenAI or Google - everything local, everything private, everything yours and 100x cheaper 5 Mac Studios, 46 watts for all of them in idle - less than one light bulb per machine what OpenAI keeps behind a subscription he ran at home with a one-time purchase in a few clicks

Noisy

21,553 Aufrufe • vor 2 Monaten

CHINESE AI TEACHER STACKED 4 NVIDIA DGX SPARK CHIPS INTO A CLUSTER - AND NOW RUNS A BUSINESS ON IT FROM HIS DESK 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else he's running a Qwen 32 35B model with 4 concurrent requests at the same time and the speed barely drops compared to a single request what used to require a $10,000+/month cloud setup now sits on his desk and costs him $40-60/month in electricity 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else the people still paying per token are financing someone else's infrastructure
1:42

CHINESE AI TEACHER STACKED 4 NVIDIA DGX SPARK CHIPS INTO A CLUSTER - AND NOW RUNS A BUSINESS ON IT FROM HIS DESK 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else he's running a Qwen 32 35B model with 4 concurrent requests at the same time and the speed barely drops compared to a single request what used to require a $10,000+/month cloud setup now sits on his desk and costs him $40-60/month in electricity 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else the people still paying per token are financing someone else's infrastructure

Sprytix

51,010 Aufrufe • vor 2 Monaten

FOUR DIFFERENT VENDORS ARE NOW SHIPPING GB10 MINI PCs WITH 128GB UNIFIED MEMORY, AND ONE MICROTIK CRS 804 SWITCH CAN CONNECT UP TO EIGHT OF THEM INTO A 1 TERABYTE LOCAL AI CLUSTER 00:00 he points at the MikroTik CRS 804, "you need some kind of switch that'll handle QSFP56 ports like these", the interconnect that makes the whole cluster possible the GB10 ecosystem is no longer just Nvidia. Dell Pro Max GB10, ASUS Ascent GX10, and MSI Edge Expert all ship the same Grace Blackwell Superchip with 128GB of coherent memory. same silicon, different cases, same 200 gigabit ports on the back the CRS 804 is what connects them at prosumer prices. four 400 gigabit QSFP56 ports on one 1U chassis, breakout cables that split each port into two 200 gigabit lanes. one switch drives eight GB10 units in parallel do the math. eight nodes at 128GB each equals 1024GB of pooled unified memory across the cluster. run vLLM, shard a frontier model across all eight, and inference happens locally on hardware that fits in half a rack the real limiter revealed in the stress test was never throttling. it was interconnect topology, exactly the layer this switch fixes at a fraction of enterprise switch pricing $400 a month for combined chatgpt pro and claude code max hits $4,800 a year per developer. a small team of five running through this cluster pays back inside eight months and never expires the article covers the buying ladder for a single desk. this post is proof of the cluster ladder that starts where the desk one ends save this before the GB10 lineup grows past four vendors and prosumer cluster switches move upmarket
1:05

FOUR DIFFERENT VENDORS ARE NOW SHIPPING GB10 MINI PCs WITH 128GB UNIFIED MEMORY, AND ONE MICROTIK CRS 804 SWITCH CAN CONNECT UP TO EIGHT OF THEM INTO A 1 TERABYTE LOCAL AI CLUSTER 00:00 he points at the MikroTik CRS 804, "you need some kind of switch that'll handle QSFP56 ports like these", the interconnect that makes the whole cluster possible the GB10 ecosystem is no longer just Nvidia. Dell Pro Max GB10, ASUS Ascent GX10, and MSI Edge Expert all ship the same Grace Blackwell Superchip with 128GB of coherent memory. same silicon, different cases, same 200 gigabit ports on the back the CRS 804 is what connects them at prosumer prices. four 400 gigabit QSFP56 ports on one 1U chassis, breakout cables that split each port into two 200 gigabit lanes. one switch drives eight GB10 units in parallel do the math. eight nodes at 128GB each equals 1024GB of pooled unified memory across the cluster. run vLLM, shard a frontier model across all eight, and inference happens locally on hardware that fits in half a rack the real limiter revealed in the stress test was never throttling. it was interconnect topology, exactly the layer this switch fixes at a fraction of enterprise switch pricing $400 a month for combined chatgpt pro and claude code max hits $4,800 a year per developer. a small team of five running through this cluster pays back inside eight months and never expires the article covers the buying ladder for a single desk. this post is proof of the cluster ladder that starts where the desk one ends save this before the GB10 lineup grows past four vendors and prosumer cluster switches move upmarket

NO1ennn

59,725 Aufrufe • vor 1 Monat

8 DGX Sparks on a desk just ran a frontier model at 400K context. A friend who owns the inference budget at a Berlin AI startup, $52K a month in API bills, sent me this at 2 a.m. with one line: "we're buying the hardware." It starts as a disaster. A mountain of cables burying $32K of NVIDIA silicon, nothing talking to anything. 54 minutes later the whole stack answers as 1 machine. 1TB of unified memory. GLM-5.2 loaded across all 8 nodes. The part that made him rewatch twice sits around the 25-minute mark, where 1 networking fix turns 8 boxes into a cluster. Most people quit exactly there. He paused it 4 times. Screenshotted the interface setup. Forwarded it to his CTO without a caption. The cloud bill was never the moat. The cables were.
54:24

8 DGX Sparks on a desk just ran a frontier model at 400K context. A friend who owns the inference budget at a Berlin AI startup, $52K a month in API bills, sent me this at 2 a.m. with one line: "we're buying the hardware." It starts as a disaster. A mountain of cables burying $32K of NVIDIA silicon, nothing talking to anything. 54 minutes later the whole stack answers as 1 machine. 1TB of unified memory. GLM-5.2 loaded across all 8 nodes. The part that made him rewatch twice sits around the 25-minute mark, where 1 networking fix turns 8 boxes into a cluster. Most people quit exactly there. He paused it 4 times. Screenshotted the interface setup. Forwarded it to his CTO without a caption. The cloud bill was never the moat. The cables were.

the lich

56,893 Aufrufe • vor 1 Monat

THE CHINESE INSTALLED 6 AI AGENTS RUNNING 24/7 FOR $11 A MONTH. THE BOX UNDER ONE DESK REPLACED $1,200 IN CLOUD SUBSCRIPTIONS. The Minisforum MS-S1 Max ships with 128GB of unified memory shared between CPU and GPU. Qwen3-Coder 30B runs at 40 to 50 tokens per second. 6 agents run simultaneously without anyone watching the meter. An inbox sorter drafts replies every 15 minutes. A research monitor watches 30+ feeds and lands a digest by 7 AM, a job that costs $15 to $20 a day on cloud APIs. The same stack on cloud subscriptions ran $800 to $1,200 a month. The hardware costs $3,000 once. Break-even hits month 3 to 4. The real shift is behavior, not savings. A meeting prep agent never would have justified per-token cost. Running it free overnight made it obvious.
0:53

THE CHINESE INSTALLED 6 AI AGENTS RUNNING 24/7 FOR $11 A MONTH. THE BOX UNDER ONE DESK REPLACED $1,200 IN CLOUD SUBSCRIPTIONS. The Minisforum MS-S1 Max ships with 128GB of unified memory shared between CPU and GPU. Qwen3-Coder 30B runs at 40 to 50 tokens per second. 6 agents run simultaneously without anyone watching the meter. An inbox sorter drafts replies every 15 minutes. A research monitor watches 30+ feeds and lands a digest by 7 AM, a job that costs $15 to $20 a day on cloud APIs. The same stack on cloud subscriptions ran $800 to $1,200 a month. The hardware costs $3,000 once. Break-even hits month 3 to 4. The real shift is behavior, not savings. A meeting prep agent never would have justified per-token cost. Running it free overnight made it obvious.

Lummox

14,093 Aufrufe • vor 2 Monaten

ONE BUILDER WIRED 4 RASPBERRY PI MODULES INTO A €450 AI CLUSTER WITH 64GB OF RAM AND KILLED HIS $200/MONTH CHATGPT SUBSCRIPTION 00:37 he points at the terminal, "loaded, loaded, loaded - 4 nodes synced over gigabit, 64 gigabytes total, llama 3.1 partitioned across the cluster" he stacked 4 raspberry pi CM5 lite modules with 16 gigabytes each onto a sipeed nanocluster board, the whole rig runs off a single 65 watt power supply and a gigabit internal switch distributed llama partitions the model across the nodes with synchronized workloads handled over the internal fabric, no quantization needed, the cluster hits about 30 tokens per second on small models a heavy developer pays $200 a month for chatgpt pro and another $200 for claude code, this rig cost €450 in parts and breaks even on the stack in month 2 most people will keep paying anthropic and openai forever, a few will spend a weekend wiring 4 raspberry pi modules and never see a subscription invoice again the window is open, follow and bookmark before it closes
0:37

ONE BUILDER WIRED 4 RASPBERRY PI MODULES INTO A €450 AI CLUSTER WITH 64GB OF RAM AND KILLED HIS $200/MONTH CHATGPT SUBSCRIPTION 00:37 he points at the terminal, "loaded, loaded, loaded - 4 nodes synced over gigabit, 64 gigabytes total, llama 3.1 partitioned across the cluster" he stacked 4 raspberry pi CM5 lite modules with 16 gigabytes each onto a sipeed nanocluster board, the whole rig runs off a single 65 watt power supply and a gigabit internal switch distributed llama partitions the model across the nodes with synchronized workloads handled over the internal fabric, no quantization needed, the cluster hits about 30 tokens per second on small models a heavy developer pays $200 a month for chatgpt pro and another $200 for claude code, this rig cost €450 in parts and breaks even on the stack in month 2 most people will keep paying anthropic and openai forever, a few will spend a weekend wiring 4 raspberry pi modules and never see a subscription invoice again the window is open, follow and bookmark before it closes

Antid

64,291 Aufrufe • vor 2 Monaten

This guy plugged a DGX Spark (the $3K Nvidia box) and a Mac Mini M4 together to run AI and what happened next surprised everyone > the Nvidia box handles the hard part - processing your prompt in milliseconds > Mac Mini M4 handles the fast part - generating the response at memory bandwidth speeds nothing else can match > together they hit 84 tokens per second on Llama - 6x faster than the Spark alone > running compute agents locally on this setup means your data never leaves your hardware > two boxes. two different architectures. one AI system that Deepseek runs at data center scale he ran it on his desk save this. the way we build local AI is about to change
18:41

This guy plugged a DGX Spark (the $3K Nvidia box) and a Mac Mini M4 together to run AI and what happened next surprised everyone > the Nvidia box handles the hard part - processing your prompt in milliseconds > Mac Mini M4 handles the fast part - generating the response at memory bandwidth speeds nothing else can match > together they hit 84 tokens per second on Llama - 6x faster than the Spark alone > running compute agents locally on this setup means your data never leaves your hardware > two boxes. two different architectures. one AI system that Deepseek runs at data center scale he ran it on his desk save this. the way we build local AI is about to change

Mr. Buzzoni

153,768 Aufrufe • vor 2 Monaten

This Chinese developer linked two $2,999 NVIDIA DGX Sparks into one box and runs the full Qwen3-235B at home, after dropping his $1,999-a-month cloud bill to zero. He wired 2 small boxes into a single computer, split a giant 235-billion-parameter model in half between them, and serves it across his own network at about 10 tokens a second, with no internet, no cloud, right there on the desk. No data center, no thousand-dollar graphics cards, no monthly cloud bill. Just him, 2 gold boxes the size of a sandwich, one cable between them, and 1 power strip. And here is the whole payoff. He used to pay the cloud $1,999 a month for the same model, and the meter ticked on every request. Now he paid $5,998 once for 2 boxes, they covered their cost in 3 months, and after that he sends as many requests as he wants for free, only electricity. The two Sparks talk over one fast cable, each holds 128GB of memory, and together they carry the whole model, about 73GB loaded per box, with the chip inside pinned near the limit at 96%. Both boxes work as one and keep trading data over the cable, with no cloud in the loop and no single word leaking out. The ready model sits on one local address, and any app on his network calls it as easily as ChatGPT. And here is how he described, in plain words, what this pair of boxes does: "this is a pair of boxes that holds the huge Qwen3-235B model and serves it to one network. the model is split in half, and each box owns its half. parts: // Box 1 (holds the first half of the model and starts the answer fast, the first word appears in under a second) // Box 2 (holds the second half and writes out the rest, about 10 tokens a second) // Cable (connects the 2 boxes and moves data between them on every step, with no lag) // Address (one local address where any app sends its request, like to a cloud model) // Test (a script that runs big prompts through and measures speed and delays) // Monitor (checks temperature, power draw, and load on both boxes every 2 seconds). the model never goes to the cloud. he only steps in when a box runs hotter than 80 degrees or the cable between them starts dropping data." So the system knows exactly what it is, what it is for, and where its limits are. It knows it has to hold the whole huge model across 2 boxes on its own. It knows it has to answer every request locally, with no meter, no limits, and no internet. It knows the human is only needed when a box overheats or the link between them stalls. → The setup runs around the clock on 2 boxes, each pulling under 60 watts → However many requests he sends, the monthly bill is $0, only electricity → The first box starts the answer in under a second → The second writes text at about 10 tokens a second → One request at a time: 838 tokens in 85 seconds, first word in 0.8s → Two requests at once: 697 tokens in 108 seconds, first word in 0.7s → Both boxes sit at 96% load and warm up to 76-78 degrees And only when a chip in a box runs hotter than 80 degrees or the cable between the 2 Sparks drops data does the system call the owner. And when he himself is out on a run or in a coffee shop, he still reaches his own model at home from his phone: sends a big prompt to the local Qwen3-235B, gets the full answer back in under a minute and a half, with no token meter ticking and no limit to hit. Here is what the test shows on his screen during one of the night runs: "one request at a time: 838 tokens in 84.9 seconds, first word in 0.8s, then 0.1s per token." "two requests at once: 697 tokens in 107.6 seconds, first word in 0.7s, then 0.15s per token." "Box 1: chip at 96% load, 76 degrees, 56 watts, 73GB used in memory." "Box 2: chip at 96% load, 78 degrees, 56 watts, the Qwen3-235B model fully loaded." And while everyone around is paying for AI by the month and bumping into limits, his top-tier model just sits on the desk and works as much as he wants: his own little power plant instead of a forever meter. He has no server rack of his own and no cloud account behind it. Just 2 DGX Spark boxes on a desk, one model split in half between them, one local address, and a folder of prompts next to it. Out of everything I have seen this year, this is the cleanest way to stop paying for AI: $5,998 of hardware on the desk once, $0 a month to the cloud, unlimited forever, and between them 2 gold boxes, 1 cable, and the full Qwen3-235B answering at home with no internet.
1:16

This Chinese developer linked two $2,999 NVIDIA DGX Sparks into one box and runs the full Qwen3-235B at home, after dropping his $1,999-a-month cloud bill to zero. He wired 2 small boxes into a single computer, split a giant 235-billion-parameter model in half between them, and serves it across his own network at about 10 tokens a second, with no internet, no cloud, right there on the desk. No data center, no thousand-dollar graphics cards, no monthly cloud bill. Just him, 2 gold boxes the size of a sandwich, one cable between them, and 1 power strip. And here is the whole payoff. He used to pay the cloud $1,999 a month for the same model, and the meter ticked on every request. Now he paid $5,998 once for 2 boxes, they covered their cost in 3 months, and after that he sends as many requests as he wants for free, only electricity. The two Sparks talk over one fast cable, each holds 128GB of memory, and together they carry the whole model, about 73GB loaded per box, with the chip inside pinned near the limit at 96%. Both boxes work as one and keep trading data over the cable, with no cloud in the loop and no single word leaking out. The ready model sits on one local address, and any app on his network calls it as easily as ChatGPT. And here is how he described, in plain words, what this pair of boxes does: "this is a pair of boxes that holds the huge Qwen3-235B model and serves it to one network. the model is split in half, and each box owns its half. parts: // Box 1 (holds the first half of the model and starts the answer fast, the first word appears in under a second) // Box 2 (holds the second half and writes out the rest, about 10 tokens a second) // Cable (connects the 2 boxes and moves data between them on every step, with no lag) // Address (one local address where any app sends its request, like to a cloud model) // Test (a script that runs big prompts through and measures speed and delays) // Monitor (checks temperature, power draw, and load on both boxes every 2 seconds). the model never goes to the cloud. he only steps in when a box runs hotter than 80 degrees or the cable between them starts dropping data." So the system knows exactly what it is, what it is for, and where its limits are. It knows it has to hold the whole huge model across 2 boxes on its own. It knows it has to answer every request locally, with no meter, no limits, and no internet. It knows the human is only needed when a box overheats or the link between them stalls. → The setup runs around the clock on 2 boxes, each pulling under 60 watts → However many requests he sends, the monthly bill is $0, only electricity → The first box starts the answer in under a second → The second writes text at about 10 tokens a second → One request at a time: 838 tokens in 85 seconds, first word in 0.8s → Two requests at once: 697 tokens in 108 seconds, first word in 0.7s → Both boxes sit at 96% load and warm up to 76-78 degrees And only when a chip in a box runs hotter than 80 degrees or the cable between the 2 Sparks drops data does the system call the owner. And when he himself is out on a run or in a coffee shop, he still reaches his own model at home from his phone: sends a big prompt to the local Qwen3-235B, gets the full answer back in under a minute and a half, with no token meter ticking and no limit to hit. Here is what the test shows on his screen during one of the night runs: "one request at a time: 838 tokens in 84.9 seconds, first word in 0.8s, then 0.1s per token." "two requests at once: 697 tokens in 107.6 seconds, first word in 0.7s, then 0.15s per token." "Box 1: chip at 96% load, 76 degrees, 56 watts, 73GB used in memory." "Box 2: chip at 96% load, 78 degrees, 56 watts, the Qwen3-235B model fully loaded." And while everyone around is paying for AI by the month and bumping into limits, his top-tier model just sits on the desk and works as much as he wants: his own little power plant instead of a forever meter. He has no server rack of his own and no cloud account behind it. Just 2 DGX Spark boxes on a desk, one model split in half between them, one local address, and a folder of prompts next to it. Out of everything I have seen this year, this is the cleanest way to stop paying for AI: $5,998 of hardware on the desk once, $0 a month to the cloud, unlimited forever, and between them 2 gold boxes, 1 cable, and the full Qwen3-235B answering at home with no internet.

Blaze

93,871 Aufrufe • vor 2 Monaten

AMD'S NEW MINI PC RUNS CLAUDE CODE OFFLINE FOR $0/MONTH AND REPLACES $200 OPENAI + $200 CLAUDE CODE SUBSCRIPTIONS amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on a single piece of silicon with 128gb of unified memory the gmktec evo-x2 runs qwen3 coder 30b at 40 to 50 tokens per second and llama 3.3 70b at 20 to 25 tokens per second locally. on linux you get 110gb of usable vram out of 128gb setup takes 90 minutes. wipe windows for linux, install ollama with one command, pull the model, point claude code at localhost. same interface, same agent loop, nothing leaves your network a heavy ai user pays $200 for claude code max, $200 for chatgpt pro and $20 for cursor. that's $5,040 a year and the box pays itself off in 5 months after break-even every month is $420 that used to leave your account. three year math is roughly $13,000 not paid to ai companies plus zero rate limits on overnight agents bookmark this and read the article below
0:52

AMD'S NEW MINI PC RUNS CLAUDE CODE OFFLINE FOR $0/MONTH AND REPLACES $200 OPENAI + $200 CLAUDE CODE SUBSCRIPTIONS amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on a single piece of silicon with 128gb of unified memory the gmktec evo-x2 runs qwen3 coder 30b at 40 to 50 tokens per second and llama 3.3 70b at 20 to 25 tokens per second locally. on linux you get 110gb of usable vram out of 128gb setup takes 90 minutes. wipe windows for linux, install ollama with one command, pull the model, point claude code at localhost. same interface, same agent loop, nothing leaves your network a heavy ai user pays $200 for claude code max, $200 for chatgpt pro and $20 for cursor. that's $5,040 a year and the box pays itself off in 5 months after break-even every month is $420 that used to leave your account. three year math is roughly $13,000 not paid to ai companies plus zero rate limits on overnight agents bookmark this and read the article below

starmex

33,516 Aufrufe • vor 2 Monaten

INTEL JUST SHIPPED A WORKSTATION CARD WITH TWO GPUs ON ONE PCB AND 48GB OF VRAM. FOUR CARDS GIVE A SINGLE MOTHERBOARD 192GB OF POOLED INFERENCE MEMORY FOR THE PRICE OF TWO RTX 5090s 00:14 he holds one card up to the camera, two GPU dies side by side under the cooler, each one running its own x8 PCIe lane back to the chipset the cluster sees eight discrete accelerators in software. intel's Battle Matrix stack shards a model across all eight, so a 235B parameter network loads in slices and answers requests in parallel what 192GB of VRAM unlocks: an entire 200B class model in memory without quantization. a vision agent reading 100 invoices at once. a research box that holds three frontier models loaded simultaneously, switching between them in under a second intel is the slow side of inference. nvidia is faster per token, that is the honest tradeoff. but the only other path to this much VRAM is a $40,000 nvidia rack or three networked Mac Studios four B60 cards plus the chassis lands under $5,000. power draw averages 800 watts, $55 a month in electricity. one engineer paying $400 a month for combined ChatGPT Pro and Claude Code Max pays the hardware off in less than a year
1:16

INTEL JUST SHIPPED A WORKSTATION CARD WITH TWO GPUs ON ONE PCB AND 48GB OF VRAM. FOUR CARDS GIVE A SINGLE MOTHERBOARD 192GB OF POOLED INFERENCE MEMORY FOR THE PRICE OF TWO RTX 5090s 00:14 he holds one card up to the camera, two GPU dies side by side under the cooler, each one running its own x8 PCIe lane back to the chipset the cluster sees eight discrete accelerators in software. intel's Battle Matrix stack shards a model across all eight, so a 235B parameter network loads in slices and answers requests in parallel what 192GB of VRAM unlocks: an entire 200B class model in memory without quantization. a vision agent reading 100 invoices at once. a research box that holds three frontier models loaded simultaneously, switching between them in under a second intel is the slow side of inference. nvidia is faster per token, that is the honest tradeoff. but the only other path to this much VRAM is a $40,000 nvidia rack or three networked Mac Studios four B60 cards plus the chassis lands under $5,000. power draw averages 800 watts, $55 a month in electricity. one engineer paying $400 a month for combined ChatGPT Pro and Claude Code Max pays the hardware off in less than a year

NO1ennn

48,054 Aufrufe • vor 1 Monat

Qwen 3.6 35B running at over 100 tokens per second on my $5,399 MacBook Pro M5 Max. This is the best local AI model I have ever run. 128GB of unified memory. No cloud. No API costs. No rate limits. Just raw local inference at speeds I didn't think were possible on a laptop. This model is more intelligent than GPT 5 on benchmarks. Running locally. On a MacBook. For free after the hardware cost. I said local AI would never compete with frontier. I'm starting to rethink that. The gap is closing faster than anyone expected.
1:04

Qwen 3.6 35B running at over 100 tokens per second on my $5,399 MacBook Pro M5 Max. This is the best local AI model I have ever run. 128GB of unified memory. No cloud. No API costs. No rate limits. Just raw local inference at speeds I didn't think were possible on a laptop. This model is more intelligent than GPT 5 on benchmarks. Running locally. On a MacBook. For free after the hardware cost. I said local AI would never compete with frontier. I'm starting to rethink that. The gap is closing faster than anyone expected.

BridgeMind

62,800 Aufrufe • vor 3 Monaten

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.
0:42

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

301,324 Aufrufe • vor 2 Tagen

THE $4,000 DGX SPARK BENCHMARKS AT 342 T/S ON PROMPT PROCESSING - THEN DROPS TO 73 WHEN IT ACTUALLY GENERATES. that clip is a dgx spark running llama-bench on qwen3-coder 30b, side by side with a mac. two numbers decide everything here: pp512, prompt processing = 342 tokens/sec tg128, token generation = 73 tokens/sec that gap is the whole lesson. local ai isn't one speed, it's two. prompt processing is reading your input: your files, your long context. the spark is compute-heavy, so it eats that fast. token generation is writing the answer, and it's bound by memory bandwidth, where the spark is modest and unified-memory macs stay close. so "which box is faster" has no single winner. feeding huge context or running agents that read a lot? prompt processing wins, the spark shines. long chat replies, one token at a time? generation matters more, and a mac holds its own. no single benchmark number, no universal "fastest box", no upgrade that wins every workload. the uncomfortable part isn't that the spark is fast. it's that "tokens per second" was always two numbers, and the hype only ever quotes the flattering one. follow RetroChainer for the benchmark truth nobody screenshots, and bookmark this before you buy on one number.
1:14

THE $4,000 DGX SPARK BENCHMARKS AT 342 T/S ON PROMPT PROCESSING - THEN DROPS TO 73 WHEN IT ACTUALLY GENERATES. that clip is a dgx spark running llama-bench on qwen3-coder 30b, side by side with a mac. two numbers decide everything here: pp512, prompt processing = 342 tokens/sec tg128, token generation = 73 tokens/sec that gap is the whole lesson. local ai isn't one speed, it's two. prompt processing is reading your input: your files, your long context. the spark is compute-heavy, so it eats that fast. token generation is writing the answer, and it's bound by memory bandwidth, where the spark is modest and unified-memory macs stay close. so "which box is faster" has no single winner. feeding huge context or running agents that read a lot? prompt processing wins, the spark shines. long chat replies, one token at a time? generation matters more, and a mac holds its own. no single benchmark number, no universal "fastest box", no upgrade that wins every workload. the uncomfortable part isn't that the spark is fast. it's that "tokens per second" was always two numbers, and the hype only ever quotes the flattering one. follow RetroChainer for the benchmark truth nobody screenshots, and bookmark this before you buy on one number.

RetroChainer

42,451 Aufrufe • vor 16 Tagen

Anthropic shipped 125 settings for Claude The official docs cover 40 One developer found the other 85 and his API bill dropped from $340 to $87 - not by using a cheaper model - not by writing shorter prompts just by moving one line in a config file to the right place > memory scoped per project → past clients never bleed into new work > Extended Thinking on Light by default → 18–25% fewer Opus tokens in week one > cache_control moved to the right line → the fix that turned a $340 bill into $87 > plugins and MCP servers toggled off when idle → saved 25–40K tokens per session > per-project model override → Haiku for docs, Sonnet for infra, Opus only where it matters most Claude users are running a $100/month tool at 30% of its actual capability
0:49

Anthropic shipped 125 settings for Claude The official docs cover 40 One developer found the other 85 and his API bill dropped from $340 to $87 - not by using a cheaper model - not by writing shorter prompts just by moving one line in a config file to the right place > memory scoped per project → past clients never bleed into new work > Extended Thinking on Light by default → 18–25% fewer Opus tokens in week one > cache_control moved to the right line → the fix that turned a $340 bill into $87 > plugins and MCP servers toggled off when idle → saved 25–40K tokens per session > per-project model override → Haiku for docs, Sonnet for infra, Opus only where it matters most Claude users are running a $100/month tool at 30% of its actual capability

Shadow Nick

380,211 Aufrufe • vor 3 Monaten

CHINESE BUSINESSMAN BOUGHT 4 MAC MINIS FOR $1,699 AND CONNECTED THEM TO HIS MACBOOK - NOW THE CORNER OF HIS DESK MAKES MORE THAN MOST OFFICES MacBook Pro 16" runs everything - assigns tasks, controls the flow, holds context - 4 Mac Minis in the corner execute inference in parallel and don't know what downtime means he connected them through EXO - an open source framework that combines multiple Apple Silicon devices into one unified cluster - and together they run models no single device could handle alone while everyone else rents cloud GPUs for $1,900/month he paid $1,699 once and the entire cluster now runs on $20/month in electricity and nothing else clients pay for inference, data never leaves the room and the cluster scales by simply placing one more Mac Mini on top the MacBook is what he works on - the Mac Minis are what work for him around the clock without days off $1,699 invested once - and a corner of a desk that makes more than most companies with a rented office
0:20

CHINESE BUSINESSMAN BOUGHT 4 MAC MINIS FOR $1,699 AND CONNECTED THEM TO HIS MACBOOK - NOW THE CORNER OF HIS DESK MAKES MORE THAN MOST OFFICES MacBook Pro 16" runs everything - assigns tasks, controls the flow, holds context - 4 Mac Minis in the corner execute inference in parallel and don't know what downtime means he connected them through EXO - an open source framework that combines multiple Apple Silicon devices into one unified cluster - and together they run models no single device could handle alone while everyone else rents cloud GPUs for $1,900/month he paid $1,699 once and the entire cluster now runs on $20/month in electricity and nothing else clients pay for inference, data never leaves the room and the cluster scales by simply placing one more Mac Mini on top the MacBook is what he works on - the Mac Minis are what work for him around the clock without days off $1,699 invested once - and a corner of a desk that makes more than most companies with a rented office

Noisy

22,391 Aufrufe • vor 2 Monaten

people burn $1,900 a month on cloud, this $2,999 NVIDIA box runs 5 AI models at once and costs $10 to power 128GB of unified memory, stack two together and you double it, enough to run models a $2,000 graphics card cannot even open in the demo it ran Qwen3 Vision, Nemotron, a voice model, text to speech and speech to text all at once, and only used half the box here is the math that makes people switch: > cloud GPUs were costing $1,900 every single month, billed forever > the box is $2,999 one time, then about $10 of electricity a month > it paid for itself in two months, $22,000 stayed in the business in year one > zero data ever leaves your desk, no rental, no rate limits most people are still renting what this owns outright the full breakdown of the box and the exact math is in the article below
1:04

people burn $1,900 a month on cloud, this $2,999 NVIDIA box runs 5 AI models at once and costs $10 to power 128GB of unified memory, stack two together and you double it, enough to run models a $2,000 graphics card cannot even open in the demo it ran Qwen3 Vision, Nemotron, a voice model, text to speech and speech to text all at once, and only used half the box here is the math that makes people switch: > cloud GPUs were costing $1,900 every single month, billed forever > the box is $2,999 one time, then about $10 of electricity a month > it paid for itself in two months, $22,000 stayed in the business in year one > zero data ever leaves your desk, no rental, no rate limits most people are still renting what this owns outright the full breakdown of the box and the exact math is in the article below

John Doe

48,135 Aufrufe • vor 2 Monaten