正在加载视频...

视频加载失败

返回首页

THIS DEVELOPER CONNECTED 8 NVIDIA DGX SPARKS INTO ONE CLUSTER - AND RAN AN 800GB MODEL THAT MADE HIM 10X MORE PRODUCTIVE 21:47 he says it straight - "this is a terabyte of VRAM - we ran Quen 3.5, 800GB on disk, a model that doesn't even fit on...

noisyb0y1's profile picture

Noisy

27,202 subscribers

186,592 次观看 • 3 个月前 •via X (Twitter)

科学技术
LIVE

Anya Rossi• Live Now

Private livecam show

36 条评论

Jared Schnelle 的头像
Jared Schnelle3 个月前

Cool video but pretty shitty of you to not even link to the dude's YouTube channel or even say his name. You're just stealing content by re-uploading his video.

Sudo su 的头像
Sudo su3 个月前

this is the spark doing what it was built for. people see one desk box, but it ships ConnectX networking specifically so you can pool them. that's why 8 hit 24 tok/s when one does 3, near linear over RDMA, not duct tape. i run one, clustering a second is the plan. own the compute, don't rent it.

Crowdsource The Truth 的头像
Crowdsource The Truth3 个月前

I think the average person, and even many sophisticated users would have a difficult time setting this up properly.

Jeremy Combs 的头像
Jeremy Combs3 个月前

You should tag and credit @digitalix if you’re going to post his content. @nikitabier is this grounds for demonetization?

GrainStats 🌾 的头像
GrainStats 🌾3 个月前

i see @digitalix video with no attribution to them 🤔

leopardracer 的头像
leopardracer3 个月前

worth watching

Noisy 的头像
Noisy3 个月前

thanks leopard you're a legend

Noisy 的头像
Noisy3 个月前

video from:

Roman 的头像
Roman3 个月前

That's about $38,000. That's based on the current Amazon price of a dgx spark. + The estimate on the cables. The cables were running 100 but they were going up last time I bought one..

Legion 的头像
Legion3 个月前

How do you win the Spark?

karla ❀ 的头像
karla ❀3 个月前

With claude is easy running the developer

Noisy 的头像
Noisy3 个月前

yeah facts, we live in simple times

Mark L Devine 的头像
Mark L Devine3 个月前

Maybe add another 50% to that so that it has a ceiling to do more than just stand up? It doesn't make sense. 24 t/s with no headroom. Just a PoC.

🇫🇷 Fearil DiabloBros 🇷🇺🌿🇵🇸 的头像
🇫🇷 Fearil DiabloBros 🇷🇺🌿🇵🇸3 个月前

8 GPU RTX PRO 6000 Blackwell 768go Vram

James Aita 的头像
James Aita3 个月前

Oh yeah no big deal just 8sparks and an insane switch. Everyone has those lying around the house.

Cryton 的头像
Cryton3 个月前

The crazy part isn't the model size. It's that local AI hardware is starting to compete with infrastructure that used to require a cloud budget.

ext87 的头像
ext873 个月前

Still nearly a 2 year ROI at 2k/mo for cloud opex vs local capex, and this assumes zero other costs like power or other cloud costs

Ventry 的头像
Ventry3 个月前

that's just incredible power

henny 的头像
henny3 个月前

Truly one of the dumbest posts I’ve ever seen. $40,000 for <20 tps?

dunik 的头像
dunik3 个月前

it’s crazy that you can now build your own AI system right at home.

qurool 的头像
qurool3 个月前

it worth my time for sure

TigerAIHQ 的头像
TigerAIHQ3 个月前

"This is actually insane… 8 DGX Sparks clustered together running an 800GB model at 24 tokens/s? A few months ago something like this would only be possible on big cloud clusters. Now regular people are just wiring up hardware at home. The fact that he even used Claude to set up the networking and SSH makes it even crazier 😂

rewind 的头像
rewind3 个月前

home clusters are getting serious

Noisy 的头像
Noisy3 个月前

yeah a cluster of 4 chips is a really powerful thing

The Foxx 的头像
The Foxx3 个月前

I barely have $10, let alone $10,000.

Ben White 的头像
Ben White3 个月前

You can just use cloud Qwen unless you need like 10 billion tokens.

Solvaix 的头像
Solvaix3 个月前

Not cheap yet, but the fact it works locally is impressive

Yeab - engg / core was in Palo Alto🌴 的头像
Yeab - engg / core was in Palo Alto🌴3 个月前

cool

Avid 的头像
Avid3 个月前

Nice

BeanBee 🐝 的头像
BeanBee 🐝3 个月前

A model that was previously cloud-only is becoming locally accessible.

starmex 的头像
starmex3 个月前

bookmarked, terabyte of vram in one cluster is wild

Billionaire Protocol 的头像
Billionaire Protocol3 个月前

24 tps = dogshit

Ramirez Eltoro 的头像
Ramirez Eltoro3 个月前

Amazing

Amarule 的头像
Amarule3 个月前

The Future That Has Already Arrived🕒

Yarchi 的头像
Yarchi3 个月前

this is the last straw, im gonna buy it rn

PLN.ALGO 的头像
PLN.ALGO3 个月前

HE SPEND SUCH MONEY ON that, and i can live it for next 10 year for that cash..

相关视频

THIS DEVELOPER JUST RAN A TRILLION PARAMETER MODEL ON 4 MAC STUDIOS - 10X FASTER AND 5X CHEAPER THAN CLOUD CODE 19:00 he says it out loud. "we just ran a trillion parameter model. 30 something tokens per second. wow." RDMA over Thunderbolt made the cluster 10x faster than before. tensor parallelism splits the model between machines in parallel instead of sequentially DeepSeek V3 full size without quantization. 183GB per machine. still 25 tokens per second. faster than most people can read Kimi K2 with trillion parameters and 256,000 context window. loaded, responded and adapted memory usage dynamically based on prompt size 4 machines consuming 66 watts total. cloud GPU for the same workload costs $1,900/month a cluster that used to cost millions now assembled from Mac Studios in a few hours
22:35

THIS DEVELOPER JUST RAN A TRILLION PARAMETER MODEL ON 4 MAC STUDIOS - 10X FASTER AND 5X CHEAPER THAN CLOUD CODE 19:00 he says it out loud. "we just ran a trillion parameter model. 30 something tokens per second. wow." RDMA over Thunderbolt made the cluster 10x faster than before. tensor parallelism splits the model between machines in parallel instead of sequentially DeepSeek V3 full size without quantization. 183GB per machine. still 25 tokens per second. faster than most people can read Kimi K2 with trillion parameters and 256,000 context window. loaded, responded and adapted memory usage dynamically based on prompt size 4 machines consuming 66 watts total. cloud GPU for the same workload costs $1,900/month a cluster that used to cost millions now assembled from Mac Studios in a few hours

Noisy

80,087 次观看 • 3 个月前

THIS CLUSTER OF 4 MAC STUDIOS USES LESS POWER THAN ONE NVIDIA CHIP - AND COSTS 5X LESS 17:30 the number hits hard - 600 watts for all four Mac Studios - less than one H200 that costs $30,000 on its own instead of one $30,000 GPU you get 4 machines with 1.5TB of memory, 3x more performance and models that simply cannot run on a single chip Apple quietly added RDMA over Thunderbolt in a beta update - latency dropped 99% and the same cluster became 3x faster without buying any new hardware before the update: 5 tokens per second. after: 15.5 - a free upgrade through a software update 8 Nvidia DGX Sparks for the same memory would cost $32,000 - this cluster costs a fraction and gets faster with every update 10x more productive at half the price - and the only thing you need is knowing where to look
20:03

THIS CLUSTER OF 4 MAC STUDIOS USES LESS POWER THAN ONE NVIDIA CHIP - AND COSTS 5X LESS 17:30 the number hits hard - 600 watts for all four Mac Studios - less than one H200 that costs $30,000 on its own instead of one $30,000 GPU you get 4 machines with 1.5TB of memory, 3x more performance and models that simply cannot run on a single chip Apple quietly added RDMA over Thunderbolt in a beta update - latency dropped 99% and the same cluster became 3x faster without buying any new hardware before the update: 5 tokens per second. after: 15.5 - a free upgrade through a software update 8 Nvidia DGX Sparks for the same memory would cost $32,000 - this cluster costs a fraction and gets faster with every update 10x more productive at half the price - and the only thing you need is knowing where to look

Noisy

106,054 次观看 • 3 个月前

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓
1:21

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

slash1s

101,317 次观看 • 3 个月前

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables
21:59

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables

Noisy

48,895 次观看 • 3 个月前

THIS DEVELOPER RAN THE LARGEST AI MODEL IN THE WORLD ON 5 MAC STUDIOS - AND IT COST 100X LESS THAN WHAT OPENAI USES 27:47 he says it after hours of setup - Llama 3.1 405B running locally on five Mac Studios - a model that normally requires 42 GPUs at $1,600 each or an entire data center 320GB unified memory combined - every Mac Studio adds its 64GB into one shared pool - and a model that needs 1TB of VRAM just distributes across all five it launched, it responded and not a single token went to OpenAI or Google - everything local, everything private, everything yours and 100x cheaper 5 Mac Studios, 46 watts for all of them in idle - less than one light bulb per machine what OpenAI keeps behind a subscription he ran at home with a one-time purchase in a few clicks
34:56

THIS DEVELOPER RAN THE LARGEST AI MODEL IN THE WORLD ON 5 MAC STUDIOS - AND IT COST 100X LESS THAN WHAT OPENAI USES 27:47 he says it after hours of setup - Llama 3.1 405B running locally on five Mac Studios - a model that normally requires 42 GPUs at $1,600 each or an entire data center 320GB unified memory combined - every Mac Studio adds its 64GB into one shared pool - and a model that needs 1TB of VRAM just distributes across all five it launched, it responded and not a single token went to OpenAI or Google - everything local, everything private, everything yours and 100x cheaper 5 Mac Studios, 46 watts for all of them in idle - less than one light bulb per machine what OpenAI keeps behind a subscription he ran at home with a one-time purchase in a few clicks

Noisy

21,553 次观看 • 3 个月前

CHINESE AI TEACHER STACKED 4 NVIDIA DGX SPARK CHIPS INTO A CLUSTER - AND NOW RUNS A BUSINESS ON IT FROM HIS DESK 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else he's running a Qwen 32 35B model with 4 concurrent requests at the same time and the speed barely drops compared to a single request what used to require a $10,000+/month cloud setup now sits on his desk and costs him $40-60/month in electricity 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else the people still paying per token are financing someone else's infrastructure
1:42

CHINESE AI TEACHER STACKED 4 NVIDIA DGX SPARK CHIPS INTO A CLUSTER - AND NOW RUNS A BUSINESS ON IT FROM HIS DESK 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else he's running a Qwen 32 35B model with 4 concurrent requests at the same time and the speed barely drops compared to a single request what used to require a $10,000+/month cloud setup now sits on his desk and costs him $40-60/month in electricity 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else the people still paying per token are financing someone else's infrastructure

Sprytix

51,132 次观看 • 4 个月前

FOUR DIFFERENT VENDORS ARE NOW SHIPPING GB10 MINI PCs WITH 128GB UNIFIED MEMORY, AND ONE MICROTIK CRS 804 SWITCH CAN CONNECT UP TO EIGHT OF THEM INTO A 1 TERABYTE LOCAL AI CLUSTER 00:00 he points at the MikroTik CRS 804, "you need some kind of switch that'll handle QSFP56 ports like these", the interconnect that makes the whole cluster possible the GB10 ecosystem is no longer just Nvidia. Dell Pro Max GB10, ASUS Ascent GX10, and MSI Edge Expert all ship the same Grace Blackwell Superchip with 128GB of coherent memory. same silicon, different cases, same 200 gigabit ports on the back the CRS 804 is what connects them at prosumer prices. four 400 gigabit QSFP56 ports on one 1U chassis, breakout cables that split each port into two 200 gigabit lanes. one switch drives eight GB10 units in parallel do the math. eight nodes at 128GB each equals 1024GB of pooled unified memory across the cluster. run vLLM, shard a frontier model across all eight, and inference happens locally on hardware that fits in half a rack the real limiter revealed in the stress test was never throttling. it was interconnect topology, exactly the layer this switch fixes at a fraction of enterprise switch pricing $400 a month for combined chatgpt pro and claude code max hits $4,800 a year per developer. a small team of five running through this cluster pays back inside eight months and never expires the article covers the buying ladder for a single desk. this post is proof of the cluster ladder that starts where the desk one ends save this before the GB10 lineup grows past four vendors and prosumer cluster switches move upmarket
1:05

FOUR DIFFERENT VENDORS ARE NOW SHIPPING GB10 MINI PCs WITH 128GB UNIFIED MEMORY, AND ONE MICROTIK CRS 804 SWITCH CAN CONNECT UP TO EIGHT OF THEM INTO A 1 TERABYTE LOCAL AI CLUSTER 00:00 he points at the MikroTik CRS 804, "you need some kind of switch that'll handle QSFP56 ports like these", the interconnect that makes the whole cluster possible the GB10 ecosystem is no longer just Nvidia. Dell Pro Max GB10, ASUS Ascent GX10, and MSI Edge Expert all ship the same Grace Blackwell Superchip with 128GB of coherent memory. same silicon, different cases, same 200 gigabit ports on the back the CRS 804 is what connects them at prosumer prices. four 400 gigabit QSFP56 ports on one 1U chassis, breakout cables that split each port into two 200 gigabit lanes. one switch drives eight GB10 units in parallel do the math. eight nodes at 128GB each equals 1024GB of pooled unified memory across the cluster. run vLLM, shard a frontier model across all eight, and inference happens locally on hardware that fits in half a rack the real limiter revealed in the stress test was never throttling. it was interconnect topology, exactly the layer this switch fixes at a fraction of enterprise switch pricing $400 a month for combined chatgpt pro and claude code max hits $4,800 a year per developer. a small team of five running through this cluster pays back inside eight months and never expires the article covers the buying ladder for a single desk. this post is proof of the cluster ladder that starts where the desk one ends save this before the GB10 lineup grows past four vendors and prosumer cluster switches move upmarket

NO1ennn

59,725 次观看 • 3 个月前

8 DGX Sparks on a desk just ran a frontier model at 400K context. A friend who owns the inference budget at a Berlin AI startup, $52K a month in API bills, sent me this at 2 a.m. with one line: "we're buying the hardware." It starts as a disaster. A mountain of cables burying $32K of NVIDIA silicon, nothing talking to anything. 54 minutes later the whole stack answers as 1 machine. 1TB of unified memory. GLM-5.2 loaded across all 8 nodes. The part that made him rewatch twice sits around the 25-minute mark, where 1 networking fix turns 8 boxes into a cluster. Most people quit exactly there. He paused it 4 times. Screenshotted the interface setup. Forwarded it to his CTO without a caption. The cloud bill was never the moat. The cables were.
54:24

8 DGX Sparks on a desk just ran a frontier model at 400K context. A friend who owns the inference budget at a Berlin AI startup, $52K a month in API bills, sent me this at 2 a.m. with one line: "we're buying the hardware." It starts as a disaster. A mountain of cables burying $32K of NVIDIA silicon, nothing talking to anything. 54 minutes later the whole stack answers as 1 machine. 1TB of unified memory. GLM-5.2 loaded across all 8 nodes. The part that made him rewatch twice sits around the 25-minute mark, where 1 networking fix turns 8 boxes into a cluster. Most people quit exactly there. He paused it 4 times. Screenshotted the interface setup. Forwarded it to his CTO without a caption. The cloud bill was never the moat. The cables were.

the lich

56,893 次观看 • 2 个月前

THE CHINESE INSTALLED 6 AI AGENTS RUNNING 24/7 FOR $11 A MONTH. THE BOX UNDER ONE DESK REPLACED $1,200 IN CLOUD SUBSCRIPTIONS. The Minisforum MS-S1 Max ships with 128GB of unified memory shared between CPU and GPU. Qwen3-Coder 30B runs at 40 to 50 tokens per second. 6 agents run simultaneously without anyone watching the meter. An inbox sorter drafts replies every 15 minutes. A research monitor watches 30+ feeds and lands a digest by 7 AM, a job that costs $15 to $20 a day on cloud APIs. The same stack on cloud subscriptions ran $800 to $1,200 a month. The hardware costs $3,000 once. Break-even hits month 3 to 4. The real shift is behavior, not savings. A meeting prep agent never would have justified per-token cost. Running it free overnight made it obvious.
0:53

THE CHINESE INSTALLED 6 AI AGENTS RUNNING 24/7 FOR $11 A MONTH. THE BOX UNDER ONE DESK REPLACED $1,200 IN CLOUD SUBSCRIPTIONS. The Minisforum MS-S1 Max ships with 128GB of unified memory shared between CPU and GPU. Qwen3-Coder 30B runs at 40 to 50 tokens per second. 6 agents run simultaneously without anyone watching the meter. An inbox sorter drafts replies every 15 minutes. A research monitor watches 30+ feeds and lands a digest by 7 AM, a job that costs $15 to $20 a day on cloud APIs. The same stack on cloud subscriptions ran $800 to $1,200 a month. The hardware costs $3,000 once. Break-even hits month 3 to 4. The real shift is behavior, not savings. A meeting prep agent never would have justified per-token cost. Running it free overnight made it obvious.

Lummox

14,093 次观看 • 3 个月前

ONE BUILDER WIRED 4 RASPBERRY PI MODULES INTO A €450 AI CLUSTER WITH 64GB OF RAM AND KILLED HIS $200/MONTH CHATGPT SUBSCRIPTION 00:37 he points at the terminal, "loaded, loaded, loaded - 4 nodes synced over gigabit, 64 gigabytes total, llama 3.1 partitioned across the cluster" he stacked 4 raspberry pi CM5 lite modules with 16 gigabytes each onto a sipeed nanocluster board, the whole rig runs off a single 65 watt power supply and a gigabit internal switch distributed llama partitions the model across the nodes with synchronized workloads handled over the internal fabric, no quantization needed, the cluster hits about 30 tokens per second on small models a heavy developer pays $200 a month for chatgpt pro and another $200 for claude code, this rig cost €450 in parts and breaks even on the stack in month 2 most people will keep paying anthropic and openai forever, a few will spend a weekend wiring 4 raspberry pi modules and never see a subscription invoice again the window is open, follow and bookmark before it closes
0:37

ONE BUILDER WIRED 4 RASPBERRY PI MODULES INTO A €450 AI CLUSTER WITH 64GB OF RAM AND KILLED HIS $200/MONTH CHATGPT SUBSCRIPTION 00:37 he points at the terminal, "loaded, loaded, loaded - 4 nodes synced over gigabit, 64 gigabytes total, llama 3.1 partitioned across the cluster" he stacked 4 raspberry pi CM5 lite modules with 16 gigabytes each onto a sipeed nanocluster board, the whole rig runs off a single 65 watt power supply and a gigabit internal switch distributed llama partitions the model across the nodes with synchronized workloads handled over the internal fabric, no quantization needed, the cluster hits about 30 tokens per second on small models a heavy developer pays $200 a month for chatgpt pro and another $200 for claude code, this rig cost €450 in parts and breaks even on the stack in month 2 most people will keep paying anthropic and openai forever, a few will spend a weekend wiring 4 raspberry pi modules and never see a subscription invoice again the window is open, follow and bookmark before it closes

Antid

64,291 次观看 • 3 个月前

FOUR BOXES ON A DESK ARE NOW A FRONTIER AI CLUSTER This is four NVIDIA DGX Spark units wired together, sitting on a single desk and running Qwen3-235B end to end. Each box holds 128GB of unified memory. Stacked together, that is half a terabyte of memory pooled across the cluster, all from hardware you could fit in a backpack. A year ago a setup like this lived in a data center behind a badge reader. Now it runs next to a monitor and a coffee cup. Here is the part people get wrong. The four boxes do not magically merge into one giant GPU. The model is sharded across all of them, and they pass data over a fast link while every box does its slice of the work. That is the whole trick. No single box could hold a 235B model, but split four ways, it runs. The machines stay quiet and barely warm while doing it, sipping a fraction of the power a rack of server GPUs would burn. The wall between you and a frontier-class model used to be a cloud bill. Now it is just desk space. Bookmark this & Watch it run ↓
1:42

FOUR BOXES ON A DESK ARE NOW A FRONTIER AI CLUSTER This is four NVIDIA DGX Spark units wired together, sitting on a single desk and running Qwen3-235B end to end. Each box holds 128GB of unified memory. Stacked together, that is half a terabyte of memory pooled across the cluster, all from hardware you could fit in a backpack. A year ago a setup like this lived in a data center behind a badge reader. Now it runs next to a monitor and a coffee cup. Here is the part people get wrong. The four boxes do not magically merge into one giant GPU. The model is sharded across all of them, and they pass data over a fast link while every box does its slice of the work. That is the whole trick. No single box could hold a 235B model, but split four ways, it runs. The machines stay quiet and barely warm while doing it, sipping a fraction of the power a rack of server GPUs would burn. The wall between you and a frontier-class model used to be a cloud bill. Now it is just desk space. Bookmark this & Watch it run ↓

slash1s

19,473 次观看 • 3 个月前

This guy plugged a DGX Spark (the $3K Nvidia box) and a Mac Mini M4 together to run AI and what happened next surprised everyone > the Nvidia box handles the hard part - processing your prompt in milliseconds > Mac Mini M4 handles the fast part - generating the response at memory bandwidth speeds nothing else can match > together they hit 84 tokens per second on Llama - 6x faster than the Spark alone > running compute agents locally on this setup means your data never leaves your hardware > two boxes. two different architectures. one AI system that Deepseek runs at data center scale he ran it on his desk save this. the way we build local AI is about to change
18:41

This guy plugged a DGX Spark (the $3K Nvidia box) and a Mac Mini M4 together to run AI and what happened next surprised everyone > the Nvidia box handles the hard part - processing your prompt in milliseconds > Mac Mini M4 handles the fast part - generating the response at memory bandwidth speeds nothing else can match > together they hit 84 tokens per second on Llama - 6x faster than the Spark alone > running compute agents locally on this setup means your data never leaves your hardware > two boxes. two different architectures. one AI system that Deepseek runs at data center scale he ran it on his desk save this. the way we build local AI is about to change

Mr. Buzzoni

154,069 次观看 • 4 个月前

This Chinese developer linked two $2,999 NVIDIA DGX Sparks into one box and runs the full Qwen3-235B at home, after dropping his $1,999-a-month cloud bill to zero. He wired 2 small boxes into a single computer, split a giant 235-billion-parameter model in half between them, and serves it across his own network at about 10 tokens a second, with no internet, no cloud, right there on the desk. No data center, no thousand-dollar graphics cards, no monthly cloud bill. Just him, 2 gold boxes the size of a sandwich, one cable between them, and 1 power strip. And here is the whole payoff. He used to pay the cloud $1,999 a month for the same model, and the meter ticked on every request. Now he paid $5,998 once for 2 boxes, they covered their cost in 3 months, and after that he sends as many requests as he wants for free, only electricity. The two Sparks talk over one fast cable, each holds 128GB of memory, and together they carry the whole model, about 73GB loaded per box, with the chip inside pinned near the limit at 96%. Both boxes work as one and keep trading data over the cable, with no cloud in the loop and no single word leaking out. The ready model sits on one local address, and any app on his network calls it as easily as ChatGPT. And here is how he described, in plain words, what this pair of boxes does: "this is a pair of boxes that holds the huge Qwen3-235B model and serves it to one network. the model is split in half, and each box owns its half. parts: // Box 1 (holds the first half of the model and starts the answer fast, the first word appears in under a second) // Box 2 (holds the second half and writes out the rest, about 10 tokens a second) // Cable (connects the 2 boxes and moves data between them on every step, with no lag) // Address (one local address where any app sends its request, like to a cloud model) // Test (a script that runs big prompts through and measures speed and delays) // Monitor (checks temperature, power draw, and load on both boxes every 2 seconds). the model never goes to the cloud. he only steps in when a box runs hotter than 80 degrees or the cable between them starts dropping data." So the system knows exactly what it is, what it is for, and where its limits are. It knows it has to hold the whole huge model across 2 boxes on its own. It knows it has to answer every request locally, with no meter, no limits, and no internet. It knows the human is only needed when a box overheats or the link between them stalls. → The setup runs around the clock on 2 boxes, each pulling under 60 watts → However many requests he sends, the monthly bill is $0, only electricity → The first box starts the answer in under a second → The second writes text at about 10 tokens a second → One request at a time: 838 tokens in 85 seconds, first word in 0.8s → Two requests at once: 697 tokens in 108 seconds, first word in 0.7s → Both boxes sit at 96% load and warm up to 76-78 degrees And only when a chip in a box runs hotter than 80 degrees or the cable between the 2 Sparks drops data does the system call the owner. And when he himself is out on a run or in a coffee shop, he still reaches his own model at home from his phone: sends a big prompt to the local Qwen3-235B, gets the full answer back in under a minute and a half, with no token meter ticking and no limit to hit. Here is what the test shows on his screen during one of the night runs: "one request at a time: 838 tokens in 84.9 seconds, first word in 0.8s, then 0.1s per token." "two requests at once: 697 tokens in 107.6 seconds, first word in 0.7s, then 0.15s per token." "Box 1: chip at 96% load, 76 degrees, 56 watts, 73GB used in memory." "Box 2: chip at 96% load, 78 degrees, 56 watts, the Qwen3-235B model fully loaded." And while everyone around is paying for AI by the month and bumping into limits, his top-tier model just sits on the desk and works as much as he wants: his own little power plant instead of a forever meter. He has no server rack of his own and no cloud account behind it. Just 2 DGX Spark boxes on a desk, one model split in half between them, one local address, and a folder of prompts next to it. Out of everything I have seen this year, this is the cleanest way to stop paying for AI: $5,998 of hardware on the desk once, $0 a month to the cloud, unlimited forever, and between them 2 gold boxes, 1 cable, and the full Qwen3-235B answering at home with no internet.
1:16

This Chinese developer linked two $2,999 NVIDIA DGX Sparks into one box and runs the full Qwen3-235B at home, after dropping his $1,999-a-month cloud bill to zero. He wired 2 small boxes into a single computer, split a giant 235-billion-parameter model in half between them, and serves it across his own network at about 10 tokens a second, with no internet, no cloud, right there on the desk. No data center, no thousand-dollar graphics cards, no monthly cloud bill. Just him, 2 gold boxes the size of a sandwich, one cable between them, and 1 power strip. And here is the whole payoff. He used to pay the cloud $1,999 a month for the same model, and the meter ticked on every request. Now he paid $5,998 once for 2 boxes, they covered their cost in 3 months, and after that he sends as many requests as he wants for free, only electricity. The two Sparks talk over one fast cable, each holds 128GB of memory, and together they carry the whole model, about 73GB loaded per box, with the chip inside pinned near the limit at 96%. Both boxes work as one and keep trading data over the cable, with no cloud in the loop and no single word leaking out. The ready model sits on one local address, and any app on his network calls it as easily as ChatGPT. And here is how he described, in plain words, what this pair of boxes does: "this is a pair of boxes that holds the huge Qwen3-235B model and serves it to one network. the model is split in half, and each box owns its half. parts: // Box 1 (holds the first half of the model and starts the answer fast, the first word appears in under a second) // Box 2 (holds the second half and writes out the rest, about 10 tokens a second) // Cable (connects the 2 boxes and moves data between them on every step, with no lag) // Address (one local address where any app sends its request, like to a cloud model) // Test (a script that runs big prompts through and measures speed and delays) // Monitor (checks temperature, power draw, and load on both boxes every 2 seconds). the model never goes to the cloud. he only steps in when a box runs hotter than 80 degrees or the cable between them starts dropping data." So the system knows exactly what it is, what it is for, and where its limits are. It knows it has to hold the whole huge model across 2 boxes on its own. It knows it has to answer every request locally, with no meter, no limits, and no internet. It knows the human is only needed when a box overheats or the link between them stalls. → The setup runs around the clock on 2 boxes, each pulling under 60 watts → However many requests he sends, the monthly bill is $0, only electricity → The first box starts the answer in under a second → The second writes text at about 10 tokens a second → One request at a time: 838 tokens in 85 seconds, first word in 0.8s → Two requests at once: 697 tokens in 108 seconds, first word in 0.7s → Both boxes sit at 96% load and warm up to 76-78 degrees And only when a chip in a box runs hotter than 80 degrees or the cable between the 2 Sparks drops data does the system call the owner. And when he himself is out on a run or in a coffee shop, he still reaches his own model at home from his phone: sends a big prompt to the local Qwen3-235B, gets the full answer back in under a minute and a half, with no token meter ticking and no limit to hit. Here is what the test shows on his screen during one of the night runs: "one request at a time: 838 tokens in 84.9 seconds, first word in 0.8s, then 0.1s per token." "two requests at once: 697 tokens in 107.6 seconds, first word in 0.7s, then 0.15s per token." "Box 1: chip at 96% load, 76 degrees, 56 watts, 73GB used in memory." "Box 2: chip at 96% load, 78 degrees, 56 watts, the Qwen3-235B model fully loaded." And while everyone around is paying for AI by the month and bumping into limits, his top-tier model just sits on the desk and works as much as he wants: his own little power plant instead of a forever meter. He has no server rack of his own and no cloud account behind it. Just 2 DGX Spark boxes on a desk, one model split in half between them, one local address, and a folder of prompts next to it. Out of everything I have seen this year, this is the cleanest way to stop paying for AI: $5,998 of hardware on the desk once, $0 a month to the cloud, unlimited forever, and between them 2 gold boxes, 1 cable, and the full Qwen3-235B answering at home with no internet.

Blaze

93,871 次观看 • 4 个月前

AMD'S NEW MINI PC RUNS CLAUDE CODE OFFLINE FOR $0/MONTH AND REPLACES $200 OPENAI + $200 CLAUDE CODE SUBSCRIPTIONS amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on a single piece of silicon with 128gb of unified memory the gmktec evo-x2 runs qwen3 coder 30b at 40 to 50 tokens per second and llama 3.3 70b at 20 to 25 tokens per second locally. on linux you get 110gb of usable vram out of 128gb setup takes 90 minutes. wipe windows for linux, install ollama with one command, pull the model, point claude code at localhost. same interface, same agent loop, nothing leaves your network a heavy ai user pays $200 for claude code max, $200 for chatgpt pro and $20 for cursor. that's $5,040 a year and the box pays itself off in 5 months after break-even every month is $420 that used to leave your account. three year math is roughly $13,000 not paid to ai companies plus zero rate limits on overnight agents bookmark this and read the article below
0:52

AMD'S NEW MINI PC RUNS CLAUDE CODE OFFLINE FOR $0/MONTH AND REPLACES $200 OPENAI + $200 CLAUDE CODE SUBSCRIPTIONS amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on a single piece of silicon with 128gb of unified memory the gmktec evo-x2 runs qwen3 coder 30b at 40 to 50 tokens per second and llama 3.3 70b at 20 to 25 tokens per second locally. on linux you get 110gb of usable vram out of 128gb setup takes 90 minutes. wipe windows for linux, install ollama with one command, pull the model, point claude code at localhost. same interface, same agent loop, nothing leaves your network a heavy ai user pays $200 for claude code max, $200 for chatgpt pro and $20 for cursor. that's $5,040 a year and the box pays itself off in 5 months after break-even every month is $420 that used to leave your account. three year math is roughly $13,000 not paid to ai companies plus zero rate limits on overnight agents bookmark this and read the article below

starmex

33,516 次观看 • 3 个月前

INTEL JUST SHIPPED A WORKSTATION CARD WITH TWO GPUs ON ONE PCB AND 48GB OF VRAM. FOUR CARDS GIVE A SINGLE MOTHERBOARD 192GB OF POOLED INFERENCE MEMORY FOR THE PRICE OF TWO RTX 5090s 00:14 he holds one card up to the camera, two GPU dies side by side under the cooler, each one running its own x8 PCIe lane back to the chipset the cluster sees eight discrete accelerators in software. intel's Battle Matrix stack shards a model across all eight, so a 235B parameter network loads in slices and answers requests in parallel what 192GB of VRAM unlocks: an entire 200B class model in memory without quantization. a vision agent reading 100 invoices at once. a research box that holds three frontier models loaded simultaneously, switching between them in under a second intel is the slow side of inference. nvidia is faster per token, that is the honest tradeoff. but the only other path to this much VRAM is a $40,000 nvidia rack or three networked Mac Studios four B60 cards plus the chassis lands under $5,000. power draw averages 800 watts, $55 a month in electricity. one engineer paying $400 a month for combined ChatGPT Pro and Claude Code Max pays the hardware off in less than a year
1:16

INTEL JUST SHIPPED A WORKSTATION CARD WITH TWO GPUs ON ONE PCB AND 48GB OF VRAM. FOUR CARDS GIVE A SINGLE MOTHERBOARD 192GB OF POOLED INFERENCE MEMORY FOR THE PRICE OF TWO RTX 5090s 00:14 he holds one card up to the camera, two GPU dies side by side under the cooler, each one running its own x8 PCIe lane back to the chipset the cluster sees eight discrete accelerators in software. intel's Battle Matrix stack shards a model across all eight, so a 235B parameter network loads in slices and answers requests in parallel what 192GB of VRAM unlocks: an entire 200B class model in memory without quantization. a vision agent reading 100 invoices at once. a research box that holds three frontier models loaded simultaneously, switching between them in under a second intel is the slow side of inference. nvidia is faster per token, that is the honest tradeoff. but the only other path to this much VRAM is a $40,000 nvidia rack or three networked Mac Studios four B60 cards plus the chassis lands under $5,000. power draw averages 800 watts, $55 a month in electricity. one engineer paying $400 a month for combined ChatGPT Pro and Claude Code Max pays the hardware off in less than a year

NO1ennn

48,247 次观看 • 3 个月前

Qwen 3.6 35B running at over 100 tokens per second on my $5,399 MacBook Pro M5 Max. This is the best local AI model I have ever run. 128GB of unified memory. No cloud. No API costs. No rate limits. Just raw local inference at speeds I didn't think were possible on a laptop. This model is more intelligent than GPT 5 on benchmarks. Running locally. On a MacBook. For free after the hardware cost. I said local AI would never compete with frontier. I'm starting to rethink that. The gap is closing faster than anyone expected.
1:04

Qwen 3.6 35B running at over 100 tokens per second on my $5,399 MacBook Pro M5 Max. This is the best local AI model I have ever run. 128GB of unified memory. No cloud. No API costs. No rate limits. Just raw local inference at speeds I didn't think were possible on a laptop. This model is more intelligent than GPT 5 on benchmarks. Running locally. On a MacBook. For free after the hardware cost. I said local AI would never compete with frontier. I'm starting to rethink that. The gap is closing faster than anyone expected.

BridgeMind

62,800 次观看 • 4 个月前

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.
0:42

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

344,670 次观看 • 1 个月前

THE $4,000 DGX SPARK BENCHMARKS AT 342 T/S ON PROMPT PROCESSING - THEN DROPS TO 73 WHEN IT ACTUALLY GENERATES. that clip is a dgx spark running llama-bench on qwen3-coder 30b, side by side with a mac. two numbers decide everything here: pp512, prompt processing = 342 tokens/sec tg128, token generation = 73 tokens/sec that gap is the whole lesson. local ai isn't one speed, it's two. prompt processing is reading your input: your files, your long context. the spark is compute-heavy, so it eats that fast. token generation is writing the answer, and it's bound by memory bandwidth, where the spark is modest and unified-memory macs stay close. so "which box is faster" has no single winner. feeding huge context or running agents that read a lot? prompt processing wins, the spark shines. long chat replies, one token at a time? generation matters more, and a mac holds its own. no single benchmark number, no universal "fastest box", no upgrade that wins every workload. the uncomfortable part isn't that the spark is fast. it's that "tokens per second" was always two numbers, and the hype only ever quotes the flattering one. follow RetroChainer for the benchmark truth nobody screenshots, and bookmark this before you buy on one number.
1:14

THE $4,000 DGX SPARK BENCHMARKS AT 342 T/S ON PROMPT PROCESSING - THEN DROPS TO 73 WHEN IT ACTUALLY GENERATES. that clip is a dgx spark running llama-bench on qwen3-coder 30b, side by side with a mac. two numbers decide everything here: pp512, prompt processing = 342 tokens/sec tg128, token generation = 73 tokens/sec that gap is the whole lesson. local ai isn't one speed, it's two. prompt processing is reading your input: your files, your long context. the spark is compute-heavy, so it eats that fast. token generation is writing the answer, and it's bound by memory bandwidth, where the spark is modest and unified-memory macs stay close. so "which box is faster" has no single winner. feeding huge context or running agents that read a lot? prompt processing wins, the spark shines. long chat replies, one token at a time? generation matters more, and a mac holds its own. no single benchmark number, no universal "fastest box", no upgrade that wins every workload. the uncomfortable part isn't that the spark is fast. it's that "tokens per second" was always two numbers, and the hype only ever quotes the flattering one. follow RetroChainer for the benchmark truth nobody screenshots, and bookmark this before you buy on one number.

RetroChainer

42,451 次观看 • 2 个月前

single most useful thing for my local ai setup that made life easy is my 2x DGX Spark becoming one serving box for every device i own here is how i do it: > 1. the two sparks run one vLLM server together with one endpoint, i use dgx sparks, you can use any nodes that can run an llm > 2. install tailscale to have all your nodes and machines on the same tailnet, the endpoint lives there so every device reaches it from anywhere and no port is open to the internet > 3. that one tailnet endpoint with the exposed port is your base url for everything, it speaks OpenAI chat, OpenAI Responses and Anthropic Messages, so any chat app, coding agent or phone bot you point at it talks to the same model > 4. the server also answers to the name "local", so clients can ask for "local" instead of a model name and when i swap the model on the servers nothing on the devices needs a config change. right now it's serving GLM 5.3-Flash, next week it can be something else this video below is the whole thing running end to end, orange dots are requests going in, white and green are tokens coming back it takes however many requests you throw at it, your config decides how many run at once and the rest wait their turn, when several run together each request's tok/s drops a bit while the box moves more tokens in total. mine is set to one at a time right now, so when every device fires together they queue up this setup really improved my quality of life with local ai, if you get stuck anywhere setting it up leave a comment and i'll help you
1:04

single most useful thing for my local ai setup that made life easy is my 2x DGX Spark becoming one serving box for every device i own here is how i do it: > 1. the two sparks run one vLLM server together with one endpoint, i use dgx sparks, you can use any nodes that can run an llm > 2. install tailscale to have all your nodes and machines on the same tailnet, the endpoint lives there so every device reaches it from anywhere and no port is open to the internet > 3. that one tailnet endpoint with the exposed port is your base url for everything, it speaks OpenAI chat, OpenAI Responses and Anthropic Messages, so any chat app, coding agent or phone bot you point at it talks to the same model > 4. the server also answers to the name "local", so clients can ask for "local" instead of a model name and when i swap the model on the servers nothing on the devices needs a config change. right now it's serving GLM 5.3-Flash, next week it can be something else this video below is the whole thing running end to end, orange dots are requests going in, white and green are tokens coming back it takes however many requests you throw at it, your config decides how many run at once and the rest wait their turn, when several run together each request's tok/s drops a bit while the box moves more tokens in total. mine is set to one at a time right now, so when every device fires together they queue up this setup really improved my quality of life with local ai, if you get stuck anywhere setting it up leave a comment and i'll help you

Sudo su

32,379 次观看 • 9 天前

Anthropic shipped 125 settings for Claude The official docs cover 40 One developer found the other 85 and his API bill dropped from $340 to $87 - not by using a cheaper model - not by writing shorter prompts just by moving one line in a config file to the right place > memory scoped per project → past clients never bleed into new work > Extended Thinking on Light by default → 18–25% fewer Opus tokens in week one > cache_control moved to the right line → the fix that turned a $340 bill into $87 > plugins and MCP servers toggled off when idle → saved 25–40K tokens per session > per-project model override → Haiku for docs, Sonnet for infra, Opus only where it matters most Claude users are running a $100/month tool at 30% of its actual capability
0:49

Anthropic shipped 125 settings for Claude The official docs cover 40 One developer found the other 85 and his API bill dropped from $340 to $87 - not by using a cheaper model - not by writing shorter prompts just by moving one line in a config file to the right place > memory scoped per project → past clients never bleed into new work > Extended Thinking on Light by default → 18–25% fewer Opus tokens in week one > cache_control moved to the right line → the fix that turned a $340 bill into $87 > plugins and MCP servers toggled off when idle → saved 25–40K tokens per session > per-project model override → Haiku for docs, Sonnet for infra, Opus only where it matters most Claude users are running a $100/month tool at 30% of its actual capability

Shadow Nick

380,456 次观看 • 4 个月前