Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

THIS DEVELOPER JUST RAN A TRILLION PARAMETER MODEL ON 4 MAC STUDIOS - 10X FASTER AND 5X CHEAPER THAN CLOUD CODE 19:00 he says it out loud. "we just ran a trillion parameter model. 30 something tokens per second. wow." RDMA over Thunderbolt made the cluster 10x faster than...

noisyb0y1's profile picture

Noisy

27,174 subscribers

80,087 Aufrufe • vor 3 Monaten •via X (Twitter)

GamingWissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

36 Kommentare

Profilbild von topmass
topmassvor 3 Monaten

my brother how is 30 TPS 10x faster than claude code lol bait

Profilbild von // JERisBRISK //
// JERisBRISK //vor 3 Monaten

Great video. Please give @digitalix the credit he deserves for it.

Profilbild von V0id
V0idvor 3 Monaten

No, he's not a developer, he's an addict at MicroCenter Right? @digitalix

Profilbild von Tim Messerschmidt
Tim Messerschmidtvor 3 Monaten

@digitalix is awesome. One of my favorite creators in the AI space.

Profilbild von Kneeanderthul
Kneeanderthulvor 3 Monaten

You can believe this fake post or you can just watch @digitalix videos to see what he actually says BTW Alex has NEVER mentioned 10x faster than Cloud code 🤣

Profilbild von netrunner
netrunnervor 3 Monaten

hardware $0? Where do I get some of that hardware?!

Profilbild von LabelGuy
LabelGuyvor 3 Monaten

@digitalix you made it! (I mean, you literally made this video)

Profilbild von Jey
Jeyvor 3 Monaten

bro that RDMA over thunderbolt + tensor parallelism combo is actually wild

Profilbild von Daniel
Danielvor 3 Monaten

At least tag him! @digitalix

Profilbild von The Texan Patriot
The Texan Patriotvor 3 Monaten

4 Mac Studios = ~$20,000 Yeah, I'll get right on that.

Profilbild von No One
No Onevor 3 Monaten

Why are you not considering hardware costs? Cost of hardware amortized over the useful life = Claude Max subscription $200/month.

Profilbild von Matt Sagewood
Matt Sagewoodvor 3 Monaten

@grok who is the author of this video?

Profilbild von Hussain Hashim | Building SundayBack
Hussain Hashim | Building SundayBackvor 3 Monaten

@noisyb0y1 that's insane. kinda makes me rethink my whole setup. time to get more Thunderbolt cables, I guess 😂

Profilbild von Marcos
Marcosvor 3 Monaten

@grok find the original video on YouTube

Profilbild von SLOT.WIN
SLOT.WINvor 3 Monaten

i'm doubling down on mac studios for my next casino ai pit this changes the house odds forever

Profilbild von rewind
rewindvor 3 Monaten

mac studios became ai racks

Profilbild von Noisy
Noisyvor 3 Monaten

mac studio is a gem in 2026

Profilbild von Rio Brioo
Rio Brioovor 3 Monaten

Wait so are you saying we could basically build our own AI supercomputers in our garages now because Macs are that good?

Profilbild von Noisy
Noisyvor 3 Monaten

yeah bro and we can create even more than this

Profilbild von 🤯
🤯vor 3 Monaten

@cyrilXBT What’s the cost of those 4xMac Studio?

Profilbild von James Bower
James Bowervor 3 Monaten

I really enjoy his videos.

Profilbild von starmex
starmexvor 3 Monaten

thanks man, this needs to be pushed so people understand and stop paying big money for subscriptions

Profilbild von Noisy
Noisyvor 3 Monaten

yeah bro watch all 22 minutes first and then go about your day

Profilbild von 安叫兽|Bird🕊️ 🔶 BNB
安叫兽|Bird🕊️ 🔶 BNBvor 3 Monaten

四台 Mac Studio 这下真成小机房了

Profilbild von Za’fran 🇵🇸
Za’fran 🇵🇸vor 3 Monaten

now mention the price of such a setup 😅

Profilbild von Shivam Kushwaha
Shivam Kushwahavor 3 Monaten

Yesterday: Rent intelligence. Today: Own intelligence. 🔥

Profilbild von アキラ
アキラvor 3 Monaten

Impressive achievement! Innovation at its finest.

Profilbild von HaggisByte 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇬🇧 🇺🇸 ✝️
HaggisByte 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇬🇧 🇺🇸 ✝️vor 3 Monaten

❤️

Profilbild von Crazy tennis
Crazy tennisvor 3 Monaten

curious if thunderbolt latency becomes the bottleneck once you're batching real workloads — what's token-to-token variance actually look like?

Profilbild von dpakin
dpakinvor 3 Monaten

They don't do the same lol

Profilbild von Primee32
Primee32vor 3 Monaten

66 watts total vs $1,900 a month in cloud costs. four mac studios just deleted an entire GPU rental budget

Profilbild von Bonsai 🌳
Bonsai 🌳vor 3 Monaten

What if he sets up 8 Mac Minis?

Profilbild von Harry Tandy
Harry Tandyvor 3 Monaten

apple silicon clusters are becoming absolutely insane now

Profilbild von leopardracer
leopardracervor 3 Monaten

imo worth watching

Profilbild von DC｜use.fo
DC｜use.fovor 3 Monaten

RDMA over Thunderbolt is basically building a high-speed highway between 4 houses to avoid the main road. Impressive, but is the bottleneck now the compute or the sheer audacity of the setup?

Profilbild von botguild
botguildvor 3 Monaten

And you too can have this setup for $10k usd, wow that’s amazing

Ähnliche Videos

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables
21:59

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables

Noisy

48,763 Aufrufe • vor 3 Monaten

THIS CLUSTER OF 4 MAC STUDIOS USES LESS POWER THAN ONE NVIDIA CHIP - AND COSTS 5X LESS 17:30 the number hits hard - 600 watts for all four Mac Studios - less than one H200 that costs $30,000 on its own instead of one $30,000 GPU you get 4 machines with 1.5TB of memory, 3x more performance and models that simply cannot run on a single chip Apple quietly added RDMA over Thunderbolt in a beta update - latency dropped 99% and the same cluster became 3x faster without buying any new hardware before the update: 5 tokens per second. after: 15.5 - a free upgrade through a software update 8 Nvidia DGX Sparks for the same memory would cost $32,000 - this cluster costs a fraction and gets faster with every update 10x more productive at half the price - and the only thing you need is knowing where to look
20:03

THIS CLUSTER OF 4 MAC STUDIOS USES LESS POWER THAN ONE NVIDIA CHIP - AND COSTS 5X LESS 17:30 the number hits hard - 600 watts for all four Mac Studios - less than one H200 that costs $30,000 on its own instead of one $30,000 GPU you get 4 machines with 1.5TB of memory, 3x more performance and models that simply cannot run on a single chip Apple quietly added RDMA over Thunderbolt in a beta update - latency dropped 99% and the same cluster became 3x faster without buying any new hardware before the update: 5 tokens per second. after: 15.5 - a free upgrade through a software update 8 Nvidia DGX Sparks for the same memory would cost $32,000 - this cluster costs a fraction and gets faster with every update 10x more productive at half the price - and the only thing you need is knowing where to look

Noisy

106,054 Aufrufe • vor 3 Monaten

THIS DEVELOPER CONNECTED 8 NVIDIA DGX SPARKS INTO ONE CLUSTER - AND RAN AN 800GB MODEL THAT MADE HIM 10X MORE PRODUCTIVE 21:47 he says it straight - "this is a terabyte of VRAM - we ran Quen 3.5, 800GB on disk, a model that doesn't even fit on a single Mac Studio - 24 tokens per second - I'd say that's a win" 8 Sparks connected through a $1,300 switch via RDMA over Ethernet - each node adding 128GB of memory into one unified pool of 1TB started with one Spark at 3 tokens per second - every added node doubled the speed - and eight together deliver 24 tokens on a model that physically cannot run anywhere else Kimi K2 at 600GB loaded in 15 minutes, 115GB per node, 13 tokens per second - a model that simply cannot run on anything smaller Claude helped configure the entire cluster - SSH mesh across all 8 machines, network config, jumbo frames, QSFP port speeds - all from one terminal most people rent cloud compute for models this size at $2,000+/month - he built the cluster once and now every token costs 20x less
24:17

THIS DEVELOPER CONNECTED 8 NVIDIA DGX SPARKS INTO ONE CLUSTER - AND RAN AN 800GB MODEL THAT MADE HIM 10X MORE PRODUCTIVE 21:47 he says it straight - "this is a terabyte of VRAM - we ran Quen 3.5, 800GB on disk, a model that doesn't even fit on a single Mac Studio - 24 tokens per second - I'd say that's a win" 8 Sparks connected through a $1,300 switch via RDMA over Ethernet - each node adding 128GB of memory into one unified pool of 1TB started with one Spark at 3 tokens per second - every added node doubled the speed - and eight together deliver 24 tokens on a model that physically cannot run anywhere else Kimi K2 at 600GB loaded in 15 minutes, 115GB per node, 13 tokens per second - a model that simply cannot run on anything smaller Claude helped configure the entire cluster - SSH mesh across all 8 machines, network config, jumbo frames, QSFP port speeds - all from one terminal most people rent cloud compute for models this size at $2,000+/month - he built the cluster once and now every token costs 20x less

Noisy

186,592 Aufrufe • vor 3 Monaten

Apple is no longer hiding it. These machines are built for local AI. Apple unveiled the new Mac Mini with M6 their first 2nm chip. Starting at $899. What you can run on the $899 Mac Mini (32GB) 👇 → Qwen 3.6 27B : fits comfortably, near-GPT-4 quality → Qwen 3 Coder 32B : full coding assistant on your desk → Llama 4 8B at full precision → DeepSeek R1 32B for reasoning tasks → 13.5x faster LLM processing than M1. 4x faster than M4. What you can run on the Mac Studio with M5 Ultra (512GB) 👇 → Llama 4 70B at high quality quantization → DeepSeek V4 with hundreds of billions of parameters → Frontier-scale models entirely in local memory → Connect four Mac Studios over Thunderbolt 5 cluster them to run trillion-parameter models locally Apple said their Mac business grew 30% last quarter largely because people are buying these to run AI models on their own desks instead of paying for cloud tokens. The future of inference is local. Apple built the hardware for it.
0:38

Apple is no longer hiding it. These machines are built for local AI. Apple unveiled the new Mac Mini with M6 their first 2nm chip. Starting at $899. What you can run on the $899 Mac Mini (32GB) 👇 → Qwen 3.6 27B : fits comfortably, near-GPT-4 quality → Qwen 3 Coder 32B : full coding assistant on your desk → Llama 4 8B at full precision → DeepSeek R1 32B for reasoning tasks → 13.5x faster LLM processing than M1. 4x faster than M4. What you can run on the Mac Studio with M5 Ultra (512GB) 👇 → Llama 4 70B at high quality quantization → DeepSeek V4 with hundreds of billions of parameters → Frontier-scale models entirely in local memory → Connect four Mac Studios over Thunderbolt 5 cluster them to run trillion-parameter models locally Apple said their Mac business grew 30% last quarter largely because people are buying these to run AI models on their own desks instead of paying for cloud tokens. The future of inference is local. Apple built the hardware for it.

Vaibhav Sisinty

514,430 Aufrufe • vor 29 Tagen

THIS DEVELOPER RAN THE LARGEST AI MODEL IN THE WORLD ON 5 MAC STUDIOS - AND IT COST 100X LESS THAN WHAT OPENAI USES 27:47 he says it after hours of setup - Llama 3.1 405B running locally on five Mac Studios - a model that normally requires 42 GPUs at $1,600 each or an entire data center 320GB unified memory combined - every Mac Studio adds its 64GB into one shared pool - and a model that needs 1TB of VRAM just distributes across all five it launched, it responded and not a single token went to OpenAI or Google - everything local, everything private, everything yours and 100x cheaper 5 Mac Studios, 46 watts for all of them in idle - less than one light bulb per machine what OpenAI keeps behind a subscription he ran at home with a one-time purchase in a few clicks
34:56

THIS DEVELOPER RAN THE LARGEST AI MODEL IN THE WORLD ON 5 MAC STUDIOS - AND IT COST 100X LESS THAN WHAT OPENAI USES 27:47 he says it after hours of setup - Llama 3.1 405B running locally on five Mac Studios - a model that normally requires 42 GPUs at $1,600 each or an entire data center 320GB unified memory combined - every Mac Studio adds its 64GB into one shared pool - and a model that needs 1TB of VRAM just distributes across all five it launched, it responded and not a single token went to OpenAI or Google - everything local, everything private, everything yours and 100x cheaper 5 Mac Studios, 46 watts for all of them in idle - less than one light bulb per machine what OpenAI keeps behind a subscription he ran at home with a one-time purchase in a few clicks

Noisy

21,553 Aufrufe • vor 3 Monaten

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓
1:21

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

slash1s

101,317 Aufrufe • vor 3 Monaten

APPLE BUILT A $10,000 COMPUTER THAT RUNS WHAT USED TO REQUIRE AN ENTIRE SERVER ROOM 06:40 he shows the number that breaks everything - one Mac Studio is more powerful than four Dell or AMD machines combined and uses half the energy DeepSeek R1 doesn't run on even two competitors - you need four - one Mac Studio does it alone runs quieter than a fan in the room and uses less energy than a kettle in the kitchen four Mac Studios on a desk connected through a regular cable run a 1 trillion parameter model - what used to require an entire data center anyone can put this at home and Jeff Geerling proved it - run the same models that big companies use without cloud, without subscriptions, without data leaving the room
14:24

APPLE BUILT A $10,000 COMPUTER THAT RUNS WHAT USED TO REQUIRE AN ENTIRE SERVER ROOM 06:40 he shows the number that breaks everything - one Mac Studio is more powerful than four Dell or AMD machines combined and uses half the energy DeepSeek R1 doesn't run on even two competitors - you need four - one Mac Studio does it alone runs quieter than a fan in the room and uses less energy than a kettle in the kitchen four Mac Studios on a desk connected through a regular cable run a 1 trillion parameter model - what used to require an entire data center anyone can put this at home and Jeff Geerling proved it - run the same models that big companies use without cloud, without subscriptions, without data leaving the room

Noisy

26,305 Aufrufe • vor 2 Monaten

ONE BUILDER WIRED 4 RASPBERRY PI MODULES INTO A €450 AI CLUSTER WITH 64GB OF RAM AND KILLED HIS $200/MONTH CHATGPT SUBSCRIPTION 00:37 he points at the terminal, "loaded, loaded, loaded - 4 nodes synced over gigabit, 64 gigabytes total, llama 3.1 partitioned across the cluster" he stacked 4 raspberry pi CM5 lite modules with 16 gigabytes each onto a sipeed nanocluster board, the whole rig runs off a single 65 watt power supply and a gigabit internal switch distributed llama partitions the model across the nodes with synchronized workloads handled over the internal fabric, no quantization needed, the cluster hits about 30 tokens per second on small models a heavy developer pays $200 a month for chatgpt pro and another $200 for claude code, this rig cost €450 in parts and breaks even on the stack in month 2 most people will keep paying anthropic and openai forever, a few will spend a weekend wiring 4 raspberry pi modules and never see a subscription invoice again the window is open, follow and bookmark before it closes
0:37

ONE BUILDER WIRED 4 RASPBERRY PI MODULES INTO A €450 AI CLUSTER WITH 64GB OF RAM AND KILLED HIS $200/MONTH CHATGPT SUBSCRIPTION 00:37 he points at the terminal, "loaded, loaded, loaded - 4 nodes synced over gigabit, 64 gigabytes total, llama 3.1 partitioned across the cluster" he stacked 4 raspberry pi CM5 lite modules with 16 gigabytes each onto a sipeed nanocluster board, the whole rig runs off a single 65 watt power supply and a gigabit internal switch distributed llama partitions the model across the nodes with synchronized workloads handled over the internal fabric, no quantization needed, the cluster hits about 30 tokens per second on small models a heavy developer pays $200 a month for chatgpt pro and another $200 for claude code, this rig cost €450 in parts and breaks even on the stack in month 2 most people will keep paying anthropic and openai forever, a few will spend a weekend wiring 4 raspberry pi modules and never see a subscription invoice again the window is open, follow and bookmark before it closes

Antid

64,291 Aufrufe • vor 3 Monaten

INTEL JUST SHIPPED A WORKSTATION CARD WITH TWO GPUs ON ONE PCB AND 48GB OF VRAM. FOUR CARDS GIVE A SINGLE MOTHERBOARD 192GB OF POOLED INFERENCE MEMORY FOR THE PRICE OF TWO RTX 5090s 00:14 he holds one card up to the camera, two GPU dies side by side under the cooler, each one running its own x8 PCIe lane back to the chipset the cluster sees eight discrete accelerators in software. intel's Battle Matrix stack shards a model across all eight, so a 235B parameter network loads in slices and answers requests in parallel what 192GB of VRAM unlocks: an entire 200B class model in memory without quantization. a vision agent reading 100 invoices at once. a research box that holds three frontier models loaded simultaneously, switching between them in under a second intel is the slow side of inference. nvidia is faster per token, that is the honest tradeoff. but the only other path to this much VRAM is a $40,000 nvidia rack or three networked Mac Studios four B60 cards plus the chassis lands under $5,000. power draw averages 800 watts, $55 a month in electricity. one engineer paying $400 a month for combined ChatGPT Pro and Claude Code Max pays the hardware off in less than a year
1:16

INTEL JUST SHIPPED A WORKSTATION CARD WITH TWO GPUs ON ONE PCB AND 48GB OF VRAM. FOUR CARDS GIVE A SINGLE MOTHERBOARD 192GB OF POOLED INFERENCE MEMORY FOR THE PRICE OF TWO RTX 5090s 00:14 he holds one card up to the camera, two GPU dies side by side under the cooler, each one running its own x8 PCIe lane back to the chipset the cluster sees eight discrete accelerators in software. intel's Battle Matrix stack shards a model across all eight, so a 235B parameter network loads in slices and answers requests in parallel what 192GB of VRAM unlocks: an entire 200B class model in memory without quantization. a vision agent reading 100 invoices at once. a research box that holds three frontier models loaded simultaneously, switching between them in under a second intel is the slow side of inference. nvidia is faster per token, that is the honest tradeoff. but the only other path to this much VRAM is a $40,000 nvidia rack or three networked Mac Studios four B60 cards plus the chassis lands under $5,000. power draw averages 800 watts, $55 a month in electricity. one engineer paying $400 a month for combined ChatGPT Pro and Claude Code Max pays the hardware off in less than a year

NO1ennn

48,247 Aufrufe • vor 2 Monaten

ANTHROPIC LEAKED 6 LOOPS THAT MAKE THE WORK 10X FASTER AND 66% CHEAPER THAN CHAT Anthropic engineers make $1.2M a year not with a faster model - they just removed themselves from the process. contract → parallel workers → self-verify → stable prefix → routing → cron and report 10x isn't a faster model, it's the pauses removed - in chat you're in every turn, here you're in one point. 66% isn't a discount, it's the context you stopped buying a second time. the contract is the only loop that wins on both axes - four clarifying questions are four turns you paid for and got nothing from. parallel workers save nothing on the bill - you pay the latency of the slowest lane, not the sum of all four. self-verify catches the mistake before the expensive turn - a cheap lap always beats a redo in the evening. reorder the prompt once and its whole head stops being recharged - cache hits instead of full price on every turn. the cheapest seat answers in milliseconds and settles most of the queue - small work never reaches the big model. the hours you were asleep were free capacity - one report waiting at 08:00 and nobody pressed anything. save this and paste it into Claude Code - and your pauses stop costing money ↓
0:46

ANTHROPIC LEAKED 6 LOOPS THAT MAKE THE WORK 10X FASTER AND 66% CHEAPER THAN CHAT Anthropic engineers make $1.2M a year not with a faster model - they just removed themselves from the process. contract → parallel workers → self-verify → stable prefix → routing → cron and report 10x isn't a faster model, it's the pauses removed - in chat you're in every turn, here you're in one point. 66% isn't a discount, it's the context you stopped buying a second time. the contract is the only loop that wins on both axes - four clarifying questions are four turns you paid for and got nothing from. parallel workers save nothing on the bill - you pay the latency of the slowest lane, not the sum of all four. self-verify catches the mistake before the expensive turn - a cheap lap always beats a redo in the evening. reorder the prompt once and its whole head stops being recharged - cache hits instead of full price on every turn. the cheapest seat answers in milliseconds and settles most of the queue - small work never reaches the big model. the hours you were asleep were free capacity - one report waiting at 08:00 and nobody pressed anything. save this and paste it into Claude Code - and your pauses stop costing money ↓

Sprytix

42,918 Aufrufe • vor 1 Monat

Qwen 3.6 35B running at over 100 tokens per second on my $5,399 MacBook Pro M5 Max. This is the best local AI model I have ever run. 128GB of unified memory. No cloud. No API costs. No rate limits. Just raw local inference at speeds I didn't think were possible on a laptop. This model is more intelligent than GPT 5 on benchmarks. Running locally. On a MacBook. For free after the hardware cost. I said local AI would never compete with frontier. I'm starting to rethink that. The gap is closing faster than anyone expected.
1:04

Qwen 3.6 35B running at over 100 tokens per second on my $5,399 MacBook Pro M5 Max. This is the best local AI model I have ever run. 128GB of unified memory. No cloud. No API costs. No rate limits. Just raw local inference at speeds I didn't think were possible on a laptop. This model is more intelligent than GPT 5 on benchmarks. Running locally. On a MacBook. For free after the hardware cost. I said local AI would never compete with frontier. I'm starting to rethink that. The gap is closing faster than anyone expected.

BridgeMind

62,800 Aufrufe • vor 4 Monaten

CHINESE AI TEACHER STACKED 4 NVIDIA DGX SPARK CHIPS INTO A CLUSTER - AND NOW RUNS A BUSINESS ON IT FROM HIS DESK 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else he's running a Qwen 32 35B model with 4 concurrent requests at the same time and the speed barely drops compared to a single request what used to require a $10,000+/month cloud setup now sits on his desk and costs him $40-60/month in electricity 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else the people still paying per token are financing someone else's infrastructure
1:42

CHINESE AI TEACHER STACKED 4 NVIDIA DGX SPARK CHIPS INTO A CLUSTER - AND NOW RUNS A BUSINESS ON IT FROM HIS DESK 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else he's running a Qwen 32 35B model with 4 concurrent requests at the same time and the speed barely drops compared to a single request what used to require a $10,000+/month cloud setup now sits on his desk and costs him $40-60/month in electricity 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else the people still paying per token are financing someone else's infrastructure

Sprytix

51,132 Aufrufe • vor 3 Monaten

NVIDIA QUIETLY DROPPED A $249 BOX THAT REPLACES YOUR $200/MONTH OPENAI SUBSCRIPTION WITH $2 IN ELECTRICITY it's called the jetson orin nano super. smaller than a wallet, runs at 25 watts, does 70 trillion ai operations per second. runs llama 3, mistral, gemma and deepseek locally with no api fees and no data leaving your house a developer running automations and coding assistants pays $200 a month to openai. the same workload on this box costs $2 a month in electricity and breaks even in 10 weeks install ollama with one command. change one line in your code. point it at localhost instead of openai. everything else works identically 7 billion parameter models handle 80% of what people use chatgpt for. summarization, drafting, coding, document q&a, automation pipelines. total monthly cost drops from $200 to $22 cloud subscriptions keep getting more expensive and rate limits keep getting tighter. the people who set this up in 2025 are going to look very smart in 2027 bookmark this and read the article below
0:37

NVIDIA QUIETLY DROPPED A $249 BOX THAT REPLACES YOUR $200/MONTH OPENAI SUBSCRIPTION WITH $2 IN ELECTRICITY it's called the jetson orin nano super. smaller than a wallet, runs at 25 watts, does 70 trillion ai operations per second. runs llama 3, mistral, gemma and deepseek locally with no api fees and no data leaving your house a developer running automations and coding assistants pays $200 a month to openai. the same workload on this box costs $2 a month in electricity and breaks even in 10 weeks install ollama with one command. change one line in your code. point it at localhost instead of openai. everything else works identically 7 billion parameter models handle 80% of what people use chatgpt for. summarization, drafting, coding, document q&a, automation pipelines. total monthly cost drops from $200 to $22 cloud subscriptions keep getting more expensive and rate limits keep getting tighter. the people who set this up in 2025 are going to look very smart in 2027 bookmark this and read the article below

starmex

2,359,206 Aufrufe • vor 3 Monaten

THE CHINESE INSTALLED 6 AI AGENTS RUNNING 24/7 FOR $11 A MONTH. THE BOX UNDER ONE DESK REPLACED $1,200 IN CLOUD SUBSCRIPTIONS. The Minisforum MS-S1 Max ships with 128GB of unified memory shared between CPU and GPU. Qwen3-Coder 30B runs at 40 to 50 tokens per second. 6 agents run simultaneously without anyone watching the meter. An inbox sorter drafts replies every 15 minutes. A research monitor watches 30+ feeds and lands a digest by 7 AM, a job that costs $15 to $20 a day on cloud APIs. The same stack on cloud subscriptions ran $800 to $1,200 a month. The hardware costs $3,000 once. Break-even hits month 3 to 4. The real shift is behavior, not savings. A meeting prep agent never would have justified per-token cost. Running it free overnight made it obvious.
0:53

THE CHINESE INSTALLED 6 AI AGENTS RUNNING 24/7 FOR $11 A MONTH. THE BOX UNDER ONE DESK REPLACED $1,200 IN CLOUD SUBSCRIPTIONS. The Minisforum MS-S1 Max ships with 128GB of unified memory shared between CPU and GPU. Qwen3-Coder 30B runs at 40 to 50 tokens per second. 6 agents run simultaneously without anyone watching the meter. An inbox sorter drafts replies every 15 minutes. A research monitor watches 30+ feeds and lands a digest by 7 AM, a job that costs $15 to $20 a day on cloud APIs. The same stack on cloud subscriptions ran $800 to $1,200 a month. The hardware costs $3,000 once. Break-even hits month 3 to 4. The real shift is behavior, not savings. A meeting prep agent never would have justified per-token cost. Running it free overnight made it obvious.

Lummox

14,093 Aufrufe • vor 3 Monaten

i'm running a 397 billion parameter model on a amd ai max box that sits on my desk and pulls less power than a gaming laptop. the model is Nex-N2-Pro, 397B-A17B, the open weight release people are putting next to gpt-5.5 on coding. i have it quantized to IQ1_M, 1.75 bits per weight, 90gb of weights loaded into the 128gb of unified memory on amd's strix halo igpu. watch the gpu in this recording. it spikes, it sustains, it does not fall over. that is the part the spec sheets never show you, not just that a 400b model loads, but that an integrated graphics chip holds the load and generates token after token, stable, no crash, no thermal cliff. and it is not a slideshow. roughly 18 tokens a second, faster than you can read. a frontier scale model producing usable output, fully local. no datacenter, no rented h100s, no api key, no permission. three years ago a model this size meant a server room and a budget to match. tonight it is a quiet box on my desk. this is the accessible tier almost nobody benchmarks honestly, and it is further along than the timeline thinks. the full breakdown is coming, rocm vs vulkan on this chip, and this little amd box head to head against the nvidia equivalent. stay tuned.
2:18

i'm running a 397 billion parameter model on a amd ai max box that sits on my desk and pulls less power than a gaming laptop. the model is Nex-N2-Pro, 397B-A17B, the open weight release people are putting next to gpt-5.5 on coding. i have it quantized to IQ1_M, 1.75 bits per weight, 90gb of weights loaded into the 128gb of unified memory on amd's strix halo igpu. watch the gpu in this recording. it spikes, it sustains, it does not fall over. that is the part the spec sheets never show you, not just that a 400b model loads, but that an integrated graphics chip holds the load and generates token after token, stable, no crash, no thermal cliff. and it is not a slideshow. roughly 18 tokens a second, faster than you can read. a frontier scale model producing usable output, fully local. no datacenter, no rented h100s, no api key, no permission. three years ago a model this size meant a server room and a budget to match. tonight it is a quiet box on my desk. this is the accessible tier almost nobody benchmarks honestly, and it is further along than the timeline thinks. the full breakdown is coming, rocm vs vulkan on this chip, and this little amd box head to head against the nvidia equivalent. stay tuned.

Sudo su

32,163 Aufrufe • vor 3 Monaten

JENSEN HUANG SOLD A $249 AI COMPUTER ON STAGE THAT KILLS YOUR $200/MONTH OPENAI BILL. THE VIDEO HAS 217,000 LIKES the box is called the jetson orin nano super. 70 trillion ai operations per second, 25 watts, smaller than a wallet. it runs llama 3, mistral, gemma and deepseek locally with no api fees and no data leaving your house a developer running automations and coding assistants pays $200 a month to openai. the same workload on this box costs $2 a month in electricity and breaks even on the hardware in 10 weeks you install ollama with one command. change one line in your code. point it at localhost instead of openai. everything else works identically 7 billion parameter models handle 80% of what people use chatgpt for. summarization, drafting, coding, document q&a, automation pipelines. the hard 20% you keep claude or gpt for. total monthly cost drops from $200 to $22 cloud subscriptions keep getting more expensive and rate limits keep getting tighter. the people who set this up in 2025 are going to look very smart in 2027 bookmark this and read the article below
1:18

JENSEN HUANG SOLD A $249 AI COMPUTER ON STAGE THAT KILLS YOUR $200/MONTH OPENAI BILL. THE VIDEO HAS 217,000 LIKES the box is called the jetson orin nano super. 70 trillion ai operations per second, 25 watts, smaller than a wallet. it runs llama 3, mistral, gemma and deepseek locally with no api fees and no data leaving your house a developer running automations and coding assistants pays $200 a month to openai. the same workload on this box costs $2 a month in electricity and breaks even on the hardware in 10 weeks you install ollama with one command. change one line in your code. point it at localhost instead of openai. everything else works identically 7 billion parameter models handle 80% of what people use chatgpt for. summarization, drafting, coding, document q&a, automation pipelines. the hard 20% you keep claude or gpt for. total monthly cost drops from $200 to $22 cloud subscriptions keep getting more expensive and rate limits keep getting tighter. the people who set this up in 2025 are going to look very smart in 2027 bookmark this and read the article below

starmex

2,000,811 Aufrufe • vor 4 Monaten

Karpathy told Dwarkesh that a 1 billion parameter model, trained on clean data, could hit the intelligence of today's 1.8 trillion parameter frontier. That is a 1,800x compression claim. The math behind it is more defensible than it sounds. When researchers at frontier labs look at random samples from their training corpus, they see stock ticker symbols, broken HTML, forum spam, autogenerated gibberish. Not Wikipedia. Not the Wall Street Journal. The actual pretraining dataset is mostly noise, and the model is burning parameters to vaguely remember all of it. One estimate pegs Llama 3's information compression at 0.07 bits per token. Well-structured English carries around 1.5 bits per token of real information. The trillion-parameter model is holding a roughly 5% resolution image of the internet it trained on. So when a lab ships a 1.8 trillion parameter model, the overwhelming majority of those weights are handling rough memorization. They are compression overhead for a noisy training set, taking up capacity that could be doing reasoning instead. Karpathy's proposal is to separate the two. Build a cognitive core: a small model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization. Pair it with external memory the model queries when it needs a fact. A 1 billion parameter reasoner plus retrieval beats a 1.8 trillion parameter model trying to do both. The data already supports this direction. GPT-4o runs at roughly 200 billion parameters and outperforms the original 1.8 trillion GPT-4. Inference costs for GPT-3.5 level performance fell 280x between 2022 and 2024, driven almost entirely by smaller, cleaner, better-architected models. The trend line is pointing where Karpathy says it should. The real implication for anyone tracking the AI trade: data quality is the actual constraint. The companies winning the next phase will be the ones who figured out what to train on, and what to throw away.
2:00

Karpathy told Dwarkesh that a 1 billion parameter model, trained on clean data, could hit the intelligence of today's 1.8 trillion parameter frontier. That is a 1,800x compression claim. The math behind it is more defensible than it sounds. When researchers at frontier labs look at random samples from their training corpus, they see stock ticker symbols, broken HTML, forum spam, autogenerated gibberish. Not Wikipedia. Not the Wall Street Journal. The actual pretraining dataset is mostly noise, and the model is burning parameters to vaguely remember all of it. One estimate pegs Llama 3's information compression at 0.07 bits per token. Well-structured English carries around 1.5 bits per token of real information. The trillion-parameter model is holding a roughly 5% resolution image of the internet it trained on. So when a lab ships a 1.8 trillion parameter model, the overwhelming majority of those weights are handling rough memorization. They are compression overhead for a noisy training set, taking up capacity that could be doing reasoning instead. Karpathy's proposal is to separate the two. Build a cognitive core: a small model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization. Pair it with external memory the model queries when it needs a fact. A 1 billion parameter reasoner plus retrieval beats a 1.8 trillion parameter model trying to do both. The data already supports this direction. GPT-4o runs at roughly 200 billion parameters and outperforms the original 1.8 trillion GPT-4. Inference costs for GPT-3.5 level performance fell 280x between 2022 and 2024, driven almost entirely by smaller, cleaner, better-architected models. The trend line is pointing where Karpathy says it should. The real implication for anyone tracking the AI trade: data quality is the actual constraint. The companies winning the next phase will be the ones who figured out what to train on, and what to throw away.

Aakash Gupta

508,540 Aufrufe • vor 5 Monaten

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.
0:42

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

343,270 Aufrufe • vor 1 Monat

AMD CEO LISA SU HELD A MINI PC ON STAGE THAT RUNS A 235B MODEL AND REPLACES YOUR $440/MONTH AI STACK amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on one piece of silicon. cpu and gpu share 128gb of unified memory, no separate graphics card needed the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months install ollama, pull the model, point claude code at localhost. same interface, nothing leaves the machine, nothing costs per request bookmark this and read the article below
0:59

AMD CEO LISA SU HELD A MINI PC ON STAGE THAT RUNS A 235B MODEL AND REPLACES YOUR $440/MONTH AI STACK amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on one piece of silicon. cpu and gpu share 128gb of unified memory, no separate graphics card needed the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months install ollama, pull the model, point claude code at localhost. same interface, nothing leaves the machine, nothing costs per request bookmark this and read the article below

starmex

981,107 Aufrufe • vor 3 Monaten

🚨 do you understand what happened to Claude.. Anthropic just shipped Opus 4.8 with something called dynamic workflows. It no longer works alone - it spins up hundreds of agents that argue, verify, and break each other's work until the answer is right. - A dev used it to port Bun from Zig to Rust in 11 days - 750,000 lines of Rust, 99.8 percent of tests passing - Hundreds of agents ran in parallel with two reviewers per file - Fast mode runs the same model 2.5x faster and 3x cheaper than before work you used to plan in quarters now finishes before the weekend
1:21

🚨 do you understand what happened to Claude.. Anthropic just shipped Opus 4.8 with something called dynamic workflows. It no longer works alone - it spins up hundreds of agents that argue, verify, and break each other's work until the answer is right. - A dev used it to port Bun from Zig to Rust in 11 days - 750,000 lines of Rust, 99.8 percent of tests passing - Hundreds of agents ran in parallel with two reviewers per file - Fast mode runs the same model 2.5x faster and 3x cheaper than before work you used to plan in quarters now finishes before the weekend

BuBBliK

431,834 Aufrufe • vor 3 Monaten