Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

M4 Mac AI Coding Cluster Uses EXO Labs to run LLMs (here Qwen 2.5 Coder 32B at 18 tok/sec) distributed across 4 M4 Mac Minis (Thunderbolt 5 80Gbps) and a MacBook Pro M4 Max. Local alternative to Cursor (benchmark comparison soon).

alexocheema's profile picture

Alex Cheema

52,295 subscribers

517,686 Aufrufe • vor 1 Jahr •via X (Twitter)

GamingWissenschaft & TechnologieBildung
LIVE

Anya Rossi• Live Now

Private livecam show

10 Kommentare

Profilbild von Alex Cheema - e/acc
Alex Cheema - e/accvor 1 Jahr

The Open-Source AI code editor is @zeddotdev with the @exolabs cluster set as a custom endpoint. exo is open source:

Profilbild von Alex Cheema - e/acc
Alex Cheema - e/accvor 1 Jahr

@exolabs Benchmarks for this setup and more will be released in the open!

Profilbild von ELLIE X
ELLIE Xvor 1 Jahr

@exolabs @cursor_ai digital sovereignty tool kit: handful of mac minis, 32 restaked ETH, 1BTC, starlink mini, 800w solar, 800ah batteries (working towards the first three still ;)

Profilbild von Alex Cheema - e/acc
Alex Cheema - e/accvor 1 Jahr

@exolabs @cursor_ai yo lets set this up. we can take it to the dessert.

Profilbild von Reza Sayar
Reza Sayarvor 1 Jahr

@exolabs @cursor_ai What are the specs here? Do we need 5 devices to run this medium sized model? @ivanfioravanti is running it on a single M4 Max MacBook Pro with Q4 MLX using like 18GBs of RAM :

Profilbild von Charlie Greenman
Charlie Greenmanvor 1 Jahr

@exolabs @cursor_ai that's cool. something about this just seems aesthetically right. Like this is how building/training AI locally is supposed to work. Next up, training AI

Profilbild von Firworks
Firworksvor 1 Jahr

@exolabs @cursor_ai How is a cluster like this effected by the "slowest gazzelle"? If I buy a 64GB M4 Pro Mini and link it up to my wife's 24GB M3 Air to be able to fit a larger model will the performance fall off a cliff? It seems like memory size/bandwidth are the primary drivers here not FLOPS?

Profilbild von Alex Cheema - e/acc
Alex Cheema - e/accvor 1 Jahr

@exolabs @cursor_ai Depends on the networking. Over TB5 it should still be a speedup since we use Tensor Parallelism (which is not merged yet, pending PR).

Profilbild von Ben Fleming
Ben Flemingvor 1 Jahr

@exolabs @cursor_ai so fire… m4 Mac minis pack a punch for the size! how many for llama 405B at reasonable speed?

Profilbild von Imrat
Imratvor 1 Jahr

@exolabs @cursor_ai does it scale up linearly? eg if i wanted to double the tokens/s does it require double the mac 4mini;s? And whats the bottleneck on each node at the moment? The arm CPUs or memory?

Ähnliche Videos

Running GLM-4.7-Flash on 4 x M4 Pro Mac Minis using EXO Labs. Uses tensor parallelism with RDMA over Thunderbolt & MLX backend (h/t Awni Hannun). Runs at 100 tok/sec. We're working on optimizing this at EXO Labs. Aiming to hit ~200 tok/sec on this setup soon.
0:22

Running GLM-4.7-Flash on 4 x M4 Pro Mac Minis using EXO Labs. Uses tensor parallelism with RDMA over Thunderbolt & MLX backend (h/t Awni Hannun). Runs at 100 tok/sec. We're working on optimizing this at EXO Labs. Aiming to hit ~200 tok/sec on this setup soon.

Alex Cheema

62,221 Aufrufe • vor 6 Monaten

Qwen 2.5 Coder Q4 M4 Max Inference test. Apple MLX vs Ollama: - MLX: 23.97 toks/sec 🥇🔥 - Ollama: 18.33 toks/sec 🥈 Here a video to show results
1:00

Qwen 2.5 Coder Q4 M4 Max Inference test. Apple MLX vs Ollama: - MLX: 23.97 toks/sec 🥇🔥 - Ollama: 18.33 toks/sec 🥈 Here a video to show results

Ivan Fioravanti ᯅ

34,881 Aufrufe • vor 1 Jahr

THIS GUY STACKED MAC MINIS TO RUN AI LOCALLY AND IT CODES FOR FREE no API. no subscription. no cloud. what he's running: - EXO across multiple Mac Minis - Qwen 2.5 Coder 7B locally - Zed editor connected to the model - GPU usage visible in real time the math: - Mac Mini M4: $599 one-time - electricity: $3/month - API costs: $0 vs $459/month in subscriptions if you want Claude Code to run at this level ClaudeKit gives it the structure to do that ( full breakdown in the article below
0:55

THIS GUY STACKED MAC MINIS TO RUN AI LOCALLY AND IT CODES FOR FREE no API. no subscription. no cloud. what he's running: - EXO across multiple Mac Minis - Qwen 2.5 Coder 7B locally - Zed editor connected to the model - GPU usage visible in real time the math: - Mac Mini M4: $599 one-time - electricity: $3/month - API costs: $0 vs $459/month in subscriptions if you want Claude Code to run at this level ClaudeKit gives it the structure to do that ( full breakdown in the article below

Madni Aghadi

26,421 Aufrufe • vor 1 Monat

Running GLM-4.7-Flash with OpenCode locally on M4 Max MacBook Pro. 4-bit model runs at 82 tok/sec. Prefill will get ~4x faster with M5 Max MacBook Pro (~28 Jan). EXO will also support disaggregating prefill and decode across devices, e.g. DGX Spark.
0:43

Running GLM-4.7-Flash with OpenCode locally on M4 Max MacBook Pro. 4-bit model runs at 82 tok/sec. Prefill will get ~4x faster with M5 Max MacBook Pro (~28 Jan). EXO will also support disaggregating prefill and decode across devices, e.g. DGX Spark.

Alex Cheema

128,404 Aufrufe • vor 6 Monaten

Qwen 32B (4-bit) generates at >40 toks/sec on an M4 Max with assisted decoding and Qwen 0.5B as the draft model. Coming soon to mlx-lm. Compare regular decoding (left) to assisted decoding (right):
0:23

Qwen 32B (4-bit) generates at >40 toks/sec on an M4 Max with assisted decoding and Qwen 0.5B as the draft model. Coming soon to mlx-lm. Compare regular decoding (left) to assisted decoding (right):

Awni Hannun

50,353 Aufrufe • vor 1 Jahr

I bought the M4 Max 48GB RAM and first thing I did is to test local LLMs. It can barely run llama3.3 70b, but qwen2.5-coder 32b works very well. I'll test with MLX next ✌️
0:31

I bought the M4 Max 48GB RAM and first thing I did is to test local LLMs. It can barely run llama3.3 70b, but qwen2.5-coder 32b works very well. I'll test with MLX next ✌️

Daniel Nguyen

315,827 Aufrufe • vor 1 Jahr

What does it take to run 3, 5, or even 10 concurrent instances of Gemma 4 locally? We've open-sourced a demo letting you run multiple models side-by-side on your hardware. Gemma 4 26B A4B easily runs 10+ concurrent requests on a MacBook Pro M4 Max at 18 tokens/sec per request.
0:48

What does it take to run 3, 5, or even 10 concurrent instances of Gemma 4 locally? We've open-sourced a demo letting you run multiple models side-by-side on your hardware. Gemma 4 26B A4B easily runs 10+ concurrent requests on a MacBook Pro M4 Max at 18 tokens/sec per request.

Google Gemma

913,685 Aufrufe • vor 3 Monaten

Apple CEO : "Apple just made the Mac Mini small enough to disappear on your desk, but powerful enough to become your AI workstation." The new M4 Mac Mini starts at $599, comes with Apple Silicon, and pushes desktop compute into a box most people could hide behind a monitor. It is the next cheap entry point for local AI, automation, coding, content, and private workflows. Mac Mini + M4 + local models + low power = the new desk setup. Worth more than another $500 productivity course.
5:32

Apple CEO : "Apple just made the Mac Mini small enough to disappear on your desk, but powerful enough to become your AI workstation." The new M4 Mac Mini starts at $599, comes with Apple Silicon, and pushes desktop compute into a box most people could hide behind a monitor. It is the next cheap entry point for local AI, automation, coding, content, and private workflows. Mac Mini + M4 + local models + low power = the new desk setup. Worth more than another $500 productivity course.

Lummox

262,119 Aufrufe • vor 1 Monat

A 23-YEARS-OLD MADE $22,500 USING MAC MINI M4 THAT CUT DEVELOPER AI CREDITS BY 80 PERCENT A typical developer spending $150 to $200 a month on Claude or OpenAI API credits can route routine work to local models and keep cloud models for hard reasoning. The stack is simple : Mac Mini M4, Ollama, LiteLLM, Open WebUI, Qwen, Llama, Gemma or Phi. The math is the hook. Electricity costs $3 to $5 a month, while cron jobs, summaries, formatting, code reviews and draft emails stop burning paid tokens. The edge is not replacing Claude. It is using Claude only where Claude actually matters.
0:33

A 23-YEARS-OLD MADE $22,500 USING MAC MINI M4 THAT CUT DEVELOPER AI CREDITS BY 80 PERCENT A typical developer spending $150 to $200 a month on Claude or OpenAI API credits can route routine work to local models and keep cloud models for hard reasoning. The stack is simple : Mac Mini M4, Ollama, LiteLLM, Open WebUI, Qwen, Llama, Gemma or Phi. The math is the hook. Electricity costs $3 to $5 a month, while cron jobs, summaries, formatting, code reviews and draft emails stop burning paid tokens. The edge is not replacing Claude. It is using Claude only where Claude actually matters.

BeingInvested

11,805 Aufrufe • vor 1 Monat

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization, it requires ~700GB of memory to run the model. EXO puts half of the layers on each device, combining their memory. EXO uses MLX distributed with TB5 interconnect, optimized for Apple Silicon. If we need higher throughput, adding two more devices lets us serve more users at once. EXO Labs handles all of this seamlessly - adding more devices to the cluster for linear scaling as we need it. The new EXO 1.0 will be open-source soonTM
0:29

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization, it requires ~700GB of memory to run the model. EXO puts half of the layers on each device, combining their memory. EXO uses MLX distributed with TB5 interconnect, optimized for Apple Silicon. If we need higher throughput, adding two more devices lets us serve more users at once. EXO Labs handles all of this seamlessly - adding more devices to the cluster for linear scaling as we need it. The new EXO 1.0 will be open-source soonTM

Matt Beton

158,485 Aufrufe • vor 11 Monaten

a developer posted his Claude Code bill on Reddit: $170 in 10 days. someone replied: "I bought a Mac Mini M4. Haven't paid Anthropic since." most people scrolled past that reply. the dev community didn't. Apple Stores ran out of Mac Minis the next month. not from launch hype from devs running the math. the math: / Claude Code Max + ChatGPT Pro + Cursor + Copilot = $459/month, $5,508/year / one $599 Mac Mini M4 runs the same workflows locally / one env var (ANTHROPIC_BASE_URL= redirects Claude Code to your hardware / $3/month in electricity. one box. silent. 5-inch square. the workflow stays identical. the bill stops. that's the trade. the developers who already made this swap aren't loud about it. they're billing the same clients on cheaper infrastructure and pocketing the spread.
0:54

a developer posted his Claude Code bill on Reddit: $170 in 10 days. someone replied: "I bought a Mac Mini M4. Haven't paid Anthropic since." most people scrolled past that reply. the dev community didn't. Apple Stores ran out of Mac Minis the next month. not from launch hype from devs running the math. the math: / Claude Code Max + ChatGPT Pro + Cursor + Copilot = $459/month, $5,508/year / one $599 Mac Mini M4 runs the same workflows locally / one env var (ANTHROPIC_BASE_URL= redirects Claude Code to your hardware / $3/month in electricity. one box. silent. 5-inch square. the workflow stays identical. the bill stops. that's the trade. the developers who already made this swap aren't loud about it. they're billing the same clients on cheaper infrastructure and pocketing the spread.

dunik

199,995 Aufrufe • vor 2 Monaten

This guy plugged a DGX Spark (the $3K Nvidia box) and a Mac Mini M4 together to run AI and what happened next surprised everyone > the Nvidia box handles the hard part - processing your prompt in milliseconds > Mac Mini M4 handles the fast part - generating the response at memory bandwidth speeds nothing else can match > together they hit 84 tokens per second on Llama - 6x faster than the Spark alone > running compute agents locally on this setup means your data never leaves your hardware > two boxes. two different architectures. one AI system that Deepseek runs at data center scale he ran it on his desk save this. the way we build local AI is about to change
18:41

This guy plugged a DGX Spark (the $3K Nvidia box) and a Mac Mini M4 together to run AI and what happened next surprised everyone > the Nvidia box handles the hard part - processing your prompt in milliseconds > Mac Mini M4 handles the fast part - generating the response at memory bandwidth speeds nothing else can match > together they hit 84 tokens per second on Llama - 6x faster than the Spark alone > running compute agents locally on this setup means your data never leaves your hardware > two boxes. two different architectures. one AI system that Deepseek runs at data center scale he ran it on his desk save this. the way we build local AI is about to change

Mr. Buzzoni

153,711 Aufrufe • vor 2 Monaten

A GIRL BOUGHT A $599 APPLE BOX AND CUT HER AI COSTS FROM $459/MONTH TO $23/MONTH. THE MAC MINI M4 IS QUIETLY BECOMING THE CHEAPEST AI SETUP IN 2026 she didn’t buy it because it looked nice on her desk. she bought it because paying for claude, chatgpt, cursor and api usage every month was getting ridiculous the setup is simple. mac mini m4, ollama, open webui, and local models like qwen, deepseek and llama. for most daily work, that’s enough to write, code, summarize, search notes, and run private workflows without sending everything to the cloud that’s why the math looks so good. a heavy ai stack can hit $459 a month, or $5,508 a year. the mac mini starts at $599 and uses around $3 a month in electricity. if it handles even 70 to 80% of the workload, it pays for itself fast install ollama, point your tools to localhost, and the workflow changes immediately. no token stress, no rate limits, no wondering where your files are going you still keep one cloud model for the hardest tasks but once a small box on your desk does most of the work, paying full price for everything starts to feel stupid
0:23

A GIRL BOUGHT A $599 APPLE BOX AND CUT HER AI COSTS FROM $459/MONTH TO $23/MONTH. THE MAC MINI M4 IS QUIETLY BECOMING THE CHEAPEST AI SETUP IN 2026 she didn’t buy it because it looked nice on her desk. she bought it because paying for claude, chatgpt, cursor and api usage every month was getting ridiculous the setup is simple. mac mini m4, ollama, open webui, and local models like qwen, deepseek and llama. for most daily work, that’s enough to write, code, summarize, search notes, and run private workflows without sending everything to the cloud that’s why the math looks so good. a heavy ai stack can hit $459 a month, or $5,508 a year. the mac mini starts at $599 and uses around $3 a month in electricity. if it handles even 70 to 80% of the workload, it pays for itself fast install ollama, point your tools to localhost, and the workflow changes immediately. no token stress, no rate limits, no wondering where your files are going you still keep one cloud model for the hardest tasks but once a small box on your desk does most of the work, paying full price for everything starts to feel stupid

Gipp 🦅

457,929 Aufrufe • vor 2 Monaten

Alibaba just released a coding model that hits 82 percent on SWE-Bench Verified. That is the highest score ever published for an open-source model. The weights are free. The license is Apache 2.0. You can run it today. The model is Qwen 4 Coder 32B. Here is what 82 percent on SWE-Bench Verified actually means. SWE-Bench Verified tests whether an AI can autonomously resolve real bugs pulled from real production GitHub repositories. Not synthetic exercises. Real open-source projects that real teams depend on. A model gets a bug report, reads the code, writes a fix, and either passes the test suite or it does not. At 82 percent, Qwen 4 Coder 32B resolves 82 out of every 100 real production bugs it is given. Without a human guiding it. On code it has never seen before. For comparison: Qwen 4 Coder 32B: 82 percent SWE-Bench Verified. Open source. Apache 2.0. Claude Fable 5: 80.3 percent SWE-Bench Pro. $10 input / $50 output per million tokens. Currently suspended. GPT-5.6 Sol: Competitive on Terminal-Bench. $5 input / $30 output per million tokens. An open-weight model that you can download and run for free just beat both of them on the benchmark designed to measure real software engineering capability. Here is the architecture. Qwen 4 Coder 32B is a 32 billion parameter dense model. Not a Mixture-of-Experts. Every parameter is active on every request. This matters for inference: a dense 32B model runs on 22 gigabytes of VRAM, which fits on a single high-end consumer GPU or a MacBook Pro with 64GB of unified memory. The smaller variant, Qwen 4 Coder 4B, runs at approximately 135 tokens per second on an M5 Max and fits inside 8 gigabytes of RAM. For a model with usable coding capability, that is a new bar for what fits in a single laptop. The training methodology continued Alibaba's approach of reinforcement learning on verifiable coding tasks. The model gets rewarded when its code passes tests. It gets penalized when it fails. Over millions of training steps, the model learns to write code that actually runs rather than code that looks plausible. License: Apache 2.0. Full commercial use. No attribution requirement. No revenue threshold. No monthly active user ceiling. Weights: Hugging Face, available today. Runs on: vLLM, Ollama, SGLang, and any standard GGUF-compatible inference engine. Qwen 4 32B also runs at approximately 135 tokens per second on an M5 Max chip, setting a new bar for what a sub-8GB model can do on Apple Silicon. The open-source coding model just beat the best closed-source model in the world on the benchmark designed to test whether AI can actually do software engineering. The weights are free. The subscription is optional. Source: Autom8Labs AI Insight July 2026, State of Open Source LLMs June 2026, Kunal Ganglani blog June 2026.
0:25

Alibaba just released a coding model that hits 82 percent on SWE-Bench Verified. That is the highest score ever published for an open-source model. The weights are free. The license is Apache 2.0. You can run it today. The model is Qwen 4 Coder 32B. Here is what 82 percent on SWE-Bench Verified actually means. SWE-Bench Verified tests whether an AI can autonomously resolve real bugs pulled from real production GitHub repositories. Not synthetic exercises. Real open-source projects that real teams depend on. A model gets a bug report, reads the code, writes a fix, and either passes the test suite or it does not. At 82 percent, Qwen 4 Coder 32B resolves 82 out of every 100 real production bugs it is given. Without a human guiding it. On code it has never seen before. For comparison: Qwen 4 Coder 32B: 82 percent SWE-Bench Verified. Open source. Apache 2.0. Claude Fable 5: 80.3 percent SWE-Bench Pro. $10 input / $50 output per million tokens. Currently suspended. GPT-5.6 Sol: Competitive on Terminal-Bench. $5 input / $30 output per million tokens. An open-weight model that you can download and run for free just beat both of them on the benchmark designed to measure real software engineering capability. Here is the architecture. Qwen 4 Coder 32B is a 32 billion parameter dense model. Not a Mixture-of-Experts. Every parameter is active on every request. This matters for inference: a dense 32B model runs on 22 gigabytes of VRAM, which fits on a single high-end consumer GPU or a MacBook Pro with 64GB of unified memory. The smaller variant, Qwen 4 Coder 4B, runs at approximately 135 tokens per second on an M5 Max and fits inside 8 gigabytes of RAM. For a model with usable coding capability, that is a new bar for what fits in a single laptop. The training methodology continued Alibaba's approach of reinforcement learning on verifiable coding tasks. The model gets rewarded when its code passes tests. It gets penalized when it fails. Over millions of training steps, the model learns to write code that actually runs rather than code that looks plausible. License: Apache 2.0. Full commercial use. No attribution requirement. No revenue threshold. No monthly active user ceiling. Weights: Hugging Face, available today. Runs on: vLLM, Ollama, SGLang, and any standard GGUF-compatible inference engine. Qwen 4 32B also runs at approximately 135 tokens per second on an M5 Max chip, setting a new bar for what a sub-8GB model can do on Apple Silicon. The open-source coding model just beat the best closed-source model in the world on the benchmark designed to test whether AI can actually do software engineering. The weights are free. The subscription is optional. Source: Autom8Labs AI Insight July 2026, State of Open Source LLMs June 2026, Kunal Ganglani blog June 2026.

Harman

41,278 Aufrufe • vor 1 Monat

THE NEXT AI WORKER MAY COST $500 AND SIT BEHIND YOUR MONITOR. Mini PCs are becoming the cheapest path to private AI. A Mac Mini M4 starts at $599, Geekom AX8 Pro starts near $529, and MinisForum pushes up to 128GB RAM for local AI workloads. Ollama turns that hardware into a personal model server. No cloud fee. No data storage bill. No waiting for another SaaS product to decide your limits. Market shift is simple -> computer is moving back to the desk. Owning a mini PC is starting to look less like buying hardware and more like hiring a quiet AI operator.
0:21

THE NEXT AI WORKER MAY COST $500 AND SIT BEHIND YOUR MONITOR. Mini PCs are becoming the cheapest path to private AI. A Mac Mini M4 starts at $599, Geekom AX8 Pro starts near $529, and MinisForum pushes up to 128GB RAM for local AI workloads. Ollama turns that hardware into a personal model server. No cloud fee. No data storage bill. No waiting for another SaaS product to decide your limits. Market shift is simple -> computer is moving back to the desk. Owning a mini PC is starting to look less like buying hardware and more like hiring a quiet AI operator.

Lummox

394,268 Aufrufe • vor 1 Monat

For the first time, the latest LLMs run on the Apple Neural Engine — and NexaSDK is the only framework that makes it possible, powered by the NexaML engine. Last year, our two co-founders were invited by Apple DMLI team (Data & Machine Learning Innovation) to share their research about on-device multimodal model for local AI agents. One of the big questions in the room was: “Can the newest LLMs actually run on ANE?” At the time, nobody had a clear path. Today, that path exists. NexaSDK now runs Granite-4.0 (IBM), Qwen3 (Qwen), Gemma3 (Google), and Parakeet-v3 (NVIDIA) fully on Apple’s NPU — unlocking low-power, always-on, fast inference across Mac and iPhone. A new wave of NPU-first local AI apps is coming to Apple devices. Start with one line of code on Mac. iOS SDK coming soon.
1:42

For the first time, the latest LLMs run on the Apple Neural Engine — and NexaSDK is the only framework that makes it possible, powered by the NexaML engine. Last year, our two co-founders were invited by Apple DMLI team (Data & Machine Learning Innovation) to share their research about on-device multimodal model for local AI agents. One of the big questions in the room was: “Can the newest LLMs actually run on ANE?” At the time, nobody had a clear path. Today, that path exists. NexaSDK now runs Granite-4.0 (IBM), Qwen3 (Qwen), Gemma3 (Google), and Parakeet-v3 (NVIDIA) fully on Apple’s NPU — unlocking low-power, always-on, fast inference across Mac and iPhone. A new wave of NPU-first local AI apps is coming to Apple devices. Start with one line of code on Mac. iOS SDK coming soon.

NEXA AI

30,213 Aufrufe • vor 8 Monaten

Qwen 3.6 35B running at over 100 tokens per second on my $5,399 MacBook Pro M5 Max. This is the best local AI model I have ever run. 128GB of unified memory. No cloud. No API costs. No rate limits. Just raw local inference at speeds I didn't think were possible on a laptop. This model is more intelligent than GPT 5 on benchmarks. Running locally. On a MacBook. For free after the hardware cost. I said local AI would never compete with frontier. I'm starting to rethink that. The gap is closing faster than anyone expected.
1:04

Qwen 3.6 35B running at over 100 tokens per second on my $5,399 MacBook Pro M5 Max. This is the best local AI model I have ever run. 128GB of unified memory. No cloud. No API costs. No rate limits. Just raw local inference at speeds I didn't think were possible on a laptop. This model is more intelligent than GPT 5 on benchmarks. Running locally. On a MacBook. For free after the hardware cost. I said local AI would never compete with frontier. I'm starting to rethink that. The gap is closing faster than anyone expected.

BridgeMind

62,800 Aufrufe • vor 2 Monaten

EVERYONE BUYING A $2,000 MAC MINI M4 PRO FOR LOCAL AI MISSED THAT A SERVER RESELLER IS UNLOADING DEAD HP PROLIANT DL380 GEN 10 BOXES WITH 5 YEAR WARRANTY, A $250 USED TESLA P40 BOLTED IN RUNS LLAMA 3.3 70B AT HOME 00:19 he stands behind a pallet, "these 10 HP Proliant DL380 Gen 10 servers, they're a few generations older, you guys will be surprised who these servers are going to" a DL380 Gen 10 with dual Xeon Gold 6248 and 256GB of DDR4 ECC sells refurbished for $380-500 in 2026, the same chassis cost $14,000 new in 2018 and got written off the books by every fortune 500 by 2024 drop a $250 used Tesla P40 with 24GB VRAM into one PCIe slot, total bill $650, llama 3.3 70B quantized to 4 bit runs at 9 tokens per second locally, the article calls $2,000 mac mini m4 pro the sweet spot, this stack hits it for a third the seller in the video gives a 5 year warranty on hardware that hyperscalers threw away, the same companies the article warns will jack subscription prices are selling you the rack they used to charge $200 a month to access a 2018 enterprise chassis pulls 280 watts at idle and screams at 65 decibels, the mac mini pulls 10 watts and runs silent, the tradeoff is real but $1,350 cheaper buys a lot of electricity and earplugs over 4 years bookmark this and read the article below
1:23

EVERYONE BUYING A $2,000 MAC MINI M4 PRO FOR LOCAL AI MISSED THAT A SERVER RESELLER IS UNLOADING DEAD HP PROLIANT DL380 GEN 10 BOXES WITH 5 YEAR WARRANTY, A $250 USED TESLA P40 BOLTED IN RUNS LLAMA 3.3 70B AT HOME 00:19 he stands behind a pallet, "these 10 HP Proliant DL380 Gen 10 servers, they're a few generations older, you guys will be surprised who these servers are going to" a DL380 Gen 10 with dual Xeon Gold 6248 and 256GB of DDR4 ECC sells refurbished for $380-500 in 2026, the same chassis cost $14,000 new in 2018 and got written off the books by every fortune 500 by 2024 drop a $250 used Tesla P40 with 24GB VRAM into one PCIe slot, total bill $650, llama 3.3 70B quantized to 4 bit runs at 9 tokens per second locally, the article calls $2,000 mac mini m4 pro the sweet spot, this stack hits it for a third the seller in the video gives a 5 year warranty on hardware that hyperscalers threw away, the same companies the article warns will jack subscription prices are selling you the rack they used to charge $200 a month to access a 2018 enterprise chassis pulls 280 watts at idle and screams at 65 decibels, the mac mini pulls 10 watts and runs silent, the tradeoff is real but $1,350 cheaper buys a lot of electricity and earplugs over 4 years bookmark this and read the article below

starmex

107,995 Aufrufe • vor 1 Monat

A used Mac Pro off Craigslist for $800, three weeks of upgrades and a $2,070 total build cost. 192GB RAM, an Afterburner card, 8TB NVMe and an eGPU enclosure later, video exports dropped from 18 minutes to 4, three local LLMs run simultaneously without breaking a sweat and models that normally need cloud compute run entirely locally. Apple wants $12,000 for similar specs. The used market sold him the same hardware for a fraction of that, and the $400 to $600 in monthly API savings paid the whole build back in under 4 months.
0:58

A used Mac Pro off Craigslist for $800, three weeks of upgrades and a $2,070 total build cost. 192GB RAM, an Afterburner card, 8TB NVMe and an eGPU enclosure later, video exports dropped from 18 minutes to 4, three local LLMs run simultaneously without breaking a sweat and models that normally need cloud compute run entirely locally. Apple wants $12,000 for similar specs. The used market sold him the same hardware for a fraction of that, and the $400 to $600 in monthly API savings paid the whole build back in under 4 months.

0xMarioNawfal

137,627 Aufrufe • vor 23 Tagen

llama3 8B (not quantized) running on an heterogeneous home cluster made of: - iPhone 15 Pro Max - iPad Pro (not sure which version XD) - MacBook Pro ( M1 Max ) - NVIDIA GeForce 3080 (not visible in video) - 2x NVIDIA Titan X Pascal Very soon also supporting Android (I *have* to also add my NVIDIA Shield GPU!!!!!). Single code base, single model format (reduced and optimally distributed to every node to save space). Everything (including iOS code) is open here ... it would be really nice, with the help of the community, taking this project to the next level in terms of optimization and support. My vision is about a distributed inference server that can run any model on any backend in any cluster topology - let's fight programmed obsolescence and democratize inference!
0:26

llama3 8B (not quantized) running on an heterogeneous home cluster made of: - iPhone 15 Pro Max - iPad Pro (not sure which version XD) - MacBook Pro ( M1 Max ) - NVIDIA GeForce 3080 (not visible in video) - 2x NVIDIA Titan X Pascal Very soon also supporting Android (I *have* to also add my NVIDIA Shield GPU!!!!!). Single code base, single model format (reduced and optimally distributed to every node to save space). Everything (including iOS code) is open here ... it would be really nice, with the help of the community, taking this project to the next level in terms of optimization and support. My vision is about a distributed inference server that can run any model on any backend in any cluster topology - let's fight programmed obsolescence and democratize inference!

Simone Margaritelli

304,072 Aufrufe • vor 2 Jahren