M4 Mac AI Coding Cluster Uses EXO Labs to run LLMs (here Qwen 2.5 Coder 32B at 18 tok/sec) distributed across 4 M4 Mac Minis (Thunderbolt 5 80Gbps) and a MacBook Pro M4 Max. Local alternative to Cursor (benchmark comparison soon).

The Open-Source AI code editor is @zeddotdev with the @exolabs cluster set as a custom endpoint. exo is open source:

@exolabs Benchmarks for this setup and more will be released in the open!

@exolabs @cursor_ai digital sovereignty tool kit: handful of mac minis, 32 restaked ETH, 1BTC, starlink mini, 800w solar, 800ah batteries (working towards the first three still ;)

@exolabs @cursor_ai yo lets set this up. we can take it to the dessert.

@exolabs @cursor_ai What are the specs here? Do we need 5 devices to run this medium sized model? @ivanfioravanti is running it on a single M4 Max MacBook Pro with Q4 MLX using like 18GBs of RAM :

@exolabs @cursor_ai that's cool. something about this just seems aesthetically right. Like this is how building/training AI locally is supposed to work. Next up, training AI

@exolabs @cursor_ai How is a cluster like this effected by the "slowest gazzelle"? If I buy a 64GB M4 Pro Mini and link it up to my wife's 24GB M3 Air to be able to fit a larger model will the performance fall off a cliff? It seems like memory size/bandwidth are the primary drivers here not FLOPS?

@exolabs @cursor_ai Depends on the networking. Over TB5 it should still be a speedup since we use Tensor Parallelism (which is not merged yet, pending PR).

@exolabs @cursor_ai so fire… m4 Mac minis pack a punch for the size! how many for llama 405B at reasonable speed?

@exolabs @cursor_ai does it scale up linearly? eg if i wanted to double the tokens/s does it require double the mac 4mini;s? And whats the bottleneck on each node at the moment? The arm CPUs or memory?