Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Speed running my home AI cluster running distributed inference across 2 MacBooks and 2 Mac Minis. exo intern displays a real-time network topology as devices discover each other over the local network. Code is open source 👇

alexocheema's profile picture

Alex Cheema

44,115 subscribers

110,105 Aufrufe • vor 2 Jahren •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

11 Kommentare

Profilbild von Alex Cheema - e/acc
Alex Cheema - e/accvor 2 Jahren

The code is in the @exolabs_ GitHub repo here:

Profilbild von Jean-Philippe DIEL 🧬 e/acc 🇫🇷🇳🇿 琼-菲利普
Jean-Philippe DIEL 🧬 e/acc 🇫🇷🇳🇿 琼-菲利普vor 2 Jahren

@mo_baioumy @exolabs_ To me this proves that compute will come back to local on edge particularly for sensitive data - the era of 100% cloud is cycling out and now we are going to have hybrid computing between edge and cloud.

Profilbild von Alex Cheema - e/acc
Alex Cheema - e/accvor 2 Jahren

@mo_baioumy @exolabs_ This thesis is why we started @exolabs_

Profilbild von EXO Labs
EXO Labsvor 2 Jahren

The new live network topology view also shows which device is active in the inference in real-time.

Profilbild von vtes
vtesvor 2 Jahren

@exolabs_ Man , no matter how many times i see this, this is still so fkin cool

Profilbild von Alex Cheema - e/acc
Alex Cheema - e/accvor 2 Jahren

@exolabs_ Did you notice it highlights which device is active (in red) in real-time? :)

Profilbild von EXO Labs
EXO Labsvor 2 Jahren

Uses MLX, Apple’s open source ML library h/t @awnihannun

Profilbild von Hrishbh Dalal
Hrishbh Dalalvor 2 Jahren

@exolabs_ is there an api possibility on it? like openai or anthropic compatible? this project is so cool. wow. could we connect my macs ipad and linux with 3 4090s to create a frankestein cluster. i can borrow my gf devices as well haha

Profilbild von Alex Cheema - e/acc
Alex Cheema - e/accvor 2 Jahren

@exolabs_ Already supports ChatGPT-compliant API endpoint:

Profilbild von Mauro Sicard
Mauro Sicardvor 2 Jahren

@exolabs_ Are there plans on supporting Thunderbolt connection to increase the inference speed in between the Macs? I think that would be a game changer!

Profilbild von Alex Cheema - e/acc
Alex Cheema - e/accvor 2 Jahren

@exolabs_ Should work out of the box, made an issue here to double check:

Ähnliche Videos

2 MacBooks is all you need. Llama 3.1 405B running distributed across 2 MacBooks using exo intern home AI cluster
0:25

2 MacBooks is all you need. Llama 3.1 405B running distributed across 2 MacBooks using exo intern home AI cluster

Alex Cheema

1,289,007 Aufrufe • vor 2 Jahren

Llama 3.1 70b beamed to my iPhone from my exo intern home AI cluster of 2 MacBooks and 1 Mac Studio My own private GPT-4 assistant at home / on the go
0:21

Llama 3.1 70b beamed to my iPhone from my exo intern home AI cluster of 2 MacBooks and 1 Mac Studio My own private GPT-4 assistant at home / on the go

Alex Cheema

243,431 Aufrufe • vor 2 Jahren

I trained models across MacBooks using Apple's AirDrop protocol. grove is a distributed training library for Apple Silicon. Devices discover each other over AWDL, a direct radio link. If there's a shared WiFi network it upgrades to that for speed, otherwise everything goes over the direct link. No router, no cloud, no setup. grove start -n 4 grove join
0:34

I trained models across MacBooks using Apple's AirDrop protocol. grove is a distributed training library for Apple Silicon. Devices discover each other over AWDL, a direct radio link. If there's a shared WiFi network it upgrades to that for speed, otherwise everything goes over the direct link. No router, no cloud, no setup. grove start -n 4 grove join

Swarnim Jain

667,999 Aufrufe • vor 5 Monaten

Running Llama-3-70B at home with exo intern Combines the compute of all these devices to make one big GPU: - iPhone 15 Pro Max - iPad Pro M4 - Galaxy S24 Ultra - MacBook Pro M2 and M3 Pro - 2 x MSI NVIDIA GeForce RTX 4090 SUPRIM Code is open source 👇
0:21

Running Llama-3-70B at home with exo intern Combines the compute of all these devices to make one big GPU: - iPhone 15 Pro Max - iPad Pro M4 - Galaxy S24 Ultra - MacBook Pro M2 and M3 Pro - 2 x MSI NVIDIA GeForce RTX 4090 SUPRIM Code is open source 👇

Alex Cheema

197,476 Aufrufe • vor 2 Jahren

4 MAC MINIS RUNNING A 671B PARAMETER MODEL AS A CLUSTER No data center, no Cloud, no expensive hardware and not a single API call.. Just 4 Mac minis connected through EXO running DeepSeek v3.1 671b locally and actually fast. The part nobody talks about is that you don’t need one monster machine, you can cluster old computers you already have and split the load between them. The full breakdown of tools that make this possible is in the article below. Save and Read it today ↓
0:29

4 MAC MINIS RUNNING A 671B PARAMETER MODEL AS A CLUSTER No data center, no Cloud, no expensive hardware and not a single API call.. Just 4 Mac minis connected through EXO running DeepSeek v3.1 671b locally and actually fast. The part nobody talks about is that you don’t need one monster machine, you can cluster old computers you already have and split the load between them. The full breakdown of tools that make this possible is in the article below. Save and Read it today ↓

slash1s

58,724 Aufrufe • vor 2 Monaten

INCREDIBLE SPEED running Claude Code w/ local models on my own GPUs at home > SGLang serving MiniMax-M2.1 > on 8x RTX 3090s > nvtop showing live GPU load > Claude Code generating code + docs > end-2-end on my AI cluster MiniMax-M2.1 is my favorite model to run locally nowadays
3:28

INCREDIBLE SPEED running Claude Code w/ local models on my own GPUs at home > SGLang serving MiniMax-M2.1 > on 8x RTX 3090s > nvtop showing live GPU load > Claude Code generating code + docs > end-2-end on my AI cluster MiniMax-M2.1 is my favorite model to run locally nowadays

Ahmad

593,078 Aufrufe • vor 6 Monaten

THIS GUY STACKED MAC MINIS TO RUN AI LOCALLY AND IT CODES FOR FREE no API. no subscription. no cloud. what he's running: - EXO across multiple Mac Minis - Qwen 2.5 Coder 7B locally - Zed editor connected to the model - GPU usage visible in real time the math: - Mac Mini M4: $599 one-time - electricity: $3/month - API costs: $0 vs $459/month in subscriptions if you want Claude Code to run at this level ClaudeKit gives it the structure to do that ( full breakdown in the article below
0:55

THIS GUY STACKED MAC MINIS TO RUN AI LOCALLY AND IT CODES FOR FREE no API. no subscription. no cloud. what he's running: - EXO across multiple Mac Minis - Qwen 2.5 Coder 7B locally - Zed editor connected to the model - GPU usage visible in real time the math: - Mac Mini M4: $599 one-time - electricity: $3/month - API costs: $0 vs $459/month in subscriptions if you want Claude Code to run at this level ClaudeKit gives it the structure to do that ( full breakdown in the article below

Madni Aghadi

26,421 Aufrufe • vor 2 Monaten

AI inference is already live on UOMI Router. Access frontier open-source models through a decentralized inference network with: • Low-cost inference • OpenAI-compatible APIs • Verifiable compute • On-chain settlement Start building: And if you have idle GPUs, there's now a second way to participate. The UOMI Provider whitelist is officially open. Instead of letting your hardware sit unused, you can contribute GPU power to the UOMI Inference Network and earn from real AI demand. Join the Provider whitelist:
0:34

AI inference is already live on UOMI Router. Access frontier open-source models through a decentralized inference network with: • Low-cost inference • OpenAI-compatible APIs • Verifiable compute • On-chain settlement Start building: And if you have idle GPUs, there's now a second way to participate. The UOMI Provider whitelist is officially open. Instead of letting your hardware sit unused, you can contribute GPU power to the UOMI Inference Network and earn from real AI demand. Join the Provider whitelist:

Uomi

18,943 Aufrufe • vor 2 Monaten

Google just quietly dropped something big for agent builders It's called SAM: Sovereign Agent Mesh A peer-to-peer network where AI agents auto-discover each other, authenticate every packet and call tools across the mesh without a central server Think of it like BitTorrent, but for your agents Right now every agent stack you build is a walled garden. Claude Code talks to its MCPs. Codex talks to its MCPs. If you want agents running on your laptop, your VPS, and your Mac Studio to share tools with each other, you're duct-taping SSH tunnels and REST endpoints together like it's 2015 SAM makes that whole layer disappear You run a sam-node on each machine and the nodes find each other automatically. Every connection is cryptographically identified, so your agents plug in and start calling tools across the whole network The identity is portable too. Same node, same keys, whether it's on GCP, your laptop, or a Raspberry Pi in your closet If you're serious about running agents across multiple devices, get on the testnet this weekend: 1. Install sam-node (binary or Docker) 2. Point it at 3. Connect Claude or Gemini to your local MCP endpoint 4. Watch your agent discover tools running on other people's nodes Still early. Not officially supported by Google. This is where the agent stack is heading. Get on it early Repo:
2:01

Google just quietly dropped something big for agent builders It's called SAM: Sovereign Agent Mesh A peer-to-peer network where AI agents auto-discover each other, authenticate every packet and call tools across the mesh without a central server Think of it like BitTorrent, but for your agents Right now every agent stack you build is a walled garden. Claude Code talks to its MCPs. Codex talks to its MCPs. If you want agents running on your laptop, your VPS, and your Mac Studio to share tools with each other, you're duct-taping SSH tunnels and REST endpoints together like it's 2015 SAM makes that whole layer disappear You run a sam-node on each machine and the nodes find each other automatically. Every connection is cryptographically identified, so your agents plug in and start calling tools across the whole network The identity is portable too. Same node, same keys, whether it's on GCP, your laptop, or a Raspberry Pi in your closet If you're serious about running agents across multiple devices, get on the testnet this weekend: 1. Install sam-node (binary or Docker) 2. Point it at 3. Connect Claude or Gemini to your local MCP endpoint 4. Watch your agent discover tools running on other people's nodes Still early. Not officially supported by Google. This is where the agent stack is heading. Get on it early Repo:

Hasan Toor

33,937 Aufrufe • vor 2 Tagen

llama3 8B (not quantized) running on an heterogeneous home cluster made of: - iPhone 15 Pro Max - iPad Pro (not sure which version XD) - MacBook Pro ( M1 Max ) - NVIDIA GeForce 3080 (not visible in video) - 2x NVIDIA Titan X Pascal Very soon also supporting Android (I *have* to also add my NVIDIA Shield GPU!!!!!). Single code base, single model format (reduced and optimally distributed to every node to save space). Everything (including iOS code) is open here ... it would be really nice, with the help of the community, taking this project to the next level in terms of optimization and support. My vision is about a distributed inference server that can run any model on any backend in any cluster topology - let's fight programmed obsolescence and democratize inference!
0:26

llama3 8B (not quantized) running on an heterogeneous home cluster made of: - iPhone 15 Pro Max - iPad Pro (not sure which version XD) - MacBook Pro ( M1 Max ) - NVIDIA GeForce 3080 (not visible in video) - 2x NVIDIA Titan X Pascal Very soon also supporting Android (I *have* to also add my NVIDIA Shield GPU!!!!!). Single code base, single model format (reduced and optimally distributed to every node to save space). Everything (including iOS code) is open here ... it would be really nice, with the help of the community, taking this project to the next level in terms of optimization and support. My vision is about a distributed inference server that can run any model on any backend in any cluster topology - let's fight programmed obsolescence and democratize inference!

Simone Margaritelli

304,072 Aufrufe • vor 2 Jahren

CHINESE BUSINESSMAN BOUGHT 4 MAC MINIS FOR $1,699 AND CONNECTED THEM TO HIS MACBOOK - NOW THE CORNER OF HIS DESK MAKES MORE THAN MOST OFFICES MacBook Pro 16" runs everything - assigns tasks, controls the flow, holds context - 4 Mac Minis in the corner execute inference in parallel and don't know what downtime means he connected them through EXO - an open source framework that combines multiple Apple Silicon devices into one unified cluster - and together they run models no single device could handle alone while everyone else rents cloud GPUs for $1,900/month he paid $1,699 once and the entire cluster now runs on $20/month in electricity and nothing else clients pay for inference, data never leaves the room and the cluster scales by simply placing one more Mac Mini on top the MacBook is what he works on - the Mac Minis are what work for him around the clock without days off $1,699 invested once - and a corner of a desk that makes more than most companies with a rented office
0:20

CHINESE BUSINESSMAN BOUGHT 4 MAC MINIS FOR $1,699 AND CONNECTED THEM TO HIS MACBOOK - NOW THE CORNER OF HIS DESK MAKES MORE THAN MOST OFFICES MacBook Pro 16" runs everything - assigns tasks, controls the flow, holds context - 4 Mac Minis in the corner execute inference in parallel and don't know what downtime means he connected them through EXO - an open source framework that combines multiple Apple Silicon devices into one unified cluster - and together they run models no single device could handle alone while everyone else rents cloud GPUs for $1,900/month he paid $1,699 once and the entire cluster now runs on $20/month in electricity and nothing else clients pay for inference, data never leaves the room and the cluster scales by simply placing one more Mac Mini on top the MacBook is what he works on - the Mac Minis are what work for him around the clock without days off $1,699 invested once - and a corner of a desk that makes more than most companies with a rented office

Noisy

22,391 Aufrufe • vor 2 Monaten

Fully local Code Assistant running on NVIDIA GPU! In this tutorial, I'll show you how to run Llama3 using TensorRT and Nvidia's Triton Inference Server to use it as a Code Assistant in VSCode In this thread 🧵, I'll walk you through the integration process, explaining each step simply and clearly 👇 NVIDIA AI NVIDIA AIDev AI at Meta
0:30

Fully local Code Assistant running on NVIDIA GPU! In this tutorial, I'll show you how to run Llama3 using TensorRT and Nvidia's Triton Inference Server to use it as a Code Assistant in VSCode In this thread 🧵, I'll walk you through the integration process, explaining each step simply and clearly 👇 NVIDIA AI NVIDIA AIDev AI at Meta

Daniel San

42,154 Aufrufe • vor 2 Jahren

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push up the frontier specifically for local hardware. Better kernels, inference tricks like spec decoding, quantizing models, model routing. 2. Usability / UX. To use Local AI today, you need to install CLIs, select a model, a quantization, a harness that works best with that model, and you need to configure your inference engine. It's not very accessible. More work needs to be done on making this seamless, so the user doesn't even know the AI is running locally.
1:52

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push up the frontier specifically for local hardware. Better kernels, inference tricks like spec decoding, quantizing models, model routing. 2. Usability / UX. To use Local AI today, you need to install CLIs, select a model, a quantization, a harness that works best with that model, and you need to configure your inference engine. It's not very accessible. More work needs to be done on making this seamless, so the user doesn't even know the AI is running locally.

Alex Cheema

30,800 Aufrufe • vor 1 Monat

How 2 Escape: Lost Submarine launches June 24 on Xbox! Work with a partner across two devices to locate a missing submarine and stop the outbreak of global war. Time is running out. 🎮 Preorder now:
1:32

How 2 Escape: Lost Submarine launches June 24 on Xbox! Work with a partner across two devices to locate a missing submarine and stop the outbreak of global war. Time is running out. 🎮 Preorder now:

Maximum Entertainment

119,242 Aufrufe • vor 1 Jahr

I think I accidentally built the coolest way to watch AI agents work. This is what agent orchestration should look like. Live galaxy of 1,700 jobs across my repos, running agents pulse, delegations flow as packets, click any node to read the prompt and the diff. Open source and local on Gitmoot
1:08

I think I accidentally built the coolest way to watch AI agents work. This is what agent orchestration should look like. Live galaxy of 1,700 jobs across my repos, running agents pulse, delegations flow as packets, click any node to read the prompt and the diff. Open source and local on Gitmoot

Jerry the Martian

10,943 Aufrufe • vor 1 Monat

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization, it requires ~700GB of memory to run the model. EXO puts half of the layers on each device, combining their memory. EXO uses MLX distributed with TB5 interconnect, optimized for Apple Silicon. If we need higher throughput, adding two more devices lets us serve more users at once. EXO Labs handles all of this seamlessly - adding more devices to the cluster for linear scaling as we need it. The new EXO 1.0 will be open-source soonTM
0:29

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization, it requires ~700GB of memory to run the model. EXO puts half of the layers on each device, combining their memory. EXO uses MLX distributed with TB5 interconnect, optimized for Apple Silicon. If we need higher throughput, adding two more devices lets us serve more users at once. EXO Labs handles all of this seamlessly - adding more devices to the cluster for linear scaling as we need it. The new EXO 1.0 will be open-source soonTM

Matt Beton

158,485 Aufrufe • vor 1 Jahr

GMI Studio is officially here. We’ve moved beyond multimodal hubs to create a dedicated workflow engine for high-end AI creation. - Long-form video (2 min+). - 1 image, 10 video styles, cinematic output. - 1.4x faster generation speed, powered by GMI inference. - Social feed: Discover and like. The beta is over. The studio is open.
1:10

GMI Studio is officially here. We’ve moved beyond multimodal hubs to create a dedicated workflow engine for high-end AI creation. - Long-form video (2 min+). - 1 image, 10 video styles, cinematic output. - 1.4x faster generation speed, powered by GMI inference. - Social feed: Discover and like. The beta is over. The studio is open.

GMI Cloud

971,337 Aufrufe • vor 7 Monaten

Most system monitors give you graphs. I made one that gives you a cat. Zoomies is a tiny macOS menu bar app that turns CPU usage into a running animal. The harder your Mac works, the faster it runs. Free. Open source. ~2 MB. Looking for early users and feedback 👇
0:26

Most system monitors give you graphs. I made one that gives you a cat. Zoomies is a tiny macOS menu bar app that turns CPU usage into a running animal. The harder your Mac works, the faster it runs. Free. Open source. ~2 MB. Looking for early users and feedback 👇

Kartik

76,818 Aufrufe • vor 2 Monaten

Fun weekend project: using AI at Meta Sam3 to segment objects inside a SpAItial AI scene. Pipeline: -> Generate a world on SpAItial AI app and export as .ply -> PlayCanvas viewer to render splat + a uniform spatial grid over splat centers -> Capture frames + camera pose, POST to a local SAM3 server -> Lift each 2D mask → 3D: frustum-cull splats via the grid, reproject with the pose -> Recolor / isolate the segmented Gaussians This is all running locally on my Mac. Static scene, segment once, reuse persisted mask. All open source (link in comment), you can clone and build on top of this 👇🏽
1:17

Fun weekend project: using AI at Meta Sam3 to segment objects inside a SpAItial AI scene. Pipeline: -> Generate a world on SpAItial AI app and export as .ply -> PlayCanvas viewer to render splat + a uniform spatial grid over splat centers -> Capture frames + camera pose, POST to a local SAM3 server -> Lift each 2D mask → 3D: frustum-cull splats via the grid, reproject with the pose -> Recolor / isolate the segmented Gaussians This is all running locally on my Mac. Static scene, segment once, reuse persisted mask. All open source (link in comment), you can clone and build on top of this 👇🏽

RO2⚡️

22,119 Aufrufe • vor 2 Monaten