Today we are shipping dnet, a distributed inference framework that lets Apple Silicon clusters run models that exceed their physical memory. We fuse pipelined-ring parallelism, disk streaming and UMA-aware scheduling so “out of memory” stops being the limit.

Local AI today is great for short generations, but breaks down when you want big models, long context, high throughput, and heterogeneous devices. dnet addresses that gap head-on.

Built on @Apple MLX, dnet focuses on: --> Distributed execution strategies --> Automatic device/model profiling --> A heterogeneity-aware solver --> A drop-in OpenAI-style API.

Under the hood, every run goes through: Strategy → Profiling → Scheduling —> Strategy: Choose how to distribute work (pipelined ring, tensor-parallel, long-context) —> Profiling: Measure FLOPs, memory, KV cache, latency, and disk to know each device's limits —> Scheduling: Solve layer/tensor placement that fits mixed devices and avoids stalls

In alpha, we ship a pipelined-ring strategy inspired by PRIMA.CPP. dnet’s solver (distilp) extends it so devices can punch above memory: layers stream from disk mid-round and overlap with compute, so total model size can exceed total cluster RAM.

Apple Silicon’s unified memory is a blessing and a trap. CPU and GPU share one pool; naïve mmap will overcommit and start swapping. dnet is UMA-aware: memory pressure buffers, Apple-specific loaders, and repacked weights keep the ring moving instead of thrashing.

dnet is designed as a plugin architecture: --Solver ---API Adapter ----Topology Adapter To add a new strategy (for example tensor parallel), you implement a solver in distilp and an adapter in dnet. Runtime, KV cache, and API stay the same.

What’s next: -->128K context on home clusters --> higher throughput with faster comms and RDMA --> a unified backend where Apple Silicon, NVIDIA, and AMD share a single cluster and scheduling layer. Local AI without datacenter assumptions.

We’re building dnet because we think local clusters will be the backbone for agentic workloads. If you want to run frontier-scale models from your desk, start with the alpha: 🔗

top top work. I guess future would be webgpu type stuff to enable any device

@awnihannun Nice

Yo this is huge! We're working on something similar for a grant for self-driving vehicles, but this is huge work and should not be understated, fantastic work guys! Is it gonna be open source?

This seems far more plausible @Scobleizer

How does it compare with @exolabs ?

holy