Video yükleniyor...
Video Yüklenemedi
Today we are shipping dnet, a distributed inference framework that lets Apple Silicon clusters run models that exceed their physical memory. We fuse pipelined-ring parallelism, disk streaming and UMA-aware scheduling so “out of memory” stops being the limit.
26,159 görüntüleme • 11 ay önce •via X (Twitter)
14 Yorum
Local AI today is great for short generations, but breaks down when you want big models, long context, high throughput, and heterogeneous devices. dnet addresses that gap head-on.
Built on @Apple MLX, dnet focuses on: --> Distributed execution strategies --> Automatic device/model profiling --> A heterogeneity-aware solver --> A drop-in OpenAI-style API.
Under the hood, every run goes through: Strategy → Profiling → Scheduling —> Strategy: Choose how to distribute work (pipelined ring, tensor-parallel, long-context) —> Profiling: Measure FLOPs, memory, KV cache, latency, and disk to know each device's limits —> Scheduling: Solve layer/tensor placement that fits mixed devices and avoids stalls
In alpha, we ship a pipelined-ring strategy inspired by PRIMA.CPP. dnet’s solver (distilp) extends it so devices can punch above memory: layers stream from disk mid-round and overlap with compute, so total model size can exceed total cluster RAM.
Apple Silicon’s unified memory is a blessing and a trap. CPU and GPU share one pool; naïve mmap will overcommit and start swapping. dnet is UMA-aware: memory pressure buffers, Apple-specific loaders, and repacked weights keep the ring moving instead of thrashing.
dnet is designed as a plugin architecture: --Solver ---API Adapter ----Topology Adapter To add a new strategy (for example tensor parallel), you implement a solver in distilp and an adapter in dnet. Runtime, KV cache, and API stay the same.
What’s next: -->128K context on home clusters --> higher throughput with faster comms and RDMA --> a unified backend where Apple Silicon, NVIDIA, and AMD share a single cluster and scheduling layer. Local AI without datacenter assumptions.
We’re building dnet because we think local clusters will be the backbone for agentic workloads. If you want to run frontier-scale models from your desk, start with the alpha: 🔗
top top work. I guess future would be webgpu type stuff to enable any device
@awnihannun Nice
Yo this is huge! We're working on something similar for a grant for self-driving vehicles, but this is huge work and should not be understated, fantastic work guys! Is it gonna be open source?
This seems far more plausible @Scobleizer
How does it compare with @exolabs ?
holy
![Dylan Patel of SemiAnalysis says a worse GPU with better storage and memory now beats the best chip without them, so buying the newest GPU alone no longer wins inference. So, an AMD GPU with more memory can outperform Nvidia in some cases. "So what we have is we have over $80 million of compute, GPUs from Nvidia, AMD, TPUs from Google, Trainium from Amazon, and we run this benchmark constantly on the newest inference engine, newest drivers, newest PyTorch version, whatever it is." "Every day it runs on an automated CI, and we run it on all the latest Chinese models, from GLM, Zhipu, Moonshot, Kimi, Alibaba, all these models we run." "Initially, when we were benchmarking the difference between these chips and different engines, different schemes for parallelism, we were just running it fixed context length." "But now with Agent X, we've analyzed over $5 million worth of Claude Code traces. This is real production traffic that people have donated to us as well as internally generated. Now we know what the actual agent workload looks like." "And then as we implement that and run those benchmarks, it turns out yes, the chip you're using is very important, but now even more important is how are you handling this memory offload?" "And so while an Nvidia GPU is faster than an AMD GPU in most cases, because AMD GPUs have more memory, they actually end up outperforming in some cases." "Or you can have a worse GPU, but a much better storage solution, and now you can outperform what the best GPU can do without those solutions. So just buying the newest and latest GPU alone doesn't get you the best inference economics." "Actually, you need to layer in all these other innovations including storage and memory." [ Who's the top player on your chart? ] "That really is a difficult multivariable problem. And generally that means you need to have, yes, you need to have the best GPU, a GB300, but you also need to have the best storage solutions. And so I won't spoil who's the best right here, but I will say that storage solutions matter a lot and memory solutions matter a lot, as does your front-end networking. That matters a lot."](https://image.24vids.com/tw-2081394445682114906/amplify_video_thumb/2081392924122812417/img/uhhbrpB5pwBZ1aFF.jpg)
