fastest way to move llms between machines for distributed inference

Maybe it’s a stupid question but do you need to have the models on all four machines in order to distribute the computing?

yes

I remember moving a megabyte or two file onto a floppy 💾 disk and waiting. Or burning a few hundred megabytes on a CD 💿 ROM. Now we’re doing hundreds of Gigabytes in a minute lol

HAYAI!

bruh at least mark this NSFW

TB5: 374 GB / 50 s ≈ 7.5 GB/s (~60 Gbps) over Finder's SMB 3.x. rsync/rclone or NFSv4 + jumbo frames add ~30%, pushing closer to ~80 Gbps. 10GbE: NVMe write-back cache on LVM RAID-5 NAS → TB-scale, NVMe-speed, low-cost model loads at a fraction of PCIe 5.0 NVMe cost #MLOps

Or I guess, plug that Thunderbolt 5 cable into the other Mac directly.

Awesome👌🏾

Good man. Now slap 2 on there and use RAID 😈

'move' glm4.7? just delete it

Have you never heard of rsync?

Insane how far we’ve come is there a moored law for file transfer speed 😂

cool, can you send the the 3 other hardware to my addy when you're done hah

This is my sign to finally invest in TB5 enclosures

Very nice 👍

This is freaking awesome.

@Stocktrader nice tip

heads up buddy

You gonna use exo?

Does this model interface well with openclaw? Wondering if the local models really can replace the api calls

@digitalix bittorent sync. easy.

Why you don’t use NAS?

I've noticed that transferring large models can be a major bottleneck in distributed inference, and I've seen it bring my app to a standstill more times than I care to admit. When I was trying to optimize my workflow, I ended up using Supabase's cloud storage integration to upload and download models directly from my workflow, which sped things up significantly.

yep, that’s why i transfer to models to each node before serving

Front ports are also T5??

I have a question. Is it possible to distribute thr computing power on each Mac Studio accross other. Like basically combining them together into 1 powerful machine?

Did he just spend 40k

But how many 🦞 are you running?

@Abe_Froman_SKC all!

Insane setup!

Do they not support 10G ethernet? would be cool to see that kind of set up because you shouldnt rely on SSD but it is cool how fast we have gotten in terms of data transfer

😲

it’s faster than even using the TB5 networking interface and sharing over a direct connected network? should be just as fast and skip the external drive you gotta swap between machines I setup an rsync script to do this but I never benchmarked against an external drive