Loading video...

Video Failed to Load

Go Home

fastest way to move llms between machines for distributed inference

74,754 views • 9 months ago •via X (Twitter)

33 Comments

Mohand's profile picture
Mohand9 months ago

Maybe it’s a stupid question but do you need to have the models on all four machines in order to distribute the computing?

Alex Ziskind's profile picture
Alex Ziskind9 months ago

yes

Wihl Silent H 🇵🇷🇺🇸's profile picture
Wihl Silent H 🇵🇷🇺🇸9 months ago

I remember moving a megabyte or two file onto a floppy 💾 disk and waiting. Or burning a few hundred megabytes on a CD 💿 ROM. Now we’re doing hundreds of Gigabytes in a minute lol

Dreams of Code's profile picture
Dreams of Code9 months ago

HAYAI!

Tom Osman 🐦‍⬛'s profile picture
Tom Osman 🐦‍⬛9 months ago

bruh at least mark this NSFW

Affinity Techworks's profile picture
Affinity Techworks9 months ago

TB5: 374 GB / 50 s ≈ 7.5 GB/s (~60 Gbps) over Finder's SMB 3.x. rsync/rclone or NFSv4 + jumbo frames add ~30%, pushing closer to ~80 Gbps. 10GbE: NVMe write-back cache on LVM RAID-5 NAS → TB-scale, NVMe-speed, low-cost model loads at a fraction of PCIe 5.0 NVMe cost #MLOps

Henk Poley's profile picture
Henk Poley9 months ago

Or I guess, plug that Thunderbolt 5 cable into the other Mac directly.

Be3b's profile picture
Be3b9 months ago

Awesome👌🏾

Zach Mueller's profile picture
Zach Mueller9 months ago

Good man. Now slap 2 on there and use RAID 😈

SirJaɱzAlot's profile picture
SirJaɱzAlot9 months ago

'move' glm4.7? just delete it

Traceix's profile picture
Traceix9 months ago

Have you never heard of rsync?

Intrepid Sponge 𐇐𐇑𐇒𐇓𐇔𐇕𐇖𐇗𐇘𐇙𐇚's profile picture
Intrepid Sponge 𐇐𐇑𐇒𐇓𐇔𐇕𐇖𐇗𐇘𐇙𐇚9 months ago

Insane how far we’ve come is there a moored law for file transfer speed 😂

Yeab - engg / core was in Palo Alto🌴's profile picture
Yeab - engg / core was in Palo Alto🌴9 months ago

cool, can you send the the 3 other hardware to my addy when you're done hah

Sam Figueroa's profile picture
Sam Figueroa9 months ago

This is my sign to finally invest in TB5 enclosures

Kenrik March's profile picture
Kenrik March9 months ago

Very nice 👍

Kiran Adimatyam's profile picture
Kiran Adimatyam9 months ago

This is freaking awesome.

Doge Codes's profile picture
Doge Codes9 months ago

@Stocktrader nice tip

Sir Nullptr's profile picture
Sir Nullptr9 months ago

heads up buddy

Simple Guy's profile picture
Simple Guy9 months ago

You gonna use exo?

Simulated Requirements's profile picture
Simulated Requirements9 months ago

Does this model interface well with openclaw? Wondering if the local models really can replace the api calls

bonbon's profile picture
bonbon9 months ago

@digitalix bittorent sync. easy.

Bartłomiej Stachera's profile picture
Bartłomiej Stachera9 months ago

Why you don’t use NAS?

Gregor's profile picture
Gregor9 months ago

I've noticed that transferring large models can be a major bottleneck in distributed inference, and I've seen it bring my app to a standstill more times than I care to admit. When I was trying to optimize my workflow, I ended up using Supabase's cloud storage integration to upload and download models directly from my workflow, which sped things up significantly.

Alex Ziskind's profile picture
Alex Ziskind9 months ago

yep, that’s why i transfer to models to each node before serving

Jan Schurch's profile picture
Jan Schurch9 months ago

Front ports are also T5??

Ammy's profile picture
Ammy9 months ago

I have a question. Is it possible to distribute thr computing power on each Mac Studio accross other. Like basically combining them together into 1 powerful machine?

OneDayRich's profile picture
OneDayRich9 months ago

Did he just spend 40k

Abe's profile picture
Abe9 months ago

But how many 🦞 are you running?

Alex Ziskind's profile picture
Alex Ziskind9 months ago

@Abe_Froman_SKC all!

Tommy Ferris's profile picture
Tommy Ferris9 months ago

Insane setup!

GoingAFK's profile picture
GoingAFK9 months ago

Do they not support 10G ethernet? would be cool to see that kind of set up because you shouldnt rely on SSD but it is cool how fast we have gotten in terms of data transfer

vivek nakalgaonkar's profile picture
vivek nakalgaonkar9 months ago

😲

seslly's profile picture
seslly9 months ago

it’s faster than even using the TB5 networking interface and sharing over a direct connected network? should be just as fast and skip the external drive you gotta swap between machines I setup an rsync script to do this but I never benchmarked against an external drive

Related Videos