正在加载视频...

视频加载失败

fastest way to move llms between machines for distributed inference

74,754 次观看 • 9 个月前 •via X (Twitter)

33 条评论

Mohand 的头像
Mohand9 个月前

Maybe it’s a stupid question but do you need to have the models on all four machines in order to distribute the computing?

Alex Ziskind 的头像
Alex Ziskind9 个月前

yes

Wihl Silent H 🇵🇷🇺🇸 的头像
Wihl Silent H 🇵🇷🇺🇸9 个月前

I remember moving a megabyte or two file onto a floppy 💾 disk and waiting. Or burning a few hundred megabytes on a CD 💿 ROM. Now we’re doing hundreds of Gigabytes in a minute lol

Dreams of Code 的头像
Dreams of Code9 个月前

HAYAI!

Tom Osman 🐦‍⬛ 的头像
Tom Osman 🐦‍⬛9 个月前

bruh at least mark this NSFW

Affinity Techworks 的头像
Affinity Techworks9 个月前

TB5: 374 GB / 50 s ≈ 7.5 GB/s (~60 Gbps) over Finder's SMB 3.x. rsync/rclone or NFSv4 + jumbo frames add ~30%, pushing closer to ~80 Gbps. 10GbE: NVMe write-back cache on LVM RAID-5 NAS → TB-scale, NVMe-speed, low-cost model loads at a fraction of PCIe 5.0 NVMe cost #MLOps

Henk Poley 的头像
Henk Poley9 个月前

Or I guess, plug that Thunderbolt 5 cable into the other Mac directly.

Be3b 的头像
Be3b9 个月前

Awesome👌🏾

Zach Mueller 的头像
Zach Mueller9 个月前

Good man. Now slap 2 on there and use RAID 😈

SirJaɱzAlot 的头像
SirJaɱzAlot9 个月前

'move' glm4.7? just delete it

Traceix 的头像
Traceix9 个月前

Have you never heard of rsync?

Intrepid Sponge 𐇐𐇑𐇒𐇓𐇔𐇕𐇖𐇗𐇘𐇙𐇚 的头像
Intrepid Sponge 𐇐𐇑𐇒𐇓𐇔𐇕𐇖𐇗𐇘𐇙𐇚9 个月前

Insane how far we’ve come is there a moored law for file transfer speed 😂

Yeab - engg / core was in Palo Alto🌴 的头像
Yeab - engg / core was in Palo Alto🌴9 个月前

cool, can you send the the 3 other hardware to my addy when you're done hah

Sam Figueroa 的头像
Sam Figueroa9 个月前

This is my sign to finally invest in TB5 enclosures

Kenrik March 的头像
Kenrik March9 个月前

Very nice 👍

Kiran Adimatyam 的头像
Kiran Adimatyam9 个月前

This is freaking awesome.

Doge Codes 的头像
Doge Codes9 个月前

@Stocktrader nice tip

Sir Nullptr 的头像
Sir Nullptr9 个月前

heads up buddy

Simple Guy 的头像
Simple Guy9 个月前

You gonna use exo?

Simulated Requirements 的头像
Simulated Requirements9 个月前

Does this model interface well with openclaw? Wondering if the local models really can replace the api calls

bonbon 的头像
bonbon9 个月前

@digitalix bittorent sync. easy.

Bartłomiej Stachera 的头像
Bartłomiej Stachera9 个月前

Why you don’t use NAS?

Gregor 的头像
Gregor9 个月前

I've noticed that transferring large models can be a major bottleneck in distributed inference, and I've seen it bring my app to a standstill more times than I care to admit. When I was trying to optimize my workflow, I ended up using Supabase's cloud storage integration to upload and download models directly from my workflow, which sped things up significantly.

Alex Ziskind 的头像
Alex Ziskind9 个月前

yep, that’s why i transfer to models to each node before serving

Jan Schurch 的头像
Jan Schurch9 个月前

Front ports are also T5??

Ammy 的头像
Ammy9 个月前

I have a question. Is it possible to distribute thr computing power on each Mac Studio accross other. Like basically combining them together into 1 powerful machine?

OneDayRich 的头像
OneDayRich9 个月前

Did he just spend 40k

Abe 的头像
Abe9 个月前

But how many 🦞 are you running?

Alex Ziskind 的头像
Alex Ziskind9 个月前

@Abe_Froman_SKC all!

Tommy Ferris 的头像
Tommy Ferris9 个月前

Insane setup!

GoingAFK 的头像
GoingAFK9 个月前

Do they not support 10G ethernet? would be cool to see that kind of set up because you shouldnt rely on SSD but it is cool how fast we have gotten in terms of data transfer

vivek nakalgaonkar 的头像
vivek nakalgaonkar9 个月前

😲

seslly 的头像
seslly9 个月前

it’s faster than even using the TB5 networking interface and sharing over a direct connected network? should be just as fast and skip the external drive you gotta swap between machines I setup an rsync script to do this but I never benchmarked against an external drive

相关视频