GLM-5.2 8bit running on two M3 Ultra 512GB with MLX distributed? Here it is! 🚀
@ivanfioravanti4.4万 位关注者
34,230 次观看
32 条评论
评论来自 X (Twitter) 上的原帖

@hotschmoe
@pcuenq Kinda crazy to see SOTA model run on two small desktop machines ngl
4 个赞

@ivanfioravanti
@pcuenq Small mmm 🤣
4 个赞

@hotschmoe
@pcuenq Compared to a server rack of day they're small! Hah

@h1kz0r
@pcuenq Good tech POC but the economics are still not there for large models on consumer hardware. API with ZDR and data residency is faster and cheaper. Small models yes, large models not yet.
2 个赞

@0xToxSec
@pcuenq GLM 5.2 has been killing it. i think it’s a wake up call on how far local models have come. they aren’t very far behind frontier models realistically.
2 个赞

@wu89_j
@pcuenq Imagine Apple release M5 ultra with connectX7 instead of Thunderbolt. Imagine Apple Mac Pro can have 4x M5 ultra. Apple is under the bad leadership.
2 个赞

@ksemenenk0
@pcuenq Is this a good model? Is it better in frontend and ui then gpt5.5 ?
1 个赞

@ivanfioravanti
@pcuenq Yes on Design and UI is much better than many out there, especially of GPT 5.5, on pure coding side I think is more slightly below Opus 4.7. GPT 5.5 is another level on backend, planning, architecture side.
2 个赞

@MartinSkorunka
@pcuenq That seems unusable, no?
1 个赞

@ivanfioravanti
@pcuenq Make it run during the night to do PR Review, Security Review or looping on its own to implement something. Not bad having such powerful local AI running freely, privately and securely on your machines.
1 个赞

@MartinSkorunka
@pcuenq I found a listing for 33k for one. 66k for 17t/s. 24*365*60*60*17 = 536M tokens a year max. 4.4usd per 1M is api price from zhipu. So ROI is at absolute best 33years. 100y with 8h shift.

@yume_arasaki
@pcuenq Future for local ai looks bright. Imagine thinking like drukenmiller and thinking 18 months from now…
1 个赞

@ivanfioravanti
@pcuenq Yes it is! The limit is the hardware, I bet it will become incredibly expensive running local AI.
3 个赞

@yume_arasaki
@pcuenq You mean inexpensive right? I hope haha
1 个赞

@ivanfioravanti
@pcuenq No no, I bet the opposite. Looking at hardware prices going up like there is no limit 😢
2 个赞

@yume_arasaki
@pcuenq Oh yeah right now…looking like near future it might actually go up and up and up lol…
1 个赞

@ChrisMacFarland
@pcuenq Amazon and nice work!!
1 个赞

@ivanfioravanti
@pcuenq 🙌

@vibecoder_dc
@pcuenq Running GLM-5.2 on M3 Ultras is like buying a fleet of luxury SUVs to move a single piano. Impressive specs, but the memory overhead is just a fancy way of saying the weights are barely fitting.
1 个赞

@twicedd
@pcuenq I was just testing EXO using this PR to patch my local mlx-lm version, seems to work!!! 🔥🔥🔥🔥
1 个赞









