跳过导航
GLM-5.2 8bit running on two M3 Ultra 512GB with MLX distributed? Here it is! 🚀
0:00 / 0:24

GLM-5.2 8bit running on two M3 Ultra 512GB with MLX distributed? Here it is! 🚀

Ivan Fioravanti

@ivanfioravanti4.4万 位关注者

在 X 上查看
280 个赞下载原帖

34,230 次观看

GLM-5.2 8bit running on two M3 Ultra 512GB with MLX distributed? Here it is! 🚀 Decode speed: 17.9 tokens/sec 🔥 Memory used: ~ 760GB 👀 Again keep in mind it's a preliminary PR by super Pedro Cuenca still a WIP!

32 条评论

评论来自 X (Twitter) 上的原帖

  • StrongEngineer_ 的头像

    @hotschmoe

    @pcuenq Kinda crazy to see SOTA model run on two small desktop machines ngl

    4 个赞

  • Ivan Fioravanti 的头像

    @ivanfioravanti

    @pcuenq Small mmm 🤣

    4 个赞

  • StrongEngineer_ 的头像

    @hotschmoe

    @pcuenq Compared to a server rack of day they're small! Hah

  • MJ 的头像

    @h1kz0r

    @pcuenq Good tech POC but the economics are still not there for large models on consumer hardware. API with ZDR and data residency is faster and cheaper. Small models yes, large models not yet.

    2 个赞

  • ToxSec 的头像

    @0xToxSec

    @pcuenq GLM 5.2 has been killing it. i think it’s a wake up call on how far local models have come. they aren’t very far behind frontier models realistically.

    2 个赞

  • Jun Wu 的头像

    @wu89_j

    @pcuenq Imagine Apple release M5 ultra with connectX7 instead of Thunderbolt. Imagine Apple Mac Pro can have 4x M5 ultra. Apple is under the bad leadership.

    2 个赞

  • KSemenenko 🇺🇦 的头像

    @ksemenenk0

    @pcuenq Is this a good model? Is it better in frontend and ui then gpt5.5 ?

    1 个赞

  • Ivan Fioravanti 的头像

    @ivanfioravanti

    @pcuenq Yes on Design and UI is much better than many out there, especially of GPT 5.5, on pure coding side I think is more slightly below Opus 4.7. GPT 5.5 is another level on backend, planning, architecture side.

    2 个赞

  • Martin Skorunka 的头像

    @MartinSkorunka

    @pcuenq That seems unusable, no?

    1 个赞

  • Ivan Fioravanti 的头像

    @ivanfioravanti

    @pcuenq Make it run during the night to do PR Review, Security Review or looping on its own to implement something. Not bad having such powerful local AI running freely, privately and securely on your machines.

    1 个赞

  • Martin Skorunka 的头像

    @MartinSkorunka

    @pcuenq I found a listing for 33k for one. 66k for 17t/s. 24*365*60*60*17 = 536M tokens a year max. 4.4usd per 1M is api price from zhipu. So ROI is at absolute best 33years. 100y with 8h shift.

  • Yume_X 的头像

    @yume_arasaki

    @pcuenq Future for local ai looks bright. Imagine thinking like drukenmiller and thinking 18 months from now…

    1 个赞

  • Ivan Fioravanti 的头像

    @ivanfioravanti

    @pcuenq Yes it is! The limit is the hardware, I bet it will become incredibly expensive running local AI.

    3 个赞

  • Yume_X 的头像

    @yume_arasaki

    @pcuenq You mean inexpensive right? I hope haha

    1 个赞

  • Ivan Fioravanti 的头像

    @ivanfioravanti

    @pcuenq No no, I bet the opposite. Looking at hardware prices going up like there is no limit 😢

    2 个赞

  • Yume_X 的头像

    @yume_arasaki

    @pcuenq Oh yeah right now…looking like near future it might actually go up and up and up lol…

    1 个赞

  • Chris MacFarland 的头像

    @ChrisMacFarland

    @pcuenq Amazon and nice work!!

    1 个赞

  • Ivan Fioravanti 的头像

    @ivanfioravanti

    @pcuenq 🙌

  • DC|use.fo 的头像

    @vibecoder_dc

    @pcuenq Running GLM-5.2 on M3 Ultras is like buying a fleet of luxury SUVs to move a single piano. Impressive specs, but the memory overhead is just a fancy way of saying the weights are barely fitting.

    1 个赞

  • Cédric Moreau 的头像

    @twicedd

    @pcuenq I was just testing EXO using this PR to patch my local mlx-lm version, seems to work!!! 🔥🔥🔥🔥

    1 个赞