Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

GLM 5.2, Q2_K routed experts (effectively ~2.6 bits) running with SSD streaming on an M5 Max 128GB computer.

antirez's profile picture

antirez

75,205 subscribers

318,190 Aufrufe • vor 1 Monat •via X (Twitter)

Wissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

DeepSeek v4 PRO running via SSD streaming on my 128GB MacBook m5 max. 1.6 trillion parameters.
4:15

DeepSeek v4 PRO running via SSD streaming on my 128GB MacBook m5 max. 1.6 trillion parameters.

antirez

274,843 Aufrufe • vor 2 Monaten

That's a bit slow. Streaming directly the K3 official hugging face 1.6TB of weights in mxfp4 in an m5 max 128gb.
1:06

That's a bit slow. Streaming directly the K3 official hugging face 1.6TB of weights in mxfp4 in an m5 max 128gb.

antirez

291,306 Aufrufe • vor 12 Tagen

~ 6.5 - 6.7 t/s for GLM 5.1 on M5 Max 128GB Added “Dense” model export, now model load is only 5s ! Experts are streaming from SSD, so we do not pre-load it. Added direct SSD->Slot memory path, removed prefetch... Many dead end experiments. See Export a “dense-only GGUF” and “Fast path ” in tools/flashmob-sidecar/README.md WIP branch for Flash-MoE-SSD
1:28

~ 6.5 - 6.7 t/s for GLM 5.1 on M5 Max 128GB Added “Dense” model export, now model load is only 5s ! Experts are streaming from SSD, so we do not pre-load it. Added direct SSD->Slot memory path, removed prefetch... Many dead end experiments. See Export a “dense-only GGUF” and “Fast path ” in tools/flashmob-sidecar/README.md WIP branch for Flash-MoE-SSD

Anemll

29,421 Aufrufe • vor 4 Monaten

Running a Hermes Agent with GLM 5.2 on a Computer the size of a Deck of Playing Cards 🔥🔥🔥
2:41

Running a Hermes Agent with GLM 5.2 on a Computer the size of a Deck of Playing Cards 🔥🔥🔥

Tech2Wild

19,294 Aufrufe • vor 29 Tagen

made this game with GLM-5.2 (claude code) running on my 32x RTX 4090s
0:20

made this game with GLM-5.2 (claude code) running on my 32x RTX 4090s

Ning

26,104 Aufrufe • vor 1 Monat

GLM 5.2, open frontier-scale intelligence on Microsoft Foundry with AMD MI300X. Running a Codex goal with an open model never felt this good!
0:54

GLM 5.2, open frontier-scale intelligence on Microsoft Foundry with AMD MI300X. Running a Codex goal with an open model never felt this good!

Alvaro Bartolome

27,277 Aufrufe • vor 1 Monat

Much better at 17 tokens/sec in M3 Max, fusing a few operations. 128gb m3 max.
0:49

Much better at 17 tokens/sec in M3 Max, fusing a few operations. 128gb m3 max.

antirez

25,147 Aufrufe • vor 3 Monaten

Now video on oMLX + Codex + qwen3.6-35b-a3b 6bit on an M5 Max 128GB. Video normal speed! Command used: onlx launch codex Local AI rocks 💪
0:56

Now video on oMLX + Codex + qwen3.6-35b-a3b 6bit on an M5 Max 128GB. Video normal speed! Command used: onlx launch codex Local AI rocks 💪

Ivan Fioravanti ᯅ

11,461 Aufrufe • vor 2 Monaten

Running GLM-4.7-Flash with OpenCode locally on M4 Max MacBook Pro. 4-bit model runs at 82 tok/sec. Prefill will get ~4x faster with M5 Max MacBook Pro (~28 Jan). EXO will also support disaggregating prefill and decode across devices, e.g. DGX Spark.
0:43

Running GLM-4.7-Flash with OpenCode locally on M4 Max MacBook Pro. 4-bit model runs at 82 tok/sec. Prefill will get ~4x faster with M5 Max MacBook Pro (~28 Jan). EXO will also support disaggregating prefill and decode across devices, e.g. DGX Spark.

Alex Cheema

128,404 Aufrufe • vor 6 Monaten

The first M5 max arrived! Many many thanks to our sponsors ⿻ Audrey Tang 唐鳳 and Niels Gron, the next will arrive on Monday.
0:42

The first M5 max arrived! Many many thanks to our sponsors ⿻ Audrey Tang 唐鳳 and Niels Gron, the next will arrive on Monday.

antirez

27,840 Aufrufe • vor 2 Monaten

First kinda working implementation of GLM 5.2 in DwarfStar. Will take some time to be good enough, but it is a promising start. 433 GB GGUF file. M3 Ultra 512GB.
0:36

First kinda working implementation of GLM 5.2 in DwarfStar. Will take some time to be good enough, but it is a promising start. 433 GB GGUF file. M3 Ultra 512GB.

antirez

69,508 Aufrufe • vor 1 Monat

WIP: First attempt to speed up prefill for Flash-MoE. Original repo did token-by-token without streamed experts. Added: Batched linear attention + batched full attention (Flash Attention style) with custom Metal kernels. Without experts: 6.2x faster prefill (11 -> 68 tok/s) With experts at full-attn layers only: 1.9x faster (11 -> 20.5 tok/s) — same output quality Qwen3.5-397B, 4-bit, 209GB, M5 Max 128GB 1/3
0:57

WIP: First attempt to speed up prefill for Flash-MoE. Original repo did token-by-token without streamed experts. Added: Batched linear attention + batched full attention (Flash Attention style) with custom Metal kernels. Without experts: 6.2x faster prefill (11 -> 68 tok/s) With experts at full-attn layers only: 1.9x faster (11 -> 20.5 tok/s) — same output quality Qwen3.5-397B, 4-bit, 209GB, M5 Max 128GB 1/3

Anemll

19,572 Aufrufe • vor 4 Monaten

Video on LM Studio MLX v.1.8.5 + Codex + qwen3.6-35b-a3b 6bit on an M5 Max 128GB. Video is in normal speed to give you an idea of the experience. Command used: codex --oss -m qwen/qwen3.6-35b-a3b Local AI is becoming powerful 💪
1:18

Video on LM Studio MLX v.1.8.5 + Codex + qwen3.6-35b-a3b 6bit on an M5 Max 128GB. Video is in normal speed to give you an idea of the experience. Command used: codex --oss -m qwen/qwen3.6-35b-a3b Local AI is becoming powerful 💪

Ivan Fioravanti ᯅ

40,021 Aufrufe • vor 2 Monaten

Timelapse #156 (36 hrs) - Worked with the tiny corp on getting GLM 5.2 running on 8xMI300X (sglang won here) - Launched KernelBench-Mega and updated Kernelbench-Hard with h100 and b200 sweeps - Took care of boring business stuff - Did some training sweeps for specialized technical vocab audio model - Some bugs with putting kernelbench-mega and hard on cloud instances so had to do some reruns. Learned a lot though - Setting up my own local rl infra and profiling concurrency 128 rollouts with vllm. Became clear to me that I need to serve in nvfp4, use MoE only for throughput, reap so training doesn’t OOM, dig into the vllm kernel graph itself to not underutilize my hardware from poor flashinfer/cutlass selections for my rtx pro 6000 sm120 architecture - Might do online distillation from glm 5.2 but for now taking it one step at a time - Slept for a bit then woke up and showered - Fixed an issue with SGLang tensor parallel deadlock on GLM 5.2 architecture with MTP enabled - GLM 5.2 inference is 2-3x faster than coding plans and running on amd boxes - Spent time with family - Hung out with some friends - Recorded some yoctogpt lectures with the revamped notebook (high taste btw) - Setting up dflash training for GLM 5.2
0:30

Timelapse #156 (36 hrs) - Worked with the tiny corp on getting GLM 5.2 running on 8xMI300X (sglang won here) - Launched KernelBench-Mega and updated Kernelbench-Hard with h100 and b200 sweeps - Took care of boring business stuff - Did some training sweeps for specialized technical vocab audio model - Some bugs with putting kernelbench-mega and hard on cloud instances so had to do some reruns. Learned a lot though - Setting up my own local rl infra and profiling concurrency 128 rollouts with vllm. Became clear to me that I need to serve in nvfp4, use MoE only for throughput, reap so training doesn’t OOM, dig into the vllm kernel graph itself to not underutilize my hardware from poor flashinfer/cutlass selections for my rtx pro 6000 sm120 architecture - Might do online distillation from glm 5.2 but for now taking it one step at a time - Slept for a bit then woke up and showered - Fixed an issue with SGLang tensor parallel deadlock on GLM 5.2 architecture with MTP enabled - GLM 5.2 inference is 2-3x faster than coding plans and running on amd boxes - Spent time with family - Hung out with some friends - Recorded some yoctogpt lectures with the revamped notebook (high taste btw) - Setting up dflash training for GLM 5.2

Elliot Arledge

45,447 Aufrufe • vor 1 Monat

Llama 3.3 70B 4-bit running on a 128GB M4 Max with MLX LM (~12 toks/sec) with mactop on the right. "Good enough", local, private and open models are now more accessible than ever!
0:29

Llama 3.3 70B 4-bit running on a 128GB M4 Max with MLX LM (~12 toks/sec) with mactop on the right. "Good enough", local, private and open models are now more accessible than ever!

Ivan Fioravanti ᯅ

43,937 Aufrufe • vor 1 Jahr

Running GLM 4.7 Flash (8-bit) with Tensor Parallel / RDMA on 2 M4 Pro Mac Minis at 60 tok/sec. mlx-lm 0.30.5 features huge speedups for GLM 4.7 Flash for long context (h/t N8 Programs & Awni Hannun). M5 Pro (~28 Jan) will have ~4x faster prefill and ~1.3x faster decode.
0:32

Running GLM 4.7 Flash (8-bit) with Tensor Parallel / RDMA on 2 M4 Pro Mac Minis at 60 tok/sec. mlx-lm 0.30.5 features huge speedups for GLM 4.7 Flash for long context (h/t N8 Programs & Awni Hannun). M5 Pro (~28 Jan) will have ~4x faster prefill and ~1.3x faster decode.

Alex Cheema

56,555 Aufrufe • vor 6 Monaten

"YE3" - pretty heavy scene running real-time on my base M4 Pro Macbook. These things are beasts for 3D work already. Imagine an M5 Max + 128 GB… new era for 3D graphics? Apple
1:16

"YE3" - pretty heavy scene running real-time on my base M4 Pro Macbook. These things are beasts for 3D work already. Imagine an M5 Max + 128 GB… new era for 3D graphics? Apple

Marcel Deneuve

17,119 Aufrufe • vor 5 Monaten

GLM-5.2 8bit running on two M3 Ultra 512GB with MLX distributed? Here it is! 🚀 Decode speed: 17.9 tokens/sec 🔥 Memory used: ~ 760GB 👀 Again keep in mind it's a preliminary PR by super Pedro Cuenca still a WIP!
0:24

GLM-5.2 8bit running on two M3 Ultra 512GB with MLX distributed? Here it is! 🚀 Decode speed: 17.9 tokens/sec 🔥 Memory used: ~ 760GB 👀 Again keep in mind it's a preliminary PR by super Pedro Cuenca still a WIP!

Ivan Fioravanti ᯅ

34,175 Aufrufe • vor 1 Monat

ParoQuant just got a big upgrade 🚀 ✅ Supports the new Qwen3.5 models ⚡ Now runs on MLX (fast local inference on Apple Silicon) 🧠 Preserves reasoning quality with 4-bit quantization We also built an agent demo running locally on my 4-year-old M2 Max. Can't wait to upgrade to an M5 Max and see what kind of magic we can do. ✨
0:46

ParoQuant just got a big upgrade 🚀 ✅ Supports the new Qwen3.5 models ⚡ Now runs on MLX (fast local inference on Apple Silicon) 🧠 Preserves reasoning quality with 4-bit quantization We also built an agent demo running locally on my 4-year-old M2 Max. Can't wait to upgrade to an M5 Max and see what kind of magic we can do. ✨

Zhijian Liu

49,442 Aufrufe • vor 5 Monaten

Running Qwen3.6 35B (vision) on 2 x M5 Max MacBook Pro with RDMA over Thunderbolt 5. It describes the image and identifies Apple Park correctly, but misidentifies John Ternus as Jeff Williams. Near instant response with prefix caching.
0:28

Running Qwen3.6 35B (vision) on 2 x M5 Max MacBook Pro with RDMA over Thunderbolt 5. It describes the image and identifies Apple Park correctly, but misidentifies John Ternus as Jeff Williams. Near instant response with prefix caching.

Alex Cheema

99,474 Aufrufe • vor 3 Monaten