Загрузка видео...

Не удалось загрузить видео

На главную

GLM 5.2, Q2_K routed experts (effectively ~2.6 bits) running with SSD streaming on an M5 Max 128GB computer.

antirez's profile picture

antirez

77,759 subscribers

318,484 просмотров • 1 месяц назад •via X (Twitter)

Наука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

DeepSeek v4 PRO running via SSD streaming on my 128GB MacBook m5 max. 1.6 trillion parameters.
4:15

DeepSeek v4 PRO running via SSD streaming on my 128GB MacBook m5 max. 1.6 trillion parameters.

antirez

275,588 просмотров • 2 месяцев назад

That's a bit slow. Streaming directly the K3 official hugging face 1.6TB of weights in mxfp4 in an m5 max 128gb.
1:06

That's a bit slow. Streaming directly the K3 official hugging face 1.6TB of weights in mxfp4 in an m5 max 128gb.

antirez

292,739 просмотров • 19 дней назад

~ 6.5 - 6.7 t/s for GLM 5.1 on M5 Max 128GB Added “Dense” model export, now model load is only 5s ! Experts are streaming from SSD, so we do not pre-load it. Added direct SSD->Slot memory path, removed prefetch... Many dead end experiments. See Export a “dense-only GGUF” and “Fast path ” in tools/flashmob-sidecar/README.md WIP branch for Flash-MoE-SSD
1:28

~ 6.5 - 6.7 t/s for GLM 5.1 on M5 Max 128GB Added “Dense” model export, now model load is only 5s ! Experts are streaming from SSD, so we do not pre-load it. Added direct SSD->Slot memory path, removed prefetch... Many dead end experiments. See Export a “dense-only GGUF” and “Fast path ” in tools/flashmob-sidecar/README.md WIP branch for Flash-MoE-SSD

Anemll

29,421 просмотров • 4 месяцев назад

Running a Hermes Agent with GLM 5.2 on a Computer the size of a Deck of Playing Cards 🔥🔥🔥
2:41

Running a Hermes Agent with GLM 5.2 on a Computer the size of a Deck of Playing Cards 🔥🔥🔥

Tech2Wild

19,294 просмотров • 1 месяц назад

made this game with GLM-5.2 (claude code) running on my 32x RTX 4090s
0:20

made this game with GLM-5.2 (claude code) running on my 32x RTX 4090s

Ning

26,104 просмотров • 1 месяц назад

GLM 5.2, open frontier-scale intelligence on Microsoft Foundry with AMD MI300X. Running a Codex goal with an open model never felt this good!
0:54

GLM 5.2, open frontier-scale intelligence on Microsoft Foundry with AMD MI300X. Running a Codex goal with an open model never felt this good!

Alvaro Bartolome

27,277 просмотров • 1 месяц назад

Much better at 17 tokens/sec in M3 Max, fusing a few operations. 128gb m3 max.
0:49

Much better at 17 tokens/sec in M3 Max, fusing a few operations. 128gb m3 max.

antirez

25,153 просмотров • 3 месяцев назад

Now video on oMLX + Codex + qwen3.6-35b-a3b 6bit on an M5 Max 128GB. Video normal speed! Command used: onlx launch codex Local AI rocks 💪
0:56

Now video on oMLX + Codex + qwen3.6-35b-a3b 6bit on an M5 Max 128GB. Video normal speed! Command used: onlx launch codex Local AI rocks 💪

Ivan Fioravanti ᯅ

11,461 просмотров • 2 месяцев назад

Running GLM-4.7-Flash with OpenCode locally on M4 Max MacBook Pro. 4-bit model runs at 82 tok/sec. Prefill will get ~4x faster with M5 Max MacBook Pro (~28 Jan). EXO will also support disaggregating prefill and decode across devices, e.g. DGX Spark.
0:43

Running GLM-4.7-Flash with OpenCode locally on M4 Max MacBook Pro. 4-bit model runs at 82 tok/sec. Prefill will get ~4x faster with M5 Max MacBook Pro (~28 Jan). EXO will also support disaggregating prefill and decode across devices, e.g. DGX Spark.

Alex Cheema

128,404 просмотров • 6 месяцев назад

The first M5 max arrived! Many many thanks to our sponsors ⿻ Audrey Tang 唐鳳 and Niels Gron, the next will arrive on Monday.
0:42

The first M5 max arrived! Many many thanks to our sponsors ⿻ Audrey Tang 唐鳳 and Niels Gron, the next will arrive on Monday.

antirez

27,840 просмотров • 2 месяцев назад

First kinda working implementation of GLM 5.2 in DwarfStar. Will take some time to be good enough, but it is a promising start. 433 GB GGUF file. M3 Ultra 512GB.
0:36

First kinda working implementation of GLM 5.2 in DwarfStar. Will take some time to be good enough, but it is a promising start. 433 GB GGUF file. M3 Ultra 512GB.

antirez

69,508 просмотров • 1 месяц назад

WIP: First attempt to speed up prefill for Flash-MoE. Original repo did token-by-token without streamed experts. Added: Batched linear attention + batched full attention (Flash Attention style) with custom Metal kernels. Without experts: 6.2x faster prefill (11 -> 68 tok/s) With experts at full-attn layers only: 1.9x faster (11 -> 20.5 tok/s) — same output quality Qwen3.5-397B, 4-bit, 209GB, M5 Max 128GB 1/3
0:57

WIP: First attempt to speed up prefill for Flash-MoE. Original repo did token-by-token without streamed experts. Added: Batched linear attention + batched full attention (Flash Attention style) with custom Metal kernels. Without experts: 6.2x faster prefill (11 -> 68 tok/s) With experts at full-attn layers only: 1.9x faster (11 -> 20.5 tok/s) — same output quality Qwen3.5-397B, 4-bit, 209GB, M5 Max 128GB 1/3

Anemll

19,572 просмотров • 4 месяцев назад

Video on LM Studio MLX v.1.8.5 + Codex + qwen3.6-35b-a3b 6bit on an M5 Max 128GB. Video is in normal speed to give you an idea of the experience. Command used: codex --oss -m qwen/qwen3.6-35b-a3b Local AI is becoming powerful 💪
1:18

Video on LM Studio MLX v.1.8.5 + Codex + qwen3.6-35b-a3b 6bit on an M5 Max 128GB. Video is in normal speed to give you an idea of the experience. Command used: codex --oss -m qwen/qwen3.6-35b-a3b Local AI is becoming powerful 💪

Ivan Fioravanti ᯅ

40,565 просмотров • 2 месяцев назад

Timelapse #156 (36 hrs) - Worked with the tiny corp on getting GLM 5.2 running on 8xMI300X (sglang won here) - Launched KernelBench-Mega and updated Kernelbench-Hard with h100 and b200 sweeps - Took care of boring business stuff - Did some training sweeps for specialized technical vocab audio model - Some bugs with putting kernelbench-mega and hard on cloud instances so had to do some reruns. Learned a lot though - Setting up my own local rl infra and profiling concurrency 128 rollouts with vllm. Became clear to me that I need to serve in nvfp4, use MoE only for throughput, reap so training doesn’t OOM, dig into the vllm kernel graph itself to not underutilize my hardware from poor flashinfer/cutlass selections for my rtx pro 6000 sm120 architecture - Might do online distillation from glm 5.2 but for now taking it one step at a time - Slept for a bit then woke up and showered - Fixed an issue with SGLang tensor parallel deadlock on GLM 5.2 architecture with MTP enabled - GLM 5.2 inference is 2-3x faster than coding plans and running on amd boxes - Spent time with family - Hung out with some friends - Recorded some yoctogpt lectures with the revamped notebook (high taste btw) - Setting up dflash training for GLM 5.2
0:30

Timelapse #156 (36 hrs) - Worked with the tiny corp on getting GLM 5.2 running on 8xMI300X (sglang won here) - Launched KernelBench-Mega and updated Kernelbench-Hard with h100 and b200 sweeps - Took care of boring business stuff - Did some training sweeps for specialized technical vocab audio model - Some bugs with putting kernelbench-mega and hard on cloud instances so had to do some reruns. Learned a lot though - Setting up my own local rl infra and profiling concurrency 128 rollouts with vllm. Became clear to me that I need to serve in nvfp4, use MoE only for throughput, reap so training doesn’t OOM, dig into the vllm kernel graph itself to not underutilize my hardware from poor flashinfer/cutlass selections for my rtx pro 6000 sm120 architecture - Might do online distillation from glm 5.2 but for now taking it one step at a time - Slept for a bit then woke up and showered - Fixed an issue with SGLang tensor parallel deadlock on GLM 5.2 architecture with MTP enabled - GLM 5.2 inference is 2-3x faster than coding plans and running on amd boxes - Spent time with family - Hung out with some friends - Recorded some yoctogpt lectures with the revamped notebook (high taste btw) - Setting up dflash training for GLM 5.2

Elliot Arledge

45,447 просмотров • 1 месяц назад

Llama 3.3 70B 4-bit running on a 128GB M4 Max with MLX LM (~12 toks/sec) with mactop on the right. "Good enough", local, private and open models are now more accessible than ever!
0:29

Llama 3.3 70B 4-bit running on a 128GB M4 Max with MLX LM (~12 toks/sec) with mactop on the right. "Good enough", local, private and open models are now more accessible than ever!

Ivan Fioravanti ᯅ

43,937 просмотров • 1 год назад

Running GLM 4.7 Flash (8-bit) with Tensor Parallel / RDMA on 2 M4 Pro Mac Minis at 60 tok/sec. mlx-lm 0.30.5 features huge speedups for GLM 4.7 Flash for long context (h/t N8 Programs & Awni Hannun). M5 Pro (~28 Jan) will have ~4x faster prefill and ~1.3x faster decode.
0:32

Running GLM 4.7 Flash (8-bit) with Tensor Parallel / RDMA on 2 M4 Pro Mac Minis at 60 tok/sec. mlx-lm 0.30.5 features huge speedups for GLM 4.7 Flash for long context (h/t N8 Programs & Awni Hannun). M5 Pro (~28 Jan) will have ~4x faster prefill and ~1.3x faster decode.

Alex Cheema

56,555 просмотров • 6 месяцев назад

"YE3" - pretty heavy scene running real-time on my base M4 Pro Macbook. These things are beasts for 3D work already. Imagine an M5 Max + 128 GB… new era for 3D graphics? Apple
1:16

"YE3" - pretty heavy scene running real-time on my base M4 Pro Macbook. These things are beasts for 3D work already. Imagine an M5 Max + 128 GB… new era for 3D graphics? Apple

Marcel Deneuve

17,119 просмотров • 5 месяцев назад

GLM-5.2 8bit running on two M3 Ultra 512GB with MLX distributed? Here it is! 🚀 Decode speed: 17.9 tokens/sec 🔥 Memory used: ~ 760GB 👀 Again keep in mind it's a preliminary PR by super Pedro Cuenca still a WIP!
0:24

GLM-5.2 8bit running on two M3 Ultra 512GB with MLX distributed? Here it is! 🚀 Decode speed: 17.9 tokens/sec 🔥 Memory used: ~ 760GB 👀 Again keep in mind it's a preliminary PR by super Pedro Cuenca still a WIP!

Ivan Fioravanti ᯅ

34,175 просмотров • 1 месяц назад

ParoQuant just got a big upgrade 🚀 ✅ Supports the new Qwen3.5 models ⚡ Now runs on MLX (fast local inference on Apple Silicon) 🧠 Preserves reasoning quality with 4-bit quantization We also built an agent demo running locally on my 4-year-old M2 Max. Can't wait to upgrade to an M5 Max and see what kind of magic we can do. ✨
0:46

ParoQuant just got a big upgrade 🚀 ✅ Supports the new Qwen3.5 models ⚡ Now runs on MLX (fast local inference on Apple Silicon) 🧠 Preserves reasoning quality with 4-bit quantization We also built an agent demo running locally on my 4-year-old M2 Max. Can't wait to upgrade to an M5 Max and see what kind of magic we can do. ✨

Zhijian Liu

49,504 просмотров • 5 месяцев назад

Running Qwen3.6 35B (vision) on 2 x M5 Max MacBook Pro with RDMA over Thunderbolt 5. It describes the image and identifies Apple Park correctly, but misidentifies John Ternus as Jeff Williams. Near instant response with prefix caching.
0:28

Running Qwen3.6 35B (vision) on 2 x M5 Max MacBook Pro with RDMA over Thunderbolt 5. It describes the image and identifies Apple Park correctly, but misidentifies John Ternus as Jeff Williams. Near instant response with prefix caching.

Alex Cheema

99,474 просмотров • 3 месяцев назад