Загрузка видео...

Не удалось загрузить видео

На главную

Running GLM 4.7 Flash (8-bit) with Tensor Parallel / RDMA on 2 M4 Pro Mac Minis at 60 tok/sec. mlx-lm 0.30.5 features huge speedups for GLM 4.7 Flash for long context (h/t N8 Programs & Awni Hannun). M5 Pro (~28 Jan) will have ~4x faster prefill and ~1.3x faster decode.

alexocheema's profile picture

Alex Cheema

54,858 subscribers

56,741 просмотров • 9 месяцев назад •via X (Twitter)

ИгрыОбразованиеНаука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 11

Фото профиля Ivan Fioravanti
Ivan Fioravanti9 месяцев назад

@N8Programs @awnihannun You did it! This release fixed all issues I was facing with my two M3 Ultra! 🙌🏻

Фото профиля Alex Cheema
Alex Cheema9 месяцев назад

@N8Programs @awnihannun 💛

Фото профиля EXO Labs
EXO Labs9 месяцев назад

@N8Programs @awnihannun Pretty fast for 2 Mac Minis.

Фото профиля Sterling Cooley
Sterling Cooley9 месяцев назад

@N8Programs @awnihannun Wait so.. M5 Pro - are you getting an early preview or something ? I would *so* grab a few M5 Pro Mac Mini's :D

Фото профиля Dan Sănduleac
Dan Sănduleac9 месяцев назад

@N8Programs @awnihannun Wait, 2 mac minis, not 2 mac studio??

Фото профиля MAGUID
MAGUID9 месяцев назад

@N8Programs @awnihannun GLM 4.7 Flash = insane throughput

Фото профиля Carnivore JoeHos
Carnivore JoeHos9 месяцев назад

@N8Programs @awnihannun I’m having great success with GLM 4.7 and a couple other local LLMs running on Ubuntu and Moltbot is having a good time playing in its sandbox.

Фото профиля Le TechLead🔰
Le TechLead🔰9 месяцев назад

@N8Programs @awnihannun Not why it’s still bad on @ollama Hallucinations and slow

Фото профиля Allison
Allison9 месяцев назад

@exolabs @N8Programs @awnihannun @grok cost of this?

Фото профиля Rahul Raghav
Rahul Raghav9 месяцев назад

@N8Programs @awnihannun curious how the M5 Pro's performance will scale with larger models

Фото профиля Cody Avant
Cody Avant9 месяцев назад

@N8Programs @awnihannun I think Decode will be way faster than 1.3x if/when exo is optimized for the TensorOps / Metal Performance Primatives in Metal 4. The matmul accelerators on each gpu core is a big step up over m4.

Похожие видео

Running GLM-4.7-Flash with OpenCode locally on M4 Max MacBook Pro. 4-bit model runs at 82 tok/sec. Prefill will get ~4x faster with M5 Max MacBook Pro (~28 Jan). EXO will also support disaggregating prefill and decode across devices, e.g. DGX Spark.
0:43

Running GLM-4.7-Flash with OpenCode locally on M4 Max MacBook Pro. 4-bit model runs at 82 tok/sec. Prefill will get ~4x faster with M5 Max MacBook Pro (~28 Jan). EXO will also support disaggregating prefill and decode across devices, e.g. DGX Spark.

Alex Cheema

128,404 просмотров • 9 месяцев назад

Running GLM-4.7-Flash on 4 x M4 Pro Mac Minis using EXO Labs. Uses tensor parallelism with RDMA over Thunderbolt & MLX backend (h/t Awni Hannun). Runs at 100 tok/sec. We're working on optimizing this at EXO Labs. Aiming to hit ~200 tok/sec on this setup soon.
0:22

Running GLM-4.7-Flash on 4 x M4 Pro Mac Minis using EXO Labs. Uses tensor parallelism with RDMA over Thunderbolt & MLX backend (h/t Awni Hannun). Runs at 100 tok/sec. We're working on optimizing this at EXO Labs. Aiming to hit ~200 tok/sec on this setup soon.

Alex Cheema

62,320 просмотров • 9 месяцев назад

GLM 4.7 Flash is supported in mlx-lm 0.30.3 (h/t Ivan Fioravanti ᯅ) The 4-bit runs fast (43 tok/s generation, ~800 tok/s prefill) on a base M5 32GB laptop.
0:37

GLM 4.7 Flash is supported in mlx-lm 0.30.3 (h/t Ivan Fioravanti ᯅ) The 4-bit runs fast (43 tok/s generation, ~800 tok/s prefill) on a base M5 32GB laptop.

Awni Hannun

141,194 просмотров • 9 месяцев назад

For even higher throughput and lower latency: batch generation + tensor parallel with mlx-lm + and mlx.distributed. Here it's generating at 63 tok/sec (throughput) with GLM 4.7 in 6-bit and batch size 4 on 4 M3 Ultras:
0:41

For even higher throughput and lower latency: batch generation + tensor parallel with mlx-lm + and mlx.distributed. Here it's generating at 63 tok/sec (throughput) with GLM 4.7 in 6-bit and batch size 4 on 4 M3 Ultras:

Awni Hannun

22,721 просмотров • 10 месяцев назад

oMLX 0.7.1.dev1 is out! This dev release adds decision models (Clef, OpenJev), batched prefill for concurrent requests, a refreshed web UI, and faster Qwen3.8, Qwen3.6, GLM-5.3-Flash and DeepSeek V4.1. Speedups in this release - Qwen3.6-35B-A3B decode on M5 Max: 156 -> 205 tok/s (+31%). - GLM-5.3-Flash decode on M3 Ultra: 31.3 -> 40.6 tok/s (+30%). The fused decode kernels now run on M3 and M4, not only M5. - Qwen3.8-Flash-Next with expert offload, 32K prefill on M3 Ultra: 396 -> 636 tok/s (+61%). - Qwen3.8-Flash-Next, 8 prompts arriving 0.4 s apart on M5 Max: time to first token 7.41 -> 5.75 s (-22%) with batched prefill. What's new - Decision models. Clef, Clef-Flash and OpenJev answer typed questions with a probability for each option through the new /v1/systemone endpoint. - Batched prefill. Prompts that prefill at the same time now share one forward pass, so MoE models read their expert weights once. - A refreshed web UI with a log table, a settings rail and a customizable dashboard. - Native video input for Qwen3.5, 3.6 and 3.8. - More accurate oQe quantization, with 11-15% lower KL to the original model. - Fewer memory guard rejections in long sessions. Most of the speedups came from contributors again. Thanks to everyone who sent PRs and tested the dev builds!
0:33

oMLX 0.7.1.dev1 is out! This dev release adds decision models (Clef, OpenJev), batched prefill for concurrent requests, a refreshed web UI, and faster Qwen3.8, Qwen3.6, GLM-5.3-Flash and DeepSeek V4.1. Speedups in this release - Qwen3.6-35B-A3B decode on M5 Max: 156 -> 205 tok/s (+31%). - GLM-5.3-Flash decode on M3 Ultra: 31.3 -> 40.6 tok/s (+30%). The fused decode kernels now run on M3 and M4, not only M5. - Qwen3.8-Flash-Next with expert offload, 32K prefill on M3 Ultra: 396 -> 636 tok/s (+61%). - Qwen3.8-Flash-Next, 8 prompts arriving 0.4 s apart on M5 Max: time to first token 7.41 -> 5.75 s (-22%) with batched prefill. What's new - Decision models. Clef, Clef-Flash and OpenJev answer typed questions with a probability for each option through the new /v1/systemone endpoint. - Batched prefill. Prompts that prefill at the same time now share one forward pass, so MoE models read their expert weights once. - A refreshed web UI with a log table, a settings rail and a customizable dashboard. - Native video input for Qwen3.5, 3.6 and 3.8. - More accurate oQe quantization, with 11-15% lower KL to the original model. - Fewer memory guard rejections in long sessions. Most of the speedups came from contributors again. Thanks to everyone who sent PRs and tested the dev builds!

Jun Kim

25,205 просмотров • 24 дней назад

GLM-5.1-478B-NVFP4 Running on: - 4x RTX Pro 6000 - Sglang - 370,000 max tokens (1.75x full context) - p10 27.7 | p90 45.6 tok/s decode (gen) - 1340 tok/s prefill I could get 2x decode if I limit to 64k context (100 tok/s) In this video it operates Figma (:
1:00

GLM-5.1-478B-NVFP4 Running on: - 4x RTX Pro 6000 - Sglang - 370,000 max tokens (1.75x full context) - p10 27.7 | p90 45.6 tok/s decode (gen) - 1340 tok/s prefill I could get 2x decode if I limit to 64k context (100 tok/s) In this video it operates Figma (:

0xSero

78,863 просмотров • 6 месяцев назад

Qwen3-Coder-Flash runs quite fast on an M4 Max with mlx-lm. Running the 4-bit here, generated 4,467 tokens at >107 tokens/sec:
0:21

Qwen3-Coder-Flash runs quite fast on an M4 Max with mlx-lm. Running the 4-bit here, generated 4,467 tokens at >107 tokens/sec:

Awni Hannun

196,606 просмотров • 1 год назад

Llama 3 running locally on iPhone with MLX Built by exo intern team Mohamed Baioumy h/t Awni Hannun MLX & Prince Canuma for the port
0:24

Llama 3 running locally on iPhone with MLX Built by exo intern team Mohamed Baioumy h/t Awni Hannun MLX & Prince Canuma for the port

Alex Cheema

295,873 просмотров • 2 лет назад

GLM 5.2 flash Q2 (generated by myself with similar recipe of DS4 Flash of DwarfStar GGUF) running on an M5 Max with 128GB. The Q4 weights also work on tensor parallel inference in two M5 Max systems. Soon on GitHub. Still testing + CUDA + ROCm ...
0:39

GLM 5.2 flash Q2 (generated by myself with similar recipe of DS4 Flash of DwarfStar GGUF) running on an M5 Max with 128GB. The Q4 weights also work on tensor parallel inference in two M5 Max systems. Soon on GitHub. Still testing + CUDA + ROCm ...

antirez

29,548 просмотров • 2 месяцев назад

$GOOGL launched Gemini 3.5 Flash which it says beats Gemini 3.1 Pro across coding, real-world agentic tasks & scaled tool use. Artificial Analysis shows it running at 289 output tokens/sec which is more than 4x faster than Claude Opus 4.7 & GPT-5.5.
1:32

$GOOGL launched Gemini 3.5 Flash which it says beats Gemini 3.1 Pro across coding, real-world agentic tasks & scaled tool use. Artificial Analysis shows it running at 289 output tokens/sec which is more than 4x faster than Claude Opus 4.7 & GPT-5.5.

Shay Boloor

56,276 просмотров • 5 месяцев назад

GLM-4.7 runs quite well on an M3 Ultra with mlx-lm, even at a near lossless precision (6-bit here). It generated the best space invaders game I've seen yet for a local model (even included sound effects!). Generated 6600 tokens and ran at 16 tok/s.
0:53

GLM-4.7 runs quite well on an M3 Ultra with mlx-lm, even at a near lossless precision (6-bit here). It generated the best space invaders game I've seen yet for a local model (even included sound effects!). Generated 6600 tokens and ran at 16 tok/s.

Awni Hannun

237,191 просмотров • 10 месяцев назад

MiniMax M2.1 vs Opus 4.5 vs GLM-4.7 built an interactive 3D solar system from scratch. Running on 3 Claude Code in parallel. The winner is: 🥇 GLM-4.7! 🥈 Opus 4.5 (slowest) 🥉 M2.1 (fastest) GLM-4.7 design capabilities are another level vs 4.6!
0:48

MiniMax M2.1 vs Opus 4.5 vs GLM-4.7 built an interactive 3D solar system from scratch. Running on 3 Claude Code in parallel. The winner is: 🥇 GLM-4.7! 🥈 Opus 4.5 (slowest) 🥉 M2.1 (fastest) GLM-4.7 design capabilities are another level vs 4.6!

Ivan Fioravanti

81,804 просмотров • 10 месяцев назад

GLM-4.7-8bit (350GB) running at 19 toks/s on two M3 Ultra 512GB using Tensor Parallelism with EXO - MLX, versus 14 toks/s with single node. 🚀 Now context benchmarking & then OpenCode tests 🔥 Note: this is from sources, I had to change things to run it.
0:50

GLM-4.7-8bit (350GB) running at 19 toks/s on two M3 Ultra 512GB using Tensor Parallelism with EXO - MLX, versus 14 toks/s with single node. 🚀 Now context benchmarking & then OpenCode tests 🔥 Note: this is from sources, I had to change things to run it.

Ivan Fioravanti

328,425 просмотров • 10 месяцев назад

GLM-5.3-Flash running at up to 300 tok/s single stream, and 1000 tok/s with 8 concurrent sessions. On average it's working at 150 tok/s per stream. Phenomenal model, really something special. On 2x Sparks we get 60 tok/s and 4x 100 tok/s 300B params is the sweet spot.
1:10

GLM-5.3-Flash running at up to 300 tok/s single stream, and 1000 tok/s with 8 concurrent sessions. On average it's working at 150 tok/s per stream. Phenomenal model, really something special. On 2x Sparks we get 60 tok/s and 4x 100 tok/s 300B params is the sweet spot.

0xSero

30,672 просмотров • 2 месяцев назад

WIP: First attempt to speed up prefill for Flash-MoE. Original repo did token-by-token without streamed experts. Added: Batched linear attention + batched full attention (Flash Attention style) with custom Metal kernels. Without experts: 6.2x faster prefill (11 -> 68 tok/s) With experts at full-attn layers only: 1.9x faster (11 -> 20.5 tok/s) — same output quality Qwen3.5-397B, 4-bit, 209GB, M5 Max 128GB 1/3
0:57

WIP: First attempt to speed up prefill for Flash-MoE. Original repo did token-by-token without streamed experts. Added: Batched linear attention + batched full attention (Flash Attention style) with custom Metal kernels. Without experts: 6.2x faster prefill (11 -> 68 tok/s) With experts at full-attn layers only: 1.9x faster (11 -> 20.5 tok/s) — same output quality Qwen3.5-397B, 4-bit, 209GB, M5 Max 128GB 1/3

Anemll

19,572 просмотров • 7 месяцев назад

GLM 5.3 Flash running locally makes a Minecraft black hole mod We ran GLM 5.3 Flash Q4 locally on a rented 4x RTX PRO 6000 box and asked it to make a lightning gun mod for the real game using the Fabric API Output: -7.6M tokens -Time: ~9 hours -Decode: ~96 tok/s The mod adds a black hole rifle, when shot it spawns a black hole that pulls in blocks, with light rings that shrink all the way into the singularity, then collapses into an explosion crater that wipes out several chunks Run GLM 5.3 Flash locally in
0:22

GLM 5.3 Flash running locally makes a Minecraft black hole mod We ran GLM 5.3 Flash Q4 locally on a rented 4x RTX PRO 6000 box and asked it to make a lightning gun mod for the real game using the Fabric API Output: -7.6M tokens -Time: ~9 hours -Decode: ~96 tok/s The mod adds a black hole rifle, when shot it spawns a black hole that pulls in blocks, with light rings that shrink all the way into the singularity, then collapses into an explosion crater that wipes out several chunks Run GLM 5.3 Flash locally in

atomic.chat

32,515 просмотров • 2 месяцев назад

I didn't expect DeepSeek v4 PRO (not Flash) to run well on the Mac Studio M3 Ultra with 512GB of RAM. This is 2 bit quantized with the same DwarfStar recipe used for Flash. 433GB GGUF file. 130 t/s prefill, 13 t/s generation. Prefill in the video is low because small prompt.
1:24

I didn't expect DeepSeek v4 PRO (not Flash) to run well on the Mac Studio M3 Ultra with 512GB of RAM. This is 2 bit quantized with the same DwarfStar recipe used for Flash. 433GB GGUF file. 130 t/s prefill, 13 t/s generation. Prefill in the video is low because small prompt.

antirez

173,075 просмотров • 5 месяцев назад

I put MCDMA through its paces today and tested Qwen 3.8 Flash Next on my dual Spark / Mac Studio cluster. My first test was disaggregated prefill across my two Sparks, then decode onto my Studio. PP 2,100 tok/s Decode 80 tok/s at 20k context, higher on short replies. That's faster than either machine manages on its own: the Studio alone prefills at about 1,200 tok/s, and the Sparks alone decode at 35 to 65. 🤯 Mac prefill has always been way behind the Sparks; now I get the best of both worlds with the hardware I currently have.
1:10

I put MCDMA through its paces today and tested Qwen 3.8 Flash Next on my dual Spark / Mac Studio cluster. My first test was disaggregated prefill across my two Sparks, then decode onto my Studio. PP 2,100 tok/s Decode 80 tok/s at 20k context, higher on short replies. That's faster than either machine manages on its own: the Studio alone prefills at about 1,200 tok/s, and the Sparks alone decode at 35 to 65. 🤯 Mac prefill has always been way behind the Sparks; now I get the best of both worlds with the hardware I currently have.

Ash Hart

13,240 просмотров • 1 месяц назад

Qwen 3.8 vs GLM 5.3 vs Gemini 3.8 Flash vs DeepSeek v4.1 Flash Prompt: Build an Angry Birds clone Qwen 3.8 took $0.49 in 3 min GLM 5.3 took $0.46 in 2 min Gemini 3.8 Flash took $0.29 in 2 min 23 sec DeepSeek v4.1 Flash took $0.11 in 1 min 28 sec Which one wins for you?
0:24

Qwen 3.8 vs GLM 5.3 vs Gemini 3.8 Flash vs DeepSeek v4.1 Flash Prompt: Build an Angry Birds clone Qwen 3.8 took $0.49 in 3 min GLM 5.3 took $0.46 in 2 min Gemini 3.8 Flash took $0.29 in 2 min 23 sec DeepSeek v4.1 Flash took $0.11 in 1 min 28 sec Which one wins for you?

Bhavy☄️

27,799 просмотров • 1 месяц назад

Built with GLM 4.7 Z.ai 🤝 Factory
1:00

Built with GLM 4.7 Z.ai 🤝 Factory

CloudAI-X

22,365 просмотров • 10 месяцев назад