Загрузка видео...

Не удалось загрузить видео

На главную

A long time coming but new mlx-lm is here with better batching support in the server and Gemma 4. pip install -U mlx-lm Here is a video where a single M3 Ultra serves 5 opencode sessions with Gemma 4 26B that process ~130k tokens in ~1.5 minutes.

angeloskath's profile picture

Angelos Katharopoulos

4,752 subscribers

66,202 просмотров • 5 месяцев назад •via X (Twitter)

ОбразованиеНаука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

The new Deep Seek V3 0324 in 4-bit runs at > 20 toks/sec on a 512GB M3 Ultra with mlx-lm!
0:31

The new Deep Seek V3 0324 in 4-bit runs at > 20 toks/sec on a 512GB M3 Ultra with mlx-lm!

Awni Hannun

168,896 просмотров • 1 год назад

MiniMax M2.1 in 4-bit cruises on an M3 Ultra with mlx-lm. Generated a space invaders game using 5098 tokens at 47.2 tok/sec:
0:23

MiniMax M2.1 in 4-bit cruises on an M3 Ultra with mlx-lm. Generated a space invaders game using 5098 tokens at 47.2 tok/sec:

Awni Hannun

94,109 просмотров • 8 месяцев назад

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!
0:52

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Prince Canuma

82,349 просмотров • 4 месяцев назад

A perfect coding model for MLX on Apple silicon.. Qwen delivered again. Runs quite fast on an M3 Ultra. Running the 4-bit quantized with mlx-lm:
0:50

A perfect coding model for MLX on Apple silicon.. Qwen delivered again. Runs quite fast on an M3 Ultra. Running the 4-bit quantized with mlx-lm:

Awni Hannun

186,772 просмотров • 1 год назад

Just for fun, here's what 32 simultaneous long-context generations with Qwen3 Next 80B looks like on an M3 Ultra. Using the new batch generation in mlx-lm. Context size for each is about 5k tokens:
0:41

Just for fun, here's what 32 simultaneous long-context generations with Qwen3 Next 80B looks like on an M3 Ultra. Using the new batch generation in mlx-lm. Context size for each is about 5k tokens:

Awni Hannun

50,272 просмотров • 11 месяцев назад

Unplugging completely! No WiFi and zero notifications. A great way to get deep focus on a project. Here is a walkthrough showing how to run Gemma 4 (26B A4B) fully offline with LM Studio & OpenCode to parse PDFs, ask questions, and build sites 100% locally.
15:06

Unplugging completely! No WiFi and zero notifications. A great way to get deep focus on a project. Here is a walkthrough showing how to run Gemma 4 (26B A4B) fully offline with LM Studio & OpenCode to parse PDFs, ask questions, and build sites 100% locally.

Google Gemma

236,602 просмотров • 4 месяцев назад

First results are in. Llama 4 Maverick 17B active / 400B total is blazing fast with MLX on an M3 Ultra. Here is the 4-bit model generating 1100 tokens at 50 tok/sec:
0:20

First results are in. Llama 4 Maverick 17B active / 400B total is blazing fast with MLX on an M3 Ultra. Here is the 4-bit model generating 1100 tokens at 50 tok/sec:

Awni Hannun

149,855 просмотров • 1 год назад

Running Ring 1T reasoning model on a single M3 Ultra with mlx-lm. It's quantized to 3.5 bits-per-weight. Uses 440GB and generated ~6k tokens at 18.2 toks/sec. Getting closer to GPT-5 at home.
0:59

Running Ring 1T reasoning model on a single M3 Ultra with mlx-lm. It's quantized to 3.5 bits-per-weight. Uses 440GB and generated ~6k tokens at 18.2 toks/sec. Getting closer to GPT-5 at home.

Awni Hannun

55,138 просмотров • 11 месяцев назад

Llama 3.3 70B 4-bit runs nicely on a 64GB M3 Max with in MLX LM (~10 toks/sec). Would be even faster on an M4 Max. Yesterday's server-only 405B is today's laptop 70B:
0:33

Llama 3.3 70B 4-bit runs nicely on a 64GB M3 Max with in MLX LM (~10 toks/sec). Would be even faster on an M4 Max. Yesterday's server-only 405B is today's laptop 70B:

Awni Hannun

110,534 просмотров • 1 год назад

The video is a Llama v1 7B model implemented in MLX and running on an M2 Ultra. More here: * Train a Transformer LM or fine-tune with LoRA * Text generation with Mistral * Image generation with Stable Diffusion * Speech recognition with Whisper
1:35

The video is a Llama v1 7B model implemented in MLX and running on an M2 Ultra. More here: * Train a Transformer LM or fine-tune with LoRA * Text generation with Mistral * Image generation with Stable Diffusion * Speech recognition with Whisper

Awni Hannun

66,585 просмотров • 2 лет назад

DeepSeek R1 (the 680B MOE) is ~20% faster in the latest mlx / mlx-lm. 4-bit model on 3 M2 Ultras generates 4k tokens at a respectable 15 toks/sec. Plus some QoL improvements: - Only downloads the local shard (much faster startup) - Distributed launcher ships with MLX
0:50

DeepSeek R1 (the 680B MOE) is ~20% faster in the latest mlx / mlx-lm. 4-bit model on 3 M2 Ultras generates 4k tokens at a respectable 15 toks/sec. Plus some QoL improvements: - Only downloads the local shard (much faster startup) - Distributed launcher ships with MLX

Awni Hannun

86,921 просмотров • 1 год назад

For even higher throughput and lower latency: batch generation + tensor parallel with mlx-lm + and mlx.distributed. Here it's generating at 63 tok/sec (throughput) with GLM 4.7 in 6-bit and batch size 4 on 4 M3 Ultras:
0:41

For even higher throughput and lower latency: batch generation + tensor parallel with mlx-lm + and mlx.distributed. Here it's generating at 63 tok/sec (throughput) with GLM 4.7 in 6-bit and batch size 4 on 4 M3 Ultras:

Awni Hannun

22,721 просмотров • 8 месяцев назад

The new Kimi K2 1T model (4-bit quant) runs on 2 512GB M3 Ultras with mlx-lm and mx.distributed. 1 trillion params, at a speed that's actually quite usable:
0:37

The new Kimi K2 1T model (4-bit quant) runs on 2 512GB M3 Ultras with mlx-lm and mx.distributed. 1 trillion params, at a speed that's actually quite usable:

Awni Hannun

238,187 просмотров • 1 год назад

Qwen 3.5 397B prompt processing on M3 Ultra (with MLX distributed + JACCL) - 3.4× speedup on 4 chips - scaling improves as context increases Really fun to use with opencode; generated a playable Asteroids clone in ~4 minutes (real time, including me playing it a bit).
4:13

Qwen 3.5 397B prompt processing on M3 Ultra (with MLX distributed + JACCL) - 3.4× speedup on 4 chips - scaling improves as context increases Really fun to use with opencode; generated a playable Asteroids clone in ~4 minutes (real time, including me playing it a bit).

Angelos Katharopoulos

23,552 просмотров • 6 месяцев назад

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.
1:11

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.

Awni Hannun

60,599 просмотров • 6 месяцев назад

Qwen3.5-397B-A17B with Vision running on a Single M3 Ultra 🚀 Shout out to the Qwen team because they cooked with this model! Get started today: > uv pip install -U mlx-vlm Model collection 👇🏽 Repo:
0:59

Qwen3.5-397B-A17B with Vision running on a Single M3 Ultra 🚀 Shout out to the Qwen team because they cooked with this model! Get started today: > uv pip install -U mlx-vlm Model collection 👇🏽 Repo:

Prince Canuma

11,386 просмотров • 6 месяцев назад

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx
0:40

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx

raullen

22,765 просмотров • 3 месяцев назад

GLM-4.7 runs quite well on an M3 Ultra with mlx-lm, even at a near lossless precision (6-bit here). It generated the best space invaders game I've seen yet for a local model (even included sound effects!). Generated 6600 tokens and ran at 16 tok/s.
0:53

GLM-4.7 runs quite well on an M3 Ultra with mlx-lm, even at a near lossless precision (6-bit here). It generated the best space invaders game I've seen yet for a local model (even included sound effects!). Generated 6600 tokens and ran at 16 tok/s.

Awni Hannun

237,191 просмотров • 8 месяцев назад

Qwen3.5 runs quite well in mlx-lm. Awesome that we have a frontier-level hybrid model. The context gets longer but the inference speed and memory use barely change. Here's the Q4 generating a space invaders game on an M3 Ultra. Generated 4,120 tokens at 37.6 tok/s.
0:28

Qwen3.5 runs quite well in mlx-lm. Awesome that we have a frontier-level hybrid model. The context gets longer but the inference speed and memory use barely change. Here's the Q4 generating a space invaders game on an M3 Ultra. Generated 4,120 tokens at 37.6 tok/s.

Awni Hannun

48,673 просмотров • 6 месяцев назад

Early WIP port of Gemma 4 multi-token prediction (MTP) on MLX Swift With MTP, Gemma 31B is 30-40% faster on M5 Max and with zero quality degradation A significant speedup by just adding a 900MB MTP drafter model
0:20

Early WIP port of Gemma 4 multi-token prediction (MTP) on MLX Swift With MTP, Gemma 31B is 30-40% faster on M5 Max and with zero quality degradation A significant speedup by just adding a 900MB MTP drafter model

Adrien Grondin

35,875 просмотров • 3 месяцев назад