Loading video...

Video Failed to Load

Go Home

Qwen 3.5 397B prompt processing on M3 Ultra (with MLX distributed + JACCL) - 3.4× speedup on 4 chips - scaling improves as context increases Really fun to use with opencode; generated a playable Asteroids clone in ~4 minutes (real time, including me playing it a bit).

angeloskath's profile picture

Angelos Katharopoulos

4,973 subscribers

23,664 views • 6 months ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

4 Comments

Angelos Katharopoulos's profile picture
Angelos Katharopoulos6 months ago

Fun fact, the clipping of the video and the plot were actually done with opencode and Qwen 3.5 as well (in parallel actually). I know they are trivial (as is the asteroids game) but when you have it you use it which is pretty fun.

moskstraumen's profile picture
moskstraumen6 months ago

That's almost linear speedup. Impressive.

Coding at Night's profile picture
Coding at Night6 months ago

the scaling improving with context length is the interesting part. interconnect overhead gets amortized over longer sequences, which is exactly the regime where local inference matters most.

LightShift Studio's profile picture
LightShift Studio6 months ago

🚀 Just whipped up an Asteroids clone 🤖!

Related Videos

A perfect coding model for MLX on Apple silicon.. Qwen delivered again. Runs quite fast on an M3 Ultra. Running the 4-bit quantized with mlx-lm:
0:50

A perfect coding model for MLX on Apple silicon.. Qwen delivered again. Runs quite fast on an M3 Ultra. Running the 4-bit quantized with mlx-lm:

Awni Hannun

186,772 views • 1 year ago

MiniMax M2.1 in 4-bit cruises on an M3 Ultra with mlx-lm. Generated a space invaders game using 5098 tokens at 47.2 tok/sec:
0:23

MiniMax M2.1 in 4-bit cruises on an M3 Ultra with mlx-lm. Generated a space invaders game using 5098 tokens at 47.2 tok/sec:

Awni Hannun

94,165 views • 9 months ago

A long time coming but new mlx-lm is here with better batching support in the server and Gemma 4. pip install -U mlx-lm Here is a video where a single M3 Ultra serves 5 opencode sessions with Gemma 4 26B that process ~130k tokens in ~1.5 minutes.
2:03

A long time coming but new mlx-lm is here with better batching support in the server and Gemma 4. pip install -U mlx-lm Here is a video where a single M3 Ultra serves 5 opencode sessions with Gemma 4 26B that process ~130k tokens in ~1.5 minutes.

Angelos Katharopoulos

66,202 views • 5 months ago

Step-3.5-Flash in action on MLX with OpenCode on a single (distributed testing in progress!) M3 Ultra to create a snake game! 🔥 6bit quantization. Perfect tool calling. Fast & powerful coding model! Recommended Inference Settings: Temperature: 1.0 Top-p: 0.95 Top-k: 40 🧵
0:27

Step-3.5-Flash in action on MLX with OpenCode on a single (distributed testing in progress!) M3 Ultra to create a snake game! 🔥 6bit quantization. Perfect tool calling. Fast & powerful coding model! Recommended Inference Settings: Temperature: 1.0 Top-p: 0.95 Top-k: 40 🧵

Ivan Fioravanti ᯅ

17,158 views • 8 months ago

The new Deep Seek V3 0324 in 4-bit runs at > 20 toks/sec on a 512GB M3 Ultra with mlx-lm!
0:31

The new Deep Seek V3 0324 in 4-bit runs at > 20 toks/sec on a 512GB M3 Ultra with mlx-lm!

Awni Hannun

168,896 views • 1 year ago

Nemotron 3 Nano runs nicely with mlx-lm on an M4 Max. Could be a great model for local use on Mac: MoE + hybrid attention make it fast even for very long context. Generating in realtime with 4-bit model:
0:29

Nemotron 3 Nano runs nicely with mlx-lm on an M4 Max. Could be a great model for local use on Mac: MoE + hybrid attention make it fast even for very long context. Generating in realtime with 4-bit model:

Awni Hannun

51,091 views • 9 months ago

🤯This is crazy... Qwen 3.8 Flash Next IS really good... it basically one shot this, then I just told it to add the MLX-Serve / Qwen structure after. (mixed 4-8bit version) (~90gb ram, it did use full 256k ctx) Prompt: build me a minecraft clone using threejs, npm, typescript, test files, linting, and better graphics than minecraft, make it look good. Just simple block building with the basic minecraft blocks, but make it look really good with amazing graphics. Generated textures and sounds. Benchmaxxing ? Maybe, needs more testing, but I have been seeing some good results on Youtube also.
1:47

🤯This is crazy... Qwen 3.8 Flash Next IS really good... it basically one shot this, then I just told it to add the MLX-Serve / Qwen structure after. (mixed 4-8bit version) (~90gb ram, it did use full 256k ctx) Prompt: build me a minecraft clone using threejs, npm, typescript, test files, linting, and better graphics than minecraft, make it look good. Just simple block building with the basic minecraft blocks, but make it look really good with amazing graphics. Generated textures and sounds. Benchmaxxing ? Maybe, needs more testing, but I have been seeing some good results on Youtube also.

David Dalcu

26,075 views • 1 month ago

DeepSeek R1 (the full 680B model) runs nicely in higher quality 4-bit on 3 M2 Ultras with MLX. Asked it a coding question and it thought for ~2k tokens and generated 3500 tokens overall:
1:28

DeepSeek R1 (the full 680B model) runs nicely in higher quality 4-bit on 3 M2 Ultras with MLX. Asked it a coding question and it thought for ~2k tokens and generated 3500 tokens overall:

Awni Hannun

997,314 views • 1 year ago

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.
1:11

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.

Awni Hannun

60,599 views • 7 months ago

Very quick comparison between Ornith 1.5 35B MoE and Qwen 3.8 27B Dense. 📣 Clearly it's not a fair one, but let's in any case see how it went. 397B download in progress! In the Videos below: - Brick SA -> Qwen 3.8 - Lego Streets -> Ornith 1.5 Context: - Pi agent used in both cases, prompt below - M3 Ultra 512GB - Ornith-1.5-35B-A3B-oQ8e MTP hosted on oMLX - mlx-community/Qwen3.8-27B-8bit DFlash 2 hosted on mlx-dspark - Ornith has been incredibly fast with speed from 75 t/s to 45 t/s (160K+ context) - Qwen3.8 suffered context much more reaching 5 t/s above 160K context, but it can be due to engine tested still work in progress - Prompt: using threejs, and cdn, create a lego like game that's inspired by gta san andreas, with beautiful aesthetics and graphics and ability to steal cars. it should be 3d and have a nice and large map and areas. the graphics should be decent and nice. it should feature iconic things from gta san andreas Notes: - Qwen 3.8 result is 0-shot, while Ornith 1.5 required 6 iterations - Ornith 1.5 is not at the same level of autonomy as Qwen 3.8 27B honestly. To try getting the same results I'm constantly nudging, steering and 🤬 at it. - Qwen 3.8 took 6 hours to complete, but it was more a problem of timeout of Chrome headless used for testing, real prefill/decode TBD. I'll test again now with oMLX - Pi agent has a nearly perfect Cache Hit ratio that for local models is MEGA important!
0:31

Very quick comparison between Ornith 1.5 35B MoE and Qwen 3.8 27B Dense. 📣 Clearly it's not a fair one, but let's in any case see how it went. 397B download in progress! In the Videos below: - Brick SA -> Qwen 3.8 - Lego Streets -> Ornith 1.5 Context: - Pi agent used in both cases, prompt below - M3 Ultra 512GB - Ornith-1.5-35B-A3B-oQ8e MTP hosted on oMLX - mlx-community/Qwen3.8-27B-8bit DFlash 2 hosted on mlx-dspark - Ornith has been incredibly fast with speed from 75 t/s to 45 t/s (160K+ context) - Qwen3.8 suffered context much more reaching 5 t/s above 160K context, but it can be due to engine tested still work in progress - Prompt: using threejs, and cdn, create a lego like game that's inspired by gta san andreas, with beautiful aesthetics and graphics and ability to steal cars. it should be 3d and have a nice and large map and areas. the graphics should be decent and nice. it should feature iconic things from gta san andreas Notes: - Qwen 3.8 result is 0-shot, while Ornith 1.5 required 6 iterations - Ornith 1.5 is not at the same level of autonomy as Qwen 3.8 27B honestly. To try getting the same results I'm constantly nudging, steering and 🤬 at it. - Qwen 3.8 took 6 hours to complete, but it was more a problem of timeout of Chrome headless used for testing, real prefill/decode TBD. I'll test again now with oMLX - Pi agent has a nearly perfect Cache Hit ratio that for local models is MEGA important!

Ivan Fioravanti ᯅ

12,150 views • 1 month ago

Introducing MTPLX V2.12: The fastest way to run local AI now even faster and with more models for lower memory macs and the new M5 Ultra. Upgrades: - 100 TPS Qwen Flash Next sustained on long coding (M5 max). - 2x Prefill speeds Qwen Flash Next: 872 --> 1695 TPS. - Requests with images now use the compiled kernels for max speeds. New Models: - Qwen Flash Next MTPLX Optimized Quality: 8 bit model built for the M5 Ultra and M3 Ultra. - Bonsai 2 27B MTPLX Optimized Speed: Works on 16gb and 24gb macs with compiled kernels and MTP running as fast as possible. 64 TPS. - MiMo V2.6 Qwen 3.5 9B MTPLX Optimized Speed: My personal favourite. Mimos magic distilled into a 9B model easily runnable at high speeds on a 16gb mac. 107 TPS. +40 bug fixes and improvements in real world coding.
0:24

Introducing MTPLX V2.12: The fastest way to run local AI now even faster and with more models for lower memory macs and the new M5 Ultra. Upgrades: - 100 TPS Qwen Flash Next sustained on long coding (M5 max). - 2x Prefill speeds Qwen Flash Next: 872 --> 1695 TPS. - Requests with images now use the compiled kernels for max speeds. New Models: - Qwen Flash Next MTPLX Optimized Quality: 8 bit model built for the M5 Ultra and M3 Ultra. - Bonsai 2 27B MTPLX Optimized Speed: Works on 16gb and 24gb macs with compiled kernels and MTP running as fast as possible. 64 TPS. - MiMo V2.6 Qwen 3.5 9B MTPLX Optimized Speed: My personal favourite. Mimos magic distilled into a 9B model easily runnable at high speeds on a 16gb mac. 107 TPS. +40 bug fixes and improvements in real world coding.

Youssof Al Toukhi

21,124 views • 7 days ago

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!
0:52

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Prince Canuma

82,349 views • 5 months ago

NVIDIA sent us 2 DGX Sparks. For a while we wondered what we would do with them. The memory bandwidth is 273GB/s making it 3x slower than an M3 Ultra (819GB/s) for batch_size=1 inference. But it has 4x more FLOPS (100 TFLOPS compared to 26 TFLOPS). So we thought, what if we could combine the DGX Spark & M3 Ultra, and make use of both the massive compute on the DGX Spark and the massive memory-bandwidth on the M3 Ultra. We came up with a way to split inference across both devices and achieve a speedup of up to 4x for long prompts compared to the M3 Ultra on its own. Full details in the blog post linked below.
1:13

NVIDIA sent us 2 DGX Sparks. For a while we wondered what we would do with them. The memory bandwidth is 273GB/s making it 3x slower than an M3 Ultra (819GB/s) for batch_size=1 inference. But it has 4x more FLOPS (100 TFLOPS compared to 26 TFLOPS). So we thought, what if we could combine the DGX Spark & M3 Ultra, and make use of both the massive compute on the DGX Spark and the massive memory-bandwidth on the M3 Ultra. We came up with a way to split inference across both devices and achieve a speedup of up to 4x for long prompts compared to the M3 Ultra on its own. Full details in the blog post linked below.

Alex Cheema

281,225 views • 11 months ago

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization, it requires ~700GB of memory to run the model. EXO puts half of the layers on each device, combining their memory. EXO uses MLX distributed with TB5 interconnect, optimized for Apple Silicon. If we need higher throughput, adding two more devices lets us serve more users at once. EXO Labs handles all of this seamlessly - adding more devices to the cluster for linear scaling as we need it. The new EXO 1.0 will be open-source soonTM
0:29

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization, it requires ~700GB of memory to run the model. EXO puts half of the layers on each device, combining their memory. EXO uses MLX distributed with TB5 interconnect, optimized for Apple Silicon. If we need higher throughput, adding two more devices lets us serve more users at once. EXO Labs handles all of this seamlessly - adding more devices to the cluster for linear scaling as we need it. The new EXO 1.0 will be open-source soonTM

Matt Beton

158,485 views • 1 year ago

👾Watch a multi-agent system built with Gemini 3.6 Flash iterate on playable game design in real time. It uses Gemini 3.5 Flash-Lite to design, construct, and verify puzzle challenges based on live player actions balancing speed and reasoning in a fast-paced environment.
0:25

👾Watch a multi-agent system built with Gemini 3.6 Flash iterate on playable game design in real time. It uses Gemini 3.5 Flash-Lite to design, construct, and verify puzzle challenges based on live player actions balancing speed and reasoning in a fast-paced environment.

Google AI Developers

30,630 views • 2 months ago

update: qwen 3.6 27b dense q4 just one shotted octopus invaders game on a single 3090. hermes agent drove the whole thing, ~41 tok/s gen 21gb vram at full 262k context, thinking mode on. one prompt in and the canonical multi-file space shooter benchmark out, the same exact prompt i ran on qwen 3.5 27b dense back in march on the same card. 3.5 needed one external scope bug fix before the game would even load on first play. 3.6 needed nothing. 11 of 11 files written, 2411 lines of code, zero steering interventions, zero external fixes, playable on first load. 16 minutes 41 seconds wall clock from prompt to playable. consumer tier king on a single 3090 is locked tonight, and the silicon underneath my desk did not change between march and now. the open source ecosystem just moved the floor. watch it ship itself, the full 16 minutes 41 seconds sped to 3 minutes 45, no human touched the keyboard between the first prompt and the final frame.
3:45

update: qwen 3.6 27b dense q4 just one shotted octopus invaders game on a single 3090. hermes agent drove the whole thing, ~41 tok/s gen 21gb vram at full 262k context, thinking mode on. one prompt in and the canonical multi-file space shooter benchmark out, the same exact prompt i ran on qwen 3.5 27b dense back in march on the same card. 3.5 needed one external scope bug fix before the game would even load on first play. 3.6 needed nothing. 11 of 11 files written, 2411 lines of code, zero steering interventions, zero external fixes, playable on first load. 16 minutes 41 seconds wall clock from prompt to playable. consumer tier king on a single 3090 is locked tonight, and the silicon underneath my desk did not change between march and now. the open source ecosystem just moved the floor. watch it ship itself, the full 16 minutes 41 seconds sped to 3 minutes 45, no human touched the keyboard between the first prompt and the final frame.

Sudo su

126,670 views • 4 months ago

Made a Twitch chat integrated "stock market" which allows users to buy and sell stocks using watch time with real time updates on screen as a fun bit but we instead recreated the 2008 financial crisis within a matter of minutes
1:30

Made a Twitch chat integrated "stock market" which allows users to buy and sell stocks using watch time with real time updates on screen as a fun bit but we instead recreated the 2008 financial crisis within a matter of minutes

Crub

314,578 views • 2 years ago