Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Apple’s “LLM in a Flash” is definitely worth checking out. Going to 2-bit for the shared-expert MLP means disk I/O is no longer dominant. 14–15 tok/s from SSD is still wild for a ~400B MoE model streamed from storage. Qwen3.5-397B-A17B Credit: Dan Woods

Anemll

3,955 subscribers

35,664 views • 3 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

WIP: First attempt to speed up prefill for Flash-MoE. Original repo did token-by-token without streamed experts. Added: Batched linear attention + batched full attention (Flash Attention style) with custom Metal kernels. Without experts: 6.2x faster prefill (11 -> 68 tok/s) With experts at full-attn layers only: 1.9x faster (11 -> 20.5 tok/s) — same output quality Qwen3.5-397B, 4-bit, 209GB, M5 Max 128GB 1/3

WIP: First attempt to speed up prefill for Flash-MoE. Original repo did token-by-token without streamed experts. Added: Batched linear attention + batched full attention (Flash Attention style) with custom Metal kernels. Without experts: 6.2x faster prefill (11 -> 68 tok/s) With experts at full-attn layers only: 1.9x faster (11 -> 20.5 tok/s) — same output quality Qwen3.5-397B, 4-bit, 209GB, M5 Max 128GB 1/3

Anemll

19,572 views • 3 months ago

Running 400B model on iPhone! 0.6 t/s Credit Dan Woods @alexintosh Daniel Isaac Anemll

Running 400B model on iPhone! 0.6 t/s Credit Dan Woods @alexintosh Daniel Isaac Anemll

Anemll

291,615 views • 3 months ago

~ 6.5 - 6.7 t/s for GLM 5.1 on M5 Max 128GB Added “Dense” model export, now model load is only 5s ! Experts are streaming from SSD, so we do not pre-load it. Added direct SSD->Slot memory path, removed prefetch... Many dead end experiments. See Export a “dense-only GGUF” and “Fast path ” in tools/flashmob-sidecar/README.md WIP branch for Flash-MoE-SSD

~ 6.5 - 6.7 t/s for GLM 5.1 on M5 Max 128GB Added “Dense” model export, now model load is only 5s ! Experts are streaming from SSD, so we do not pre-load it. Added direct SSD->Slot memory path, removed prefetch... Many dead end experiments. See Export a “dense-only GGUF” and “Fast path ” in tools/flashmob-sidecar/README.md WIP branch for Flash-MoE-SSD

Anemll

29,421 views • 2 months ago

Extending on Dan Woods (and Claude's) repo I managed to get Qwen3.5-397B-A17B-4bit (224GB) running comfortably on my new M5 Max Laptop. Shout out to Carsen Klock's MacTop too. Over 10 Tok/sec!!!

Extending on Dan Woods (and Claude's) repo I managed to get Qwen3.5-397B-A17B-4bit (224GB) running comfortably on my new M5 Max Laptop. Shout out to Carsen Klock's MacTop too. Over 10 Tok/sec!!!

John T Davies 🇪🇺

17,807 views • 3 months ago

It's optimization time. Qwen3.5 35B at 13.1 tok/sec on iPhone 17 (no-pro). 2.3x from baseline of 2 days ago. Experimental features: --- Fused attention CMD1+CMD2 Merge Fused Expert Kernel Expert Prefetch I/O Fanout (credit Anemll) cc Dan Woods Daniel Isaac

It's optimization time. Qwen3.5 35B at 13.1 tok/sec on iPhone 17 (no-pro). 2.3x from baseline of 2 days ago. Experimental features: --- Fused attention CMD1+CMD2 Merge Fused Expert Kernel Expert Prefetch I/O Fanout (credit Anemll) cc Dan Woods Daniel Isaac

alexintosh

102,536 views • 3 months ago

Qwen3.5-397B-A17B with Vision running on a Single M3 Ultra 🚀 Shout out to the Qwen team because they cooked with this model! Get started today: > uv pip install -U mlx-vlm Model collection 👇🏽 Repo:

Qwen3.5-397B-A17B with Vision running on a Single M3 Ultra 🚀 Shout out to the Qwen team because they cooked with this model! Get started today: > uv pip install -U mlx-vlm Model collection 👇🏽 Repo:

Prince Canuma

11,386 views • 4 months ago

🥊 Time for a new fight Qwen3.5 397B A17B vs Qwen3.6 27B 🌸 "Cherry Blossom" (↓ prompt below) Using OpenRouter for 397B Running 27B locally on a single RTX 5080 Wow! 🤯 I'm sure this one is going to divide opinions. 397B feels cinematic with the flare and shadows it added, while 27B is just beautiful; it has a completely different color palette and the leaves are falling from the branches as asked. It's really tough to decide which model is the best. I was about to call it a tie, but then I remembered that 397B needs 180GB VRAM to run locally 😅 Don't get me wrong the model is amazing, but right now the VRAM it requires doesn't feel justified. It's simply not 10× better than 27B. We'll have to dig deeper with more prompts to be sure, let's see what each one has in store for us. What do you think?

🥊 Time for a new fight Qwen3.5 397B A17B vs Qwen3.6 27B 🌸 "Cherry Blossom" (↓ prompt below) Using OpenRouter for 397B Running 27B locally on a single RTX 5080 Wow! 🤯 I'm sure this one is going to divide opinions. 397B feels cinematic with the flare and shadows it added, while 27B is just beautiful; it has a completely different color palette and the leaves are falling from the branches as asked. It's really tough to decide which model is the best. I was about to call it a tie, but then I remembered that 397B needs 180GB VRAM to run locally 😅 Don't get me wrong the model is amazing, but right now the VRAM it requires doesn't feel justified. It's simply not 10× better than 27B. We'll have to dig deeper with more prompts to be sure, let's see what each one has in store for us. What do you think?

left curve dev

21,075 views • 1 month ago

The RTX 3090 is a 5-year-old GPU and it still runs a 27B model at 20 tok/s I tested Qwen3.5:27b across 3 generations of NVIDIA: 5090 → ~60 tok/s 4090 → ~40 tok/s 3090 → ~20 tok/s Perfectly linear scaling. Double the generation, double the speed.

The RTX 3090 is a 5-year-old GPU and it still runs a 27B model at 20 tok/s I tested Qwen3.5:27b across 3 generations of NVIDIA: 5090 → ~60 tok/s 4090 → ~40 tok/s 3090 → ~20 tok/s Perfectly linear scaling. Double the generation, double the speed.

stevibe

142,379 views • 3 months ago

Qwen3.5-35B with only 3B active parameters This MoE model runs FASTER than most 7B dense models. Tested on 3 generations of NVIDIA: - 5090: 137 tok/s - 4090: 112 tok/s - 3090: 78 tok/s The surprise? The 4090 <> 5090 gap is only 22%. With a 3B active MoE, even old GPUs fly.

Qwen3.5-35B with only 3B active parameters This MoE model runs FASTER than most 7B dense models. Tested on 3 generations of NVIDIA: - 5090: 137 tok/s - 4090: 112 tok/s - 3090: 78 tok/s The surprise? The 4090 <> 5090 gap is only 22%. With a 3B active MoE, even old GPUs fly.

stevibe

69,501 views • 3 months ago

OpenCode + MLX + Qwen3.5-397B-A17B-4bit. Video is 8x, but the goal is showing that It works! This is something unimaginable just few months ago. MLX Team is pushing like crazy and M5 Ultra will do the rest 🚀

OpenCode + MLX + Qwen3.5-397B-A17B-4bit. Video is 8x, but the goal is showing that It works! This is something unimaginable just few months ago. MLX Team is pushing like crazy and M5 Ultra will do the rest 🚀

Ivan Fioravanti ᯅ

48,692 views • 4 months ago

Running Qwen3.5-397B-A17B (4bit quants, 177 GB) on two DGX Sparks using llama.cpp with RPC and RDMA:

Running Qwen3.5-397B-A17B (4bit quants, 177 GB) on two DGX Sparks using llama.cpp with RPC and RDMA:

Radoslav Gerganov

25,622 views • 1 month ago

Chud The Builder tires to make people feel sorry for him. No one is buying it. My guy is still chimpin out from a Jail cell. Wild.

Chud The Builder tires to make people feel sorry for him. No one is buying it. My guy is still chimpin out from a Jail cell. Wild.

Rodney

202,243 views • 16 days ago

🎊 Kudos to the teams at Qwen on the launch of Qwen 3.5 with Qwen3.5-397B-A17B. 🙌 Developers can start building today for free: Or download and customize it with NVIDIA NeMo:

🎊 Kudos to the teams at Qwen on the launch of Qwen 3.5 with Qwen3.5-397B-A17B. 🙌 Developers can start building today for free: Or download and customize it with NVIDIA NeMo:

NVIDIA AI Developer

26,904 views • 4 months ago

Doing some testing of SS1 reading PS1 data directly from a disk without dumping the data first. It’s much better than I expected. That being said, copying games to the SD or SSD is still the optimal strategy for performance. It’s a one-time process and better for random access.

Doing some testing of SS1 reading PS1 data directly from a disk without dumping the data first. It’s much better than I expected. That being said, copying games to the SD or SSD is still the optimal strategy for performance. It’s a one-time process and better for random access.

Taki Udon

110,104 views • 11 months ago

The Democratic Party today, a party I no longer recognize, is the fulcrum of the transformation from a bottom-up model to centralized control.

The Democratic Party today, a party I no longer recognize, is the fulcrum of the transformation from a bottom-up model to centralized control.

Robert F. Kennedy Jr

418,807 views • 1 year ago

First results are in. Llama 4 Maverick 17B active / 400B total is blazing fast with MLX on an M3 Ultra. Here is the 4-bit model generating 1100 tokens at 50 tok/sec:

First results are in. Llama 4 Maverick 17B active / 400B total is blazing fast with MLX on an M3 Ultra. Here is the 4-bit model generating 1100 tokens at 50 tok/sec:

Awni Hannun

149,853 views • 1 year ago

According to the expert this gold bar is from the 1500s! Find out houch it's worth!

According to the expert this gold bar is from the 1500s! Find out houch it's worth!

LynneP

757,515 views • 6 months ago

AI animation is going to enable people to tell stories from all over the world. Here’s one from my neck of the woods! Made in partnership with Invideo . Look out for a post about how I made it with tomorrow 👀

AI animation is going to enable people to tell stories from all over the world. Here’s one from my neck of the woods! Made in partnership with Invideo . Look out for a post about how I made it with tomorrow 👀

Uncanny Harry AI

14,103 views • 3 months ago

This means that Vivek is no longer legally a citizen, and is ineligible to run for any kind of office. He should be immediately removed from the race in Ohio. He was an anchor-baby, born in America to non-citizens (his father is still not a citizen, after 40+ years in America).

This means that Vivek is no longer legally a citizen, and is ineligible to run for any kind of office. He should be immediately removed from the race in Ohio. He was an anchor-baby, born in America to non-citizens (his father is still not a citizen, after 40+ years in America).

Sólionath

556,368 views • 1 year ago