Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Running Qwen3 8B thinking on an iPhone Air with MLX. The model is quantized to 4-bit and runs pretty well.

awnihannun's profile picture

Awni Hannun

44,767 subscribers

215,529 Aufrufe • vor 10 Monaten •via X (Twitter)

Wissenschaft & TechnologieNachrichten & PolitikBildung
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

A perfect coding model for MLX on Apple silicon.. Qwen delivered again. Runs quite fast on an M3 Ultra. Running the 4-bit quantized with mlx-lm:
0:50

A perfect coding model for MLX on Apple silicon.. Qwen delivered again. Runs quite fast on an M3 Ultra. Running the 4-bit quantized with mlx-lm:

Awni Hannun

186,641 Aufrufe • vor 1 Jahr

Qwen3-Coder-Flash runs quite fast on an M4 Max with mlx-lm. Running the 4-bit here, generated 4,467 tokens at >107 tokens/sec:
0:21

Qwen3-Coder-Flash runs quite fast on an M4 Max with mlx-lm. Running the 4-bit here, generated 4,467 tokens at >107 tokens/sec:

Awni Hannun

196,508 Aufrufe • vor 11 Monaten

Can an iPhone hold its own against a Mac? 💻📱 Ollama is running DeepSeek R1 Distill Llama 8B on a 16GB M2 MacBook Air, while the iPhone 15 Pro (8GB A17 Pro) runs the same model. Same model. Same sampling temperature. Same system prompt. While the iPhone is a bit slower, the 3-bit OmniQuant model reasons just as well as the 4-bit RTN model on the Mac. 🤯
1:07

Can an iPhone hold its own against a Mac? 💻📱 Ollama is running DeepSeek R1 Distill Llama 8B on a 16GB M2 MacBook Air, while the iPhone 15 Pro (8GB A17 Pro) runs the same model. Same model. Same sampling temperature. Same system prompt. While the iPhone is a bit slower, the 3-bit OmniQuant model reasons just as well as the 4-bit RTN model on the Mac. 🤯

Private LLM

26,544 Aufrufe • vor 1 Jahr

Quantized Gemma 2B runs pretty fast on my iPhone 15 pro in MLX Swift. code & docs: Comparable to GPT 3.5 turbo and Mixtral 8x7B in LMSYS Org benchmarks but runs efficiently on an iPhone. Pretty wild.
0:25

Quantized Gemma 2B runs pretty fast on my iPhone 15 pro in MLX Swift. code & docs: Comparable to GPT 3.5 turbo and Mixtral 8x7B in LMSYS Org benchmarks but runs efficiently on an iPhone. Pretty wild.

Awni Hannun

79,702 Aufrufe • vor 1 Jahr

Qwen3 235B MoE (22B active) runs so fast on an M2 Ultra with mlx-lm. - 4-bit model uses ~132GB - Generated 580 tokens at ~28 toks/sec
0:29

Qwen3 235B MoE (22B active) runs so fast on an M2 Ultra with mlx-lm. - 4-bit model uses ~132GB - Generated 580 tokens at ~28 toks/sec

Awni Hannun

117,763 Aufrufe • vor 1 Jahr

1-bit Bonsai 8B running locally on an M4 Pro (MLX) alongside a standard 16-bit 8B model. Same class of model, very different deployment profile: far lower memory use and substantially higher throughput.
0:32

1-bit Bonsai 8B running locally on an M4 Pro (MLX) alongside a standard 16-bit 8B model. Same class of model, very different deployment profile: far lower memory use and substantially higher throughput.

PrismML

93,021 Aufrufe • vor 3 Monaten

The new Qwen 3.5 4B runs incredibly well on M5. The model is close to GPT-4o in benchmarks. Running fully on-device with MLX.
0:20

The new Qwen 3.5 4B runs incredibly well on M5. The model is close to GPT-4o in benchmarks. Running fully on-device with MLX.

Adrien Grondin

230,466 Aufrufe • vor 4 Monaten

The new Qwen 3.5 by Qwen running on-device on iPhone 17 Pro. Qwen 3.5 beats models 4 times its size, has strong visual understanding, and can toggle reasoning on or off. The 2B 6-bit model here is running with MLX optimized for Apple Silicon.
0:25

The new Qwen 3.5 by Qwen running on-device on iPhone 17 Pro. Qwen 3.5 beats models 4 times its size, has strong visual understanding, and can toggle reasoning on or off. The 2B 6-bit model here is running with MLX optimized for Apple Silicon.

Adrien Grondin

3,555,049 Aufrufe • vor 4 Monaten

Translating a full-length news article on iPhone 17 Pro with TranslateGemma from Google DeepMind The model works incredibly well on long text Running fully on-device with MLX
0:46

Translating a full-length news article on iPhone 17 Pro with TranslateGemma from Google DeepMind The model works incredibly well on long text Running fully on-device with MLX

Adrien Grondin

15,493 Aufrufe • vor 5 Monaten

Running Qwen2 VL 72B 4bit 🇨🇳 quantized model on Mac Mini M4 Pro 64GB. It works very well and delivers good accuracy, suitable for practical use cases. Integrating MLX and MLX-VLM backend into Sparrow as main solution for local inference and data extraction. Kudos to Prince Canuma for porting Qwen2 VL to MLX.
0:23

Running Qwen2 VL 72B 4bit 🇨🇳 quantized model on Mac Mini M4 Pro 64GB. It works very well and delivers good accuracy, suitable for practical use cases. Integrating MLX and MLX-VLM backend into Sparrow as main solution for local inference and data extraction. Kudos to Prince Canuma for porting Qwen2 VL to MLX.

Andrej Baranovskij

13,742 Aufrufe • vor 1 Jahr

You can now vibecode your own WisprFlow or Monologue alternative that runs completely locally on Apple Silicon using MLX-Audio-Swift 🔥 Check out this live transcription of Dwarkesh Patel interview with Andrej Karpathy using Qwen3-ASR-0.6B quantized to 4bit on a M3 Max. It also runs in realtime on a iPhone 15 Pro and iPad Pro M1. No cloud. No API keys.
0:21

You can now vibecode your own WisprFlow or Monologue alternative that runs completely locally on Apple Silicon using MLX-Audio-Swift 🔥 Check out this live transcription of Dwarkesh Patel interview with Andrej Karpathy using Qwen3-ASR-0.6B quantized to 4bit on a M3 Max. It also runs in realtime on a iPhone 15 Pro and iPad Pro M1. No cloud. No API keys.

Prince Canuma

60,974 Aufrufe • vor 4 Monaten

Next level: QLoRA fine-tuning 4-bit Llama 3 8B on iPhone 15 pro. Incoming (Q)LoRA MLX Swift example by David Koski: works with lot's of models (Mistral, Gemma, Phi-2, etc)
1:19

Next level: QLoRA fine-tuning 4-bit Llama 3 8B on iPhone 15 pro. Incoming (Q)LoRA MLX Swift example by David Koski: works with lot's of models (Mistral, Gemma, Phi-2, etc)

Awni Hannun

581,723 Aufrufe • vor 2 Jahren

The new Kimi K2 1T model (4-bit quant) runs on 2 512GB M3 Ultras with mlx-lm and mx.distributed. 1 trillion params, at a speed that's actually quite usable:
0:37

The new Kimi K2 1T model (4-bit quant) runs on 2 512GB M3 Ultras with mlx-lm and mx.distributed. 1 trillion params, at a speed that's actually quite usable:

Awni Hannun

238,057 Aufrufe • vor 1 Jahr

Nemotron 3 Nano runs nicely with mlx-lm on an M4 Max. Could be a great model for local use on Mac: MoE + hybrid attention make it fast even for very long context. Generating in realtime with 4-bit model:
0:29

Nemotron 3 Nano runs nicely with mlx-lm on an M4 Max. Could be a great model for local use on Mac: MoE + hybrid attention make it fast even for very long context. Generating in realtime with 4-bit model:

Awni Hannun

51,070 Aufrufe • vor 7 Monaten

You can now have an AI researcher running on your laptop 24/7 for free! Running Qwen3-35B-A3B with llama.cpp and a 4-bit quant from Unsloth
0:30

You can now have an AI researcher running on your laptop 24/7 for free! Running Qwen3-35B-A3B with llama.cpp and a 4-bit quant from Unsloth

Lewis Tunstall

118,444 Aufrufe • vor 2 Monaten

First results are in. Llama 4 Maverick 17B active / 400B total is blazing fast with MLX on an M3 Ultra. Here is the 4-bit model generating 1100 tokens at 50 tok/sec:
0:20

First results are in. Llama 4 Maverick 17B active / 400B total is blazing fast with MLX on an M3 Ultra. Here is the 4-bit model generating 1100 tokens at 50 tok/sec:

Awni Hannun

149,855 Aufrufe • vor 1 Jahr

Running Ring 1T reasoning model on a single M3 Ultra with mlx-lm. It's quantized to 3.5 bits-per-weight. Uses 440GB and generated ~6k tokens at 18.2 toks/sec. Getting closer to GPT-5 at home.
0:59

Running Ring 1T reasoning model on a single M3 Ultra with mlx-lm. It's quantized to 3.5 bits-per-weight. Uses 440GB and generated ~6k tokens at 18.2 toks/sec. Getting closer to GPT-5 at home.

Awni Hannun

55,138 Aufrufe • vor 9 Monaten

DeepSeek R1 (the full 680B model) runs nicely in higher quality 4-bit on 3 M2 Ultras with MLX. Asked it a coding question and it thought for ~2k tokens and generated 3500 tokens overall:
1:28

DeepSeek R1 (the full 680B model) runs nicely in higher quality 4-bit on 3 M2 Ultras with MLX. Asked it a coding question and it thought for ~2k tokens and generated 3500 tokens overall:

Awni Hannun

997,198 Aufrufe • vor 1 Jahr

On-device realtime transcription on iPhone 15 Pro max 🚀 Using MLX-Audio-Swift + Qwen3-ASR-0.6B by Qwen It’s much faster and more consistent with the latest adjustments. Almost ready to push to GH.
1:46

On-device realtime transcription on iPhone 15 Pro max 🚀 Using MLX-Audio-Swift + Qwen3-ASR-0.6B by Qwen It’s much faster and more consistent with the latest adjustments. Almost ready to push to GH.

Prince Canuma

44,103 Aufrufe • vor 5 Monaten