Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Quantized Gemma 2B runs pretty fast on my iPhone 15 pro in MLX Swift. code & docs: Comparable to GPT 3.5 turbo and Mixtral 8x7B in LMSYS Org benchmarks but runs efficiently on an iPhone. Pretty wild.

Awni Hannun

44,966 subscribers

79,979 views • 2 years ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

10 Comments

Logan Kilpatrick2 years ago

@lmsysorg Cost of intelligence takes another hit today : )

Christian Schoppe2 years ago

@lmsysorg I have the 6 bit quantized version running on my Pixel. Not quite as fast as yours but still quite usable. After a few initial tests, I still prefer Phi-3-mini.

Eric Hartford2 years ago

@lmsysorg that's awesome!

Kirito (e/acc) 🏴‍☠️2 years ago

@lmsysorg Great work we all saw it coming - privacy and intelligence at the palm of your hand

Rami El-Masri2 years ago

@lmsysorg Running advanced models like Gemma 2B efficiently on mobile devices is a game-changing milestone.

Tris Warkentin2 years ago

@lmsysorg What an incredible demo -- speed and quality are very impressive. Now to work on accessibility =)

NFTPerks 🇵🇹2 years ago

@lmsysorg awesome

Stavros Kassinos2 years ago

@lmsysorg 🚀🚀

Mani2 years ago

@lmsysorg Is it 4bit quantized?

Awni Hannun2 years ago

@lmsysorg Yes

Related Videos

Running Qwen3 8B thinking on an iPhone Air with MLX. The model is quantized to 4-bit and runs pretty well.

Running Qwen3 8B thinking on an iPhone Air with MLX. The model is quantized to 4-bit and runs pretty well.

Awni Hannun

215,597 views • 1 year ago

The new Qwen 3.5 4B runs incredibly well on M5. The model is close to GPT-4o in benchmarks. Running fully on-device with MLX.

The new Qwen 3.5 4B runs incredibly well on M5. The model is close to GPT-4o in benchmarks. Running fully on-device with MLX.

Adrien Grondin

230,466 views • 6 months ago

You can now vibecode your own WisprFlow or Monologue alternative that runs completely locally on Apple Silicon using MLX-Audio-Swift 🔥 Check out this live transcription of Dwarkesh Patel interview with Andrej Karpathy using Qwen3-ASR-0.6B quantized to 4bit on a M3 Max. It also runs in realtime on a iPhone 15 Pro and iPad Pro M1. No cloud. No API keys.

You can now vibecode your own WisprFlow or Monologue alternative that runs completely locally on Apple Silicon using MLX-Audio-Swift 🔥 Check out this live transcription of Dwarkesh Patel interview with Andrej Karpathy using Qwen3-ASR-0.6B quantized to 4bit on a M3 Max. It also runs in realtime on a iPhone 15 Pro and iPad Pro M1. No cloud. No API keys.

Prince Canuma

60,974 views • 7 months ago

GLM 4.6 runs quite fast on an M3 Ultra with mlx-lm even at higher precision. Pretty remarkable that it benchmarks competitive to the just-released Sonnet 4.5. Hope those benchmarks hold-up in day-to-day use. Here's a run using 5.5 bpw quantized model, generating 5.3k tokens at 17+ tok/sec using 244 GB. What prompts should I test?

GLM 4.6 runs quite fast on an M3 Ultra with mlx-lm even at higher precision. Pretty remarkable that it benchmarks competitive to the just-released Sonnet 4.5. Hope those benchmarks hold-up in day-to-day use. Here's a run using 5.5 bpw quantized model, generating 5.3k tokens at 17+ tok/sec using 244 GB. What prompts should I test?

Awni Hannun

68,539 views • 11 months ago

GPT-4o level intelligence running on your phone! MiniCPM-V 4.5 delivers enterprise-grade AI performance in just 8B parameters, outperforming models like GPT-4o, Gemini-2.0 Pro on vision and language tasks. - 30+ language support - Runs smoothly on iPhone/iPad 100% open-source!

GPT-4o level intelligence running on your phone! MiniCPM-V 4.5 delivers enterprise-grade AI performance in just 8B parameters, outperforming models like GPT-4o, Gemini-2.0 Pro on vision and language tasks. - 30+ language support - Runs smoothly on iPhone/iPad 100% open-source!

Akshay 🚀

84,720 views • 1 year ago

Woah, DeepSeek R1 1.5B runs perfectly on my iPhone 15 pro with MLX You can run unlimited SOTA models for free forever, in your pocket More importantly, this change the game for mobile AI apps pricing strategy: - Previously, You literally can’t sell an LLM powered app at $9.99 due to API cost - But now you can Making a tutorial of how to do this, stay tuned…

Woah, DeepSeek R1 1.5B runs perfectly on my iPhone 15 pro with MLX You can run unlimited SOTA models for free forever, in your pocket More importantly, this change the game for mobile AI apps pricing strategy: - Previously, You literally can’t sell an LLM powered app at $9.99 due to API cost - But now you can Making a tutorial of how to do this, stay tuned…

Jason Zhou

12,086 views • 1 year ago

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.

Awni Hannun

60,599 views • 7 months ago

The latest Qwen 3 VL by Qwen running on iPhone 17 Pro with MLX Qwen 3 VL brings upgraded visual understanding, recognition, and OCR capabilities without sacrificing text performance like previous models The 4B model here is close to Qwen 2.5 VL 72B in many benchmarks

The latest Qwen 3 VL by Qwen running on iPhone 17 Pro with MLX Qwen 3 VL brings upgraded visual understanding, recognition, and OCR capabilities without sacrificing text performance like previous models The 4B model here is close to Qwen 2.5 VL 72B in many benchmarks

Adrien Grondin

109,700 views • 11 months ago

I open sourced my custom tab bar and window splits library for Swift. Say hi to Bonsplit. CoreAnimation-driven animations, pretty flexible/configurable, feels right at home on the Mac and showcased in a demo app. Read the docs/see examples/get the code

I open sourced my custom tab bar and window splits library for Swift. Say hi to Bonsplit. CoreAnimation-driven animations, pretty flexible/configurable, feels right at home on the Mac and showcased in a demo app. Read the docs/see examples/get the code

almonk

28,474 views • 8 months ago

Let’s check in on the Bridgeport Islanders-OH MY GOD Fair to say that the rivalry between Bridgeport and Hartford runs pretty deep when a skater and goalie drop the gloves

Let’s check in on the Bridgeport Islanders-OH MY GOD Fair to say that the rivalry between Bridgeport and Hartford runs pretty deep when a skater and goalie drop the gloves

Nicole Shirman

265,794 views • 6 months ago

Congratulations to the Cohere team on the release of Cohere Transcribe Arabic! 🎉 Runs natively on mlx-audio (Python + Swift) from day-0 🚀 What's inside: → 2B params, Conformer encoder-decoder (audio-in, text-out) → Large Conformer encoder for acoustic representations + lightweight Transformer decoder for token generation → Built for Arabic dialects and Arabic-English code-switching → Auto-resampling to 16kHz + stereo→mono handling built into preprocessing → Apache 2.0 — fully open for community research Currently topping the Open Universal Arabic ASR Leaderboard 🥇 Get started today: 🐍 Python uv pip install -U mlx-audio 🍎 Swift

Congratulations to the Cohere team on the release of Cohere Transcribe Arabic! 🎉 Runs natively on mlx-audio (Python + Swift) from day-0 🚀 What's inside: → 2B params, Conformer encoder-decoder (audio-in, text-out) → Large Conformer encoder for acoustic representations + lightweight Transformer decoder for token generation → Built for Arabic dialects and Arabic-English code-switching → Auto-resampling to 16kHz + stereo→mono handling built into preprocessing → Apache 2.0 — fully open for community research Currently topping the Open Universal Arabic ASR Leaderboard 🥇 Get started today: 🐍 Python uv pip install -U mlx-audio 🍎 Swift

Prince Canuma

16,680 views • 2 months ago

Just got to try the AirPods Pro 3 and its new real-time, in-ear heart rate sensor with an outdoor walk workout. Here's a video of my heart rate being monitored in the Fitness app on iPhone 17 Pro #AppleEvent

Just got to try the AirPods Pro 3 and its new real-time, in-ear heart rate sensor with an outdoor walk workout. Here's a video of my heart rate being monitored in the Fitness app on iPhone 17 Pro #AppleEvent

Ray Wong

412,403 views • 1 year ago

New lighting tech in BAD Game (code name). 100% real time and runs at 240Hz 4K on my RTX 4090. 8x faster than my initial 16x SDF-ray per pixel proto. 1 ray per pixel and a new 6x6 bilateral filter (shared between SSAO and SDF rays). Starting to look pretty good :)

New lighting tech in BAD Game (code name). 100% real time and runs at 240Hz 4K on my RTX 4090. 8x faster than my initial 16x SDF-ray per pixel proto. 1 ray per pixel and a new 6x6 bilateral filter (shared between SSAO and SDF rays). Starting to look pretty good :)

Sebastian Aaltonen

18,646 views • 11 days ago

llama3 8B (not quantized) running on an heterogeneous home cluster made of: - iPhone 15 Pro Max - iPad Pro (not sure which version XD) - MacBook Pro ( M1 Max ) - NVIDIA GeForce 3080 (not visible in video) - 2x NVIDIA Titan X Pascal Very soon also supporting Android (I *have* to also add my NVIDIA Shield GPU!!!!!). Single code base, single model format (reduced and optimally distributed to every node to save space). Everything (including iOS code) is open here ... it would be really nice, with the help of the community, taking this project to the next level in terms of optimization and support. My vision is about a distributed inference server that can run any model on any backend in any cluster topology - let's fight programmed obsolescence and democratize inference!

llama3 8B (not quantized) running on an heterogeneous home cluster made of: - iPhone 15 Pro Max - iPad Pro (not sure which version XD) - MacBook Pro ( M1 Max ) - NVIDIA GeForce 3080 (not visible in video) - 2x NVIDIA Titan X Pascal Very soon also supporting Android (I have to also add my NVIDIA Shield GPU!!!!!). Single code base, single model format (reduced and optimally distributed to every node to save space). Everything (including iOS code) is open here ... it would be really nice, with the help of the community, taking this project to the next level in terms of optimization and support. My vision is about a distributed inference server that can run any model on any backend in any cluster topology - let's fight programmed obsolescence and democratize inference!

Simone Margaritelli

304,161 views • 2 years ago

WebGPU + Compute Shaders = 🤯 >500K stars runs at 60fps on my M1 Pro ⚡️ In my latest blog post, I walk through how to build this interactive galaxy simulation 🌌 Source code and live demo link in comments 👇🏻 #threejs #webgl #webgpu #shaders

WebGPU + Compute Shaders = 🤯 >500K stars runs at 60fps on my M1 Pro ⚡️ In my latest blog post, I walk through how to build this interactive galaxy simulation 🌌 Source code and live demo link in comments 👇🏻 #threejs #webgl #webgpu #shaders

Dan Greenheck

18,629 views • 10 months ago

Apple used "15" iPhone 17 Pro Max units to broadcast a Soccer match LIVE on Apple TV… WOW 15 iPhones placed across an 84,000-square-foot pitch capturing cinematic angles, closeups and tracking shots in real time. This is beyond smartphone cameras now!!!

Apple used "15" iPhone 17 Pro Max units to broadcast a Soccer match LIVE on Apple TV… WOW 15 iPhones placed across an 84,000-square-foot pitch capturing cinematic angles, closeups and tracking shots in real time. This is beyond smartphone cameras now!!!

TechDroider

1,448,984 views • 4 months ago

A 27B parameter model used to need a server room. Now it runs on: • iPhone • Android • Mac PrismML's Bonsai makes it possible. • 1-bit weights • 27B params in just 3.9GB • ~90% of full precision quality (PrismML evals) • Better than the 2-bit version at less than half the size The entire Bonsai family is now live in RunAnywhere. • 1.7B to 27B models • 1-bit + 2-bit ternary • Thinking mode On iOS and Mac, Bonsai runs through ggml's llama.cpp and Apple's MLX. On Android, Bonsai runs through ggml's llama.cpp and directly on the Qualcomm Hexagon NPU through QHexRT, our proprietary inference runtime. We built custom silicon kernels to make 1-bit inference on an NPU possible for the first time. Two years ago this needed a data center. Now it thinks in airplane mode. Now available in the RunAnywhere app on the App Store and Google Play.

A 27B parameter model used to need a server room. Now it runs on: • iPhone • Android • Mac PrismML's Bonsai makes it possible. • 1-bit weights • 27B params in just 3.9GB • ~90% of full precision quality (PrismML evals) • Better than the 2-bit version at less than half the size The entire Bonsai family is now live in RunAnywhere. • 1.7B to 27B models • 1-bit + 2-bit ternary • Thinking mode On iOS and Mac, Bonsai runs through ggml's llama.cpp and Apple's MLX. On Android, Bonsai runs through ggml's llama.cpp and directly on the Qualcomm Hexagon NPU through QHexRT, our proprietary inference runtime. We built custom silicon kernels to make 1-bit inference on an NPU possible for the first time. Two years ago this needed a data center. Now it thinks in airplane mode. Now available in the RunAnywhere app on the App Store and Google Play.

RunAnywhere (YC W26)

17,587 views • 2 months ago

Here’s how I wake my Mac up remotely from my iPhone and connect in from anywhere. This is new in Astropad Workbench 1.1. We’ve made sleep handling much smarter. Small thing, but very useful when you need to check on a machine fast.

Here’s how I wake my Mac up remotely from my iPhone and connect in from anywhere. This is new in Astropad Workbench 1.1. We’ve made sleep handling much smarter. Small thing, but very useful when you need to check on a machine fast.

Matt Ronge

79,644 views • 5 months ago

Sitting in the back of an Uber (XL Black) 6 minutes away from my destination And I decided to do a $100,000 bonus buy from my iPhone 17 pro on Stake.com… Even the driver was screaming when the bananas connected

Sitting in the back of an Uber (XL Black) 6 minutes away from my destination And I decided to do a $100,000 bonus buy from my iPhone 17 pro on Stake.com… Even the driver was screaming when the bananas connected

BTCs

116,537 views • 2 months ago