Загрузка видео...

Не удалось загрузить видео

На главную

Quantized Gemma 2B runs pretty fast on my iPhone 15 pro in MLX Swift. code & docs: Comparable to GPT 3.5 turbo and Mixtral 8x7B in LMSYS Org benchmarks but runs efficiently on an iPhone. Pretty wild.

awnihannun's profile picture

Awni Hannun

44,966 subscribers

79,979 просмотров • 2 лет назад •via X (Twitter)

Наука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 10

Фото профиля Logan Kilpatrick
Logan Kilpatrick2 лет назад

@lmsysorg Cost of intelligence takes another hit today : )

Фото профиля Christian Schoppe
Christian Schoppe2 лет назад

@lmsysorg I have the 6 bit quantized version running on my Pixel. Not quite as fast as yours but still quite usable. After a few initial tests, I still prefer Phi-3-mini.

Фото профиля Eric Hartford
Eric Hartford2 лет назад

@lmsysorg that's awesome!

Фото профиля Kirito (e/acc) 🏴‍☠️
Kirito (e/acc) 🏴‍☠️2 лет назад

@lmsysorg Great work we all saw it coming - privacy and intelligence at the palm of your hand

Фото профиля Rami El-Masri
Rami El-Masri2 лет назад

@lmsysorg Running advanced models like Gemma 2B efficiently on mobile devices is a game-changing milestone.

Фото профиля Tris Warkentin
Tris Warkentin2 лет назад

@lmsysorg What an incredible demo -- speed and quality are very impressive. Now to work on accessibility =)

Фото профиля NFTPerks 🇵🇹
NFTPerks 🇵🇹2 лет назад

@lmsysorg awesome

Фото профиля Stavros Kassinos
Stavros Kassinos2 лет назад

@lmsysorg 🚀🚀

Фото профиля Mani
Mani2 лет назад

@lmsysorg Is it 4bit quantized?

Фото профиля Awni Hannun
Awni Hannun2 лет назад

@lmsysorg Yes

Похожие видео

Running Qwen3 8B thinking on an iPhone Air with MLX. The model is quantized to 4-bit and runs pretty well.
0:45

Running Qwen3 8B thinking on an iPhone Air with MLX. The model is quantized to 4-bit and runs pretty well.

Awni Hannun

215,597 просмотров • 1 год назад

The new Qwen 3.5 4B runs incredibly well on M5. The model is close to GPT-4o in benchmarks. Running fully on-device with MLX.
0:20

The new Qwen 3.5 4B runs incredibly well on M5. The model is close to GPT-4o in benchmarks. Running fully on-device with MLX.

Adrien Grondin

230,466 просмотров • 6 месяцев назад

You can now vibecode your own WisprFlow or Monologue alternative that runs completely locally on Apple Silicon using MLX-Audio-Swift 🔥 Check out this live transcription of Dwarkesh Patel interview with Andrej Karpathy using Qwen3-ASR-0.6B quantized to 4bit on a M3 Max. It also runs in realtime on a iPhone 15 Pro and iPad Pro M1. No cloud. No API keys.
0:21

You can now vibecode your own WisprFlow or Monologue alternative that runs completely locally on Apple Silicon using MLX-Audio-Swift 🔥 Check out this live transcription of Dwarkesh Patel interview with Andrej Karpathy using Qwen3-ASR-0.6B quantized to 4bit on a M3 Max. It also runs in realtime on a iPhone 15 Pro and iPad Pro M1. No cloud. No API keys.

Prince Canuma

60,974 просмотров • 7 месяцев назад

GLM 4.6 runs quite fast on an M3 Ultra with mlx-lm even at higher precision. Pretty remarkable that it benchmarks competitive to the just-released Sonnet 4.5. Hope those benchmarks hold-up in day-to-day use. Here's a run using 5.5 bpw quantized model, generating 5.3k tokens at 17+ tok/sec using 244 GB. What prompts should I test?
0:54

GLM 4.6 runs quite fast on an M3 Ultra with mlx-lm even at higher precision. Pretty remarkable that it benchmarks competitive to the just-released Sonnet 4.5. Hope those benchmarks hold-up in day-to-day use. Here's a run using 5.5 bpw quantized model, generating 5.3k tokens at 17+ tok/sec using 244 GB. What prompts should I test?

Awni Hannun

68,539 просмотров • 11 месяцев назад

GPT-4o level intelligence running on your phone! MiniCPM-V 4.5 delivers enterprise-grade AI performance in just 8B parameters, outperforming models like GPT-4o, Gemini-2.0 Pro on vision and language tasks. - 30+ language support - Runs smoothly on iPhone/iPad 100% open-source!
0:28

GPT-4o level intelligence running on your phone! MiniCPM-V 4.5 delivers enterprise-grade AI performance in just 8B parameters, outperforming models like GPT-4o, Gemini-2.0 Pro on vision and language tasks. - 30+ language support - Runs smoothly on iPhone/iPad 100% open-source!

Akshay 🚀

84,720 просмотров • 1 год назад

Woah, DeepSeek R1 1.5B runs perfectly on my iPhone 15 pro with MLX You can run unlimited SOTA models for free forever, in your pocket More importantly, this change the game for mobile AI apps pricing strategy: - Previously, You literally can’t sell an LLM powered app at $9.99 due to API cost - But now you can Making a tutorial of how to do this, stay tuned…
0:44

Woah, DeepSeek R1 1.5B runs perfectly on my iPhone 15 pro with MLX You can run unlimited SOTA models for free forever, in your pocket More importantly, this change the game for mobile AI apps pricing strategy: - Previously, You literally can’t sell an LLM powered app at $9.99 due to API cost - But now you can Making a tutorial of how to do this, stay tuned…

Jason Zhou

12,086 просмотров • 1 год назад

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.
1:11

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.

Awni Hannun

60,599 просмотров • 7 месяцев назад

The latest Qwen 3 VL by Qwen running on iPhone 17 Pro with MLX Qwen 3 VL brings upgraded visual understanding, recognition, and OCR capabilities without sacrificing text performance like previous models The 4B model here is close to Qwen 2.5 VL 72B in many benchmarks
0:21

The latest Qwen 3 VL by Qwen running on iPhone 17 Pro with MLX Qwen 3 VL brings upgraded visual understanding, recognition, and OCR capabilities without sacrificing text performance like previous models The 4B model here is close to Qwen 2.5 VL 72B in many benchmarks

Adrien Grondin

109,700 просмотров • 11 месяцев назад

I open sourced my custom tab bar and window splits library for Swift. Say hi to Bonsplit. CoreAnimation-driven animations, pretty flexible/configurable, feels right at home on the Mac and showcased in a demo app. Read the docs/see examples/get the code
0:24

I open sourced my custom tab bar and window splits library for Swift. Say hi to Bonsplit. CoreAnimation-driven animations, pretty flexible/configurable, feels right at home on the Mac and showcased in a demo app. Read the docs/see examples/get the code

almonk

28,474 просмотров • 8 месяцев назад

Let’s check in on the Bridgeport Islanders-OH MY GOD Fair to say that the rivalry between Bridgeport and Hartford runs pretty deep when a skater and goalie drop the gloves
0:29

Let’s check in on the Bridgeport Islanders-OH MY GOD Fair to say that the rivalry between Bridgeport and Hartford runs pretty deep when a skater and goalie drop the gloves

Nicole Shirman

265,794 просмотров • 6 месяцев назад

Congratulations to the Cohere team on the release of Cohere Transcribe Arabic! 🎉 Runs natively on mlx-audio (Python + Swift) from day-0 🚀 What's inside: → 2B params, Conformer encoder-decoder (audio-in, text-out) → Large Conformer encoder for acoustic representations + lightweight Transformer decoder for token generation → Built for Arabic dialects and Arabic-English code-switching → Auto-resampling to 16kHz + stereo→mono handling built into preprocessing → Apache 2.0 — fully open for community research Currently topping the Open Universal Arabic ASR Leaderboard 🥇 Get started today: 🐍 Python uv pip install -U mlx-audio 🍎 Swift
1:02

Congratulations to the Cohere team on the release of Cohere Transcribe Arabic! 🎉 Runs natively on mlx-audio (Python + Swift) from day-0 🚀 What's inside: → 2B params, Conformer encoder-decoder (audio-in, text-out) → Large Conformer encoder for acoustic representations + lightweight Transformer decoder for token generation → Built for Arabic dialects and Arabic-English code-switching → Auto-resampling to 16kHz + stereo→mono handling built into preprocessing → Apache 2.0 — fully open for community research Currently topping the Open Universal Arabic ASR Leaderboard 🥇 Get started today: 🐍 Python uv pip install -U mlx-audio 🍎 Swift

Prince Canuma

16,680 просмотров • 2 месяцев назад

Just got to try the AirPods Pro 3 and its new real-time, in-ear heart rate sensor with an outdoor walk workout. Here's a video of my heart rate being monitored in the Fitness app on iPhone 17 Pro #AppleEvent
1:39

Just got to try the AirPods Pro 3 and its new real-time, in-ear heart rate sensor with an outdoor walk workout. Here's a video of my heart rate being monitored in the Fitness app on iPhone 17 Pro #AppleEvent

Ray Wong

412,403 просмотров • 1 год назад

New lighting tech in BAD Game (code name). 100% real time and runs at 240Hz 4K on my RTX 4090. 8x faster than my initial 16x SDF-ray per pixel proto. 1 ray per pixel and a new 6x6 bilateral filter (shared between SSAO and SDF rays). Starting to look pretty good :)
0:48

New lighting tech in BAD Game (code name). 100% real time and runs at 240Hz 4K on my RTX 4090. 8x faster than my initial 16x SDF-ray per pixel proto. 1 ray per pixel and a new 6x6 bilateral filter (shared between SSAO and SDF rays). Starting to look pretty good :)

Sebastian Aaltonen

18,646 просмотров • 11 дней назад

llama3 8B (not quantized) running on an heterogeneous home cluster made of: - iPhone 15 Pro Max - iPad Pro (not sure which version XD) - MacBook Pro ( M1 Max ) - NVIDIA GeForce 3080 (not visible in video) - 2x NVIDIA Titan X Pascal Very soon also supporting Android (I *have* to also add my NVIDIA Shield GPU!!!!!). Single code base, single model format (reduced and optimally distributed to every node to save space). Everything (including iOS code) is open here ... it would be really nice, with the help of the community, taking this project to the next level in terms of optimization and support. My vision is about a distributed inference server that can run any model on any backend in any cluster topology - let's fight programmed obsolescence and democratize inference!
0:26

llama3 8B (not quantized) running on an heterogeneous home cluster made of: - iPhone 15 Pro Max - iPad Pro (not sure which version XD) - MacBook Pro ( M1 Max ) - NVIDIA GeForce 3080 (not visible in video) - 2x NVIDIA Titan X Pascal Very soon also supporting Android (I *have* to also add my NVIDIA Shield GPU!!!!!). Single code base, single model format (reduced and optimally distributed to every node to save space). Everything (including iOS code) is open here ... it would be really nice, with the help of the community, taking this project to the next level in terms of optimization and support. My vision is about a distributed inference server that can run any model on any backend in any cluster topology - let's fight programmed obsolescence and democratize inference!

Simone Margaritelli

304,161 просмотров • 2 лет назад

WebGPU + Compute Shaders = 🤯 >500K stars runs at 60fps on my M1 Pro ⚡️ In my latest blog post, I walk through how to build this interactive galaxy simulation 🌌 Source code and live demo link in comments 👇🏻 #threejs #webgl #webgpu #shaders
0:46

WebGPU + Compute Shaders = 🤯 >500K stars runs at 60fps on my M1 Pro ⚡️ In my latest blog post, I walk through how to build this interactive galaxy simulation 🌌 Source code and live demo link in comments 👇🏻 #threejs #webgl #webgpu #shaders

Dan Greenheck

18,629 просмотров • 10 месяцев назад

Apple used "15" iPhone 17 Pro Max units to broadcast a Soccer match LIVE on Apple TV… WOW 15 iPhones placed across an 84,000-square-foot pitch capturing cinematic angles, closeups and tracking shots in real time. This is beyond smartphone cameras now!!!
0:39

Apple used "15" iPhone 17 Pro Max units to broadcast a Soccer match LIVE on Apple TV… WOW 15 iPhones placed across an 84,000-square-foot pitch capturing cinematic angles, closeups and tracking shots in real time. This is beyond smartphone cameras now!!!

TechDroider

1,448,984 просмотров • 4 месяцев назад

A 27B parameter model used to need a server room. Now it runs on: • iPhone • Android • Mac PrismML's Bonsai makes it possible. • 1-bit weights • 27B params in just 3.9GB • ~90% of full precision quality (PrismML evals) • Better than the 2-bit version at less than half the size The entire Bonsai family is now live in RunAnywhere. • 1.7B to 27B models • 1-bit + 2-bit ternary • Thinking mode On iOS and Mac, Bonsai runs through ggml's llama.cpp and Apple's MLX. On Android, Bonsai runs through ggml's llama.cpp and directly on the Qualcomm Hexagon NPU through QHexRT, our proprietary inference runtime. We built custom silicon kernels to make 1-bit inference on an NPU possible for the first time. Two years ago this needed a data center. Now it thinks in airplane mode. Now available in the RunAnywhere app on the App Store and Google Play.
0:27

A 27B parameter model used to need a server room. Now it runs on: • iPhone • Android • Mac PrismML's Bonsai makes it possible. • 1-bit weights • 27B params in just 3.9GB • ~90% of full precision quality (PrismML evals) • Better than the 2-bit version at less than half the size The entire Bonsai family is now live in RunAnywhere. • 1.7B to 27B models • 1-bit + 2-bit ternary • Thinking mode On iOS and Mac, Bonsai runs through ggml's llama.cpp and Apple's MLX. On Android, Bonsai runs through ggml's llama.cpp and directly on the Qualcomm Hexagon NPU through QHexRT, our proprietary inference runtime. We built custom silicon kernels to make 1-bit inference on an NPU possible for the first time. Two years ago this needed a data center. Now it thinks in airplane mode. Now available in the RunAnywhere app on the App Store and Google Play.

RunAnywhere (YC W26)

17,587 просмотров • 2 месяцев назад

Here’s how I wake my Mac up remotely from my iPhone and connect in from anywhere. This is new in Astropad Workbench 1.1. We’ve made sleep handling much smarter. Small thing, but very useful when you need to check on a machine fast.
0:36

Here’s how I wake my Mac up remotely from my iPhone and connect in from anywhere. This is new in Astropad Workbench 1.1. We’ve made sleep handling much smarter. Small thing, but very useful when you need to check on a machine fast.

Matt Ronge

79,644 просмотров • 5 месяцев назад

Sitting in the back of an Uber (XL Black) 6 minutes away from my destination And I decided to do a $100,000 bonus buy from my iPhone 17 pro on Stake.com… Even the driver was screaming when the bananas connected
1:34

Sitting in the back of an Uber (XL Black) 6 minutes away from my destination And I decided to do a $100,000 bonus buy from my iPhone 17 pro on Stake.com… Even the driver was screaming when the bananas connected

BTCs

116,537 просмотров • 2 месяцев назад