Loading video...

Video Failed to Load

Go Home

I’ve been using the new MacBook Pro M5 since launch and it’s a beast, especially compared to my old M1! It’s good for running the smaller AI models locally but also everything you do is just so fast. Here’s Gemma 3 QAT 12B running with MLX:

adrgrondin's profile picture

Adrien Grondin

5,668 subscribers

101,950 views • 8 months ago •via X (Twitter)

EducationScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx
0:40

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx

raullen

22,760 views • 2 months ago

Subagents running locally and simultaneously on MacBook Pro M5 with Codex CLI + LM Studio to review code and find bugs using Qwen 3.6 Powered by the updated MLX engine with batching in beta in the app The batching speed boost is noticeable
1:40

Subagents running locally and simultaneously on MacBook Pro M5 with Codex CLI + LM Studio to review code and find bugs using Qwen 3.6 Powered by the updated MLX engine with batching in beta in the app The batching speed boost is noticeable

Adrien Grondin

75,549 views • 2 months ago

This afternoon I picked up a new Nvidia DGX Spark computer with the goal of trying to run Gemma 4 31b (4bit) on it locally as a server. Just 1.5 hours later, it’s working! Using Open WebUI on my MacBook as the interface and it’s connecting to my DGX Spark running as a Gemma 4 server.
0:46

This afternoon I picked up a new Nvidia DGX Spark computer with the goal of trying to run Gemma 4 31b (4bit) on it locally as a server. Just 1.5 hours later, it’s working! Using Open WebUI on my MacBook as the interface and it’s connecting to my DGX Spark running as a Gemma 4 server.

Dave Lee

139,246 views • 4 months ago

ParoQuant just got a big upgrade 🚀 ✅ Supports the new Qwen3.5 models ⚡ Now runs on MLX (fast local inference on Apple Silicon) 🧠 Preserves reasoning quality with 4-bit quantization We also built an agent demo running locally on my 4-year-old M2 Max. Can't wait to upgrade to an M5 Max and see what kind of magic we can do. ✨
0:46

ParoQuant just got a big upgrade 🚀 ✅ Supports the new Qwen3.5 models ⚡ Now runs on MLX (fast local inference on Apple Silicon) 🧠 Preserves reasoning quality with 4-bit quantization We also built an agent demo running locally on my 4-year-old M2 Max. Can't wait to upgrade to an M5 Max and see what kind of magic we can do. ✨

Zhijian Liu

49,442 views • 5 months ago

Running Gemma 4 12B locally at 28T/s. Facing issues with vision in LM Studio, but text performance is quite solid. Asked it to create a landing page for a tech event. Video is fast forwarded for the demo.
0:40

Running Gemma 4 12B locally at 28T/s. Facing issues with vision in LM Studio, but text performance is quite solid. Asked it to create a landing page for a tech event. Video is fast forwarded for the demo.

AshutoshShrivastava

15,691 views • 2 months ago

I tested Gemma 4 12b on my Macbook Pro M5 Max with 128GB of unified memory. It runs VERY slow for a 12b parameter model. 44 tokens/second. The lava lamp that it produced was also terrible. I would stick to Qwen 3.6 27b or Qwen 3.6 35b. Google is still way behind, even with their local models.
1:36

I tested Gemma 4 12b on my Macbook Pro M5 Max with 128GB of unified memory. It runs VERY slow for a 12b parameter model. 44 tokens/second. The lava lamp that it produced was also terrible. I would stick to Qwen 3.6 27b or Qwen 3.6 35b. Google is still way behind, even with their local models.

BridgeMind

70,626 views • 2 months ago

A voice agent powered by gpt-oss. Running locally on my macBook. Demo recorded in a Waymo with WiFi turned off. I'm still on my space game voice AI kick, obviously. Code link below. For conversational voice AI, you want to set the gpt-oss reasoning behavior to "low". (The default is "medium".) Notes on how to do that and a jinja template you can use are in the repo. The LLM in the demo video is the big, 120B version of gpt-oss. You can use the smaller, 20B model for this, of course. But OpenAI really did a cool thing here designing the 120B model to run in "just" 80GB of VRAM. And the llama.cpp mlx inference is fast: ~250ms TTFT. Running a big model on-device feels like a time warp into the future of AI.
4:35

A voice agent powered by gpt-oss. Running locally on my macBook. Demo recorded in a Waymo with WiFi turned off. I'm still on my space game voice AI kick, obviously. Code link below. For conversational voice AI, you want to set the gpt-oss reasoning behavior to "low". (The default is "medium".) Notes on how to do that and a jinja template you can use are in the repo. The LLM in the demo video is the big, 120B version of gpt-oss. You can use the smaller, 20B model for this, of course. But OpenAI really did a cool thing here designing the 120B model to run in "just" 80GB of VRAM. And the llama.cpp mlx inference is fast: ~250ms TTFT. Running a big model on-device feels like a time warp into the future of AI.

kwindla

202,147 views • 1 year ago

The top question I got was does Halo support local LLM running on a Mac. Here's an example of the app using gemma-4-26b-a4b-qat via LM Studio running on a base M5/32GB. It's giving an average of 40 tok/s and in my brief testing is able to execute all tool calls correctly. Very usable with this app.
0:37

The top question I got was does Halo support local LLM running on a Mac. Here's an example of the app using gemma-4-26b-a4b-qat via LM Studio running on a base M5/32GB. It's giving an average of 40 tok/s and in my brief testing is able to execute all tool calls correctly. Very usable with this app.

Tanmay

11,682 views • 1 month ago

people think running AI locally requires: → $3,000 MacBook Pro → RTX 4090 → $20/month cloud subscription nvidia just dropped a $249 computer. 67 TOPS. runs llama 3.1-8B locally. no internet. no API. no monthly fee. ever. smaller than your router. costs the same as AirPods. runs the same models you pay $240/year to access via ChatGPT. the local AI era just got a price tag. $249.
5:54

people think running AI locally requires: → $3,000 MacBook Pro → RTX 4090 → $20/month cloud subscription nvidia just dropped a $249 computer. 67 TOPS. runs llama 3.1-8B locally. no internet. no API. no monthly fee. ever. smaller than your router. costs the same as AirPods. runs the same models you pay $240/year to access via ChatGPT. the local AI era just got a price tag. $249.

self.dll

321,624 views • 2 months ago

It’s been a while since my last post, but I just wanted to say that the Cropdeck launch went pretty well! I’ve been busy preparing patches for critical bugs and balance issues, but it’s getting calmer now 😁
0:34

It’s been a while since my last post, but I just wanted to say that the Cropdeck launch went pretty well! I’ve been busy preparing patches for critical bugs and balance issues, but it’s getting calmer now 😁

Piotrek - Wishlist Cropdeck 🌱 on Steam!

11,106 views • 2 months ago

The P211 GTO Is here! And yes, it takes 320 Mags! I feel like I’ve been waiting 17 years for this but I’m thrilled to announce that it’s officially on its way to you. This of course is extra special for me since I’ve been running this platform since I was 15 years old.
2:34

The P211 GTO Is here! And yes, it takes 320 Mags! I feel like I’ve been waiting 17 years for this but I’m thrilled to announce that it’s officially on its way to you. This of course is extra special for me since I’ve been running this platform since I was 15 years old.

Max Michel

15,917 views • 1 year ago

We've partnered to bring more Gemma 3 quantized models to you! 🚀 We worked with Georgi Gerganov llama.cpp, LM Studio, MLX, ollama to make sure you can run it using your favorite tool! Gemma models optimized with QAT, reduce memory requirements while keeping quality! All models checkpoints are available on Hugging Face and Kaggle. 🤗 What does this mean? - Gemma 3 27B (int4): Fits on NVIDIA RTX 3090 (24GB VRAM) or similar. - Gemma 3 12B (int4): Only needs a NVIDIA RTX 4060 (8GB VRAM) or similar. - Gemma 3 4B, 1B (int4): Run anything with more than 2.5GB Memory. Want to see it in action? Video below shows how easy it is to get started using LMStudio:
0:20

We've partnered to bring more Gemma 3 quantized models to you! 🚀 We worked with Georgi Gerganov llama.cpp, LM Studio, MLX, ollama to make sure you can run it using your favorite tool! Gemma models optimized with QAT, reduce memory requirements while keeping quality! All models checkpoints are available on Hugging Face and Kaggle. 🤗 What does this mean? - Gemma 3 27B (int4): Fits on NVIDIA RTX 3090 (24GB VRAM) or similar. - Gemma 3 12B (int4): Only needs a NVIDIA RTX 4060 (8GB VRAM) or similar. - Gemma 3 4B, 1B (int4): Run anything with more than 2.5GB Memory. Want to see it in action? Video below shows how easy it is to get started using LMStudio:

Philipp Schmid

15,996 views • 1 year ago

🗓️ Release Week Recap Big week. mlx-audio and mlx-vlm are now among some of the fastest-growing OSS projects. Here’s what we shipped last week. Gemma 4 on Apple Silicon Two awesome releases by our partner Google DeepMind & Google Gemma : > Gemma 4 12B — their new dense, unified multimodal model. It uses an encoder free audio path and simplified vision encoder. > Gemma 4 QAT — quantization-aware training checkpoints, optimized to run locally on consumer GPUs and edge devices, compressing the model while preserving the quality you expect from Gemma 4. On the audio 🎧 side we added support for 15+ new TTS, ASR & VAD models, faster long-form transcription, and an expanded OpenAI-compatible audio server. All local on Apple Silicon. Huge thanks to every contributor and my co-maintainer Lucas Newman. 🙏🏽
1:34

🗓️ Release Week Recap Big week. mlx-audio and mlx-vlm are now among some of the fastest-growing OSS projects. Here’s what we shipped last week. Gemma 4 on Apple Silicon Two awesome releases by our partner Google DeepMind & Google Gemma : > Gemma 4 12B — their new dense, unified multimodal model. It uses an encoder free audio path and simplified vision encoder. > Gemma 4 QAT — quantization-aware training checkpoints, optimized to run locally on consumer GPUs and edge devices, compressing the model while preserving the quality you expect from Gemma 4. On the audio 🎧 side we added support for 15+ new TTS, ASR & VAD models, faster long-form transcription, and an expanded OpenAI-compatible audio server. All local on Apple Silicon. Huge thanks to every contributor and my co-maintainer Lucas Newman. 🙏🏽

Prince Canuma

11,731 views • 2 months ago

Hot tip for anyone doing AI dev: Use Ollama to easily run models like Deepseek-r1 or Gemma locally on your machine. It downloads them and spins up a server with an OpenAI SDK compatible API The smaller models are fast and good enough to work on new features or debug streaming without having to pay for API requests
2:08

Hot tip for anyone doing AI dev: Use Ollama to easily run models like Deepseek-r1 or Gemma locally on your machine. It downloads them and spins up a server with an OpenAI SDK compatible API The smaller models are fast and good enough to work on new features or debug streaming without having to pay for API requests

Wes Bos

153,499 views • 1 year ago

just built a local AI waifu on my two macbooks by using the oss parallax inference library!!! not super smooth yet but parallax is still the easiest way to run llms locally on top of mlx - and no bigco can steal my waifu. still need folks to help her see/hear/draw since mlx has no legit wrapper yet. i’m running a campaign giving away mac minis + dgx spark. please please share your localhost ai model setup by using parallax (you don't need a gpu to try self-hosting you can just start with an extra macbook you borrowed from your friend like i did) or any fun app built on top of yourself host model to link below. come build sovereign intelligence with me 🫶
7:41

just built a local AI waifu on my two macbooks by using the oss parallax inference library!!! not super smooth yet but parallax is still the easiest way to run llms locally on top of mlx - and no bigco can steal my waifu. still need folks to help her see/hear/draw since mlx has no legit wrapper yet. i’m running a campaign giving away mac minis + dgx spark. please please share your localhost ai model setup by using parallax (you don't need a gpu to try self-hosting you can just start with an extra macbook you borrowed from your friend like i did) or any fun app built on top of yourself host model to link below. come build sovereign intelligence with me 🫶

Asuka Zheng🎀@ICML

178,998 views • 8 months ago

It’s official: I’m running for Congress. I’ve spent my career holding the powerful accountable as a political and legal journalist. I’m running for Congress to tell you the truth, to fight for New York…and to Unf*ck Our Country. Join my campaign:
2:19

It’s official: I’m running for Congress. I’ve spent my career holding the powerful accountable as a political and legal journalist. I’m running for Congress to tell you the truth, to fight for New York…and to Unf*ck Our Country. Join my campaign:

Mike Sacks

333,809 views • 1 year ago

It’s wild watching AI Agents work with software. Here’s ChatGPT connected to Box running a deep research task on market analysis documents. The future is just giving AI Agents the data and systems to comb through and having them work for you in the background.
1:33

It’s wild watching AI Agents work with software. Here’s ChatGPT connected to Box running a deep research task on market analysis documents. The future is just giving AI Agents the data and systems to comb through and having them work for you in the background.

Aaron Levie

235,980 views • 1 year ago

Qwen 3.6 35B running at over 100 tokens per second on my $5,399 MacBook Pro M5 Max. This is the best local AI model I have ever run. 128GB of unified memory. No cloud. No API costs. No rate limits. Just raw local inference at speeds I didn't think were possible on a laptop. This model is more intelligent than GPT 5 on benchmarks. Running locally. On a MacBook. For free after the hardware cost. I said local AI would never compete with frontier. I'm starting to rethink that. The gap is closing faster than anyone expected.
1:04

Qwen 3.6 35B running at over 100 tokens per second on my $5,399 MacBook Pro M5 Max. This is the best local AI model I have ever run. 128GB of unified memory. No cloud. No API costs. No rate limits. Just raw local inference at speeds I didn't think were possible on a laptop. This model is more intelligent than GPT 5 on benchmarks. Running locally. On a MacBook. For free after the hardware cost. I said local AI would never compete with frontier. I'm starting to rethink that. The gap is closing faster than anyone expected.

BridgeMind

62,800 views • 2 months ago

hoping for a fast recovery to you my leeknow, so you can start running again and do the things you love ( but please get all the rest you need first)
1:49

hoping for a fast recovery to you my leeknow, so you can start running again and do the things you love ( but please get all the rest you need first)

minhotron 3000

61,734 views • 8 months ago