Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

OpenCode + MLX + Qwen3.5-397B-A17B-4bit. Video is 8x, but the goal is showing that It works! This is something unimaginable just few months ago. MLX Team is pushing like crazy and M5 Ultra will do the rest 🚀

ivanfioravanti's profile picture

Ivan Fioravanti ᯅ

38,156 subscribers

48,692 Aufrufe • vor 6 Monaten •via X (Twitter)

BildungWissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Qwen3.5-397B-A17B with Vision running on a Single M3 Ultra 🚀 Shout out to the Qwen team because they cooked with this model! Get started today: > uv pip install -U mlx-vlm Model collection 👇🏽 Repo:
0:59

Qwen3.5-397B-A17B with Vision running on a Single M3 Ultra 🚀 Shout out to the Qwen team because they cooked with this model! Get started today: > uv pip install -U mlx-vlm Model collection 👇🏽 Repo:

Prince Canuma

11,386 Aufrufe • vor 6 Monaten

A long time coming but new mlx-lm is here with better batching support in the server and Gemma 4. pip install -U mlx-lm Here is a video where a single M3 Ultra serves 5 opencode sessions with Gemma 4 26B that process ~130k tokens in ~1.5 minutes.
2:03

A long time coming but new mlx-lm is here with better batching support in the server and Gemma 4. pip install -U mlx-lm Here is a video where a single M3 Ultra serves 5 opencode sessions with Gemma 4 26B that process ~130k tokens in ~1.5 minutes.

Angelos Katharopoulos

66,202 Aufrufe • vor 4 Monaten

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx
0:40

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx

raullen

22,760 Aufrufe • vor 2 Monaten

DAAUUUMMMM! Deep Seek R1 - 4bit on a single Mac Studio 512gb. 18.26 Tokens per second with MLX. Took over a minute to load the model but I sped that up. Generation was great! thanks Awni Hannun mlx is the future.
1:30

DAAUUUMMMM! Deep Seek R1 - 4bit on a single Mac Studio 512gb. 18.26 Tokens per second with MLX. Took over a minute to load the model but I sped that up. Generation was great! thanks Awni Hannun mlx is the future.

Austin Vance

174,289 Aufrufe • vor 1 Jahr

oMLX is working really well as single machine inference engine for coding agents! Caching is managed perfectly (it can use a ton of disk space, be aware!) and oQ quantization delivers great results. Behind the scenes it uses the standard MLX building blocks (75% created by Prince Canuma 🙏): - mlx-lm - mlx-vlm - mlx-embeddings - mlx-audio I tested Qwen3.6-35B-A3B-oQ6 on M5 Max with two pi instances and it was fast and furious and leveraging cache like crazy as you can see in the video. Let me try to create some oQ versions (2,4,6?) of MiniMax M2.7 now and then I'll pass to distributed inference. I must win! 💪
3:57

oMLX is working really well as single machine inference engine for coding agents! Caching is managed perfectly (it can use a ton of disk space, be aware!) and oQ quantization delivers great results. Behind the scenes it uses the standard MLX building blocks (75% created by Prince Canuma 🙏): - mlx-lm - mlx-vlm - mlx-embeddings - mlx-audio I tested Qwen3.6-35B-A3B-oQ6 on M5 Max with two pi instances and it was fast and furious and leveraging cache like crazy as you can see in the video. Let me try to create some oQ versions (2,4,6?) of MiniMax M2.7 now and then I'll pass to distributed inference. I must win! 💪

Ivan Fioravanti ᯅ

19,420 Aufrufe • vor 4 Monaten

Qwen3.5 is here 🚀 397B params, just 17B active. Native multimodal agents for coding, reasoning, GUI + video. 200+ languages. Open weights. Real scale. The next frontier is open. 🔗
1:15

Qwen3.5 is here 🚀 397B params, just 17B active. Native multimodal agents for coding, reasoning, GUI + video. 200+ languages. Open weights. Real scale. The next frontier is open. 🔗

Hugging Models

107,931 Aufrufe • vor 6 Monaten

SAM-2 is apache license - you can indeed just build crazy apps with them without fear. This weekend I ported SAM-2 models to run optimally on Apple Silicon. I have also quantized all the SAM models to 16bit, 8bit, and 4bit variants (collection below) Repo is open source and pip installable. Comes with all you'll need for interactive video segmentation - - bidirectional propagation, - positive and negative clicks, - bounding boxes, - streaming masks back to user, - lightweight fast-api server that you can extend I basically asked Codex to port over the original Facebook sam2 codebase to mlx and optimize it for mlx. And it just worked. This demo for example is a 4bit base-plus model (~100MB download) To speed up, you can also do spatial or temporal downsampling. That speeds inference on my old macbook by ~3x and the hit in parity is negligible.
0:53

SAM-2 is apache license - you can indeed just build crazy apps with them without fear. This weekend I ported SAM-2 models to run optimally on Apple Silicon. I have also quantized all the SAM models to 16bit, 8bit, and 4bit variants (collection below) Repo is open source and pip installable. Comes with all you'll need for interactive video segmentation - - bidirectional propagation, - positive and negative clicks, - bounding boxes, - streaming masks back to user, - lightweight fast-api server that you can extend I basically asked Codex to port over the original Facebook sam2 codebase to mlx and optimize it for mlx. And it just worked. This demo for example is a 4bit base-plus model (~100MB download) To speed up, you can also do spatial or temporal downsampling. That speeds inference on my old macbook by ~3x and the hit in parity is negligible.

AVB

17,396 Aufrufe • vor 3 Monaten

I did it! It works! Using GLM-4.7-4bit with mlx_lm.server and opencode to fix real code locally! 🔥 Here single M3 Ultra 512GB, nex step phase will be 2 using Tensor Parallelism and then apply same changes to exo. Prefill is slow on a single machine, but generation is good.
1:44

I did it! It works! Using GLM-4.7-4bit with mlx_lm.server and opencode to fix real code locally! 🔥 Here single M3 Ultra 512GB, nex step phase will be 2 using Tensor Parallelism and then apply same changes to exo. Prefill is slow on a single machine, but generation is good.

Ivan Fioravanti ᯅ

44,000 Aufrufe • vor 7 Monaten

This is MiniMax-M2.5 MLX running in LM Studio on an Apple Mac Studio M3 Ultra 512GB. Fast enough out of the box for hosting OpenClaw, n8n workflows, and Open WebUI for the team.
0:53

This is MiniMax-M2.5 MLX running in LM Studio on an Apple Mac Studio M3 Ultra 512GB. Fast enough out of the box for hosting OpenClaw, n8n workflows, and Open WebUI for the team.

Patrick J Kennedy

76,574 Aufrufe • vor 6 Monaten

I’ve been using the new MacBook Pro M5 since launch and it’s a beast, especially compared to my old M1! It’s good for running the smaller AI models locally but also everything you do is just so fast. Here’s Gemma 3 QAT 12B running with MLX:
0:35

I’ve been using the new MacBook Pro M5 since launch and it’s a beast, especially compared to my old M1! It’s good for running the smaller AI models locally but also everything you do is just so fast. Here’s Gemma 3 QAT 12B running with MLX:

Adrien Grondin

101,950 Aufrufe • vor 8 Monaten

Maxed out M3 Ultra is here! 🚀 Huge thank you to the entire MLX community for making this possible. With your support, I can dedicate myself full-time to advancing MLX as an independent developer. For the first time, I have the hardware to match the ambition—no more bottlenecks holding back the biggest models and most challenging work. Excited for what’s ahead. Let’s build something incredible together.
1:27

Maxed out M3 Ultra is here! 🚀 Huge thank you to the entire MLX community for making this possible. With your support, I can dedicate myself full-time to advancing MLX as an independent developer. For the first time, I have the hardware to match the ambition—no more bottlenecks holding back the biggest models and most challenging work. Excited for what’s ahead. Let’s build something incredible together.

Prince Canuma

27,880 Aufrufe • vor 10 Monaten

Qwen3.6 27B landed yesterday, so I ran it on 4 setups side-by-side to see how they stack up: 🔴 RTX 4090 — 45.59 tok/s, TTFT 525ms 🟢 RTX 5090 — 51.83 tok/s, TTFT 752ms ⚫️ M2 Ultra — 22.30 tok/s, TTFT 216ms 🟣 DGX Spark — 11.08 tok/s, TTFT 319ms This is a standard test: no tuning, just the out-of-the-box experience. For the NVIDIA cards I used llama.cpp with Unsloth's UD-Q4_K_XL quant. For the M2 Ultra I used MLX with Unsloth's UD-MLX-4bit quant, since MLX is the native path on Apple Silicon. Please consider this as the baseline, you can definitely squeeze more out of every one of these with fine-tuned settings.
3:49

Qwen3.6 27B landed yesterday, so I ran it on 4 setups side-by-side to see how they stack up: 🔴 RTX 4090 — 45.59 tok/s, TTFT 525ms 🟢 RTX 5090 — 51.83 tok/s, TTFT 752ms ⚫️ M2 Ultra — 22.30 tok/s, TTFT 216ms 🟣 DGX Spark — 11.08 tok/s, TTFT 319ms This is a standard test: no tuning, just the out-of-the-box experience. For the NVIDIA cards I used llama.cpp with Unsloth's UD-Q4_K_XL quant. For the M2 Ultra I used MLX with Unsloth's UD-MLX-4bit quant, since MLX is the native path on Apple Silicon. Please consider this as the baseline, you can definitely squeeze more out of every one of these with fine-tuned settings.

stevibe

104,479 Aufrufe • vor 3 Monaten

Omo may this not happen to you. He just bought this car few months ago, and now it is gone just like that😭😭😭
2:56

Omo may this not happen to you. He just bought this car few months ago, and now it is gone just like that😭😭😭

Dapsy𓃵

25,216 Aufrufe • vor 1 Jahr

GLM-4.7-8bit (350GB) running at 19 toks/s on two M3 Ultra 512GB using Tensor Parallelism with EXO - MLX, versus 14 toks/s with single node. 🚀 Now context benchmarking & then OpenCode tests 🔥 Note: this is from sources, I had to change things to run it.
0:50

GLM-4.7-8bit (350GB) running at 19 toks/s on two M3 Ultra 512GB using Tensor Parallelism with EXO - MLX, versus 14 toks/s with single node. 🚀 Now context benchmarking & then OpenCode tests 🔥 Note: this is from sources, I had to change things to run it.

Ivan Fioravanti ᯅ

327,687 Aufrufe • vor 7 Monaten

“I'm here now. And the main goal is this team, not just me. I know it's a great story and stuff like that, but my main focus is to do my job and help this team win games.” - Tennessee QB Joey Aguilar
0:24

“I'm here now. And the main goal is this team, not just me. I know it's a great story and stuff like that, but my main focus is to do my job and help this team win games.” - Tennessee QB Joey Aguilar

zach ragan

51,225 Aufrufe • vor 11 Monaten

🥊 Time for a new fight Qwen3.5 397B A17B vs Qwen3.6 27B 🌸 "Cherry Blossom" (↓ prompt below) Using OpenRouter for 397B Running 27B locally on a single RTX 5080 Wow! 🤯 I'm sure this one is going to divide opinions. 397B feels cinematic with the flare and shadows it added, while 27B is just beautiful; it has a completely different color palette and the leaves are falling from the branches as asked. It's really tough to decide which model is the best. I was about to call it a tie, but then I remembered that 397B needs 180GB VRAM to run locally 😅 Don't get me wrong the model is amazing, but right now the VRAM it requires doesn't feel justified. It's simply not 10× better than 27B. We'll have to dig deeper with more prompts to be sure, let's see what each one has in store for us. What do you think?
0:30

🥊 Time for a new fight Qwen3.5 397B A17B vs Qwen3.6 27B 🌸 "Cherry Blossom" (↓ prompt below) Using OpenRouter for 397B Running 27B locally on a single RTX 5080 Wow! 🤯 I'm sure this one is going to divide opinions. 397B feels cinematic with the flare and shadows it added, while 27B is just beautiful; it has a completely different color palette and the leaves are falling from the branches as asked. It's really tough to decide which model is the best. I was about to call it a tie, but then I remembered that 397B needs 180GB VRAM to run locally 😅 Don't get me wrong the model is amazing, but right now the VRAM it requires doesn't feel justified. It's simply not 10× better than 27B. We'll have to dig deeper with more prompts to be sure, let's see what each one has in store for us. What do you think?

left curve dev

21,224 Aufrufe • vor 3 Monaten

I bought the M4 Max 48GB RAM and first thing I did is to test local LLMs. It can barely run llama3.3 70b, but qwen2.5-coder 32b works very well. I'll test with MLX next ✌️
0:31

I bought the M4 Max 48GB RAM and first thing I did is to test local LLMs. It can barely run llama3.3 70b, but qwen2.5-coder 32b works very well. I'll test with MLX next ✌️

Daniel Nguyen

315,827 Aufrufe • vor 1 Jahr

Steven Spielberg on artificial intelligence: “The soul is unimaginable and is ineffable, the soul. And it cannot be created by any algorithm, it is just something that exists in all of us.” #AI #Colbert
3:26

Steven Spielberg on artificial intelligence: “The soul is unimaginable and is ineffable, the soul. And it cannot be created by any algorithm, it is just something that exists in all of us.” #AI #Colbert

The Late Show

301,545 Aufrufe • vor 3 Jahren

Just for fun, here's what 32 simultaneous long-context generations with Qwen3 Next 80B looks like on an M3 Ultra. Using the new batch generation in mlx-lm. Context size for each is about 5k tokens:
0:41

Just for fun, here's what 32 simultaneous long-context generations with Qwen3 Next 80B looks like on an M3 Ultra. Using the new batch generation in mlx-lm. Context size for each is about 5k tokens:

Awni Hannun

50,272 Aufrufe • vor 11 Monaten