正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Qwen 2.5 Coder Q4 M4 Max Inference test. Apple MLX vs Ollama: - MLX: 23.97 toks/sec 🥇🔥 - Ollama: 18.33 toks/sec 🥈 Here a video to show results

Ivan Fioravanti ᯅ

15,122 subscribers

34,881 次观看 • 1 年前 •via X (Twitter)

教育科学技术

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Qwen 32B (4-bit) generates at >40 toks/sec on an M4 Max with assisted decoding and Qwen 0.5B as the draft model. Coming soon to mlx-lm. Compare regular decoding (left) to assisted decoding (right):

Qwen 32B (4-bit) generates at >40 toks/sec on an M4 Max with assisted decoding and Qwen 0.5B as the draft model. Coming soon to mlx-lm. Compare regular decoding (left) to assisted decoding (right):

Awni Hannun

50,353 次观看 • 1 年前

Llama 3.3 70B 4-bit runs nicely on a 64GB M3 Max with in MLX LM (~10 toks/sec). Would be even faster on an M4 Max. Yesterday's server-only 405B is today's laptop 70B:

Llama 3.3 70B 4-bit runs nicely on a 64GB M3 Max with in MLX LM (~10 toks/sec). Would be even faster on an M4 Max. Yesterday's server-only 405B is today's laptop 70B:

Awni Hannun

110,534 次观看 • 1 年前

Llama 3.3 70B 4-bit running on a 128GB M4 Max with MLX LM (~12 toks/sec) with mactop on the right. "Good enough", local, private and open models are now more accessible than ever!

Llama 3.3 70B 4-bit running on a 128GB M4 Max with MLX LM (~12 toks/sec) with mactop on the right. "Good enough", local, private and open models are now more accessible than ever!

Ivan Fioravanti ᯅ

43,937 次观看 • 1 年前

Qwen3-Coder-Flash runs quite fast on an M4 Max with mlx-lm. Running the 4-bit here, generated 4,467 tokens at >107 tokens/sec:

Qwen3-Coder-Flash runs quite fast on an M4 Max with mlx-lm. Running the 4-bit here, generated 4,467 tokens at >107 tokens/sec:

Awni Hannun

196,508 次观看 • 1 年前

oMLX last version seems having some issues in decoding. 🧐 Here a test with Qwen3.6-35B-A3B-MLX-6bit on M5 Max using. 🥇 LM Studio MLX 1.8.5 → 100.9 toks/s 🥈 mlx-vlm 0.6.2 → 100.1 toks/s 🥉 oMLX 0.4.2 dev3 → 58.7 toks/s 👀 Avg Gen TPS: oMLX 58.7 → LM Studio 100.9 (+71.8%) I have to thank pymike00 that raised this oMLX issue after seeing my video on using it with Codex. I bet there is a bug in oMLX chat and server at the moment, because internal benchmarks are ok, video attached. I bet Jun Kim will fix this soon 💪

oMLX last version seems having some issues in decoding. 🧐 Here a test with Qwen3.6-35B-A3B-MLX-6bit on M5 Max using. 🥇 LM Studio MLX 1.8.5 → 100.9 toks/s 🥈 mlx-vlm 0.6.2 → 100.1 toks/s 🥉 oMLX 0.4.2 dev3 → 58.7 toks/s 👀 Avg Gen TPS: oMLX 58.7 → LM Studio 100.9 (+71.8%) I have to thank pymike00 that raised this oMLX issue after seeing my video on using it with Codex. I bet there is a bug in oMLX chat and server at the moment, because internal benchmarks are ok, video attached. I bet Jun Kim will fix this soon 💪

Ivan Fioravanti ᯅ

20,066 次观看 • 2 个月前

M4 Mac AI Coding Cluster Uses EXO Labs to run LLMs (here Qwen 2.5 Coder 32B at 18 tok/sec) distributed across 4 M4 Mac Minis (Thunderbolt 5 80Gbps) and a MacBook Pro M4 Max. Local alternative to Cursor (benchmark comparison soon).

M4 Mac AI Coding Cluster Uses EXO Labs to run LLMs (here Qwen 2.5 Coder 32B at 18 tok/sec) distributed across 4 M4 Mac Minis (Thunderbolt 5 80Gbps) and a MacBook Pro M4 Max. Local alternative to Cursor (benchmark comparison soon).

Alex Cheema

517,686 次观看 • 1 年前

QLoRA fine-tuning 4-bit Gemma 2B on iPhone 15 Pro with MLX Swift. A nice size for fine-tuning on device, getting 70-100 toks/sec depending on the batch. Guide here:

QLoRA fine-tuning 4-bit Gemma 2B on iPhone 15 Pro with MLX Swift. A nice size for fine-tuning on device, getting 70-100 toks/sec depending on the batch. Guide here:

Awni Hannun

19,530 次观看 • 2 年前

Reasoning on Apple Vision Pro with Apple MLX and DeepSeek R1 Qwen 7B 4bit! 🔥 14 tokens per sec! 🔥 Note: sorry for the shaking footage, I was excited to see it running 😂

Reasoning on Apple Vision Pro with Apple MLX and DeepSeek R1 Qwen 7B 4bit! 🔥 14 tokens per sec! 🔥 Note: sorry for the shaking footage, I was excited to see it running 😂

Ivan Fioravanti ᯅ

24,324 次观看 • 1 年前

Running Ring 1T reasoning model on a single M3 Ultra with mlx-lm. It's quantized to 3.5 bits-per-weight. Uses 440GB and generated ~6k tokens at 18.2 toks/sec. Getting closer to GPT-5 at home.

Running Ring 1T reasoning model on a single M3 Ultra with mlx-lm. It's quantized to 3.5 bits-per-weight. Uses 440GB and generated ~6k tokens at 18.2 toks/sec. Getting closer to GPT-5 at home.

Awni Hannun

55,138 次观看 • 10 个月前

MLX Swift example can also QLoRA fine-tune Llama 3.2. Here's the 1B fine-tuning on my iPhone 15 Pro at > 150 toks/sec. A this rate only takes a few minutes to learn some decent adapters fully on-device.

MLX Swift example can also QLoRA fine-tune Llama 3.2. Here's the 1B fine-tuning on my iPhone 15 Pro at > 150 toks/sec. A this rate only takes a few minutes to learn some decent adapters fully on-device.

Awni Hannun

91,961 次观看 • 1 年前

🔥 DeepSeek R1 671B Q4 - M3 Ultra 512GB with MLX🔥 - 18.43 tokens/sec - Generates a p5js zero-shot, tested at video's end 😱 - Video in real-time, no acceleration! - First test, and I'm blown away! Prompt: "Create an amazing animation using p5js"

🔥 DeepSeek R1 671B Q4 - M3 Ultra 512GB with MLX🔥 - 18.43 tokens/sec - Generates a p5js zero-shot, tested at video's end 😱 - Video in real-time, no acceleration! - First test, and I'm blown away! Prompt: "Create an amazing animation using p5js"

Ivan Fioravanti ᯅ

46,493 次观看 • 1 年前

oMLX is working really well as single machine inference engine for coding agents! Caching is managed perfectly (it can use a ton of disk space, be aware!) and oQ quantization delivers great results. Behind the scenes it uses the standard MLX building blocks (75% created by Prince Canuma 🙏): - mlx-lm - mlx-vlm - mlx-embeddings - mlx-audio I tested Qwen3.6-35B-A3B-oQ6 on M5 Max with two pi instances and it was fast and furious and leveraging cache like crazy as you can see in the video. Let me try to create some oQ versions (2,4,6?) of MiniMax M2.7 now and then I'll pass to distributed inference. I must win! 💪

oMLX is working really well as single machine inference engine for coding agents! Caching is managed perfectly (it can use a ton of disk space, be aware!) and oQ quantization delivers great results. Behind the scenes it uses the standard MLX building blocks (75% created by Prince Canuma 🙏): - mlx-lm - mlx-vlm - mlx-embeddings - mlx-audio I tested Qwen3.6-35B-A3B-oQ6 on M5 Max with two pi instances and it was fast and furious and leveraging cache like crazy as you can see in the video. Let me try to create some oQ versions (2,4,6?) of MiniMax M2.7 now and then I'll pass to distributed inference. I must win! 💪

Ivan Fioravanti ᯅ

19,420 次观看 • 3 个月前

MLX GLM 5.2 Distributed on two M3 Ultra 512GB 🔥 One M3 Ultra: 18.8 tokens/sec Two M3 Ultra: 23.4 tokens/sec Context: - PR by Pedro Cuenca is still open and probably there is room for improvement: - basic generation test to measure decoding performance here, I will do a full context benchmarking once PR is more mature - nvfp4 quantization used - Video alternates standard speed and x20, with one Mac first and distributed later. Enjoy! 🙌🏻

MLX GLM 5.2 Distributed on two M3 Ultra 512GB 🔥 One M3 Ultra: 18.8 tokens/sec Two M3 Ultra: 23.4 tokens/sec Context: - PR by Pedro Cuenca is still open and probably there is room for improvement: - basic generation test to measure decoding performance here, I will do a full context benchmarking once PR is more mature - nvfp4 quantization used - Video alternates standard speed and x20, with one Mac first and distributed later. Enjoy! 🙌🏻

Ivan Fioravanti ᯅ

87,805 次观看 • 1 个月前

MiniMax M3 vs GLM-5.2 vs Kimi K2.7 in Lunar Lander contest! 🔥 Last test of the day before going back to learning & build phase with Local Models, Swift and Apple Core AI 💪 I took the opportunity to test the MiniMax Code app for macOS. Video below speeded up. Followed by Lunar Lander implementation from the various models. My ranking: 🥇 GLM 5.2 🥈 MiniMax M3 🥉 Kimi K2.7 Coding Prompt and final results in this gist:

MiniMax M3 vs GLM-5.2 vs Kimi K2.7 in Lunar Lander contest! 🔥 Last test of the day before going back to learning & build phase with Local Models, Swift and Apple Core AI 💪 I took the opportunity to test the MiniMax Code app for macOS. Video below speeded up. Followed by Lunar Lander implementation from the various models. My ranking: 🥇 GLM 5.2 🥈 MiniMax M3 🥉 Kimi K2.7 Coding Prompt and final results in this gist:

Ivan Fioravanti ᯅ

21,604 次观看 • 1 个月前

Alibaba just released a coding model that hits 82 percent on SWE-Bench Verified. That is the highest score ever published for an open-source model. The weights are free. The license is Apache 2.0. You can run it today. The model is Qwen 4 Coder 32B. Here is what 82 percent on SWE-Bench Verified actually means. SWE-Bench Verified tests whether an AI can autonomously resolve real bugs pulled from real production GitHub repositories. Not synthetic exercises. Real open-source projects that real teams depend on. A model gets a bug report, reads the code, writes a fix, and either passes the test suite or it does not. At 82 percent, Qwen 4 Coder 32B resolves 82 out of every 100 real production bugs it is given. Without a human guiding it. On code it has never seen before. For comparison: Qwen 4 Coder 32B: 82 percent SWE-Bench Verified. Open source. Apache 2.0. Claude Fable 5: 80.3 percent SWE-Bench Pro. $10 input / $50 output per million tokens. Currently suspended. GPT-5.6 Sol: Competitive on Terminal-Bench. $5 input / $30 output per million tokens. An open-weight model that you can download and run for free just beat both of them on the benchmark designed to measure real software engineering capability. Here is the architecture. Qwen 4 Coder 32B is a 32 billion parameter dense model. Not a Mixture-of-Experts. Every parameter is active on every request. This matters for inference: a dense 32B model runs on 22 gigabytes of VRAM, which fits on a single high-end consumer GPU or a MacBook Pro with 64GB of unified memory. The smaller variant, Qwen 4 Coder 4B, runs at approximately 135 tokens per second on an M5 Max and fits inside 8 gigabytes of RAM. For a model with usable coding capability, that is a new bar for what fits in a single laptop. The training methodology continued Alibaba's approach of reinforcement learning on verifiable coding tasks. The model gets rewarded when its code passes tests. It gets penalized when it fails. Over millions of training steps, the model learns to write code that actually runs rather than code that looks plausible. License: Apache 2.0. Full commercial use. No attribution requirement. No revenue threshold. No monthly active user ceiling. Weights: Hugging Face, available today. Runs on: vLLM, Ollama, SGLang, and any standard GGUF-compatible inference engine. Qwen 4 32B also runs at approximately 135 tokens per second on an M5 Max chip, setting a new bar for what a sub-8GB model can do on Apple Silicon. The open-source coding model just beat the best closed-source model in the world on the benchmark designed to test whether AI can actually do software engineering. The weights are free. The subscription is optional. Source: Autom8Labs AI Insight July 2026, State of Open Source LLMs June 2026, Kunal Ganglani blog June 2026.

Alibaba just released a coding model that hits 82 percent on SWE-Bench Verified. That is the highest score ever published for an open-source model. The weights are free. The license is Apache 2.0. You can run it today. The model is Qwen 4 Coder 32B. Here is what 82 percent on SWE-Bench Verified actually means. SWE-Bench Verified tests whether an AI can autonomously resolve real bugs pulled from real production GitHub repositories. Not synthetic exercises. Real open-source projects that real teams depend on. A model gets a bug report, reads the code, writes a fix, and either passes the test suite or it does not. At 82 percent, Qwen 4 Coder 32B resolves 82 out of every 100 real production bugs it is given. Without a human guiding it. On code it has never seen before. For comparison: Qwen 4 Coder 32B: 82 percent SWE-Bench Verified. Open source. Apache 2.0. Claude Fable 5: 80.3 percent SWE-Bench Pro. $10 input / $50 output per million tokens. Currently suspended. GPT-5.6 Sol: Competitive on Terminal-Bench. $5 input / $30 output per million tokens. An open-weight model that you can download and run for free just beat both of them on the benchmark designed to measure real software engineering capability. Here is the architecture. Qwen 4 Coder 32B is a 32 billion parameter dense model. Not a Mixture-of-Experts. Every parameter is active on every request. This matters for inference: a dense 32B model runs on 22 gigabytes of VRAM, which fits on a single high-end consumer GPU or a MacBook Pro with 64GB of unified memory. The smaller variant, Qwen 4 Coder 4B, runs at approximately 135 tokens per second on an M5 Max and fits inside 8 gigabytes of RAM. For a model with usable coding capability, that is a new bar for what fits in a single laptop. The training methodology continued Alibaba's approach of reinforcement learning on verifiable coding tasks. The model gets rewarded when its code passes tests. It gets penalized when it fails. Over millions of training steps, the model learns to write code that actually runs rather than code that looks plausible. License: Apache 2.0. Full commercial use. No attribution requirement. No revenue threshold. No monthly active user ceiling. Weights: Hugging Face, available today. Runs on: vLLM, Ollama, SGLang, and any standard GGUF-compatible inference engine. Qwen 4 32B also runs at approximately 135 tokens per second on an M5 Max chip, setting a new bar for what a sub-8GB model can do on Apple Silicon. The open-source coding model just beat the best closed-source model in the world on the benchmark designed to test whether AI can actually do software engineering. The weights are free. The subscription is optional. Source: Autom8Labs AI Insight July 2026, State of Open Source LLMs June 2026, Kunal Ganglani blog June 2026.

Harman

41,278 次观看 • 1 个月前

A $40 BC250 BOARD WITH 16GB GDDR6 GETS A GITHUB FIRMWARE FLASH, THE DEFAULT 8/8 CPU/GPU SPLIT REWRITES TO 0.5/15.5 AND OPENS 15.5GB OF UNIFIED MEMORY TO OLLAMA, DEAD CRYPTO HARDWARE JUST DOUBLED ITS AI CEILING 01:08 the operator points at the chart, "you only give 512 megs to the GPU part and that's reserved, that means all the rest of the RAM is available to either the CPU or the GPU" a BC250 ships from the factory with 16GB GDDR6 split 8/8 between Oberon CPU cores and the RDNA 2 iGPU, the iGPU only ever uses 4-5GB during inference, the other 3-4GB sits stranded SEC Bolt's firmware mod on the moth-enjoyer GitHub flashes a 0.5/15.5 split, the GPU reserves 512MB and 15.5GB falls into a unified pool that ollama treats as VRAM, qwen 3.6 14B at 4 bit fits with 2GB context headroom 15.5GB on a $40 board with a $40 CH347 flasher beats the $249 Jetson Orin Nano's 8GB by nearly 2x at 1/6th the price, the firmware is open source, the dump backup process takes 4 minutes per board your map's tier zero floor was the OptiPlex at $35-50 with iGPU only, the modded BC250 sits at the same $40 mark with 15.5GB of GDDR6 and an actual RDNA 2 GPU, the buyer who flashes once unlocks a 13B class local AI host for the price of a dinner the window is open, follow and bookmark before it closes

A $40 BC250 BOARD WITH 16GB GDDR6 GETS A GITHUB FIRMWARE FLASH, THE DEFAULT 8/8 CPU/GPU SPLIT REWRITES TO 0.5/15.5 AND OPENS 15.5GB OF UNIFIED MEMORY TO OLLAMA, DEAD CRYPTO HARDWARE JUST DOUBLED ITS AI CEILING 01:08 the operator points at the chart, "you only give 512 megs to the GPU part and that's reserved, that means all the rest of the RAM is available to either the CPU or the GPU" a BC250 ships from the factory with 16GB GDDR6 split 8/8 between Oberon CPU cores and the RDNA 2 iGPU, the iGPU only ever uses 4-5GB during inference, the other 3-4GB sits stranded SEC Bolt's firmware mod on the moth-enjoyer GitHub flashes a 0.5/15.5 split, the GPU reserves 512MB and 15.5GB falls into a unified pool that ollama treats as VRAM, qwen 3.6 14B at 4 bit fits with 2GB context headroom 15.5GB on a $40 board with a $40 CH347 flasher beats the $249 Jetson Orin Nano's 8GB by nearly 2x at 1/6th the price, the firmware is open source, the dump backup process takes 4 minutes per board your map's tier zero floor was the OptiPlex at $35-50 with iGPU only, the modded BC250 sits at the same $40 mark with 15.5GB of GDDR6 and an actual RDNA 2 GPU, the buyer who flashes once unlocks a 13B class local AI host for the price of a dinner the window is open, follow and bookmark before it closes

Antid

25,619 次观看 • 1 个月前

$I nearly 2x'd the speed while only using +1GB VRAM with the new MTP update in llama.cpp 🤯 You need to add these flags to start using it: --spec-type draft-mtp \ --spec-draft-p-min 0.75 \ --spec-draft-n-max 2 My results with Qwen3.6 27B on a single RTX 5080 ↓ ⚪️ no flag (without mtp) → 54.3 tok/s with 13.26GB VRAM 🔵 --spec-draft-n-max 2 → 90.7 tok/s with 14.29GB VRAM 🔴 --spec-draft-n-max 2 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.30GB VRAM 🟢 --spec-draft-n-max 6 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.87GB VRAM Increasing to 6 draft tokens didn't help my setup for some reason. I made sure to test with a low context length to have enough headroom and eliminate risk of vram stress. From my understanding: 1) The speed gains are very task-dependent. You need to test across a wide range of tasks to get a realistic idea of the benefits 2) We’re already running heavily quantized GGUF models (Q3, Q4, Q6, etc.), so we already benefit from strong speed/performance thanks to the reduced size. That’s why some people are seeing little to no improvement compared to MLX or other quantized versions The progress over the past few days has been insane to say the least. However, MTP now consumes significantly more VRAM. Personally 16GB just isn't enough to use MTP and run it with a good context size. Time to upgrade lads, 24GB+ users are eating GOOD today 🔥 Full setup below ↓$

I nearly 2x'd the speed while only using +1GB VRAM with the new MTP update in llama.cpp 🤯 You need to add these flags to start using it: --spec-type draft-mtp \ --spec-draft-p-min 0.75 \ --spec-draft-n-max 2 My results with Qwen3.6 27B on a single RTX 5080 ↓ ⚪️ no flag (without mtp) → 54.3 tok/s with 13.26GB VRAM 🔵 --spec-draft-n-max 2 → 90.7 tok/s with 14.29GB VRAM 🔴 --spec-draft-n-max 2 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.30GB VRAM 🟢 --spec-draft-n-max 6 --spec-draft-p-min 0.75 → 93.9 tok/s with 14.87GB VRAM Increasing to 6 draft tokens didn't help my setup for some reason. I made sure to test with a low context length to have enough headroom and eliminate risk of vram stress. From my understanding: 1) The speed gains are very task-dependent. You need to test across a wide range of tasks to get a realistic idea of the benefits 2) We’re already running heavily quantized GGUF models (Q3, Q4, Q6, etc.), so we already benefit from strong speed/performance thanks to the reduced size. That’s why some people are seeing little to no improvement compared to MLX or other quantized versions The progress over the past few days has been insane to say the least. However, MTP now consumes significantly more VRAM. Personally 16GB just isn't enough to use MTP and run it with a good context size. Time to upgrade lads, 24GB+ users are eating GOOD today 🔥 Full setup below ↓

left curve dev

30,086 次观看 • 2 个月前

Nag test on chill mode, V14.3.3. The video you’re watching below is of me typing this post. So far it doesn’t seem to care about me chilling on my phone below 40mph. Seems a little worse above, but you can see I just got a blue flash at 42mph that cleared with a quick glance and hasn’t come back. Switching to sloth now. Pulling up on this red light. One sec. Not sure wtf the wheel just did there lmao. Anyway we’re on sloth now, about to make this right behind the GMC. Still not giving af. Pretty sick. I’m wearing sunglasses. Going to keep going until we hit up the ramp, then it’s mad max highway time. Still chillin out here in sloth, 35 mph. Doesn’t care. Vibes vibes vibes. I’m hype af. Ok here’s the turn toward the ramp. Cutting the video now. Bye!

Nag test on chill mode, V14.3.3. The video you’re watching below is of me typing this post. So far it doesn’t seem to care about me chilling on my phone below 40mph. Seems a little worse above, but you can see I just got a blue flash at 42mph that cleared with a quick glance and hasn’t come back. Switching to sloth now. Pulling up on this red light. One sec. Not sure wtf the wheel just did there lmao. Anyway we’re on sloth now, about to make this right behind the GMC. Still not giving af. Pretty sick. I’m wearing sunglasses. Going to keep going until we hit up the ramp, then it’s mad max highway time. Still chillin out here in sloth, 35 mph. Doesn’t care. Vibes vibes vibes. I’m hype af. Ok here’s the turn toward the ramp. Cutting the video now. Bye!

Mike P

38,618 次观看 • 2 个月前