正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

After a small patch to mlx-lm, I have tool parameter streaming \o/

Armin Ronacher ⇌

79,345 subscribers

21,484 次观看 • 2 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

I'm so in love with antirez' ds4. Patched some slop on it to get better streaming, but I can just install a pi extension on a 128GB mac and it manages everything for me. No need for mlx-lm, ollama or lm studio or finagling pi configs.

I'm so in love with antirez' ds4. Patched some slop on it to get better streaming, but I can just install a pi extension on a 128GB mac and it manages everything for me. No need for mlx-lm, ollama or lm studio or finagling pi configs.

Armin Ronacher ⇌

56,223 次观看 • 1 个月前

I wasn't expecting such an intelligence model on my local machine. Minimax 2.1 on MLX🔥 git clone cd mlx-lm && pip install -e . mlx_lm.server --model mlx-community/MiniMax-M2.1-4bit

I wasn't expecting such an intelligence model on my local machine. Minimax 2.1 on MLX🔥 git clone cd mlx-lm && pip install -e . mlx_lm.server --model mlx-community/MiniMax-M2.1-4bit

mzba

30,377 次观看 • 5 个月前

oMLX last version seems having some issues in decoding. 🧐 Here a test with Qwen3.6-35B-A3B-MLX-6bit on M5 Max using. 🥇 LM Studio MLX 1.8.5 → 100.9 toks/s 🥈 mlx-vlm 0.6.2 → 100.1 toks/s 🥉 oMLX 0.4.2 dev3 → 58.7 toks/s 👀 Avg Gen TPS: oMLX 58.7 → LM Studio 100.9 (+71.8%) I have to thank pymike00 that raised this oMLX issue after seeing my video on using it with Codex. I bet there is a bug in oMLX chat and server at the moment, because internal benchmarks are ok, video attached. I bet Jun Kim will fix this soon 💪

oMLX last version seems having some issues in decoding. 🧐 Here a test with Qwen3.6-35B-A3B-MLX-6bit on M5 Max using. 🥇 LM Studio MLX 1.8.5 → 100.9 toks/s 🥈 mlx-vlm 0.6.2 → 100.1 toks/s 🥉 oMLX 0.4.2 dev3 → 58.7 toks/s 👀 Avg Gen TPS: oMLX 58.7 → LM Studio 100.9 (+71.8%) I have to thank pymike00 that raised this oMLX issue after seeing my video on using it with Codex. I bet there is a bug in oMLX chat and server at the moment, because internal benchmarks are ok, video attached. I bet Jun Kim will fix this soon 💪

Ivan Fioravanti ᯅ

19,846 次观看 • 22 天前

A perfect coding model for MLX on Apple silicon.. Qwen delivered again. Runs quite fast on an M3 Ultra. Running the 4-bit quantized with mlx-lm:

A perfect coding model for MLX on Apple silicon.. Qwen delivered again. Runs quite fast on an M3 Ultra. Running the 4-bit quantized with mlx-lm:

Awni Hannun

186,641 次观看 • 11 个月前

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Prince Canuma

82,169 次观看 • 2 个月前

A long time coming but new mlx-lm is here with better batching support in the server and Gemma 4. pip install -U mlx-lm Here is a video where a single M3 Ultra serves 5 opencode sessions with Gemma 4 26B that process ~130k tokens in ~1.5 minutes.

A long time coming but new mlx-lm is here with better batching support in the server and Gemma 4. pip install -U mlx-lm Here is a video where a single M3 Ultra serves 5 opencode sessions with Gemma 4 26B that process ~130k tokens in ~1.5 minutes.

Angelos Katharopoulos

66,128 次观看 • 2 个月前

Pretty cool that with the new Qwen 2.5 models you can ask questions / generate using a reasonably sized code-base as context, all running on a laptop with mlx-lm. The 7B runs pretty fast on an M4 Max using the mlx-lm code base (~16k lines) as context:

Pretty cool that with the new Qwen 2.5 models you can ask questions / generate using a reasonably sized code-base as context, all running on a laptop with mlx-lm. The 7B runs pretty fast on an M4 Max using the mlx-lm code base (~16k lines) as context:

Awni Hannun

27,442 次观看 • 1 年前

Patch Tool, Clone Stamp, Brush Tool

Patch Tool, Clone Stamp, Brush Tool

Photoshop Tricks!

167,083 次观看 • 3 个月前

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx

raullen

22,680 次观看 • 22 天前

The new 1 Trillion parameter Kimi K2 Thinking model runs well on 2 M3 Ultras in its native format - no loss in quality! The model was quantization aware trained (qat) at int4. Here it generated ~3500 tokens at 15 toks/sec using pipeline-parallelism in mlx-lm:

The new 1 Trillion parameter Kimi K2 Thinking model runs well on 2 M3 Ultras in its native format - no loss in quality! The model was quantization aware trained (qat) at int4. Here it generated ~3500 tokens at 15 toks/sec using pipeline-parallelism in mlx-lm:

Awni Hannun

500,714 次观看 • 7 个月前

The new Deep Seek V3 0324 in 4-bit runs at > 20 toks/sec on a 512GB M3 Ultra with mlx-lm!

The new Deep Seek V3 0324 in 4-bit runs at > 20 toks/sec on a 512GB M3 Ultra with mlx-lm!

Awni Hannun

168,842 次观看 • 1 年前

DeepSeek R1 (the 680B MOE) is ~20% faster in the latest mlx / mlx-lm. 4-bit model on 3 M2 Ultras generates 4k tokens at a respectable 15 toks/sec. Plus some QoL improvements: - Only downloads the local shard (much faster startup) - Distributed launcher ships with MLX

DeepSeek R1 (the 680B MOE) is ~20% faster in the latest mlx / mlx-lm. 4-bit model on 3 M2 Ultras generates 4k tokens at a respectable 15 toks/sec. Plus some QoL improvements: - Only downloads the local shard (much faster startup) - Distributed launcher ships with MLX

Awni Hannun

86,921 次观看 • 1 年前

A small but significant list of changes have launched with Y10S2! For a full Patch Notes breakdown, check out the Patch Notes segment from the Warrior's Den here!

A small but significant list of changes have launched with Y10S2! For a full Patch Notes breakdown, check out the Patch Notes segment from the Warrior's Den here!

For Honor

16,605 次观看 • 9 天前

MiniMax M2.1 in 4-bit cruises on an M3 Ultra with mlx-lm. Generated a space invaders game using 5098 tokens at 47.2 tok/sec:

MiniMax M2.1 in 4-bit cruises on an M3 Ultra with mlx-lm. Generated a space invaders game using 5098 tokens at 47.2 tok/sec:

Awni Hannun

93,826 次观看 • 6 个月前

As part of our goal to make MLX a great research tool, we're expanding support to new languages like Swift and C, making experimentation on Apple silicon easier for ML researchers. Video generating text with Mistral 7B and MLX Swift 👇 MLX is an array framework for machine learning research on Apple silicon. MLX is intended for research and not for production deployment of models in apps.

As part of our goal to make MLX a great research tool, we're expanding support to new languages like Swift and C, making experimentation on Apple silicon easier for ML researchers. Video generating text with Mistral 7B and MLX Swift 👇 MLX is an array framework for machine learning research on Apple silicon. MLX is intended for research and not for production deployment of models in apps.

Awni Hannun

88,745 次观看 • 2 年前

“Can we get a new text analysis tool?” “No—we have Topic Model at home” Topic Model at home: outputs vague keywords; needs constant parameter fiddling🫠 Is there a better way? We introduce LLooM, a concept induction tool to explore text data in terms of interpretable concepts🧵

“Can we get a new text analysis tool?” “No—we have Topic Model at home” Topic Model at home: outputs vague keywords; needs constant parameter fiddling🫠 Is there a better way? We introduce LLooM, a concept induction tool to explore text data in terms of interpretable concepts🧵

Michelle Lam

38,102 次观看 • 2 年前

🔥 Thrilled to have worked with Google AI Developers on day-0 MLX support for Gemma 3 QAT! QAT optimizes models during training by simulating low-precision operations, delivering similar performance to FP16 and dramatic memory savings when quantised: • Gemma 3 27B: 54GB → 14.1GB (74% reduction) • Gemma 3 12B: 24GB → 6.6GB (72% reduction) • Gemma 3 4B: 8GB → 2.6GB (67% reduction) • Gemma 3 1B: 2GB → 0.5GB (75% reduction) Get started: > pip install mlx-vlm mlx-lm Model collection 👇🏽

🔥 Thrilled to have worked with Google AI Developers on day-0 MLX support for Gemma 3 QAT! QAT optimizes models during training by simulating low-precision operations, delivering similar performance to FP16 and dramatic memory savings when quantised: • Gemma 3 27B: 54GB → 14.1GB (74% reduction) • Gemma 3 12B: 24GB → 6.6GB (72% reduction) • Gemma 3 4B: 8GB → 2.6GB (67% reduction) • Gemma 3 1B: 2GB → 0.5GB (75% reduction) Get started: > pip install mlx-vlm mlx-lm Model collection 👇🏽

Prince Canuma

11,374 次观看 • 1 年前

oMLX is working really well as single machine inference engine for coding agents! Caching is managed perfectly (it can use a ton of disk space, be aware!) and oQ quantization delivers great results. Behind the scenes it uses the standard MLX building blocks (75% created by Prince Canuma 🙏): - mlx-lm - mlx-vlm - mlx-embeddings - mlx-audio I tested Qwen3.6-35B-A3B-oQ6 on M5 Max with two pi instances and it was fast and furious and leveraging cache like crazy as you can see in the video. Let me try to create some oQ versions (2,4,6?) of MiniMax M2.7 now and then I'll pass to distributed inference. I must win! 💪

oMLX is working really well as single machine inference engine for coding agents! Caching is managed perfectly (it can use a ton of disk space, be aware!) and oQ quantization delivers great results. Behind the scenes it uses the standard MLX building blocks (75% created by Prince Canuma 🙏): - mlx-lm - mlx-vlm - mlx-embeddings - mlx-audio I tested Qwen3.6-35B-A3B-oQ6 on M5 Max with two pi instances and it was fast and furious and leveraging cache like crazy as you can see in the video. Let me try to create some oQ versions (2,4,6?) of MiniMax M2.7 now and then I'll pass to distributed inference. I must win! 💪

Ivan Fioravanti ᯅ

19,268 次观看 • 2 个月前