正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

After a small patch to mlx-lm, I have tool parameter streaming \o/

Armin Ronacher ⇌

83,477 subscribers

21,568 次观看 • 6 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

I'm so in love with antirez' ds4. Patched some slop on it to get better streaming, but I can just install a pi extension on a 128GB mac and it manages everything for me. No need for mlx-lm, ollama or lm studio or finagling pi configs.

I'm so in love with antirez' ds4. Patched some slop on it to get better streaming, but I can just install a pi extension on a 128GB mac and it manages everything for me. No need for mlx-lm, ollama or lm studio or finagling pi configs.

Armin Ronacher ⇌

56,284 次观看 • 6 个月前

I wasn't expecting such an intelligence model on my local machine. Minimax 2.1 on MLX🔥 git clone cd mlx-lm && pip install -e . mlx_lm.server --model mlx-community/MiniMax-M2.1-4bit

I wasn't expecting such an intelligence model on my local machine. Minimax 2.1 on MLX🔥 git clone cd mlx-lm && pip install -e . mlx_lm.server --model mlx-community/MiniMax-M2.1-4bit

mzba

30,654 次观看 • 10 个月前

oMLX last version seems having some issues in decoding. 🧐 Here a test with Qwen3.6-35B-A3B-MLX-6bit on M5 Max using. 🥇 LM Studio MLX 1.8.5 → 100.9 toks/s 🥈 mlx-vlm 0.6.2 → 100.1 toks/s 🥉 oMLX 0.4.2 dev3 → 58.7 toks/s 👀 Avg Gen TPS: oMLX 58.7 → LM Studio 100.9 (+71.8%) I have to thank pymike00 that raised this oMLX issue after seeing my video on using it with Codex. I bet there is a bug in oMLX chat and server at the moment, because internal benchmarks are ok, video attached. I bet Jun Kim will fix this soon 💪

oMLX last version seems having some issues in decoding. 🧐 Here a test with Qwen3.6-35B-A3B-MLX-6bit on M5 Max using. 🥇 LM Studio MLX 1.8.5 → 100.9 toks/s 🥈 mlx-vlm 0.6.2 → 100.1 toks/s 🥉 oMLX 0.4.2 dev3 → 58.7 toks/s 👀 Avg Gen TPS: oMLX 58.7 → LM Studio 100.9 (+71.8%) I have to thank pymike00 that raised this oMLX issue after seeing my video on using it with Codex. I bet there is a bug in oMLX chat and server at the moment, because internal benchmarks are ok, video attached. I bet Jun Kim will fix this soon 💪

Ivan Fioravanti ᯅ

20,066 次观看 • 5 个月前

A perfect coding model for MLX on Apple silicon.. Qwen delivered again. Runs quite fast on an M3 Ultra. Running the 4-bit quantized with mlx-lm:

A perfect coding model for MLX on Apple silicon.. Qwen delivered again. Runs quite fast on an M3 Ultra. Running the 4-bit quantized with mlx-lm:

Awni Hannun

186,831 次观看 • 1 年前

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Prince Canuma

82,349 次观看 • 7 个月前

A long time coming but new mlx-lm is here with better batching support in the server and Gemma 4. pip install -U mlx-lm Here is a video where a single M3 Ultra serves 5 opencode sessions with Gemma 4 26B that process ~130k tokens in ~1.5 minutes.

A long time coming but new mlx-lm is here with better batching support in the server and Gemma 4. pip install -U mlx-lm Here is a video where a single M3 Ultra serves 5 opencode sessions with Gemma 4 26B that process ~130k tokens in ~1.5 minutes.

Angelos Katharopoulos

66,202 次观看 • 7 个月前

Kimi K3 (2.8T params) running locally on two Mac Studios, pi agent + mlx-lm

Kimi K3 (2.8T params) running locally on two Mac Studios, pi agent + mlx-lm

Tarjei Mandt

60,563 次观看 • 3 个月前

Patch Tool, Clone Stamp, Brush Tool

Patch Tool, Clone Stamp, Brush Tool

Photoshop Tricks!

167,905 次观看 • 8 个月前

The new 1 Trillion parameter Kimi K2 Thinking model runs well on 2 M3 Ultras in its native format - no loss in quality! The model was quantization aware trained (qat) at int4. Here it generated ~3500 tokens at 15 toks/sec using pipeline-parallelism in mlx-lm:

The new 1 Trillion parameter Kimi K2 Thinking model runs well on 2 M3 Ultras in its native format - no loss in quality! The model was quantization aware trained (qat) at int4. Here it generated ~3500 tokens at 15 toks/sec using pipeline-parallelism in mlx-lm:

Awni Hannun

501,223 次观看 • 1 年前

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx

raullen

22,776 次观看 • 5 个月前

The new Deep Seek V3 0324 in 4-bit runs at > 20 toks/sec on a 512GB M3 Ultra with mlx-lm!

The new Deep Seek V3 0324 in 4-bit runs at > 20 toks/sec on a 512GB M3 Ultra with mlx-lm!

Awni Hannun

168,942 次观看 • 1 年前

A small but significant list of changes have launched with Y10S2! For a full Patch Notes breakdown, check out the Patch Notes segment from the Warrior's Den here!

A small but significant list of changes have launched with Y10S2! For a full Patch Notes breakdown, check out the Patch Notes segment from the Warrior's Den here!

For Honor

17,474 次观看 • 4 个月前

MiniMax M2.1 in 4-bit cruises on an M3 Ultra with mlx-lm. Generated a space invaders game using 5098 tokens at 47.2 tok/sec:

MiniMax M2.1 in 4-bit cruises on an M3 Ultra with mlx-lm. Generated a space invaders game using 5098 tokens at 47.2 tok/sec:

Awni Hannun

94,165 次观看 • 10 个月前

As part of our goal to make MLX a great research tool, we're expanding support to new languages like Swift and C, making experimentation on Apple silicon easier for ML researchers. Video generating text with Mistral 7B and MLX Swift 👇 MLX is an array framework for machine learning research on Apple silicon. MLX is intended for research and not for production deployment of models in apps.

As part of our goal to make MLX a great research tool, we're expanding support to new languages like Swift and C, making experimentation on Apple silicon easier for ML researchers. Video generating text with Mistral 7B and MLX Swift 👇 MLX is an array framework for machine learning research on Apple silicon. MLX is intended for research and not for production deployment of models in apps.

Awni Hannun

88,821 次观看 • 2 年前