Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

After a small patch to mlx-lm, I have tool parameter streaming \o/

mitsuhiko's profile picture

Armin Ronacher ⇌

79,345 subscribers

21,484 görüntüleme • 1 ay önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

I'm so in love with antirez' ds4. Patched some slop on it to get better streaming, but I can just install a pi extension on a 128GB mac and it manages everything for me. No need for mlx-lm, ollama or lm studio or finagling pi configs.
1:33

I'm so in love with antirez' ds4. Patched some slop on it to get better streaming, but I can just install a pi extension on a 128GB mac and it manages everything for me. No need for mlx-lm, ollama or lm studio or finagling pi configs.

Armin Ronacher ⇌

56,033 görüntüleme • 1 ay önce

I wasn't expecting such an intelligence model on my local machine. Minimax 2.1 on MLX🔥 git clone cd mlx-lm && pip install -e . mlx_lm.server --model mlx-community/MiniMax-M2.1-4bit
1:14

I wasn't expecting such an intelligence model on my local machine. Minimax 2.1 on MLX🔥 git clone cd mlx-lm && pip install -e . mlx_lm.server --model mlx-community/MiniMax-M2.1-4bit

mzba

30,377 görüntüleme • 5 ay önce

A perfect coding model for MLX on Apple silicon.. Qwen delivered again. Runs quite fast on an M3 Ultra. Running the 4-bit quantized with mlx-lm:
0:50

A perfect coding model for MLX on Apple silicon.. Qwen delivered again. Runs quite fast on an M3 Ultra. Running the 4-bit quantized with mlx-lm:

Awni Hannun

186,641 görüntüleme • 10 ay önce

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!
0:52

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Prince Canuma

82,085 görüntüleme • 1 ay önce

A long time coming but new mlx-lm is here with better batching support in the server and Gemma 4. pip install -U mlx-lm Here is a video where a single M3 Ultra serves 5 opencode sessions with Gemma 4 26B that process ~130k tokens in ~1.5 minutes.
2:03

A long time coming but new mlx-lm is here with better batching support in the server and Gemma 4. pip install -U mlx-lm Here is a video where a single M3 Ultra serves 5 opencode sessions with Gemma 4 26B that process ~130k tokens in ~1.5 minutes.

Angelos Katharopoulos

66,005 görüntüleme • 2 ay önce

Pretty cool that with the new Qwen 2.5 models you can ask questions / generate using a reasonably sized code-base as context, all running on a laptop with mlx-lm. The 7B runs pretty fast on an M4 Max using the mlx-lm code base (~16k lines) as context:
0:43

Pretty cool that with the new Qwen 2.5 models you can ask questions / generate using a reasonably sized code-base as context, all running on a laptop with mlx-lm. The 7B runs pretty fast on an M4 Max using the mlx-lm code base (~16k lines) as context:

Awni Hannun

27,442 görüntüleme • 1 yıl önce

Patch Tool, Clone Stamp, Brush Tool
2:01

Patch Tool, Clone Stamp, Brush Tool

Photoshop Tricks!

166,721 görüntüleme • 3 ay önce

The new Deep Seek V3 0324 in 4-bit runs at > 20 toks/sec on a 512GB M3 Ultra with mlx-lm!
0:31

The new Deep Seek V3 0324 in 4-bit runs at > 20 toks/sec on a 512GB M3 Ultra with mlx-lm!

Awni Hannun

168,815 görüntüleme • 1 yıl önce

The new 1 Trillion parameter Kimi K2 Thinking model runs well on 2 M3 Ultras in its native format - no loss in quality! The model was quantization aware trained (qat) at int4. Here it generated ~3500 tokens at 15 toks/sec using pipeline-parallelism in mlx-lm:
0:41

The new 1 Trillion parameter Kimi K2 Thinking model runs well on 2 M3 Ultras in its native format - no loss in quality! The model was quantization aware trained (qat) at int4. Here it generated ~3500 tokens at 15 toks/sec using pipeline-parallelism in mlx-lm:

Awni Hannun

500,639 görüntüleme • 7 ay önce

DeepSeek R1 (the 680B MOE) is ~20% faster in the latest mlx / mlx-lm. 4-bit model on 3 M2 Ultras generates 4k tokens at a respectable 15 toks/sec. Plus some QoL improvements: - Only downloads the local shard (much faster startup) - Distributed launcher ships with MLX
0:50

DeepSeek R1 (the 680B MOE) is ~20% faster in the latest mlx / mlx-lm. 4-bit model on 3 M2 Ultras generates 4k tokens at a respectable 15 toks/sec. Plus some QoL improvements: - Only downloads the local shard (much faster startup) - Distributed launcher ships with MLX

Awni Hannun

86,921 görüntüleme • 1 yıl önce

MiniMax M2.1 in 4-bit cruises on an M3 Ultra with mlx-lm. Generated a space invaders game using 5098 tokens at 47.2 tok/sec:
0:23

MiniMax M2.1 in 4-bit cruises on an M3 Ultra with mlx-lm. Generated a space invaders game using 5098 tokens at 47.2 tok/sec:

Awni Hannun

93,826 görüntüleme • 5 ay önce

As part of our goal to make MLX a great research tool, we're expanding support to new languages like Swift and C, making experimentation on Apple silicon easier for ML researchers. Video generating text with Mistral 7B and MLX Swift 👇 MLX is an array framework for machine learning research on Apple silicon. MLX is intended for research and not for production deployment of models in apps.
0:56

As part of our goal to make MLX a great research tool, we're expanding support to new languages like Swift and C, making experimentation on Apple silicon easier for ML researchers. Video generating text with Mistral 7B and MLX Swift 👇 MLX is an array framework for machine learning research on Apple silicon. MLX is intended for research and not for production deployment of models in apps.

Awni Hannun

88,726 görüntüleme • 2 yıl önce

“Can we get a new text analysis tool?” “No—we have Topic Model at home” Topic Model at home: outputs vague keywords; needs constant parameter fiddling🫠 Is there a better way? We introduce LLooM, a concept induction tool to explore text data in terms of interpretable concepts🧵
0:52

“Can we get a new text analysis tool?” “No—we have Topic Model at home” Topic Model at home: outputs vague keywords; needs constant parameter fiddling🫠 Is there a better way? We introduce LLooM, a concept induction tool to explore text data in terms of interpretable concepts🧵

Michelle Lam

38,067 görüntüleme • 2 yıl önce

oMLX is working really well as single machine inference engine for coding agents! Caching is managed perfectly (it can use a ton of disk space, be aware!) and oQ quantization delivers great results. Behind the scenes it uses the standard MLX building blocks (75% created by Prince Canuma 🙏): - mlx-lm - mlx-vlm - mlx-embeddings - mlx-audio I tested Qwen3.6-35B-A3B-oQ6 on M5 Max with two pi instances and it was fast and furious and leveraging cache like crazy as you can see in the video. Let me try to create some oQ versions (2,4,6?) of MiniMax M2.7 now and then I'll pass to distributed inference. I must win! 💪
3:57

oMLX is working really well as single machine inference engine for coding agents! Caching is managed perfectly (it can use a ton of disk space, be aware!) and oQ quantization delivers great results. Behind the scenes it uses the standard MLX building blocks (75% created by Prince Canuma 🙏): - mlx-lm - mlx-vlm - mlx-embeddings - mlx-audio I tested Qwen3.6-35B-A3B-oQ6 on M5 Max with two pi instances and it was fast and furious and leveraging cache like crazy as you can see in the video. Let me try to create some oQ versions (2,4,6?) of MiniMax M2.7 now and then I'll pass to distributed inference. I must win! 💪

Ivan Fioravanti ᯅ

19,268 görüntüleme • 1 ay önce

🔥 Thrilled to have worked with Google AI Developers on day-0 MLX support for Gemma 3 QAT! QAT optimizes models during training by simulating low-precision operations, delivering similar performance to FP16 and dramatic memory savings when quantised: • Gemma 3 27B: 54GB → 14.1GB (74% reduction) • Gemma 3 12B: 24GB → 6.6GB (72% reduction) • Gemma 3 4B: 8GB → 2.6GB (67% reduction) • Gemma 3 1B: 2GB → 0.5GB (75% reduction) Get started: > pip install mlx-vlm mlx-lm Model collection 👇🏽
0:30

🔥 Thrilled to have worked with Google AI Developers on day-0 MLX support for Gemma 3 QAT! QAT optimizes models during training by simulating low-precision operations, delivering similar performance to FP16 and dramatic memory savings when quantised: • Gemma 3 27B: 54GB → 14.1GB (74% reduction) • Gemma 3 12B: 24GB → 6.6GB (72% reduction) • Gemma 3 4B: 8GB → 2.6GB (67% reduction) • Gemma 3 1B: 2GB → 0.5GB (75% reduction) Get started: > pip install mlx-vlm mlx-lm Model collection 👇🏽

Prince Canuma

11,374 görüntüleme • 1 yıl önce

Day 2 of 3 MLX Releases: Introducing Local Computer-Use 🚀🔥 A powerful tool built with MLX that uses Vision Language models and Voice models to control your Mac through visual understanding, planning and reasoning. Features ⚡️Automate your workflow with natural language 😎 Control your computer “hands-free” This project now supports both: 🤖 Level 1 (GUI Agent) 🧠 Level 2 (Autonomous GUI Agent) Get started: > pip install -U mlx-vlm mlx-audio mlx-whisper Please leave us a star and send a PR :)
2:23

Day 2 of 3 MLX Releases: Introducing Local Computer-Use 🚀🔥 A powerful tool built with MLX that uses Vision Language models and Voice models to control your Mac through visual understanding, planning and reasoning. Features ⚡️Automate your workflow with natural language 😎 Control your computer “hands-free” This project now supports both: 🤖 Level 1 (GUI Agent) 🧠 Level 2 (Autonomous GUI Agent) Get started: > pip install -U mlx-vlm mlx-audio mlx-whisper Please leave us a star and send a PR :)

Prince Canuma

45,867 görüntüleme • 1 yıl önce

Cinna gets emotional and reveals to her viewers why she hasn't been streaming after making a shocking discovery.. 😮👀 “I haven’t been streaming because I found out I might have autism.” 💀😭
0:30

Cinna gets emotional and reveals to her viewers why she hasn't been streaming after making a shocking discovery.. 😮👀 “I haven’t been streaming because I found out I might have autism.” 💀😭

veizau

156,103 görüntüleme • 5 ay önce