Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

oMLX last version seems having some issues in decoding. 🧐 Here a test with Qwen3.6-35B-A3B-MLX-6bit on M5 Max using. 🥇 LM Studio MLX 1.8.5 → 100.9 toks/s 🥈 mlx-vlm 0.6.2 → 100.1 toks/s 🥉 oMLX 0.4.2 dev3 → 58.7 toks/s 👀 Avg Gen TPS: oMLX 58.7 → LM Studio... show more

Ivan Fioravanti ᯅ

32,261 subscribers

19,634 görüntüleme • 13 gün önce •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Video on LM Studio MLX v.1.8.5 + Codex + qwen3.6-35b-a3b 6bit on an M5 Max 128GB. Video is in normal speed to give you an idea of the experience. Command used: codex --oss -m qwen/qwen3.6-35b-a3b Local AI is becoming powerful 💪

Video on LM Studio MLX v.1.8.5 + Codex + qwen3.6-35b-a3b 6bit on an M5 Max 128GB. Video is in normal speed to give you an idea of the experience. Command used: codex --oss -m qwen/qwen3.6-35b-a3b Local AI is becoming powerful 💪

Ivan Fioravanti ᯅ

40,021 görüntüleme • 14 gün önce

oMLX is working really well as single machine inference engine for coding agents! Caching is managed perfectly (it can use a ton of disk space, be aware!) and oQ quantization delivers great results. Behind the scenes it uses the standard MLX building blocks (75% created by Prince Canuma 🙏): - mlx-lm - mlx-vlm - mlx-embeddings - mlx-audio I tested Qwen3.6-35B-A3B-oQ6 on M5 Max with two pi instances and it was fast and furious and leveraging cache like crazy as you can see in the video. Let me try to create some oQ versions (2,4,6?) of MiniMax M2.7 now and then I'll pass to distributed inference. I must win! 💪

oMLX is working really well as single machine inference engine for coding agents! Caching is managed perfectly (it can use a ton of disk space, be aware!) and oQ quantization delivers great results. Behind the scenes it uses the standard MLX building blocks (75% created by Prince Canuma 🙏): - mlx-lm - mlx-vlm - mlx-embeddings - mlx-audio I tested Qwen3.6-35B-A3B-oQ6 on M5 Max with two pi instances and it was fast and furious and leveraging cache like crazy as you can see in the video. Let me try to create some oQ versions (2,4,6?) of MiniMax M2.7 now and then I'll pass to distributed inference. I must win! 💪

Ivan Fioravanti ᯅ

19,268 görüntüleme • 2 ay önce

Qwen 2.5 Coder Q4 M4 Max Inference test. Apple MLX vs Ollama: - MLX: 23.97 toks/sec 🥇🔥 - Ollama: 18.33 toks/sec 🥈 Here a video to show results

Qwen 2.5 Coder Q4 M4 Max Inference test. Apple MLX vs Ollama: - MLX: 23.97 toks/sec 🥇🔥 - Ollama: 18.33 toks/sec 🥈 Here a video to show results

Ivan Fioravanti ᯅ

34,881 görüntüleme • 1 yıl önce

Qwen 32B (4-bit) generates at >40 toks/sec on an M4 Max with assisted decoding and Qwen 0.5B as the draft model. Coming soon to mlx-lm. Compare regular decoding (left) to assisted decoding (right):

Qwen 32B (4-bit) generates at >40 toks/sec on an M4 Max with assisted decoding and Qwen 0.5B as the draft model. Coming soon to mlx-lm. Compare regular decoding (left) to assisted decoding (right):

Awni Hannun

50,353 görüntüleme • 1 yıl önce

The new Deep Seek V3 0324 in 4-bit runs at > 20 toks/sec on a 512GB M3 Ultra with mlx-lm!

The new Deep Seek V3 0324 in 4-bit runs at > 20 toks/sec on a 512GB M3 Ultra with mlx-lm!

Awni Hannun

168,842 görüntüleme • 1 yıl önce

GLM-4.7-8bit (350GB) running at 19 toks/s on two M3 Ultra 512GB using Tensor Parallelism with EXO - MLX, versus 14 toks/s with single node. 🚀 Now context benchmarking & then OpenCode tests 🔥 Note: this is from sources, I had to change things to run it.

GLM-4.7-8bit (350GB) running at 19 toks/s on two M3 Ultra 512GB using Tensor Parallelism with EXO - MLX, versus 14 toks/s with single node. 🚀 Now context benchmarking & then OpenCode tests 🔥 Note: this is from sources, I had to change things to run it.

Ivan Fioravanti ᯅ

327,687 görüntüleme • 5 ay önce

Llama 3.3 70B 4-bit runs nicely on a 64GB M3 Max with in MLX LM (~10 toks/sec). Would be even faster on an M4 Max. Yesterday's server-only 405B is today's laptop 70B:

Llama 3.3 70B 4-bit runs nicely on a 64GB M3 Max with in MLX LM (~10 toks/sec). Would be even faster on an M4 Max. Yesterday's server-only 405B is today's laptop 70B:

Awni Hannun

110,375 görüntüleme • 1 yıl önce

DeepSeek R1 (the 680B MOE) is ~20% faster in the latest mlx / mlx-lm. 4-bit model on 3 M2 Ultras generates 4k tokens at a respectable 15 toks/sec. Plus some QoL improvements: - Only downloads the local shard (much faster startup) - Distributed launcher ships with MLX

DeepSeek R1 (the 680B MOE) is ~20% faster in the latest mlx / mlx-lm. 4-bit model on 3 M2 Ultras generates 4k tokens at a respectable 15 toks/sec. Plus some QoL improvements: - Only downloads the local shard (much faster startup) - Distributed launcher ships with MLX

Awni Hannun

86,921 görüntüleme • 1 yıl önce

Llama 3.3 70B 4-bit running on a 128GB M4 Max with MLX LM (~12 toks/sec) with mactop on the right. "Good enough", local, private and open models are now more accessible than ever!

Llama 3.3 70B 4-bit running on a 128GB M4 Max with MLX LM (~12 toks/sec) with mactop on the right. "Good enough", local, private and open models are now more accessible than ever!

Ivan Fioravanti ᯅ

43,929 görüntüleme • 1 yıl önce

Subagents running locally and simultaneously on MacBook Pro M5 with Codex CLI + LM Studio to review code and find bugs using Qwen 3.6 Powered by the updated MLX engine with batching in beta in the app The batching speed boost is noticeable

Subagents running locally and simultaneously on MacBook Pro M5 with Codex CLI + LM Studio to review code and find bugs using Qwen 3.6 Powered by the updated MLX engine with batching in beta in the app The batching speed boost is noticeable

Adrien Grondin

74,729 görüntüleme • 1 ay önce

Trying out omlx vibes. <-- much better than vllm-studio especially for those of you interested in self hosting and have macs.

Trying out omlx vibes. <-- much better than vllm-studio especially for those of you interested in self hosting and have macs.

0xSero

20,819 görüntüleme • 2 ay önce

local LLMs on Mac are getting good. but the workflow around models is still messy. you have models in: •⁠ ⁠LM Studio •⁠ ⁠Ollama •⁠ ⁠Hugging Face cache •⁠ ⁠random folders •⁠ ⁠manual downloads and every runtime expects something slightly different. we built ModelHub to clean up this layer. it’s a native macOS menu bar app for discovering, downloading, and managing local LLMs. not a replacement for Ollama, oMLX or LM Studio. it sits beside them. Ollama = run models MLX = Apple Silicon acceleration LM Studio = local model workspace Hugging Face = model ecosystem ModelHub = model discovery + management layer we’re live on Product Hunt today. would love feedback from people running local models on Mac.

local LLMs on Mac are getting good. but the workflow around models is still messy. you have models in: •⁠ ⁠LM Studio •⁠ ⁠Ollama •⁠ ⁠Hugging Face cache •⁠ ⁠random folders •⁠ ⁠manual downloads and every runtime expects something slightly different. we built ModelHub to clean up this layer. it’s a native macOS menu bar app for discovering, downloading, and managing local LLMs. not a replacement for Ollama, oMLX or LM Studio. it sits beside them. Ollama = run models MLX = Apple Silicon acceleration LM Studio = local model workspace Hugging Face = model ecosystem ModelHub = model discovery + management layer we’re live on Product Hunt today. would love feedback from people running local models on Mac.

sabesh 📟

30,332 görüntüleme • 27 gün önce

Qwen3 235B MoE (22B active) runs so fast on an M2 Ultra with mlx-lm. - 4-bit model uses ~132GB - Generated 580 tokens at ~28 toks/sec

Qwen3 235B MoE (22B active) runs so fast on an M2 Ultra with mlx-lm. - 4-bit model uses ~132GB - Generated 580 tokens at ~28 toks/sec

Awni Hannun

117,763 görüntüleme • 1 yıl önce

A long time coming but new mlx-lm is here with better batching support in the server and Gemma 4. pip install -U mlx-lm Here is a video where a single M3 Ultra serves 5 opencode sessions with Gemma 4 26B that process ~130k tokens in ~1.5 minutes.

A long time coming but new mlx-lm is here with better batching support in the server and Gemma 4. pip install -U mlx-lm Here is a video where a single M3 Ultra serves 5 opencode sessions with Gemma 4 26B that process ~130k tokens in ~1.5 minutes.

Angelos Katharopoulos

66,095 görüntüleme • 2 ay önce

Pretty cool that with the new Qwen 2.5 models you can ask questions / generate using a reasonably sized code-base as context, all running on a laptop with mlx-lm. The 7B runs pretty fast on an M4 Max using the mlx-lm code base (~16k lines) as context:

Pretty cool that with the new Qwen 2.5 models you can ask questions / generate using a reasonably sized code-base as context, all running on a laptop with mlx-lm. The 7B runs pretty fast on an M4 Max using the mlx-lm code base (~16k lines) as context:

Awni Hannun

27,442 görüntüleme • 1 yıl önce

Mistral Vibe CLI running Devstral-small-2-2512 on M3 Ultra 512GB with LM Studio as backend 🔥 First part of the video 1x and the remaining 2x Great speed overall, 27 toks/s as average tracked on LM Studio console. Details to run it below 🧵

Mistral Vibe CLI running Devstral-small-2-2512 on M3 Ultra 512GB with LM Studio as backend 🔥 First part of the video 1x and the remaining 2x Great speed overall, 27 toks/s as average tracked on LM Studio console. Details to run it below 🧵

Ivan Fioravanti ᯅ

49,682 görüntüleme • 6 ay önce

The new 1 Trillion parameter Kimi K2 Thinking model runs well on 2 M3 Ultras in its native format - no loss in quality! The model was quantization aware trained (qat) at int4. Here it generated ~3500 tokens at 15 toks/sec using pipeline-parallelism in mlx-lm:

The new 1 Trillion parameter Kimi K2 Thinking model runs well on 2 M3 Ultras in its native format - no loss in quality! The model was quantization aware trained (qat) at int4. Here it generated ~3500 tokens at 15 toks/sec using pipeline-parallelism in mlx-lm:

Awni Hannun

500,714 görüntüleme • 7 ay önce

Running Ring 1T reasoning model on a single M3 Ultra with mlx-lm. It's quantized to 3.5 bits-per-weight. Uses 440GB and generated ~6k tokens at 18.2 toks/sec. Getting closer to GPT-5 at home.

Running Ring 1T reasoning model on a single M3 Ultra with mlx-lm. It's quantized to 3.5 bits-per-weight. Uses 440GB and generated ~6k tokens at 18.2 toks/sec. Getting closer to GPT-5 at home.

Awni Hannun

55,131 görüntüleme • 8 ay önce

$After a small patch to mlx-lm, I have tool parameter streaming \o/$

After a small patch to mlx-lm, I have tool parameter streaming \o/

Armin Ronacher ⇌

21,484 görüntüleme • 1 ay önce

GLM 4.7 Flash is supported in mlx-lm 0.30.3 (h/t Ivan Fioravanti ᯅ) The 4-bit runs fast (43 tok/s generation, ~800 tok/s prefill) on a base M5 32GB laptop.

GLM 4.7 Flash is supported in mlx-lm 0.30.3 (h/t Ivan Fioravanti ᯅ) The 4-bit runs fast (43 tok/s generation, ~800 tok/s prefill) on a base M5 32GB laptop.

Awni Hannun

141,194 görüntüleme • 5 ay önce