正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Running Qwen2 VL 72B 4bit 🇨🇳 quantized model on Mac Mini M4 Pro 64GB. It works very well and delivers good accuracy, suitable for practical use cases. Integrating MLX and MLX-VLM backend into Sparrow as main solution for local inference and data extraction. Kudos to Prince Canuma for porting... show more

Andrej Baranovskij

6,693 subscribers

13,742 次观看 • 1 年前 •via X (Twitter)

科学技术教育

Anya Rossi• Live Now

Private livecam show

2 条评论

Andrej Baranovskij 的头像

Andrej Baranovskij1 年前

Sparrow: Sparrow UI: @katana_ml

Ivan Fioravanti ᯅ 的头像

Ivan Fioravanti ᯅ1 年前

Top!!!

相关视频

oMLX is working really well as single machine inference engine for coding agents! Caching is managed perfectly (it can use a ton of disk space, be aware!) and oQ quantization delivers great results. Behind the scenes it uses the standard MLX building blocks (75% created by Prince Canuma 🙏): - mlx-lm - mlx-vlm - mlx-embeddings - mlx-audio I tested Qwen3.6-35B-A3B-oQ6 on M5 Max with two pi instances and it was fast and furious and leveraging cache like crazy as you can see in the video. Let me try to create some oQ versions (2,4,6?) of MiniMax M2.7 now and then I'll pass to distributed inference. I must win! 💪

oMLX is working really well as single machine inference engine for coding agents! Caching is managed perfectly (it can use a ton of disk space, be aware!) and oQ quantization delivers great results. Behind the scenes it uses the standard MLX building blocks (75% created by Prince Canuma 🙏): - mlx-lm - mlx-vlm - mlx-embeddings - mlx-audio I tested Qwen3.6-35B-A3B-oQ6 on M5 Max with two pi instances and it was fast and furious and leveraging cache like crazy as you can see in the video. Let me try to create some oQ versions (2,4,6?) of MiniMax M2.7 now and then I'll pass to distributed inference. I must win! 💪

Ivan Fioravanti ᯅ

19,420 次观看 • 4 个月前

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx

🔴 The Pain: Running local MLX models is incredibly fast and private. But let's be real - testing tool calling via terminal is clunky, and there's zero good UI for it. 🟢 The Fix: rapid-mlx share Just ONE command gives you a polished web chat + seamless tool calling (works beautifully with gemma-4-12b-qat). We are proud to be the ONLY MLX inference engine in the community shipping this. ⚡️ 👇 Try it now: brew install raullenchai/rapid-mlx/rapid-mlx

raullen

22,760 次观看 • 2 个月前

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Prince Canuma

82,349 次观看 • 4 个月前

SAM-2 is apache license - you can indeed just build crazy apps with them without fear. This weekend I ported SAM-2 models to run optimally on Apple Silicon. I have also quantized all the SAM models to 16bit, 8bit, and 4bit variants (collection below) Repo is open source and pip installable. Comes with all you'll need for interactive video segmentation - - bidirectional propagation, - positive and negative clicks, - bounding boxes, - streaming masks back to user, - lightweight fast-api server that you can extend I basically asked Codex to port over the original Facebook sam2 codebase to mlx and optimize it for mlx. And it just worked. This demo for example is a 4bit base-plus model (~100MB download) To speed up, you can also do spatial or temporal downsampling. That speeds inference on my old macbook by ~3x and the hit in parity is negligible.

SAM-2 is apache license - you can indeed just build crazy apps with them without fear. This weekend I ported SAM-2 models to run optimally on Apple Silicon. I have also quantized all the SAM models to 16bit, 8bit, and 4bit variants (collection below) Repo is open source and pip installable. Comes with all you'll need for interactive video segmentation - - bidirectional propagation, - positive and negative clicks, - bounding boxes, - streaming masks back to user, - lightweight fast-api server that you can extend I basically asked Codex to port over the original Facebook sam2 codebase to mlx and optimize it for mlx. And it just worked. This demo for example is a 4bit base-plus model (~100MB download) To speed up, you can also do spatial or temporal downsampling. That speeds inference on my old macbook by ~3x and the hit in parity is negligible.

AVB

17,396 次观看 • 3 个月前

local LLMs on Mac are getting good. but the workflow around models is still messy. you have models in: •⁠ ⁠LM Studio •⁠ ⁠Ollama •⁠ ⁠Hugging Face cache •⁠ ⁠random folders •⁠ ⁠manual downloads and every runtime expects something slightly different. we built ModelHub to clean up this layer. it’s a native macOS menu bar app for discovering, downloading, and managing local LLMs. not a replacement for Ollama, oMLX or LM Studio. it sits beside them. Ollama = run models MLX = Apple Silicon acceleration LM Studio = local model workspace Hugging Face = model ecosystem ModelHub = model discovery + management layer we’re live on Product Hunt today. would love feedback from people running local models on Mac.

local LLMs on Mac are getting good. but the workflow around models is still messy. you have models in: •⁠ ⁠LM Studio •⁠ ⁠Ollama •⁠ ⁠Hugging Face cache •⁠ ⁠random folders •⁠ ⁠manual downloads and every runtime expects something slightly different. we built ModelHub to clean up this layer. it’s a native macOS menu bar app for discovering, downloading, and managing local LLMs. not a replacement for Ollama, oMLX or LM Studio. it sits beside them. Ollama = run models MLX = Apple Silicon acceleration LM Studio = local model workspace Hugging Face = model ecosystem ModelHub = model discovery + management layer we’re live on Product Hunt today. would love feedback from people running local models on Mac.

sabesh 📟

31,497 次观看 • 3 个月前

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.

Awni Hannun

60,599 次观看 • 6 个月前

🎉 Congrats to MOSI + OpenMOSS on the release of MOSS-Transcribe-Diarize-0.9B — a genuinely impressive end-to-end ASR model for real-world, multi-speaker conversations. We're proud to partner with them for day-0 support on mlx-audio (Python & Swift). Now running local-first on Apple Silicon: 🎙️ transcript + speaker labels + timestamps in one pass 🗣️ unlimited-speaker diarization ⏱️ up to ~90 min of audio per input 🌍 90+ languages 🧠 128K context • Whisper-Medium encoder + Qwen3-0.6B decoder 🎯 hotword boosting for names, products & domain terms Perfect for meetings, podcasts, interviews, and long-form call analysis — no cloud, no data leaving your Mac. Get started now: 🐍 Python > uv pip install -U mlx-audio 🍎 Swift .package(url: " from: "0.1.3") Go build. 🚀

🎉 Congrats to MOSI + OpenMOSS on the release of MOSS-Transcribe-Diarize-0.9B — a genuinely impressive end-to-end ASR model for real-world, multi-speaker conversations. We're proud to partner with them for day-0 support on mlx-audio (Python & Swift). Now running local-first on Apple Silicon: 🎙️ transcript + speaker labels + timestamps in one pass 🗣️ unlimited-speaker diarization ⏱️ up to ~90 min of audio per input 🌍 90+ languages 🧠 128K context • Whisper-Medium encoder + Qwen3-0.6B decoder 🎯 hotword boosting for names, products & domain terms Perfect for meetings, podcasts, interviews, and long-form call analysis — no cloud, no data leaving your Mac. Get started now: 🐍 Python > uv pip install -U mlx-audio 🍎 Swift .package(url: " from: "0.1.3") Go build. 🚀

Prince Canuma

11,600 次观看 • 1 个月前

AI researcher breaks down why your $5,000 Mac Studio runs local AI slower than benchmarks promise: "Performance isn't about the hardware - it's about three software layers stacked on top of it." in 15 minutes he tests MLX, Ollama, llama.cpp, and the new vllm-mlx head to head no GPU needed, just the Mac you probably already own - why Ollama, the most popular tool, is actually one of the slower options on Mac - the real bottleneck nobody benchmarks: 'prefill' time before the first token even appears - the exact quantization setting (4-bit) that's the best speed-to-quality tradeoff - why unified memory makes Macs uniquely good for big models compared to Nvidia GPUs complete guide to choosing the right hardware for local AI is below 👇

AI researcher breaks down why your $5,000 Mac Studio runs local AI slower than benchmarks promise: "Performance isn't about the hardware - it's about three software layers stacked on top of it." in 15 minutes he tests MLX, Ollama, llama.cpp, and the new vllm-mlx head to head no GPU needed, just the Mac you probably already own - why Ollama, the most popular tool, is actually one of the slower options on Mac - the real bottleneck nobody benchmarks: 'prefill' time before the first token even appears - the exact quantization setting (4-bit) that's the best speed-to-quality tradeoff - why unified memory makes Macs uniquely good for big models compared to Nvidia GPUs complete guide to choosing the right hardware for local AI is below 👇

rewind

207,513 次观看 • 2 个月前

Open-weight MiniMax M3 filled out a US customs form from a driver's license photo For this test we deployed MiniMax M3 Q4 using MLX-VLM on a Mac Studio M3 Ultra 512GB RAM. The model was tasked with reading a scanned document and an ID card photo, then completing a declaration form Output: 736 tokens · Input: 1,847 tokens · Time: ~31s The model analyzed both inputs, streamed its reasoning, and then called three tools: write_field for text fields, mark for Yes/No checkboxes, and sign for the signature and date. It extracted the required information, mapped it to the correct fields and completed the form without any manual input

Open-weight MiniMax M3 filled out a US customs form from a driver's license photo For this test we deployed MiniMax M3 Q4 using MLX-VLM on a Mac Studio M3 Ultra 512GB RAM. The model was tasked with reading a scanned document and an ID card photo, then completing a declaration form Output: 736 tokens · Input: 1,847 tokens · Time: ~31s The model analyzed both inputs, streamed its reasoning, and then called three tools: write_field for text fields, mark for Yes/No checkboxes, and sign for the signature and date. It extracted the required information, mapped it to the correct fields and completed the form without any manual input

atomic.chat

109,369 次观看 • 2 个月前

🗓️ Release Week Recap Big week. mlx-audio and mlx-vlm are now among some of the fastest-growing OSS projects. Here’s what we shipped last week. Gemma 4 on Apple Silicon Two awesome releases by our partner Google DeepMind & Google Gemma : > Gemma 4 12B — their new dense, unified multimodal model. It uses an encoder free audio path and simplified vision encoder. > Gemma 4 QAT — quantization-aware training checkpoints, optimized to run locally on consumer GPUs and edge devices, compressing the model while preserving the quality you expect from Gemma 4. On the audio 🎧 side we added support for 15+ new TTS, ASR & VAD models, faster long-form transcription, and an expanded OpenAI-compatible audio server. All local on Apple Silicon. Huge thanks to every contributor and my co-maintainer Lucas Newman. 🙏🏽

🗓️ Release Week Recap Big week. mlx-audio and mlx-vlm are now among some of the fastest-growing OSS projects. Here’s what we shipped last week. Gemma 4 on Apple Silicon Two awesome releases by our partner Google DeepMind & Google Gemma : > Gemma 4 12B — their new dense, unified multimodal model. It uses an encoder free audio path and simplified vision encoder. > Gemma 4 QAT — quantization-aware training checkpoints, optimized to run locally on consumer GPUs and edge devices, compressing the model while preserving the quality you expect from Gemma 4. On the audio 🎧 side we added support for 15+ new TTS, ASR & VAD models, faster long-form transcription, and an expanded OpenAI-compatible audio server. All local on Apple Silicon. Huge thanks to every contributor and my co-maintainer Lucas Newman. 🙏🏽

Prince Canuma

11,810 次观看 • 2 个月前

Introducing Unsloth Desktop 🦥 The first desktop app to run and train models locally. • Open-source. Runs on Mac, Windows and Linux • Supports MLX, diffusion image/video, audio, GGUF • Connect Claude Code and Codex to local LLMs • 50% more accurate, self-healing tool calls + sandboxed code exec • Works for CPU + multiGPU setups - NVIDIA, AMD, Intel, Mac • Train models 2× faster with 70% less VRAM • Private web search, deep research, RAG, MCP and exports (NVFP4, GGUF) • Use Unsloth’s OpenAI-compatible API and cloud models • Securely deploy LLMs remotely and access anywhere Unsloth Desktop is now available on and GitHub. GitHub: Blog and Guide:

Introducing Unsloth Desktop 🦥 The first desktop app to run and train models locally. • Open-source. Runs on Mac, Windows and Linux • Supports MLX, diffusion image/video, audio, GGUF • Connect Claude Code and Codex to local LLMs • 50% more accurate, self-healing tool calls + sandboxed code exec • Works for CPU + multiGPU setups - NVIDIA, AMD, Intel, Mac • Train models 2× faster with 70% less VRAM • Private web search, deep research, RAG, MCP and exports (NVFP4, GGUF) • Use Unsloth’s OpenAI-compatible API and cloud models • Securely deploy LLMs remotely and access anywhere Unsloth Desktop is now available on and GitHub. GitHub: Blog and Guide:

Unsloth AI

799,378 次观看 • 20 天前

Introducing Jan-v1: 4B model for web search, an open-source alternative to Perplexity Pro. In our evals, Jan v1 delivers 91% SimpleQA accuracy, slightly outperforming Perplexity Pro while running fully locally. Use cases: - Web search - Deep Research Built on the new version of Qwen's Qwen3-4B-Thinking (up to 256k context length), fine-tuned for reasoning and tool use in Jan. You can run the model in Jan, llama.cpp, or vLLM. To enable search in Jan, go to Settings → Experimental Features → On, then Settings → MCP Servers → enable a search-related MCP such as Serper. Use the model: - Jan-v1-4B: - Jan-v1-4B-GGUF: Credit to the Qwen team for Qwen3 4B Thinking & Georgi Gerganov for llama.cpp.

Introducing Jan-v1: 4B model for web search, an open-source alternative to Perplexity Pro. In our evals, Jan v1 delivers 91% SimpleQA accuracy, slightly outperforming Perplexity Pro while running fully locally. Use cases: - Web search - Deep Research Built on the new version of Qwen's Qwen3-4B-Thinking (up to 256k context length), fine-tuned for reasoning and tool use in Jan. You can run the model in Jan, llama.cpp, or vLLM. To enable search in Jan, go to Settings → Experimental Features → On, then Settings → MCP Servers → enable a search-related MCP such as Serper. Use the model: - Jan-v1-4B: - Jan-v1-4B-GGUF: Credit to the Qwen team for Qwen3 4B Thinking & Georgi Gerganov for llama.cpp.

👋 Jan

690,853 次观看 • 1 年前

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push up the frontier specifically for local hardware. Better kernels, inference tricks like spec decoding, quantizing models, model routing. 2. Usability / UX. To use Local AI today, you need to install CLIs, select a model, a quantization, a harness that works best with that model, and you need to configure your inference engine. It's not very accessible. More work needs to be done on making this seamless, so the user doesn't even know the AI is running locally.

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push up the frontier specifically for local hardware. Better kernels, inference tricks like spec decoding, quantizing models, model routing. 2. Usability / UX. To use Local AI today, you need to install CLIs, select a model, a quantization, a harness that works best with that model, and you need to configure your inference engine. It's not very accessible. More work needs to be done on making this seamless, so the user doesn't even know the AI is running locally.

Alex Cheema

30,820 次观看 • 1 个月前

A voice agent powered by gpt-oss. Running locally on my macBook. Demo recorded in a Waymo with WiFi turned off. I'm still on my space game voice AI kick, obviously. Code link below. For conversational voice AI, you want to set the gpt-oss reasoning behavior to "low". (The default is "medium".) Notes on how to do that and a jinja template you can use are in the repo. The LLM in the demo video is the big, 120B version of gpt-oss. You can use the smaller, 20B model for this, of course. But OpenAI really did a cool thing here designing the 120B model to run in "just" 80GB of VRAM. And the llama.cpp mlx inference is fast: ~250ms TTFT. Running a big model on-device feels like a time warp into the future of AI.

A voice agent powered by gpt-oss. Running locally on my macBook. Demo recorded in a Waymo with WiFi turned off. I'm still on my space game voice AI kick, obviously. Code link below. For conversational voice AI, you want to set the gpt-oss reasoning behavior to "low". (The default is "medium".) Notes on how to do that and a jinja template you can use are in the repo. The LLM in the demo video is the big, 120B version of gpt-oss. You can use the smaller, 20B model for this, of course. But OpenAI really did a cool thing here designing the 120B model to run in "just" 80GB of VRAM. And the llama.cpp mlx inference is fast: ~250ms TTFT. Running a big model on-device feels like a time warp into the future of AI.

kwindla

202,211 次观看 • 1 年前

🐆 Rapid-MLX v0.12 is here. We’ve officially evolved from a simple chat app into a full-fledged, on-device AI studio for Apple Silicon! 🖥️✨ We didn't just push the MLX inference engine to its limits and expand support for a massive lineup of local open-source models—we are alpha-launching the highly anticipated Desktop Version. (A huge shoutout to the IoTeX community for grinding through the closed beta with us. Your feedback was incredible and helped shape this beast.) Here are the game-changing features you can run on your Mac right now, 100% free and 100% offline 👇 🚀 Blazing Fast Local LLMs Run anything from 4B up to Qwen3.5-122B completely offline. No guessing games—we recommend models matched perfectly to your Mac's actual RAM. Rich chat includes syntax highlighting, markdown tables, and honest tok/s metrics. 🎨 Local Image Generation A brand new Images tab to render directly on your machine. Pick a model (FLUX.2-klein, Z-Image-Turbo), prompt, and refine. Everything lands in a visual filmstrip. 👁️ Vision & Live Web Tools Attach an image and chat about it with local vision. Need real-time data? Our built-in web tools (weather, search, page-fetch) run mid-answer with strict, transparent privacy controls. 🤖 Plug-and-Play Coding Agents Wire up Claude Code, Codex, Cline, or Continue in seconds. One copy-paste from the Launch tab spins up a local OpenAI/Anthropic-compatible endpoint. 🔒 Private by Design Everything runs on-device. Signed, notarized, and entirely local. Your data stays yours. Turn your Mac into an AI powerhouse today. ⚡️

raullen

33,889 次观看 • 21 天前

A 27B parameter model used to need a server room. Now it runs on: • iPhone • Android • Mac PrismML's Bonsai makes it possible. • 1-bit weights • 27B params in just 3.9GB • ~90% of full precision quality (PrismML evals) • Better than the 2-bit version at less than half the size The entire Bonsai family is now live in RunAnywhere. • 1.7B to 27B models • 1-bit + 2-bit ternary • Thinking mode On iOS and Mac, Bonsai runs through ggml's llama.cpp and Apple's MLX. On Android, Bonsai runs through ggml's llama.cpp and directly on the Qualcomm Hexagon NPU through QHexRT, our proprietary inference runtime. We built custom silicon kernels to make 1-bit inference on an NPU possible for the first time. Two years ago this needed a data center. Now it thinks in airplane mode. Now available in the RunAnywhere app on the App Store and Google Play.

A 27B parameter model used to need a server room. Now it runs on: • iPhone • Android • Mac PrismML's Bonsai makes it possible. • 1-bit weights • 27B params in just 3.9GB • ~90% of full precision quality (PrismML evals) • Better than the 2-bit version at less than half the size The entire Bonsai family is now live in RunAnywhere. • 1.7B to 27B models • 1-bit + 2-bit ternary • Thinking mode On iOS and Mac, Bonsai runs through ggml's llama.cpp and Apple's MLX. On Android, Bonsai runs through ggml's llama.cpp and directly on the Qualcomm Hexagon NPU through QHexRT, our proprietary inference runtime. We built custom silicon kernels to make 1-bit inference on an NPU possible for the first time. Two years ago this needed a data center. Now it thinks in airplane mode. Now available in the RunAnywhere app on the App Store and Google Play.

RunAnywhere (YC W26)

16,892 次观看 • 1 个月前

THE NEXT AI WORKER MAY COST $500 AND SIT BEHIND YOUR MONITOR. Mini PCs are becoming the cheapest path to private AI. A Mac Mini M4 starts at $599, Geekom AX8 Pro starts near $529, and MinisForum pushes up to 128GB RAM for local AI workloads. Ollama turns that hardware into a personal model server. No cloud fee. No data storage bill. No waiting for another SaaS product to decide your limits. Market shift is simple -> computer is moving back to the desk. Owning a mini PC is starting to look less like buying hardware and more like hiring a quiet AI operator.

THE NEXT AI WORKER MAY COST $500 AND SIT BEHIND YOUR MONITOR. Mini PCs are becoming the cheapest path to private AI. A Mac Mini M4 starts at $599, Geekom AX8 Pro starts near $529, and MinisForum pushes up to 128GB RAM for local AI workloads. Ollama turns that hardware into a personal model server. No cloud fee. No data storage bill. No waiting for another SaaS product to decide your limits. Market shift is simple -> computer is moving back to the desk. Owning a mini PC is starting to look less like buying hardware and more like hiring a quiet AI operator.

Lummox

394,894 次观看 • 2 个月前

How to get OpenClaw🦞 to code for you at almost $0 costs. 1. Power your openclaw with Ollama (if you're on a mac f.e.), or with haiku or kimi k2.5 (if you're on VPS). Or pony-alpha on openrouter which I just discovered today (completely free) 2. Use the Kilo CLI for coding, since it lets you use MiniMax M2.1 for free, a very good coding model Works surprisingly well and doesn't bankrupt you

How to get OpenClaw🦞 to code for you at almost $0 costs. 1. Power your openclaw with Ollama (if you're on a mac f.e.), or with haiku or kimi k2.5 (if you're on VPS). Or pony-alpha on openrouter which I just discovered today (completely free) 2. Use the Kilo CLI for coding, since it lets you use MiniMax M2.1 for free, a very good coding model Works surprisingly well and doesn't bankrupt you

Moritz Kremb

56,200 次观看 • 6 个月前