Got continuous batching working with SSMs in mlx-lm. Here's... four OpenCode agents simultaneously running Nvidia's Nemotron Nano on 64GB M4 Max. This is a nice model for smaller machines since it's MoE + hybrid attention (small cache).show more

Running DeepSeek-V3 on M4 Mac Mini AI Cluster 671B... MoE model distributed across 8 M4 Pro 64GB Mac Minis. Apple Silicon with unified memory is a great fit for MoE.show more

No-one: But can you do 16 generations on your... M4 laptop simultaneously? MLX LM:show more

Batching for vision models is now available in Beta... with our latest MLX engine update 👾 The updated engine also brings major improvements to caching for faster inference overall. Turn on Developer Mode, choose the beta runtime channel, and select LM Studio MLX v1.8.1.show more

First steps for a specialized DeepSeek v4 Flash inference... engine focused on inference quality / stability at different quantizations, with networked API that is batching capable. This is the 2 bit quants model running on my M3 Max 128GB.show more

Qwen QwQ 32B fp16 on M4 Max and M2... Ultra powered by MLX! M2 Ultra - 10.2 toks/s M4 Max - 7.6 toks/s! "Create an amazing animation using p5js" o1-mini level local model! Note: use temp 0.7-0.75 for optimal results in coding. I did some tests and thisshow more

How much faster is the new MacBook Pro for... AI inference? M4 Max is 27% faster with 72 tok/sec compared to 56 tok/sec of the M3 Max with MLX running Gemma 2 9B (4bit). The 27% speedup is the same with Llama-3.2-1b, Llama-3.2-3b and others. Next up: EXO Labs M4 cluster.show more

NVIDIA Nemotron 3 Nano Omni, a new multimodal reasoning... model, is now live on Jetson AI Lab and unifies vision, audio, and language into a single reasoning loop. 🙌 Power your NemoClaws by running this model with Ollama, vLLM and other inference frameworks on NVIDIA Jetson hardware. Try it ➡️show more

DeepSeek-Prover (4-bit 7B) running at 114 toks/sec in MLX... LM on an M2 Ultra Thanks to for the port!show more

M4 Mac Mini AI Cluster Uses EXO Labs with... Thunderbolt 5 interconnect (80Gbps) to run LLMs distributed across 4 M4 Pro Mac Minis. The cluster is small (iPhone for reference). It’s running Nemotron 70B at 8 tok/sec and scales to Llama 405B (benchmarks soon).show more

Managed to get Ling Mini 16B (1.4B active) running... on my iPhone Air. It runs very fast with MLX. It's a DWQ of Ling Mini quantized to 3 bits-per-weight. A 16B model running on an Air at this speed is pretty awesome:show more

Sam 3 by Facebook now on MLX 🚀 Here... is a realtime object tracking running on M3 Max 96GB.show more

Kimi K3 is now live in LM Studio Bionic!... A massive, open frontier model: 2.8T MoE with 1M context window. Kimi K3 in Bionic is hosted on US-based servers, with ZDR enabled by default.show more

DeepSeek R1 Qwen 7B 4bit M2 Ultra vs M4... Max on Apple MLX 🤫 Let them think... (video 4x in center part) M2 Ultra: 114.9 tokens per sec M4 Max (14"): 88.3 tokens per secshow more

Introducing MON Protocol Partner - Hybrid Hybrid is an... Ethereum-based Layer 2 blockchain that integrates a Mixture of Experts (MoE) framework, enabling easy creation and monetization of AI agents in a plug-and-play approach. More about Hybrid here: Hybridshow more

Currently working on a retro inspired action horror game... with a small team DDDistortion Here's the test lowpoly player model I madeshow more

Qwen 3.5 0.8B, Gated DeltaNet attention is running on... Apple Neural Engine ~56 t/s in LUT6 quantization with some room for optimization left. It is CoreML, Swift and IOSurface on M4Pro. It will slow down as we increase context, but not by much. I think Private API opens the way to integrate ANE with GPU/MLX and possibly some MoE.show more

Gotta love MoEs on Apple silicon with MLX. Kimi's... new 16B (3B active) Moonshot model runs very nicely on an M4 Max. As good or better than some of the best dense 7Bs and 1.5x faster inference (154 toks/sec!):show more