Got continuous batching working with SSMs in mlx-lm.
Awni Hannun
Visit on X@awnihannun45K followers
35,231 views
Got continuous batching working with SSMs in mlx-lm.
Here's four OpenCode agents simultaneously running Nvidia's Nemotron Nano on 64GB M4 Max.
This is a nice model for smaller machines since it's MoE + hybrid attention (small cache).
Comments
Comments from the original post on X (Twitter)
No comments yet
