Running four simultaneous OpenCode agents works well with mlx_lm.server continuous batching and MiniMax M2.1 on an M3 Ultra:

Here's a guide to setup OpenCode with mlx-lm:

M2.1 works well as a cognitive core that orchestrates other programs/tools/agents. It reasons efficiently and doesn't get distracted during long-running tasks. And it's lightweight enough to run at home

It's a good model, sir

This + Distributed TP = 🔥🔥🔥

M5!Ultra is going to be something.

its gone so far in just a few months!

Okay but does it come with a side of fries tho?

memory only spiked up to 212Gb? Very cool

Really healthy tok/s for 4 entire concurrent agents. How quantized is Minimax m2.1 here? Even without much skin in the AI game, Apple hardware might be the hardware of choice for local consumer inference

this is crazy, how many tokens/sec per agent? is it 20

I have mac m1 can I run it ? 😌

Local is not the future anymore

sounds like a solid setup, nice work

cool

Hm so that would be possible in a 256GB machine?

Reeeally cool but will M3.1 locally run as smoothly on a M3 Ultra? As in how future proof is it?

~15 t/s generation an agent?

As u work in apple, do u mind donating one M3 ultra to me please?

Beautiful.

@grok what similar setup could you do with M3 pro 18gb? Can be only 2 agents working together

Stack of M4s will be the GPU in future.

the m3 ultra quietly becoming the "just works" option for local inference while everyone argues about cloud costs

best $2 ever spent

Impressive! Four agents running smoothly on an M3 Ultra is great.

Do I need to configure to use batching in mlx_lm.server. I have 2 studios with m3 ultra 512G

No config necessary. If you encounter issues, let us know.

That is an amazing, beautiful video. There is so much going on there that I didn't know that I want to understand... I'll start reading. First question though - is it reasonable to view one M3 Ultra as a multi-user LLM server?

Very reasonable in my opinion. The biggest challenge is prompt processing. It works way better if all the users share say a long system prompt because then caching helps.

Four agents and a six-panel cockpit-prompt processing is the real bottleneck here

How do you optimize the context handling for those four?

Okay tell us what those agents are doing instead of being a stinky scammer Indian

so cool

That's interesting. I'm assuming it is still using remote LLMs per agent as well?