DwarfStar DeepSeek v4 Flash 0731 pre-M5 kernel keeps improving 💪🏻 - prefill +11% - decode +10% - single request no DSpark/MTP. - 100% same logics between baseline and now. Today is QA like crazy. I hope to file PR by EOD antirez Here are results on the M3 Ultra... and a video now vs baseline using ds4-eval.show more

It’s crazy that testing has become the bottleneck and not coding… it’s not even testing but hardware being slow and taking forever to run through the automated tests. Btw you should check out llmprobe, I built it specifically for this, fast iteration, before the final big test. “npx llmprobe@latest” give it a try and let me know if you like it. On my github too.

@antirez Sure! I’ll give it a try, you’re building great things!

I'll push this beyond 40 before creating PR. I think/hope there is still room for improvement.

@antirez @ivanfioravanti On my M3 Ultra with my custom Metal implementation, I’m seeing about 45 tokens/s decode and 426 tokens/s prefill on DeepSeek-V4-Flash-0731-MXFP4-MLX.

@antirez Have you tried batching the prefill?

@antirez Can this be used over MLX Distributed? Using 2 M5Max with MLX/MTP … 30 t/s

@antirez ds4 has its own TP distributed mechanism.

@antirez Yes, did not realize but the drafter is running indeed on the 2 nodes in sync.

@antirez

prime-agent played a great role here!

@antirez What's next on the pre-m5 line?

@antirez I was actually hoping that @eigenlabs would do a contest for DSv4 targeting both the 2bit and the 4bit. I'm curious if they'd get as much participation due to the ram floor.

@antirez @eigenlabs 4bit is big, 256GB up. 2bit would be better, but still not so common.

@antirez @eigenlabs Yeah, the tiny ones are the most fun to optimize but they are not that useful outside of the 35b variants.

@antirez @eigenlabs 27b?

@antirez @eigenlabs Its memory bound and close to compute bound so its very hard. I was looking at the encoding stuff they were doing for NVFP4 on Laguna so there's probably some headway to get around the mid 80s if those pan out. There's just no way around having to read 27b weights.

logits not logics

@antirez Very cool.

@antirez Stay tuned for more! 🚀

@antirez So basically it runs on M3 Ultra with minimal quality loss?

@antirez How much ram does your m3 ultra have

@antirez @ivanfioravanti can you share your benchmarks here for the community with a PR? Thank you

@antirez Great work! 0731 is a fine model!

@antirez Try it with quantprobe, your datapoint would be interesting in my charts! I’m about to test it on my 6GB card and 16GB ram

@antirez i spent all morning debugging a kernel panic just to find out my prefill metrics were off by a factor of ten

@antirez Awesome work.

@antirez Great test, Ivan👍 If you'd like to try the cloud version later, feel free to reach out. Our DeepSeek V4 Flash 0731 is now live, and we can offer you test credits to help more people discover this great model.

@antirez Or use Dram instead of HBM to serve the KV cache?

@antirez The +10% decode on M3 Ultra is the headline. Prefill has more room to move, but squeezing 35.27 to 38.86 tok/s from a single-request path is real kernel work.

@antirez prefill and decode usually move for different reasons, one compute bound and one bandwidth bound. both landing near ten points at once suggests dispatch overhead rather than the math, which is the better kind of win because it holds across shapes.

@antirez Cool speed gains. But does it actually write better cold emails? That's what I need to know.

@antirez what is the githup repo?

@antirez Can it use SSD as the kv cache?

@antirez Strong single-request gains, but that's the easy case. How does it handle concurrent requests or batching?