- GLM 5.3 Flash EXL3 - 2x DGX Sparks - TensorFold - 4 concurrent prose streams Pay attention to tok/s and TTFT on each stream. Buttery smooth experience!

Excellent work on this recipe, Mia! 🚀

Already installed! Thanks, @ashxhart & @MiaAI_lab The numbers speak for themselves: both Sparks CC@2000, with very low power consumption. to be exact I measured 37-45W Best Package so far!!

You can see this is going to be a massive problem for Anthropic & Open AI

Great! What about intelligence? Can it keep up with NVFP4?

no. it's better.

@just_mntz Why? Isn’t nvpf4 more optimized?

@just_mntz noup

This is unbelievable. Congratulations to you and Ash

looks amazing! also the harness is looking great Mia!

thanks!

Is there anything lost by going to fp8 kv cache?

None, I've tested it

Beautiful recipe. Fast and completely fixes the issue we had on the previous Tensorfold recipe last night where the worker would overheat due to idling on the GPU. Amazing work!

Awesome stuff! Any plans to use tensorfold on ds 4.1 ?

ofc

Amazing 🔥🔥🔥

I will be joining the 2x DGX spark club shortly

还是你的工具好看，老妹儿

Ablit?

Soon

Ours, 4 streams: 42.6 tok/s each, TTFT 312–313 ms on every slot. Old vLLM Kernel was 10.9 each at ~2.6 s. Stream 4 doesn't bleed.

You get BETTER speeds than me on MY recipe?? lol

hey what can I tell ya maybe its the harness... we are running it on Pi our own version of soveryn pi.. but with out your set up I would not have any numbers to share really nice work

I really need proper images handling, 4 is not enough and aplying the image limits patch would reprocess the entire conversation, any plans for a fix for this?

Legend 🫡 Tried it out and works as expected. Noticed the following and was hoping to get your two cents on the issue: - Prompt cache is breaking with image inputs on opencode (haven’t tried with videos) - Rig is running hot during prefill, should we decrease the clock?

Four streams is the workday case.

Stability and speed are looking amazing. Congrats to everyone involved 👏

If you were buying today would you go 2x DGX Sparks or M5 Ultra 256GB?

Prefill is at ~840 tok/s, not the ~1,950 in the README. Their own benchmark tool measures the same here. any hints ?

もはやクラウド級のAI🤯

Amazing

Nice! Can you also show one where you input large text + images?

2x dgx spark is 256gb of unified memory on a desk. tensorfold's whole trick is lazy mmap streaming so you never pin all the weights at once, and exl3 shrinks the quants further. 4 concurrent streams feeling buttery makes sense, the box finally has headroom for kv cache.

The speed, and the speed of new changes, is amazing.

How about running this on a single ultra5? How is TensorFold compared with MLX?

Any chance you'll optimize MLX once 512 GB ultras get released?

great work ! trying it now and comparing with the vllm 2xsparks vs 2xsparks

This is great look at the improvement! Thanks @MiaAI_lab

I don’t think people outside of local AI comprehend what huge advancement we’ve seen in the past few months. It’s absolutely incredible. Thank you so much for all the work you’ve done.