Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

- GLM 5.3 Flash EXL3 - 2x DGX Sparks - TensorFold - 4 concurrent prose streams Pay attention to tok/s and TTFT on each stream. Buttery smooth experience!

Mia

40,234 subscribers

32,137 Aufrufe • vor 11 Tagen •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

39 Kommentare

Profilbild von Ash Hart

Ash Hartvor 11 Tagen

Excellent work on this recipe, Mia! 🚀

Profilbild von Mark Argyelan

Mark Argyelanvor 11 Tagen

Already installed! Thanks, @ashxhart & @MiaAI_lab The numbers speak for themselves: both Sparks CC@2000, with very low power consumption. to be exact I measured 37-45W Best Package so far!!

Profilbild von Matt Barrie

Matt Barrievor 11 Tagen

You can see this is going to be a massive problem for Anthropic & Open AI

Profilbild von just mntz

just mntzvor 11 Tagen

Great! What about intelligence? Can it keep up with NVFP4?

Profilbild von Mia

Miavor 11 Tagen

no. it's better.

Profilbild von demmon standafer

demmon standafervor 11 Tagen

@just_mntz Why? Isn’t nvpf4 more optimized?

Profilbild von Mia

Miavor 11 Tagen

@just_mntz noup

Profilbild von Mackross

Mackrossvor 11 Tagen

This is unbelievable. Congratulations to you and Ash

Profilbild von JCode

JCodevor 11 Tagen

looks amazing! also the harness is looking great Mia!

Profilbild von Mia

Miavor 11 Tagen

thanks!

Profilbild von Daniel Banu

Daniel Banuvor 11 Tagen

Is there anything lost by going to fp8 kv cache?

Profilbild von Mia

Miavor 11 Tagen

None, I've tested it

Profilbild von Sayyid Fareed

Sayyid Fareedvor 11 Tagen

Beautiful recipe. Fast and completely fixes the issue we had on the previous Tensorfold recipe last night where the worker would overheat due to idling on the GPU. Amazing work!

Profilbild von Norman Headings

Norman Headingsvor 11 Tagen

Awesome stuff! Any plans to use tensorfold on ds 4.1 ?

Profilbild von Mia

Miavor 11 Tagen

ofc

Profilbild von 🦁 Lion (LFMN)

🦁 Lion (LFMN)vor 11 Tagen

Amazing 🔥🔥🔥

Profilbild von Daniel Morris

Daniel Morrisvor 11 Tagen

I will be joining the 2x DGX spark club shortly

Profilbild von pd19

pd19vor 11 Tagen

还是你的工具好看，老妹儿

Profilbild von ☁️

☁️vor 11 Tagen

Ablit?

Profilbild von Mia

Miavor 11 Tagen

Soon

Profilbild von Soveryn Intelligence

Soveryn Intelligencevor 11 Tagen

Ours, 4 streams: 42.6 tok/s each, TTFT 312–313 ms on every slot. Old vLLM Kernel was 10.9 each at ~2.6 s. Stream 4 doesn't bleed.

Profilbild von Mia

Miavor 11 Tagen

You get BETTER speeds than me on MY recipe?? lol

Profilbild von Soveryn Intelligence

Soveryn Intelligencevor 11 Tagen

hey what can I tell ya maybe its the harness... we are running it on Pi our own version of soveryn pi.. but with out your set up I would not have any numbers to share really nice work

Profilbild von hhh ppp

hhh pppvor 11 Tagen

I really need proper images handling, 4 is not enough and aplying the image limits patch would reprocess the entire conversation, any plans for a fix for this?

Profilbild von Sidhanth Krishna

Sidhanth Krishnavor 11 Tagen

Legend 🫡 Tried it out and works as expected. Noticed the following and was hoping to get your two cents on the issue: - Prompt cache is breaking with image inputs on opencode (haven’t tried with videos) - Rig is running hot during prefill, should we decrease the clock?

Profilbild von Liam Nerd

Liam Nerdvor 11 Tagen

Four streams is the workday case.

Profilbild von gQ

gQvor 11 Tagen

Stability and speed are looking amazing. Congrats to everyone involved 👏

Profilbild von Cesar A.

Cesar A.vor 11 Tagen

If you were buying today would you go 2x DGX Sparks or M5 Ultra 256GB?

Profilbild von Alissa

Alissavor 11 Tagen

Prefill is at ~840 tok/s, not the ~1,950 in the README. Their own benchmark tool measures the same here. any hints ?

Profilbild von sic Fona

sic Fonavor 11 Tagen

もはやクラウド級のAI🤯

Profilbild von James Malsawm

James Malsawmvor 11 Tagen

Amazing

Profilbild von Dirk Glensk

Dirk Glenskvor 11 Tagen

Nice! Can you also show one where you input large text + images?

Profilbild von basedcapital

basedcapitalvor 11 Tagen

2x dgx spark is 256gb of unified memory on a desk. tensorfold's whole trick is lazy mmap streaming so you never pin all the weights at once, and exl3 shrinks the quants further. 4 concurrent streams feeling buttery makes sense, the box finally has headroom for kv cache.

Profilbild von Me&Doggo

Me&Doggovor 11 Tagen

The speed, and the speed of new changes, is amazing.

Profilbild von Yacine Zaitri

Yacine Zaitrivor 11 Tagen

How about running this on a single ultra5? How is TensorFold compared with MLX?

Profilbild von L1inear

L1inearvor 11 Tagen

Any chance you'll optimize MLX once 512 GB ultras get released?

Profilbild von Alissa

Alissavor 11 Tagen

great work ! trying it now and comparing with the vllm 2xsparks vs 2xsparks

Profilbild von Aditya

Adityavor 11 Tagen

This is great look at the improvement! Thanks @MiaAI_lab

Profilbild von A

Avor 11 Tagen

I don’t think people outside of local AI comprehend what huge advancement we’ve seen in the past few months. It’s absolutely incredible. Thank you so much for all the work you’ve done.

Ähnliche Videos

How prose looks like in Hermes agent using GLM 5.3 Flash EXL3 on 2x DGX Sparks

How prose looks like in Hermes agent using GLM 5.3 Flash EXL3 on 2x DGX Sparks

Mia

10,574 Aufrufe • vor 1 Monat

Full GLM 5.3 running on 3x DGX Sparks with TensorFold Currently 24-28 tok/s on prose

Full GLM 5.3 running on 3x DGX Sparks with TensorFold Currently 24-28 tok/s on prose

Mia

49,756 Aufrufe • vor 9 Tagen

GLM 5.3 Flash on 2x DGX Sparks Look at the tok/s at the top :) This is prose. Releasing soon

GLM 5.3 Flash on 2x DGX Sparks Look at the tok/s at the top :) This is prose. Releasing soon

Mia

16,605 Aufrufe • vor 1 Monat

GLM 5.3 Flash EXL3 running on 3x DGX Sparks.

GLM 5.3 Flash EXL3 running on 3x DGX Sparks.

Mia

12,535 Aufrufe • vor 27 Tagen

GLM-5.3-Flash running at up to 300 tok/s single stream, and 1000 tok/s with 8 concurrent sessions. On average it's working at 150 tok/s per stream. Phenomenal model, really something special. On 2x Sparks we get 60 tok/s and 4x 100 tok/s 300B params is the sweet spot.

GLM-5.3-Flash running at up to 300 tok/s single stream, and 1000 tok/s with 8 concurrent sessions. On average it's working at 150 tok/s per stream. Phenomenal model, really something special. On 2x Sparks we get 60 tok/s and 4x 100 tok/s 300B params is the sweet spot.

0xSero

30,628 Aufrufe • vor 1 Monat

How DeepSeek v4.1 Flash looks on 3x DGX Sparks with 4 concurrent streams

How DeepSeek v4.1 Flash looks on 3x DGX Sparks with 4 concurrent streams

Mia

19,730 Aufrufe • vor 1 Monat

GLM-5.3-Flash streaming at 32 tok/s on a single Spark via Tensorfold ⚡️ i will share the recipe later today

GLM-5.3-Flash streaming at 32 tok/s on a single Spark via Tensorfold ⚡️ i will share the recipe later today

filipe

15,103 Aufrufe • vor 10 Tagen

Check out how DeepSeek v4.1 Flash performs on 2x DGX Sparks after the update. Smooth experience!

Check out how DeepSeek v4.1 Flash performs on 2x DGX Sparks after the update. Smooth experience!

Mia

39,491 Aufrufe • vor 26 Tagen

This is what 20 tok/s looks like on 4 x DGX Sparks running GLM 5.2

This is what 20 tok/s looks like on 4 x DGX Sparks running GLM 5.2

Tech2Wild

14,140 Aufrufe • vor 3 Monaten

I ran this locally with GLM-5.3 Flash EXL3 on my DGX Sparks, and I think it beats both Fable 5.1 and GPT-6 Astra. The crazy part: it was faster to make create this on my sparks than it was with either Fable 5.1 or Astra. Live link here:

I ran this locally with GLM-5.3 Flash EXL3 on my DGX Sparks, and I think it beats both Fable 5.1 and GPT-6 Astra. The crazy part: it was faster to make create this on my sparks than it was with either Fable 5.1 or Astra. Live link here:

Mia

48,031 Aufrufe • vor 1 Monat

Warning: Do not stare too long, trust me GLM 5.3 Flash TP=4 5718 tok/s 512 Concurrent Users 6610 tok/s C=768 Engine Decode Tried c=1024, don't fit Wall of c=256 each generating unique code (couldn't fit more on browser, which is probably good thing)

Warning: Do not stare too long, trust me GLM 5.3 Flash TP=4 5718 tok/s 512 Concurrent Users 6610 tok/s C=768 Engine Decode Tried c=1024, don't fit Wall of c=256 each generating unique code (couldn't fit more on browser, which is probably good thing)

Killy

30,081 Aufrufe • vor 1 Monat

🤯 GLM-5.3-Flash 4bit on two M5 Ultras (tp=2) with TensorFold and 4x TB5 cables bonded. 🚀 114 tks decode on Prose single stream, and ~1600tks prefil. C=8 decode up to 200tps. There is headroom here as well, but this is where we landed today. A new-session TTFT with our prefix-cache lean mode is 0.06s. Alex Ziskind Thanks for letting me use the M5Us!

🤯 GLM-5.3-Flash 4bit on two M5 Ultras (tp=2) with TensorFold and 4x TB5 cables bonded. 🚀 114 tks decode on Prose single stream, and ~1600tks prefil. C=8 decode up to 200tps. There is headroom here as well, but this is where we landed today. A new-session TTFT with our prefix-cache lean mode is 0.06s. Alex Ziskind Thanks for letting me use the M5Us!

Ash Hart

26,047 Aufrufe • vor 5 Tagen

Apple Silicon + Gemma 4 fans: this is for you. Pico AI Server now supports continuous batching with MLX-Swift. 43 tok/s on 1 stream. 26 tok/s per stream on 2 concurrent streams. That’s 52 tok/s total. a 21% throughput gain on a six-year-old MacBook Pro M1 Max!

Apple Silicon + Gemma 4 fans: this is for you. Pico AI Server now supports continuous batching with MLX-Swift. 43 tok/s on 1 stream. 26 tok/s per stream on 2 concurrent streams. That’s 52 tok/s total. a 21% throughput gain on a six-year-old MacBook Pro M1 Max!

Ronald Mannak

53,392 Aufrufe • vor 5 Monaten

Loading DeepSeek v4 Flash just reminded me how good and fast this model is and how well it runs on DGX Sparks. GLM-5.2 is really awesome, but if you want fast and concurrent inference, DS4F is still a top choice IMO. Created by DS4F in single shot The coffee guy

Loading DeepSeek v4 Flash just reminded me how good and fast this model is and how well it runs on DGX Sparks. GLM-5.2 is really awesome, but if you want fast and concurrent inference, DS4F is still a top choice IMO. Created by DS4F in single shot The coffee guy

Mia

40,899 Aufrufe • vor 2 Monaten

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.

stevibe

403,356 Aufrufe • vor 5 Monaten

Qwen3.6 27B landed yesterday, so I ran it on 4 setups side-by-side to see how they stack up: 🔴 RTX 4090 — 45.59 tok/s, TTFT 525ms 🟢 RTX 5090 — 51.83 tok/s, TTFT 752ms ⚫️ M2 Ultra — 22.30 tok/s, TTFT 216ms 🟣 DGX Spark — 11.08 tok/s, TTFT 319ms This is a standard test: no tuning, just the out-of-the-box experience. For the NVIDIA cards I used llama.cpp with Unsloth's UD-Q4_K_XL quant. For the M2 Ultra I used MLX with Unsloth's UD-MLX-4bit quant, since MLX is the native path on Apple Silicon. Please consider this as the baseline, you can definitely squeeze more out of every one of these with fine-tuned settings.

Qwen3.6 27B landed yesterday, so I ran it on 4 setups side-by-side to see how they stack up: 🔴 RTX 4090 — 45.59 tok/s, TTFT 525ms 🟢 RTX 5090 — 51.83 tok/s, TTFT 752ms ⚫️ M2 Ultra — 22.30 tok/s, TTFT 216ms 🟣 DGX Spark — 11.08 tok/s, TTFT 319ms This is a standard test: no tuning, just the out-of-the-box experience. For the NVIDIA cards I used llama.cpp with Unsloth's UD-Q4_K_XL quant. For the M2 Ultra I used MLX with Unsloth's UD-MLX-4bit quant, since MLX is the native path on Apple Silicon. Please consider this as the baseline, you can definitely squeeze more out of every one of these with fine-tuned settings.

stevibe

104,692 Aufrufe • vor 5 Monaten

GLM-5.3 weights from Z.ai are live, with SGLang powering day-0 serving support! GLM-5.3 inherits every optimization and feature we battle-tested for GLM-5.2 over the past months. On real-world multi-turn agentic workloads, we measured 537.6 tok/s/user on NVFP4 and 413 tok/s/user on FP8, at BS=1 with TP8 on 8x B300. It's fast, efficient, and production-ready today on NVIDIA AI Blackwell and Hopper, and AI at AMD MI300X/325X/355X. We believe this is a big step forward for GLM-Series in agentic tasks, with a path toward mythos-class cyber capability. Can't wait to see what people build with GLM-5.3 and SGLang🚀 Cookbook👇

GLM-5.3 weights from Z.ai are live, with SGLang powering day-0 serving support! GLM-5.3 inherits every optimization and feature we battle-tested for GLM-5.2 over the past months. On real-world multi-turn agentic workloads, we measured 537.6 tok/s/user on NVFP4 and 413 tok/s/user on FP8, at BS=1 with TP8 on 8x B300. It's fast, efficient, and production-ready today on NVIDIA AI Blackwell and Hopper, and AI at AMD MI300X/325X/355X. We believe this is a big step forward for GLM-Series in agentic tasks, with a path toward mythos-class cyber capability. Can't wait to see what people build with GLM-5.3 and SGLang🚀 Cookbook👇

SGLang

99,455 Aufrufe • vor 1 Monat

GLM 5.3 Flash running locally makes a Minecraft black hole mod We ran GLM 5.3 Flash Q4 locally on a rented 4x RTX PRO 6000 box and asked it to make a lightning gun mod for the real game using the Fabric API Output: -7.6M tokens -Time: ~9 hours -Decode: ~96 tok/s The mod adds a black hole rifle, when shot it spawns a black hole that pulls in blocks, with light rings that shrink all the way into the singularity, then collapses into an explosion crater that wipes out several chunks Run GLM 5.3 Flash locally in

GLM 5.3 Flash running locally makes a Minecraft black hole mod We ran GLM 5.3 Flash Q4 locally on a rented 4x RTX PRO 6000 box and asked it to make a lightning gun mod for the real game using the Fabric API Output: -7.6M tokens -Time: ~9 hours -Decode: ~96 tok/s The mod adds a black hole rifle, when shot it spawns a black hole that pulls in blocks, with light rings that shrink all the way into the singularity, then collapses into an explosion crater that wipes out several chunks Run GLM 5.3 Flash locally in

atomic.chat

32,515 Aufrufe • vor 1 Monat

🚨 We finally cracked it. GLM-5.3, the FULL model (not the Flash), running on 4 DGX Sparks 🔥 ⚡️ 35 tok/s on real coding, peaking near 40 when it's hot 📊 Full C1 to C6 benchmark done, same speeds we were hitting on 5.2 🧠 200K context right now, tuning it higher from here 🎯 And this is only MTP4... we haven't even added DFlash yet I quantized it myself and I'm uploading the repo RIGHT NOW so everyone can go tune it with me 👀 Be tuned, it drops any minute. Let's get it 🚀

🚨 We finally cracked it. GLM-5.3, the FULL model (not the Flash), running on 4 DGX Sparks 🔥 ⚡️ 35 tok/s on real coding, peaking near 40 when it's hot 📊 Full C1 to C6 benchmark done, same speeds we were hitting on 5.2 🧠 200K context right now, tuning it higher from here 🎯 And this is only MTP4... we haven't even added DFlash yet I quantized it myself and I'm uploading the repo RIGHT NOW so everyone can go tune it with me 👀 Be tuned, it drops any minute. Let's get it 🚀

Tech2Wild

13,377 Aufrufe • vor 1 Monat

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.

stevibe

34,141 Aufrufe • vor 5 Monaten