Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

- GLM 5.3 Flash EXL3 - 2x DGX Sparks - TensorFold - 4 concurrent prose streams Pay attention to tok/s and TTFT on each stream. Buttery smooth experience!

Mia

40,234 subscribers

32,137 görüntüleme • 11 gün önce •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

39 Yorum

Ash Hart profil fotoğrafı

Ash Hart11 gün önce

Excellent work on this recipe, Mia! 🚀

Mark Argyelan profil fotoğrafı

Mark Argyelan11 gün önce

Already installed! Thanks, @ashxhart & @MiaAI_lab The numbers speak for themselves: both Sparks CC@2000, with very low power consumption. to be exact I measured 37-45W Best Package so far!!

Matt Barrie profil fotoğrafı

Matt Barrie11 gün önce

You can see this is going to be a massive problem for Anthropic & Open AI

just mntz profil fotoğrafı

just mntz11 gün önce

Great! What about intelligence? Can it keep up with NVFP4?

Mia profil fotoğrafı

Mia11 gün önce

no. it's better.

demmon standafer profil fotoğrafı

demmon standafer11 gün önce

@just_mntz Why? Isn’t nvpf4 more optimized?

Mia profil fotoğrafı

Mia11 gün önce

@just_mntz noup

Mackross profil fotoğrafı

Mackross11 gün önce

This is unbelievable. Congratulations to you and Ash

JCode profil fotoğrafı

JCode11 gün önce

looks amazing! also the harness is looking great Mia!

Mia profil fotoğrafı

Mia11 gün önce

thanks!

Daniel Banu profil fotoğrafı

Daniel Banu11 gün önce

Is there anything lost by going to fp8 kv cache?

Mia profil fotoğrafı

Mia11 gün önce

None, I've tested it

Sayyid Fareed profil fotoğrafı

Sayyid Fareed11 gün önce

Beautiful recipe. Fast and completely fixes the issue we had on the previous Tensorfold recipe last night where the worker would overheat due to idling on the GPU. Amazing work!

Norman Headings profil fotoğrafı

Norman Headings11 gün önce

Awesome stuff! Any plans to use tensorfold on ds 4.1 ?

Mia profil fotoğrafı

Mia11 gün önce

ofc

🦁 Lion (LFMN) profil fotoğrafı

🦁 Lion (LFMN)11 gün önce

Amazing 🔥🔥🔥

Daniel Morris profil fotoğrafı

Daniel Morris11 gün önce

I will be joining the 2x DGX spark club shortly

pd19 profil fotoğrafı

pd1911 gün önce

还是你的工具好看，老妹儿

☁️ profil fotoğrafı

☁️11 gün önce

Ablit?

Mia profil fotoğrafı

Mia11 gün önce

Soon

Soveryn Intelligence profil fotoğrafı

Soveryn Intelligence11 gün önce

Ours, 4 streams: 42.6 tok/s each, TTFT 312–313 ms on every slot. Old vLLM Kernel was 10.9 each at ~2.6 s. Stream 4 doesn't bleed.

Mia profil fotoğrafı

Mia11 gün önce

You get BETTER speeds than me on MY recipe?? lol

Soveryn Intelligence profil fotoğrafı

Soveryn Intelligence11 gün önce

hey what can I tell ya maybe its the harness... we are running it on Pi our own version of soveryn pi.. but with out your set up I would not have any numbers to share really nice work

hhh ppp profil fotoğrafı

hhh ppp11 gün önce

I really need proper images handling, 4 is not enough and aplying the image limits patch would reprocess the entire conversation, any plans for a fix for this?

Sidhanth Krishna profil fotoğrafı

Sidhanth Krishna11 gün önce

Legend 🫡 Tried it out and works as expected. Noticed the following and was hoping to get your two cents on the issue: - Prompt cache is breaking with image inputs on opencode (haven’t tried with videos) - Rig is running hot during prefill, should we decrease the clock?

Liam Nerd profil fotoğrafı

Liam Nerd11 gün önce

Four streams is the workday case.

gQ profil fotoğrafı

gQ11 gün önce

Stability and speed are looking amazing. Congrats to everyone involved 👏

Cesar A. profil fotoğrafı

Cesar A.11 gün önce

If you were buying today would you go 2x DGX Sparks or M5 Ultra 256GB?

Alissa profil fotoğrafı

Alissa11 gün önce

Prefill is at ~840 tok/s, not the ~1,950 in the README. Their own benchmark tool measures the same here. any hints ?

sic Fona profil fotoğrafı

sic Fona11 gün önce

もはやクラウド級のAI🤯

James Malsawm profil fotoğrafı

James Malsawm11 gün önce

Amazing

Dirk Glensk profil fotoğrafı

Dirk Glensk11 gün önce

Nice! Can you also show one where you input large text + images?

basedcapital profil fotoğrafı

basedcapital11 gün önce

2x dgx spark is 256gb of unified memory on a desk. tensorfold's whole trick is lazy mmap streaming so you never pin all the weights at once, and exl3 shrinks the quants further. 4 concurrent streams feeling buttery makes sense, the box finally has headroom for kv cache.

Me&Doggo profil fotoğrafı

Me&Doggo11 gün önce

The speed, and the speed of new changes, is amazing.

Yacine Zaitri profil fotoğrafı

Yacine Zaitri11 gün önce

How about running this on a single ultra5? How is TensorFold compared with MLX?

L1inear profil fotoğrafı

L1inear11 gün önce

Any chance you'll optimize MLX once 512 GB ultras get released?

Alissa profil fotoğrafı

Alissa11 gün önce

great work ! trying it now and comparing with the vllm 2xsparks vs 2xsparks

Aditya profil fotoğrafı

Aditya11 gün önce

This is great look at the improvement! Thanks @MiaAI_lab

A profil fotoğrafı

A11 gün önce

I don’t think people outside of local AI comprehend what huge advancement we’ve seen in the past few months. It’s absolutely incredible. Thank you so much for all the work you’ve done.

Benzer Videolar

How prose looks like in Hermes agent using GLM 5.3 Flash EXL3 on 2x DGX Sparks

How prose looks like in Hermes agent using GLM 5.3 Flash EXL3 on 2x DGX Sparks

Mia

10,574 görüntüleme • 1 ay önce

Full GLM 5.3 running on 3x DGX Sparks with TensorFold Currently 24-28 tok/s on prose

Full GLM 5.3 running on 3x DGX Sparks with TensorFold Currently 24-28 tok/s on prose

Mia

49,756 görüntüleme • 9 gün önce

GLM 5.3 Flash on 2x DGX Sparks Look at the tok/s at the top :) This is prose. Releasing soon

GLM 5.3 Flash on 2x DGX Sparks Look at the tok/s at the top :) This is prose. Releasing soon

Mia

16,605 görüntüleme • 1 ay önce

GLM 5.3 Flash EXL3 running on 3x DGX Sparks.

GLM 5.3 Flash EXL3 running on 3x DGX Sparks.

Mia

12,535 görüntüleme • 27 gün önce

GLM-5.3-Flash running at up to 300 tok/s single stream, and 1000 tok/s with 8 concurrent sessions. On average it's working at 150 tok/s per stream. Phenomenal model, really something special. On 2x Sparks we get 60 tok/s and 4x 100 tok/s 300B params is the sweet spot.

GLM-5.3-Flash running at up to 300 tok/s single stream, and 1000 tok/s with 8 concurrent sessions. On average it's working at 150 tok/s per stream. Phenomenal model, really something special. On 2x Sparks we get 60 tok/s and 4x 100 tok/s 300B params is the sweet spot.

0xSero

30,628 görüntüleme • 1 ay önce

How DeepSeek v4.1 Flash looks on 3x DGX Sparks with 4 concurrent streams

How DeepSeek v4.1 Flash looks on 3x DGX Sparks with 4 concurrent streams

Mia

19,730 görüntüleme • 1 ay önce

GLM-5.3-Flash streaming at 32 tok/s on a single Spark via Tensorfold ⚡️ i will share the recipe later today

GLM-5.3-Flash streaming at 32 tok/s on a single Spark via Tensorfold ⚡️ i will share the recipe later today

filipe

15,103 görüntüleme • 10 gün önce

Check out how DeepSeek v4.1 Flash performs on 2x DGX Sparks after the update. Smooth experience!

Check out how DeepSeek v4.1 Flash performs on 2x DGX Sparks after the update. Smooth experience!

Mia

39,491 görüntüleme • 26 gün önce

This is what 20 tok/s looks like on 4 x DGX Sparks running GLM 5.2

This is what 20 tok/s looks like on 4 x DGX Sparks running GLM 5.2

Tech2Wild

14,140 görüntüleme • 3 ay önce

I ran this locally with GLM-5.3 Flash EXL3 on my DGX Sparks, and I think it beats both Fable 5.1 and GPT-6 Astra. The crazy part: it was faster to make create this on my sparks than it was with either Fable 5.1 or Astra. Live link here:

I ran this locally with GLM-5.3 Flash EXL3 on my DGX Sparks, and I think it beats both Fable 5.1 and GPT-6 Astra. The crazy part: it was faster to make create this on my sparks than it was with either Fable 5.1 or Astra. Live link here:

Mia

48,031 görüntüleme • 1 ay önce

Warning: Do not stare too long, trust me GLM 5.3 Flash TP=4 5718 tok/s 512 Concurrent Users 6610 tok/s C=768 Engine Decode Tried c=1024, don't fit Wall of c=256 each generating unique code (couldn't fit more on browser, which is probably good thing)

Warning: Do not stare too long, trust me GLM 5.3 Flash TP=4 5718 tok/s 512 Concurrent Users 6610 tok/s C=768 Engine Decode Tried c=1024, don't fit Wall of c=256 each generating unique code (couldn't fit more on browser, which is probably good thing)

Killy

30,081 görüntüleme • 1 ay önce

🤯 GLM-5.3-Flash 4bit on two M5 Ultras (tp=2) with TensorFold and 4x TB5 cables bonded. 🚀 114 tks decode on Prose single stream, and ~1600tks prefil. C=8 decode up to 200tps. There is headroom here as well, but this is where we landed today. A new-session TTFT with our prefix-cache lean mode is 0.06s. Alex Ziskind Thanks for letting me use the M5Us!

🤯 GLM-5.3-Flash 4bit on two M5 Ultras (tp=2) with TensorFold and 4x TB5 cables bonded. 🚀 114 tks decode on Prose single stream, and ~1600tks prefil. C=8 decode up to 200tps. There is headroom here as well, but this is where we landed today. A new-session TTFT with our prefix-cache lean mode is 0.06s. Alex Ziskind Thanks for letting me use the M5Us!

Ash Hart

26,047 görüntüleme • 5 gün önce

Apple Silicon + Gemma 4 fans: this is for you. Pico AI Server now supports continuous batching with MLX-Swift. 43 tok/s on 1 stream. 26 tok/s per stream on 2 concurrent streams. That’s 52 tok/s total. a 21% throughput gain on a six-year-old MacBook Pro M1 Max!

Apple Silicon + Gemma 4 fans: this is for you. Pico AI Server now supports continuous batching with MLX-Swift. 43 tok/s on 1 stream. 26 tok/s per stream on 2 concurrent streams. That’s 52 tok/s total. a 21% throughput gain on a six-year-old MacBook Pro M1 Max!

Ronald Mannak

53,392 görüntüleme • 5 ay önce

Loading DeepSeek v4 Flash just reminded me how good and fast this model is and how well it runs on DGX Sparks. GLM-5.2 is really awesome, but if you want fast and concurrent inference, DS4F is still a top choice IMO. Created by DS4F in single shot The coffee guy

Loading DeepSeek v4 Flash just reminded me how good and fast this model is and how well it runs on DGX Sparks. GLM-5.2 is really awesome, but if you want fast and concurrent inference, DS4F is still a top choice IMO. Created by DS4F in single shot The coffee guy

Mia

40,899 görüntüleme • 2 ay önce

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.

stevibe

403,356 görüntüleme • 5 ay önce

Qwen3.6 27B landed yesterday, so I ran it on 4 setups side-by-side to see how they stack up: 🔴 RTX 4090 — 45.59 tok/s, TTFT 525ms 🟢 RTX 5090 — 51.83 tok/s, TTFT 752ms ⚫️ M2 Ultra — 22.30 tok/s, TTFT 216ms 🟣 DGX Spark — 11.08 tok/s, TTFT 319ms This is a standard test: no tuning, just the out-of-the-box experience. For the NVIDIA cards I used llama.cpp with Unsloth's UD-Q4_K_XL quant. For the M2 Ultra I used MLX with Unsloth's UD-MLX-4bit quant, since MLX is the native path on Apple Silicon. Please consider this as the baseline, you can definitely squeeze more out of every one of these with fine-tuned settings.

Qwen3.6 27B landed yesterday, so I ran it on 4 setups side-by-side to see how they stack up: 🔴 RTX 4090 — 45.59 tok/s, TTFT 525ms 🟢 RTX 5090 — 51.83 tok/s, TTFT 752ms ⚫️ M2 Ultra — 22.30 tok/s, TTFT 216ms 🟣 DGX Spark — 11.08 tok/s, TTFT 319ms This is a standard test: no tuning, just the out-of-the-box experience. For the NVIDIA cards I used llama.cpp with Unsloth's UD-Q4_K_XL quant. For the M2 Ultra I used MLX with Unsloth's UD-MLX-4bit quant, since MLX is the native path on Apple Silicon. Please consider this as the baseline, you can definitely squeeze more out of every one of these with fine-tuned settings.

stevibe

104,692 görüntüleme • 5 ay önce

GLM-5.3 weights from Z.ai are live, with SGLang powering day-0 serving support! GLM-5.3 inherits every optimization and feature we battle-tested for GLM-5.2 over the past months. On real-world multi-turn agentic workloads, we measured 537.6 tok/s/user on NVFP4 and 413 tok/s/user on FP8, at BS=1 with TP8 on 8x B300. It's fast, efficient, and production-ready today on NVIDIA AI Blackwell and Hopper, and AI at AMD MI300X/325X/355X. We believe this is a big step forward for GLM-Series in agentic tasks, with a path toward mythos-class cyber capability. Can't wait to see what people build with GLM-5.3 and SGLang🚀 Cookbook👇

GLM-5.3 weights from Z.ai are live, with SGLang powering day-0 serving support! GLM-5.3 inherits every optimization and feature we battle-tested for GLM-5.2 over the past months. On real-world multi-turn agentic workloads, we measured 537.6 tok/s/user on NVFP4 and 413 tok/s/user on FP8, at BS=1 with TP8 on 8x B300. It's fast, efficient, and production-ready today on NVIDIA AI Blackwell and Hopper, and AI at AMD MI300X/325X/355X. We believe this is a big step forward for GLM-Series in agentic tasks, with a path toward mythos-class cyber capability. Can't wait to see what people build with GLM-5.3 and SGLang🚀 Cookbook👇

SGLang

99,455 görüntüleme • 1 ay önce

GLM 5.3 Flash running locally makes a Minecraft black hole mod We ran GLM 5.3 Flash Q4 locally on a rented 4x RTX PRO 6000 box and asked it to make a lightning gun mod for the real game using the Fabric API Output: -7.6M tokens -Time: ~9 hours -Decode: ~96 tok/s The mod adds a black hole rifle, when shot it spawns a black hole that pulls in blocks, with light rings that shrink all the way into the singularity, then collapses into an explosion crater that wipes out several chunks Run GLM 5.3 Flash locally in

GLM 5.3 Flash running locally makes a Minecraft black hole mod We ran GLM 5.3 Flash Q4 locally on a rented 4x RTX PRO 6000 box and asked it to make a lightning gun mod for the real game using the Fabric API Output: -7.6M tokens -Time: ~9 hours -Decode: ~96 tok/s The mod adds a black hole rifle, when shot it spawns a black hole that pulls in blocks, with light rings that shrink all the way into the singularity, then collapses into an explosion crater that wipes out several chunks Run GLM 5.3 Flash locally in

atomic.chat

32,515 görüntüleme • 1 ay önce

🚨 We finally cracked it. GLM-5.3, the FULL model (not the Flash), running on 4 DGX Sparks 🔥 ⚡️ 35 tok/s on real coding, peaking near 40 when it's hot 📊 Full C1 to C6 benchmark done, same speeds we were hitting on 5.2 🧠 200K context right now, tuning it higher from here 🎯 And this is only MTP4... we haven't even added DFlash yet I quantized it myself and I'm uploading the repo RIGHT NOW so everyone can go tune it with me 👀 Be tuned, it drops any minute. Let's get it 🚀

🚨 We finally cracked it. GLM-5.3, the FULL model (not the Flash), running on 4 DGX Sparks 🔥 ⚡️ 35 tok/s on real coding, peaking near 40 when it's hot 📊 Full C1 to C6 benchmark done, same speeds we were hitting on 5.2 🧠 200K context right now, tuning it higher from here 🎯 And this is only MTP4... we haven't even added DFlash yet I quantized it myself and I'm uploading the repo RIGHT NOW so everyone can go tune it with me 👀 Be tuned, it drops any minute. Let's get it 🚀

Tech2Wild

13,377 görüntüleme • 1 ay önce

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.

stevibe

34,141 görüntüleme • 5 ay önce