Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

I'm optimizing tensor parallel execution across two MacBook M5 Max systems using RDMA. This is how fast DeepSeek v4 Flash Exp Vision can run in MXFP4 (full precision) right now *without* DSpark, so this is the min speed you get in practice.

antirez's profile picture

antirez

80,094 subscribers

26,238 Aufrufe • vor 3 Tagen •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

GLM 5.2 flash Q2 (generated by myself with similar recipe of DS4 Flash of DwarfStar GGUF) running on an M5 Max with 128GB. The Q4 weights also work on tensor parallel inference in two M5 Max systems. Soon on GitHub. Still testing + CUDA + ROCm ...
0:39

GLM 5.2 flash Q2 (generated by myself with similar recipe of DS4 Flash of DwarfStar GGUF) running on an M5 Max with 128GB. The Q4 weights also work on tensor parallel inference in two M5 Max systems. Soon on GitHub. Still testing + CUDA + ROCm ...

antirez

28,103 Aufrufe • vor 8 Tagen

DeepSeek v4 Flash experimental with vision running (fast) on an m5 max computer and analyzing an image. The implementation for Metal / CUDA / ROCm is finished, just doing the last tests today before releasing it.
1:19

DeepSeek v4 Flash experimental with vision running (fast) on an m5 max computer and analyzing an image. The implementation for Metal / CUDA / ROCm is finished, just doing the last tests today before releasing it.

antirez

60,095 Aufrufe • vor 4 Tagen

DeepSeek v4 PRO running via SSD streaming on my 128GB MacBook m5 max. 1.6 trillion parameters.
4:15

DeepSeek v4 PRO running via SSD streaming on my 128GB MacBook m5 max. 1.6 trillion parameters.

antirez

275,713 Aufrufe • vor 3 Monaten

DwarfStar with vision support for GLM 5.3 running on a MacBook m5max computer. The vision support is *not* quantized for max precision for now.
0:51

DwarfStar with vision support for GLM 5.3 running on a MacBook m5max computer. The vision support is *not* quantized for max precision for now.

antirez

22,187 Aufrufe • vor 7 Tagen

Initial DeepSeek v4 and v4 Flash support added to anemll-flash-llama.cpp! 🚀 M5 MAX 128GB can run full DeepSeek-V4 with 1.6T params in original FP8/FP4 weights from SSD without requantization! One-click scripts to convert HF safetensors to dense + MoE sidecar (no GGUF needed) Inference and server examples Prefill and decoding benchmarks Branch and docs here: This is WIP and I plan to add cleanup and more testing this week. Feedback welcome!
0:21

Initial DeepSeek v4 and v4 Flash support added to anemll-flash-llama.cpp! 🚀 M5 MAX 128GB can run full DeepSeek-V4 with 1.6T params in original FP8/FP4 weights from SSD without requantization! One-click scripts to convert HF safetensors to dense + MoE sidecar (no GGUF needed) Inference and server examples Prefill and decoding benchmarks Branch and docs here: This is WIP and I plan to add cleanup and more testing this week. Feedback welcome!

Anemll

32,250 Aufrufe • vor 4 Monaten

I just saw AGI, what the heck. Deepseek-v4-flash-0731 invented vision for itself, this is so incredible. Result in the next post:
4:11

I just saw AGI, what the heck. Deepseek-v4-flash-0731 invented vision for itself, this is so incredible. Result in the next post:

0xSero

174,038 Aufrufe • vor 24 Tagen

That's a bit slow. Streaming directly the K3 official hugging face 1.6TB of weights in mxfp4 in an m5 max 128gb.
1:06

That's a bit slow. Streaming directly the K3 official hugging face 1.6TB of weights in mxfp4 in an m5 max 128gb.

antirez

294,029 Aufrufe • vor 1 Monat

I didn't expect DeepSeek v4 PRO (not Flash) to run well on the Mac Studio M3 Ultra with 512GB of RAM. This is 2 bit quantized with the same DwarfStar recipe used for Flash. 433GB GGUF file. 130 t/s prefill, 13 t/s generation. Prefill in the video is low because small prompt.
1:24

I didn't expect DeepSeek v4 PRO (not Flash) to run well on the Mac Studio M3 Ultra with 512GB of RAM. This is 2 bit quantized with the same DwarfStar recipe used for Flash. 433GB GGUF file. 130 t/s prefill, 13 t/s generation. Prefill in the video is low because small prompt.

antirez

171,478 Aufrufe • vor 3 Monaten

Crazy prefill speed of DwarfStar running DeepSeek v4 PRO in the DGX Station (private branch for now) using techniques that leverage the peculiar RAM + VRAM mixed setup of the Station. 3000 t/s but even more can be reached.
0:23

Crazy prefill speed of DwarfStar running DeepSeek v4 PRO in the DGX Station (private branch for now) using techniques that leverage the peculiar RAM + VRAM mixed setup of the Station. 3000 t/s but even more can be reached.

antirez

62,001 Aufrufe • vor 18 Tagen

We've got Opus 4.8 at home! Running DeepSeek V4 Flash Vision-Exp on 2 x DGX Spark with OMP. This is a mini-benchmark firmware analysis task I've been using to test out different models/harnesses. Opus 4.8 didn't find anything in this firmware that DeepSeek didn't. Thanks Mia
0:24

We've got Opus 4.8 at home! Running DeepSeek V4 Flash Vision-Exp on 2 x DGX Spark with OMP. This is a mini-benchmark firmware analysis task I've been using to test out different models/harnesses. Opus 4.8 didn't find anything in this firmware that DeepSeek didn't. Thanks Mia

Matt Brown

25,139 Aufrufe • vor 4 Tagen

SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉 DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales. We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec. At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads. With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300. DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks DeepSeek for open-sourcing! Blog with full technical details and commands to run below 👇
0:33

SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉 DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales. We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec. At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads. With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300. DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks DeepSeek for open-sourcing! Blog with full technical details and commands to run below 👇

LMSYS Org

170,203 Aufrufe • vor 2 Monaten

DeepSeek V4 Flash 0731 in Hermes Agent and one prompt, took 32 minutes and cost 0.07$, this model is so cheap to the point where 2 dollars can last you a full day.
1:03

DeepSeek V4 Flash 0731 in Hermes Agent and one prompt, took 32 minutes and cost 0.07$, this model is so cheap to the point where 2 dollars can last you a full day.

Elshayib

689,517 Aufrufe • vor 1 Monat

If you need AI to do a search for you in the real world, ds4-agent is basically SOTA, because it can access the web sites without any limitations given that it uses your local Chrome browser (no, not in headless mode, that's the trick...), and DeepSeek v4 is great at search.
1:29

If you need AI to do a search for you in the real world, ds4-agent is basically SOTA, because it can access the web sites without any limitations given that it uses your local Chrome browser (no, not in headless mode, that's the trick...), and DeepSeek v4 is great at search.

antirez

149,921 Aufrufe • vor 2 Monaten

DeepSeek v4 PRO running in DwarfStar on a DGX Station. Can go faster, already peaks 50 t/s without DSpark: dynamic VRAM/RAM allocation of experts based on past statistics, 4k t/s prefill on big chunks. VRAM/RAM layer-major streaming. Mixed RAM/VRAM inference requires care.
0:27

DeepSeek v4 PRO running in DwarfStar on a DGX Station. Can go faster, already peaks 50 t/s without DSpark: dynamic VRAM/RAM allocation of experts based on past statistics, 4k t/s prefill on big chunks. VRAM/RAM layer-major streaming. Mixed RAM/VRAM inference requires care.

antirez

26,246 Aufrufe • vor 18 Tagen

Loading DeepSeek v4 Flash just reminded me how good and fast this model is and how well it runs on DGX Sparks. GLM-5.2 is really awesome, but if you want fast and concurrent inference, DS4F is still a top choice IMO. Created by DS4F in single shot The coffee guy
0:20

Loading DeepSeek v4 Flash just reminded me how good and fast this model is and how well it runs on DGX Sparks. GLM-5.2 is really awesome, but if you want fast and concurrent inference, DS4F is still a top choice IMO. Created by DS4F in single shot The coffee guy

Mia

39,686 Aufrufe • vor 1 Monat

For the DGX Spark owners. This is what you get with DS4 in your hardware. I want to post this to show how with fast prefill and not very fast generation, the system remains absolutely fine to use.
1:54

For the DGX Spark owners. This is what you get with DS4 in your hardware. I want to post this to show how with fast prefill and not very fast generation, the system remains absolutely fine to use.

antirez

40,085 Aufrufe • vor 3 Monaten

Timelapse #177 (14 hrs) - Picked up an RTX Pro 6000 Blackwell and wired in another 1000W PSU - running two of them now - DeepSeek v4 Flash running locally on both GPUs - Distilling DeepSeek v4 Flash down into Nanbeige (still cooking) - Kernel bench runs for DeepSeek v4 Flash and Qwen 3.8 Max - Got it into chat but couldn't reliably get it into the OMP harness yet - Private contract work - days like this are stuff I actually want to build. and when it's the high-intensity grind, i still have to show up and get it done
0:30

Timelapse #177 (14 hrs) - Picked up an RTX Pro 6000 Blackwell and wired in another 1000W PSU - running two of them now - DeepSeek v4 Flash running locally on both GPUs - Distilling DeepSeek v4 Flash down into Nanbeige (still cooking) - Kernel bench runs for DeepSeek v4 Flash and Qwen 3.8 Max - Got it into chat but couldn't reliably get it into the OMP harness yet - Private contract work - days like this are stuff I actually want to build. and when it's the high-intensity grind, i still have to show up and get it done

Elliot Arledge

17,401 Aufrufe • vor 1 Monat

DGX Station: this is just the start of what it is possible to do (and why I claim DwarfStar could be *the* Station inference engine). DeepSeek v4 PRO Q2 with routed experts split among VRAM / RAM with kernels optimized for this peculiar setup. 45 t/s but can go faster.
0:53

DGX Station: this is just the start of what it is possible to do (and why I claim DwarfStar could be *the* Station inference engine). DeepSeek v4 PRO Q2 with routed experts split among VRAM / RAM with kernels optimized for this peculiar setup. 45 t/s but can go faster.

antirez

41,545 Aufrufe • vor 19 Tagen

Where there's a will, there's a way 😎 DwarfStar DeepSeek V4 Flash 0731 mxfp4 on M3 Ultra single request no MPT/DSpark ~41 toks/s decode!!! 🚀 40 tps barrier broken! Same logits, 100% same result: this is The Mandatory Rule for any agents improving kernels that I think everyone should follow. Speed without mathematical precision is useless. Thanks 20% Opus 5 and 80% Kimi.ai K3 (what a model!) And just looking at the thinking trace of K3 you can learn tons of stuff! Here usually I have a separate chat with K3 and when I see something I want to dig deeper or ask about from the main thinking trace I do and in some cases I stop the experiment and pivot immediately to something else. Now let's see if I can apply learned lessons to the M5 kernel! 🚀 Branch for any braves soul willing to test it:
0:36

Where there's a will, there's a way 😎 DwarfStar DeepSeek V4 Flash 0731 mxfp4 on M3 Ultra single request no MPT/DSpark ~41 toks/s decode!!! 🚀 40 tps barrier broken! Same logits, 100% same result: this is The Mandatory Rule for any agents improving kernels that I think everyone should follow. Speed without mathematical precision is useless. Thanks 20% Opus 5 and 80% Kimi.ai K3 (what a model!) And just looking at the thinking trace of K3 you can learn tons of stuff! Here usually I have a separate chat with K3 and when I see something I want to dig deeper or ask about from the main thinking trace I do and in some cases I stop the experiment and pivot immediately to something else. Now let's see if I can apply learned lessons to the M5 kernel! 🚀 Branch for any braves soul willing to test it:

Ivan Fioravanti ᯅ

18,381 Aufrufe • vor 29 Tagen

anything under deepseek v4 flash basically doesn't exist to me. with <200GB of vram, i'm getting ~800K context @ 200 tokens per second. It's not quite Cerebras fast, but damn. It is a joy to use, and its basically somewhere in the opus 4.7/4.8 range. It's incredibly good at browser control too. this model is a beast. w/ qwen 3.8 27B being multimodal though, it might just be worth switching to, especially if you want to run a lot of parallel agents will 300toks/s be possible? Pretty addicting let me tell you. Imagine a computer use model this fast. watch below, this is basically human speed.
0:28

anything under deepseek v4 flash basically doesn't exist to me. with <200GB of vram, i'm getting ~800K context @ 200 tokens per second. It's not quite Cerebras fast, but damn. It is a joy to use, and its basically somewhere in the opus 4.7/4.8 range. It's incredibly good at browser control too. this model is a beast. w/ qwen 3.8 27B being multimodal though, it might just be worth switching to, especially if you want to run a lot of parallel agents will 300toks/s be possible? Pretty addicting let me tell you. Imagine a computer use model this fast. watch below, this is basically human speed.

am.will

72,729 Aufrufe • vor 25 Tagen