antirez's banner

antirez

@antirez • 64,283 subscribers

Reproducible bugs are candies. I like programming too much for not liking automatic programming.

Shorts

Iterating... Evals take time and are boring: but are a fundamental validation step of sane LLM inference. Let's try to make them as easy and fun to run as possible.

Iterating... Evals take time and are boring: but are a fundamental validation step of sane LLM inference. Let's try to make them as easy and fun to run as possible.

32,634 Aufrufe

First steps for a specialized DeepSeek v4 Flash inference engine focused on inference quality / stability at different quantizations, with networked API that is batching capable. This is the 2 bit quants model running on my M3 Max 128GB.

First steps for a specialized DeepSeek v4 Flash inference engine focused on inference quality / stability at different quantizations, with networked API that is batching capable. This is the 2 bit quants model running on my M3 Max 128GB.

14,155 Aufrufe

Videos

Anya Rossi

sweetdream.ai

SweetDream.ai•Sponsored•Livecam

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

Exclusive private shows

1.2k viewers online

Private Show

Join now for exclusive access

Free preview available • Premium content

DeepSeek v4 PRO running via SSD streaming on my 128GB MacBook m5 max. 1.6 trillion parameters.

DeepSeek v4 PRO running via SSD streaming on my 128GB MacBook m5 max. 1.6 trillion parameters.

258,651 Aufrufe • vor 2 Tagen

I didn't expect DeepSeek v4 PRO (not Flash) to run well on the Mac Studio M3 Ultra with 512GB of RAM. This is 2 bit quantized with the same DwarfStar recipe used for Flash. 433GB GGUF file. 130 t/s prefill, 13 t/s generation. Prefill in the video is low because small prompt.

I didn't expect DeepSeek v4 PRO (not Flash) to run well on the Mac Studio M3 Ultra with 512GB of RAM. This is 2 bit quantized with the same DwarfStar recipe used for Flash. 433GB GGUF file. 130 t/s prefill, 13 t/s generation. Prefill in the video is low because small prompt.

168,072 Aufrufe • vor 20 Tagen

DS4 running on DGX Spark (GB10 / CUDA), private branch for now. 12 tokens/sec, the memory bandwidth is limited in this system, at 270GB/sec. But prefill is ways more alighed to M3 Max at ~200 t/s. I'll release when more mature, but it is almost sure that it will get merged.

DS4 running on DGX Spark (GB10 / CUDA), private branch for now. 12 tokens/sec, the memory bandwidth is limited in this system, at 270GB/sec. But prefill is ways more alighed to M3 Max at ~200 t/s. I'll release when more mature, but it is almost sure that it will get merged.

83,314 Aufrufe • vor 27 Tagen

You didn't see this coming in ds4-agent, did you?

You didn't see this coming in ds4-agent, did you?

38,820 Aufrufe • vor 13 Tagen

For the DGX Spark owners. This is what you get with DS4 in your hardware. I want to post this to show how with fast prefill and not very fast generation, the system remains absolutely fine to use.

For the DGX Spark owners. This is what you get with DS4 in your hardware. I want to post this to show how with fast prefill and not very fast generation, the system remains absolutely fine to use.

38,217 Aufrufe • vor 14 Tagen

The first M5 max arrived! Many many thanks to our sponsors ⿻ Audrey Tang 唐鳳 and Niels Gron, the next will arrive on Monday.

The first M5 max arrived! Many many thanks to our sponsors ⿻ Audrey Tang 唐鳳 and Niels Gron, the next will arrive on Monday.

27,710 Aufrufe • vor 15 Tagen

In case you have doubts about the q2 quants inference of DS4 (I noticed many don't trust the README claims), here is it analyzing Picol source code using the pi agent.

In case you have doubts about the q2 quants inference of DS4 (I noticed many don't trust the README claims), here is it analyzing Picol source code using the pi agent.

44,074 Aufrufe • vor 29 Tagen

Updating you on ds4-agent

Updating you on ds4-agent

24,689 Aufrufe • vor 19 Tagen

Much better at 17 tokens/sec in M3 Max, fusing a few operations. 128gb m3 max.

Much better at 17 tokens/sec in M3 Max, fusing a few operations. 128gb m3 max.

24,817 Aufrufe • vor 1 Monat

Now we are talking. 2x audio length speed for the 1.7B Qwen3-ASR large (1.7b) transcription, on my very modest Hetzner 13th Gen i5 server. The pause you see initially is the encoding stage. -S 0 means: no segmentation, one offline pass.

Now we are talking. 2x audio length speed for the 1.7B Qwen3-ASR large (1.7b) transcription, on my very modest Hetzner 13th Gen i5 server. The pause you see initially is the encoding stage. -S 0 means: no segmentation, one offline pass.

15,987 Aufrufe • vor 3 Monaten

Keine weiteren Inhalte verfügbar