Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Apple’s “LLM in a Flash” is definitely worth checking out. Going to 2-bit for the shared-expert MLP means disk I/O is no longer dominant. 14–15 tok/s from SSD is still wild for a ~400B MoE model streamed from storage. Qwen3.5-397B-A17B Credit: Dan Woods

anemll's profile picture

Anemll

3,955 subscribers

35,664 Aufrufe • vor 4 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

WIP: First attempt to speed up prefill for Flash-MoE. Original repo did token-by-token without streamed experts. Added: Batched linear attention + batched full attention (Flash Attention style) with custom Metal kernels. Without experts: 6.2x faster prefill (11 -> 68 tok/s) With experts at full-attn layers only: 1.9x faster (11 -> 20.5 tok/s) — same output quality Qwen3.5-397B, 4-bit, 209GB, M5 Max 128GB 1/3
0:57

WIP: First attempt to speed up prefill for Flash-MoE. Original repo did token-by-token without streamed experts. Added: Batched linear attention + batched full attention (Flash Attention style) with custom Metal kernels. Without experts: 6.2x faster prefill (11 -> 68 tok/s) With experts at full-attn layers only: 1.9x faster (11 -> 20.5 tok/s) — same output quality Qwen3.5-397B, 4-bit, 209GB, M5 Max 128GB 1/3

Anemll

19,572 Aufrufe • vor 4 Monaten

Running 400B model on iPhone! 0.6 t/s Credit Dan Woods @alexintosh Daniel Isaac Anemll
3:51

Running 400B model on iPhone! 0.6 t/s Credit Dan Woods @alexintosh Daniel Isaac Anemll

Anemll

294,001 Aufrufe • vor 4 Monaten

~ 6.5 - 6.7 t/s for GLM 5.1 on M5 Max 128GB Added “Dense” model export, now model load is only 5s ! Experts are streaming from SSD, so we do not pre-load it. Added direct SSD->Slot memory path, removed prefetch... Many dead end experiments. See Export a “dense-only GGUF” and “Fast path ” in tools/flashmob-sidecar/README.md WIP branch for Flash-MoE-SSD
1:28

~ 6.5 - 6.7 t/s for GLM 5.1 on M5 Max 128GB Added “Dense” model export, now model load is only 5s ! Experts are streaming from SSD, so we do not pre-load it. Added direct SSD->Slot memory path, removed prefetch... Many dead end experiments. See Export a “dense-only GGUF” and “Fast path ” in tools/flashmob-sidecar/README.md WIP branch for Flash-MoE-SSD

Anemll

29,421 Aufrufe • vor 4 Monaten

🥊 Time for a new fight Qwen3.5 397B A17B vs Qwen3.6 27B 🌸 "Cherry Blossom" (↓ prompt below) Using OpenRouter for 397B Running 27B locally on a single RTX 5080 Wow! 🤯 I'm sure this one is going to divide opinions. 397B feels cinematic with the flare and shadows it added, while 27B is just beautiful; it has a completely different color palette and the leaves are falling from the branches as asked. It's really tough to decide which model is the best. I was about to call it a tie, but then I remembered that 397B needs 180GB VRAM to run locally 😅 Don't get me wrong the model is amazing, but right now the VRAM it requires doesn't feel justified. It's simply not 10× better than 27B. We'll have to dig deeper with more prompts to be sure, let's see what each one has in store for us. What do you think?
0:30

🥊 Time for a new fight Qwen3.5 397B A17B vs Qwen3.6 27B 🌸 "Cherry Blossom" (↓ prompt below) Using OpenRouter for 397B Running 27B locally on a single RTX 5080 Wow! 🤯 I'm sure this one is going to divide opinions. 397B feels cinematic with the flare and shadows it added, while 27B is just beautiful; it has a completely different color palette and the leaves are falling from the branches as asked. It's really tough to decide which model is the best. I was about to call it a tie, but then I remembered that 397B needs 180GB VRAM to run locally 😅 Don't get me wrong the model is amazing, but right now the VRAM it requires doesn't feel justified. It's simply not 10× better than 27B. We'll have to dig deeper with more prompts to be sure, let's see what each one has in store for us. What do you think?

left curve dev

21,224 Aufrufe • vor 3 Monaten

Initial DeepSeek v4 and v4 Flash support added to anemll-flash-llama.cpp! 🚀 M5 MAX 128GB can run full DeepSeek-V4 with 1.6T params in original FP8/FP4 weights from SSD without requantization! One-click scripts to convert HF safetensors to dense + MoE sidecar (no GGUF needed) Inference and server examples Prefill and decoding benchmarks Branch and docs here: This is WIP and I plan to add cleanup and more testing this week. Feedback welcome!
0:21

Initial DeepSeek v4 and v4 Flash support added to anemll-flash-llama.cpp! 🚀 M5 MAX 128GB can run full DeepSeek-V4 with 1.6T params in original FP8/FP4 weights from SSD without requantization! One-click scripts to convert HF safetensors to dense + MoE sidecar (no GGUF needed) Inference and server examples Prefill and decoding benchmarks Branch and docs here: This is WIP and I plan to add cleanup and more testing this week. Feedback welcome!

Anemll

32,250 Aufrufe • vor 3 Monaten

Chud The Builder tires to make people feel sorry for him. No one is buying it. My guy is still chimpin out from a Jail cell. Wild.
1:39

Chud The Builder tires to make people feel sorry for him. No one is buying it. My guy is still chimpin out from a Jail cell. Wild.

Rodney

202,756 Aufrufe • vor 2 Monaten

Doing some testing of SS1 reading PS1 data directly from a disk without dumping the data first. It’s much better than I expected. That being said, copying games to the SD or SSD is still the optimal strategy for performance. It’s a one-time process and better for random access.
1:06

Doing some testing of SS1 reading PS1 data directly from a disk without dumping the data first. It’s much better than I expected. That being said, copying games to the SD or SSD is still the optimal strategy for performance. It’s a one-time process and better for random access.

Taki Udon

110,104 Aufrufe • vor 1 Jahr

AI animation is going to enable people to tell stories from all over the world. Here’s one from my neck of the woods! Made in partnership with Invideo . Look out for a post about how I made it with tomorrow 👀
3:16

AI animation is going to enable people to tell stories from all over the world. Here’s one from my neck of the woods! Made in partnership with Invideo . Look out for a post about how I made it with tomorrow 👀

Uncanny Harry AI

14,103 Aufrufe • vor 4 Monaten

This means that Vivek is no longer legally a citizen, and is ineligible to run for any kind of office. He should be immediately removed from the race in Ohio. He was an anchor-baby, born in America to non-citizens (his father is still not a citizen, after 40+ years in America).
0:43

This means that Vivek is no longer legally a citizen, and is ineligible to run for any kind of office. He should be immediately removed from the race in Ohio. He was an anchor-baby, born in America to non-citizens (his father is still not a citizen, after 40+ years in America).

Sólionath

556,368 Aufrufe • vor 1 Jahr

For My Dad. He Passed away last night around 11pm. He was going to be 82 this year. He suffered from Guillain Barre Syndrome and his body has been slowly deteriorating for the past 15 years or so. My Dad is in a better place now. He is no longer in pain and he is no longer restricted to what he is able to do. Love You Pop. See you when I see you.
3:20

For My Dad. He Passed away last night around 11pm. He was going to be 82 this year. He suffered from Guillain Barre Syndrome and his body has been slowly deteriorating for the past 15 years or so. My Dad is in a better place now. He is no longer in pain and he is no longer restricted to what he is able to do. Love You Pop. See you when I see you.

Grumpy Old Marine

173,209 Aufrufe • vor 6 Monaten

FormaL, Nadeshot react to M8's fans calling out the bomb in a 1v1 👀😬 Nadeshot : "If I hear the crowd like that I'm checking the bomb or have no clue what is going on in the map" FormaL : "I'll take credit for that Ghosty 1v1 I helped him, but the crowd dude idk man" 🗣️🫣
0:47

FormaL, Nadeshot react to M8's fans calling out the bomb in a 1v1 👀😬 Nadeshot : "If I hear the crowd like that I'm checking the bomb or have no clue what is going on in the map" FormaL : "I'll take credit for that Ghosty 1v1 I helped him, but the crowd dude idk man" 🗣️🫣

CDL Hater Central

129,576 Aufrufe • vor 1 Monat

Added few tweaks to alexintosh iOS port. ~7 t/s on iPhone 17pro Also tested on Neo, looks like A19pro SSD is way faster! The same prompt…
1:12

Added few tweaks to alexintosh iOS port. ~7 t/s on iPhone 17pro Also tested on Neo, looks like A19pro SSD is way faster! The same prompt…

Anemll

40,461 Aufrufe • vor 4 Monaten

“Nillion aims to provide privacy to the people who have none.” John Woods CTO of Nillion joins The Defiant to unpack the “blind computer”—a federated network for private storage and compute—with NIL 2.0 weaving tokenomics directly into the tech stack. From private LLM inference to genomics, this is privacy-by-design in action. Listen now👇
30:30

“Nillion aims to provide privacy to the people who have none.” John Woods CTO of Nillion joins The Defiant to unpack the “blind computer”—a federated network for private storage and compute—with NIL 2.0 weaving tokenomics directly into the tech stack. From private LLM inference to genomics, this is privacy-by-design in action. Listen now👇

The Defiant

15,601 Aufrufe • vor 8 Monaten

"I have every confidence in the world that the President is going to meet Putin from a position of strength, that he is going to look out for Europe and Ukrainian needs to end this war honorably." Lindsey Graham looking like he is being forced to record a hostage video is a great omen for peace.
0:22

"I have every confidence in the world that the President is going to meet Putin from a position of strength, that he is going to look out for Europe and Ukrainian needs to end this war honorably." Lindsey Graham looking like he is being forced to record a hostage video is a great omen for peace.

Margarita Simonyan

122,181 Aufrufe • vor 1 Jahr

STEPHANOPOULOS: He promised it would be fixed for under $2 million. It's already cost more than $15 million with a no bid contract. The pool is going to be drained and you're putting out solicitations for new bids. So the pool is not fixed right now, is it? BURGUM: Well it's fixed in the sense that it's no longer leaking 45,000 gallons a day
1:12

STEPHANOPOULOS: He promised it would be fixed for under $2 million. It's already cost more than $15 million with a no bid contract. The pool is going to be drained and you're putting out solicitations for new bids. So the pool is not fixed right now, is it? BURGUM: Well it's fixed in the sense that it's no longer leaking 45,000 gallons a day

Aaron Rupar

439,578 Aufrufe • vor 1 Monat

Fine: When you're $40 trillion in debt and the things that you're paying for are being borrowed from your children and grandchildren, the question is no longer whether something is a good idea. The question is it worth driving us further into debt to do it.
1:11

Fine: When you're $40 trillion in debt and the things that you're paying for are being borrowed from your children and grandchildren, the question is no longer whether something is a good idea. The question is it worth driving us further into debt to do it.

Acyn

35,900 Aufrufe • vor 25 Tagen

enjoyed this little clip from kapital. i found their boro jackets (the main item feat in this clip) a bit hard to wear, but their ring coat is super easy if you get another item from the line (eg scarf or boots). very worth checking out if you're open to offbeat japanese workwear
3:45

enjoyed this little clip from kapital. i found their boro jackets (the main item feat in this clip) a bit hard to wear, but their ring coat is super easy if you get another item from the line (eg scarf or boots). very worth checking out if you're open to offbeat japanese workwear

derek guy

180,809 Aufrufe • vor 2 Jahren

All The Right Movies I love Ferris . . . But I definitely had a thing for Mia Sara from Ferris Bueller's Day Off back in the day. Most definitely, my choice is Sloan Peterson No offense to Ferris.
0:28

All The Right Movies I love Ferris . . . But I definitely had a thing for Mia Sara from Ferris Bueller's Day Off back in the day. Most definitely, my choice is Sloan Peterson No offense to Ferris.

Nostalgia . . .

63,138 Aufrufe • vor 2 Jahren