DFlash speculative decoding on Apple Silicon
37.068 görüntüleme
18 Yorum
Yorumlar X (Twitter) üzerindeki orijinal gönderiden alınmıştır

@bstnxbt
@alexocheema @ivanfioravanti
7 beğeni

@danveloper
@awnihannun This rules, please continue, and let us know if you need any support!
4 beğeni

@bstnxbt
@awnihannun Thanks, will reach out if needed!
2 beğeni

@FarAICoder
nice numbers but remember that 4k context is where the kv cache actually starts eating your ram on m-series chips, so keep an eye on those limits before you ship it
2 beğeni

@0xClandestine
dogggggg you beat me to it lol, have been up for two days working on this
2 beğeni

@TheMoonMidas
check this @Prince_Canuma bstn is implementing DFlash speculative decoding on Apple Silicon in Qwen3.5-9B 85 tok/s, 3.3× at 1024 tokens
1 beğeni

@vineelyalamarth
@danveloper Looking fwd

@nerdyinventor
@awnihannun How much ram did the Mac have?

@bstnxbt
@awnihannun M5 Max, 64GB unified

@nerdyinventor
@awnihannun That’s awesome, I can’t wait till they will sell bigger ram models
2 beğeni

@ekryski
Great to see replication of the speed increase!

@BopityBibity
holy shit, goat

@sakurayukiai
Using a diffusion model for the draft phase is a ridiculously clean way to dodge the VRAM tax of normal speculative decoding. Does the acceptance rate tank past 2k tokens?

@GFaang97609
Lossless?

@TheMoonMidas
*drools* 🤤

@discordiceris
ok wow but what do you want to achieve with potato-class llm?

@jaro3th
Running a qwen 9b on m5 is like taking a bus when you have a chauffer

@agentic_austin
what test is this?