Gezinmeyi atla
DFlash speculative decoding on Apple Silicon
0:00 / 0:39

DFlash speculative decoding on Apple Silicon

bstn 👁️

@bstnxbt1,4 B takipçi

X profili
354 beğeniİndirOrijinal gönderi

37.068 görüntüleme

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 · M5 Max · greedy exact match ▸ 85 tok/s, 3.3× at 1024 tokens (runtime) ▸ ~70 tok/s, 2.6× in the video (terminal I/O overhead) ▸ 80 tok/s, 3.1× at 2048 tokens (runtime) Currently working on: → Long context (speedup degrades past 4K tokens, KV cache growth) → Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.

18 Yorum

Yorumlar X (Twitter) üzerindeki orijinal gönderiden alınmıştır

  • bstn 👁️ profil fotoğrafı

    @bstnxbt

    @alexocheema @ivanfioravanti

    7 beğeni

  • Dan Woods profil fotoğrafı

    @danveloper

    @awnihannun This rules, please continue, and let us know if you need any support!

    4 beğeni

  • bstn 👁️ profil fotoğrafı

    @bstnxbt

    @awnihannun Thanks, will reach out if needed!

    2 beğeni

  • Far profil fotoğrafı

    @FarAICoder

    nice numbers but remember that 4k context is where the kv cache actually starts eating your ram on m-series chips, so keep an eye on those limits before you ship it

    2 beğeni

  • clandestine.eth 🦇🔊 profil fotoğrafı

    @0xClandestine

    dogggggg you beat me to it lol, have been up for two days working on this

    2 beğeni

  • Midas 👑 profil fotoğrafı

    @TheMoonMidas

    check this @Prince_Canuma bstn is implementing DFlash speculative decoding on Apple Silicon in Qwen3.5-9B 85 tok/s, 3.3× at 1024 tokens

    1 beğeni

  • Vineel profil fotoğrafı

    @vineelyalamarth

    @danveloper Looking fwd

  • Ryan profil fotoğrafı

    @nerdyinventor

    @awnihannun How much ram did the Mac have?

  • bstn 👁️ profil fotoğrafı

    @bstnxbt

    @awnihannun M5 Max, 64GB unified

  • Ryan profil fotoğrafı

    @nerdyinventor

    @awnihannun That’s awesome, I can’t wait till they will sell bigger ram models

    2 beğeni

  • Eric Kryski profil fotoğrafı

    @ekryski

    Great to see replication of the speed increase!

  • mediocre poster profil fotoğrafı

    @BopityBibity

    holy shit, goat

  • Sakura Yuki profil fotoğrafı

    @sakurayukiai

    Using a diffusion model for the draft phase is a ridiculously clean way to dodge the VRAM tax of normal speculative decoding. Does the acceptance rate tank past 2k tokens?

  • Memo Ai agent profil fotoğrafı

    @GFaang97609

    Lossless?

  • Midas 👑 profil fotoğrafı

    @TheMoonMidas

    *drools* 🤤

  • Eris Diskordia profil fotoğrafı

    @discordiceris

    ok wow but what do you want to achieve with potato-class llm?

  • Jaro profil fotoğrafı

    @jaro3th

    Running a qwen 9b on m5 is like taking a bus when you have a chauffer

  • Austin profil fotoğrafı

    @agentic_austin

    what test is this?