Skip navigation
DiffusionGemma can now run at 2000+ tokens/sec! ⚡
0:00 / 0:24

DiffusionGemma can now run at 2000+ tokens/sec! ⚡

Unsloth AI

@UnslothAI105K followers

Visit on X
1,697 likesDownloadOriginal post

182,809 views

DiffusionGemma can now run at 2000+ tokens/sec! ⚡ We made local DiffusionGemma inference 1.8× faster. Run it on 18GB RAM via Unsloth Studio. GitHub: Guide:

35 Comments

Comments from the original post on X (Twitter)

  • Philipp Schmid's profile picture

    @_philschmid

    Wow!

    13 likes

  • Tery Emilson's profile picture

    @emilson_tery

    I think batch speed is the boring win compared with always-on local AI. A model you poke once can afford to be slow. A model running continuously in the background ... watching context, briefing you before things happen ... that's where 2000 tok/s is the line between a chatbot and an ambient assistant. Well done! (again!)

    9 likes

  • Maziyar PANAHI's profile picture

    @MaziyarPanahi

    2000+ tokens/sec! wow! you know, i would die for a diffusion model like this on my iOS! ps: of course with a unsloth fine-tuning recipe! 🤩

    9 likes

  • Apollo's profile picture

    @ApollonVisual

    isnt DiffusionGemma more prone to lower quality output though? Google’s own launch post says DiffusionGemma is optimized for speed and that its overall output quality is lower than standard Gemma 4. Google also says standard Gemma 4 remains the better choice for applications that need maximum quality.

    9 likes

  • Terp's profile picture

    @OnlyTerp

    ??? im getting 170 usable tps on my 5090 compared to 500+ through vllm what's the issue ?

    8 likes

  • Le TechLead🔰's profile picture

    @LeTechLead

    @danielhanchen we need to be able to serve it though, cli and chat doesn’t cut it.

    6 likes

  • Xaden Ryan's profile picture

    @XadenRyan

    @danielhanchen Does it do tool calling?

    6 likes

  • Ankit Prateek's profile picture

    @ankit_appy

    llama-server still doesn't support diffusion model. mlx does but token gen speed is horrible.

    5 likes

  • Tarrito.rocks's profile picture

    @tarritorocks

    Nice but I didn't find UD-Q4 model version in your repo

    5 likes

  • Dariton's profile picture

    @Dariton4000

    Does this work with CPU offloading though?

    5 likes

  • Ankit Prateek's profile picture

    @ankit_appy

    This is wild

    4 likes

  • Ankit Prateek's profile picture

    @ankit_appy

    I spent ~6 hours making this diffusion model work on my mac, and that gave me 10 tokens/s because there was no llama.cpp support lol

    4 likes

  • ibrand's profile picture

    @ibrand

    It needs to run comfortably on 16gb. Hardly anyone has 18

    4 likes

  • Piyush's profile picture

    @CatAstro_Piyush

    any quantized version available that will enable it run on T4?

    4 likes

  • AACeeert's profile picture

    @AACeeert

    An abliterated version of this will have malware scripts flying around the internet in milliseconds

    4 likes

  • Eric ⚡️ Building...'s profile picture

    @outsource_

    WOW

    3 likes

  • Vabbyshabby's profile picture

    @vabbyshabby

    2000 tok/s local is the actual answer to this morning's news. nobody export-controls a gguf on your own box. this is the lane.

    3 likes

  • Emircan ERKUL's profile picture

    @EmircanErkulX

    cant you fit that into 14gb so i could use with 16vram gpu

    3 likes

  • mr_r0b0t's profile picture

    @mr_r0b0t

    Cooking with white hot 🔥🔥🔥🔥

    2 likes

  • Anis🐬Al's profile picture

    @AnisAIb6

    My sister, this is truly exhilarating news! 🌟 Seeing DiffusionGemma achieve such breathtaking speeds—surpassing 2000 tokens per second—while remaining accessible on local hardware like 18GB RAM is a masterpiece of efficiency over sheer bulk. It’s not just about the technical milestones; it's about the democratization of intelligence. By bridging the gap between high-performance research and local accessibility, you are helping to put the pulse of innovation directly into our hands. This transition from massive cloud dependency to agile, local execution is where technology truly begins to serve humanity with grace and speed. Keep pushing these boundaries! ✨

    2 likes