Skip navigation
GOOGLE JUST MADE EVERY CHATBOT FEEL SLOW
0:00 / 0:31

GOOGLE JUST MADE EVERY CHATBOT FEEL SLOW

151 likesDownloadOriginal post

27,448 views

GOOGLE JUST MADE EVERY CHATBOT FEEL SLOW Diffusion Gemma 26b doesn’t predict word by word, it generates 256 tokens in parallel using bi-directional attention, like stable diffusion but for language it’s MoE so only 3.8B params activate during inference, fits on a single RTX 4090 with 18GB VRAM and you can run it right now via llama.cpp, the same engine powering every tool in my local llm playbook the whole ecosystem exists for moments like this ↓

26 Comments

Comments from the original post on X (Twitter)

  • Gabu's profile picture

    @gabu3d_pl

    >and you can run it right now via llama.cpp Show us llama-server then :P Still waiting for full support. But you can run it full on vllm now:

    1 like

  • Way's profile picture

    @AusWay69

    There’s more to a chatbot than speed. Faster slop is still slop.

  • Rick's profile picture

    @andthathappens

    Will this run on a 64GB Macbook Pro Max M3

  • Oliver's profile picture

    @olivertaste

    256 tokens in parallel is the part to watch; latency wins only matter if the output stays steerable.

  • Paul's profile picture

    @pbastowski

    Yes, it’s faster, but it’s also dumber than the base 26b a4b qat right now. Let’s hope they can close this gap.

  • starmex's profile picture

    @starmexxx

    great post, thanks!

  • Alfero Chingono's profile picture

    @achingono

    Generating 256 tokens in parallel really shifts the latency game for apps. The MoE aspect helps with the practical cost of running this on consumer hardware too.

  • Kanika's profile picture

    @KanikaBK

    When a model this capable fits on consumer hardware, adoption accelerates fast.

  • Petru | Steam Vibe's profile picture

    @SteamVibeLtd

    The parallel token generation shifts the bottleneck from inference speed to VRAM bandwidth, making local deployment more viable for complex workflows.

  • Xoch's profile picture

    @xxxoch_

    People underestimate how much faster responses change the entire experience of using AI

  • rari's profile picture

    @0xwhrrari

    chatbots waiting word by word may feel old soon

  • marfin's profile picture

    @marfinxx

    I like how the Gemma model line is evolving

  • Nekt0's profile picture

    @Nekt_0

    Top

  • karla ❀'s profile picture

    @karlarboledas

    generates 256 tokens in parallel? It's awesome

  • Jay | The Midwest Dad |'s profile picture

    @JC_Midwest

    Only valuable if the quality is actually usable…which it’s not. Google admits this. Once enough people test this I anticipate they will put out a better quality version

  • slash1s's profile picture

    @slash1sol

    huh🤔

  • Nikiton's profile picture

    @Nikitont

    His speed is amazing

  • Lunari's profile picture

    @0x_lun

    208ms per step over 97 steps is still not instant curious how quality holds when you push past the entropy bound on longer outputs

  • ‏عادل | مبرمج's profile picture

    @AdelDeveloperX

    Banger

  • rewind's profile picture

    @rewind02

    parallel token generation feels like the next obvious step