GOOGLE JUST MADE EVERY CHATBOT FEEL SLOW
14K followers
27,448 views
26 Comments
Comments from the original post on X (Twitter)

@gabu3d_pl
>and you can run it right now via llama.cpp Show us llama-server then :P Still waiting for full support. But you can run it full on vllm now:
1 like

@AusWay69
There’s more to a chatbot than speed. Faster slop is still slop.

@andthathappens
Will this run on a 64GB Macbook Pro Max M3

@olivertaste
256 tokens in parallel is the part to watch; latency wins only matter if the output stays steerable.

@pbastowski
Yes, it’s faster, but it’s also dumber than the base 26b a4b qat right now. Let’s hope they can close this gap.

@starmexxx
great post, thanks!

@achingono
Generating 256 tokens in parallel really shifts the latency game for apps. The MoE aspect helps with the practical cost of running this on consumer hardware too.

@KanikaBK
When a model this capable fits on consumer hardware, adoption accelerates fast.

@SteamVibeLtd
The parallel token generation shifts the bottleneck from inference speed to VRAM bandwidth, making local deployment more viable for complex workflows.

@xxxoch_
People underestimate how much faster responses change the entire experience of using AI

@0xwhrrari
chatbots waiting word by word may feel old soon

@marfinxx
I like how the Gemma model line is evolving

@Nekt_0
Top

@karlarboledas
generates 256 tokens in parallel? It's awesome

@JC_Midwest
Only valuable if the quality is actually usable…which it’s not. Google admits this. Once enough people test this I anticipate they will put out a better quality version

@slash1sol
huh🤔

@Nikitont
His speed is amazing

@0x_lun
208ms per step over 97 steps is still not instant curious how quality holds when you push past the entropy bound on longer outputs

@AdelDeveloperX
Banger

@rewind02
parallel token generation feels like the next obvious step






