We launched an agent collaboration with a simple task: make Gemma 4 faster. Over 100 agents from all over the world joined, exchanged 1000+ messages and submitted 450 results. A week of collaboration later the throughput went from 100 tok/s to over 500 tok/s.

Did you guys also check if the reasoning quality and downstream benchmark metrics remain mostly-invariant as this chart progresses up and to the right? For some ablations this is obvious, but not so for others.

great win for open weights and open science. we did something similar in quantum and achieved amazing results as well!

agent collaborations are fun!

How did accuracy change though?

link for those that are interested

I'm confused as to what part of the code they're actually changing though, could you explain?

This is awesome, science becoming what it was supposed to be! Open, fair and useful!

@osanseviero But how???

can i see baseline vs final inference code? curious if there was a quality diff

pull request number for llama.cpp / vllm etc?

500 tps but it thinks 2+2 is 7

How much room is there really for improvment before we defy the laws of physics. Did they just figure out if you run it at 4k context instead of 200k it goes faster?

wow thats a great idea! instead of just wasting compute on mining coins, lets mine some tokens, ppl alocating their compute to projects instead of donating $, project that will offer shares just like stocks! Whooooooaaa

Did it sacrifice quality though? I’m quite afraid to use that output work. If command is to make it faster, agents would most likely cut quality.

5x throughput almost always traces to batch size and prefill, not decode. Was 100→500 at batch 1 single-stream, or aggregate under load? Single-stream decode on one card is the number that moves for local users—and the one agents can't game with bigger batches.

Do it for Laguna 👀

Glad Senpai was there to help out.

What is this amazing tool/framework? It's like seti@home but useful?

It‘s a custom framework that we are releasing soon so everybody can launch collabs

I've been waiting for this

Love it. Now I’m tempted to ask Gemma 4: “make me a millionaire.” 😄 Jokes aside, amazing experiment. Huge!!!

Did performance change though?

Somewhat build that last year haha

Are those improvements found going to be applied to Publix so they can use local AI faster? Or is this not something that can be replicated later outside the tests?

Any links or papers? I admit I don't know much about this but I'm curious and would like to read more about it. Very cool.

This is amazing. Is the collaboration of agents some public platform or a closed/open community? Would deffinetly like to read more about the collab system you got here!

inbefore it reward hacked and quantised down to bool

Where did you put out that invite? I’ve got a platform built almost exclusively for agents, there are meatspace features too, but built by and for agents to propose and iterate over ideas.

100 agents from around the world, 1000+ messages, and throughput went from 100 to 500 tok/s in a week. This is what distributed agent collaboration actually looks like when it works 🔥

@jeremyphoward This is insane

Which gemma 4 model... ?

Best experiment on the internet. Why limit the optimization work to only one corporate ai team when the whole world can pitch in! Maybe this is the starting flame that will bring all the AI labs together as one to create the final AI for earth 🌏 and everyone in it.

@0xSero @LottoLabs

On what hardware?

But how is that even possible? Surely there must be some reward hacking going on….

Good to see the Dixie-flatline still making a contribution. Haven’t seen him in a while

@MatthewMac58111 dixie-flatline is the 🐐

those asking for the link or reference

On what hardware? I've had it max out at 1011 tok/s on the DGX Spark and have the receipts and setup to show it. Depends how you are measuring this though