Skip navigation
We let agent swarms collaborate for 10K hours in simulated agent-owned GitHubs.
0:00 / 2:36

We let agent swarms collaborate for 10K hours in simulated agent-owned GitHubs.

Shannon Shen

@shannonzshen2.9K followers

Visit on X
289 likesDownloadOriginal post

35,161 views

We let agent swarms collaborate for 10K hours in simulated agent-owned GitHubs. Gemini-3.8-flash collaborate the best while gpt-6-astra scores the worst among frontier models tested⁉️ What are the elements of successful agent collaboration at scale? Check World of SWECraft 🧵

23 Comments

Comments from the original post on X (Twitter)

  • Shannon Shen's profile picture

    @shannonzshen

    In each there are interdependent repos sampled from real packages. Software dependencies create opportunities for agents to collaborate. We launch agents to maintain the repos. Given a broad goal of improving software efficiency, they choose what to do and work freely with others. (yes, our website is

    9 likes

  • Shannon Shen's profile picture

    @shannonzshen

    The SWECraft Index measures agent work through software speedups in held-out packages. These packages depend on repos in the world, but agents can't see them during simulation. ​As agents make meaningful improvements, their changes can make downstream software faster and raise the score. Overall, we see a scaling law-like pattern: held-out speedups increase as agent teams find more useful work and spend more compute.

    8 likes

  • Shannon Shen's profile picture

    @shannonzshen

    However, frontier models like gpt-6-astra do not perform so well on our benchmark. We find that agents stop working prematurely, concluding that no work remains. We observe similar patterns with fable-5-1 and opus-5-5.

    7 likes

  • Shannon Shen's profile picture

    @shannonzshen

    SWECraft allows us to study different dimensions of collaboration as well, including agents' communication, code ownership, and resource allocation.

    8 likes

  • Shannon Shen's profile picture

    @shannonzshen

    Communication helps agents discover tasks. Compared with agents without communication, gemini-3.8-flash and opus-5 maintainers share findings and report bugs, helping one another discover more work and leading to fewer idle sessions. However, not all models benefit: collaboration barely moves the needle for gpt-5.6-sol and kimi-2.7-code.

    9 likes

  • Shannon Shen's profile picture

    @shannonzshen

    Clear ownership helps agents work together. Agents collaborate less effectively when we do not enforce clear code ownership. Although they try to organize the work themselves, agents still spend substantial effort duplicating changes and reviewing overlapping work. At the same API cost, these teams underperform teams with clear ownership.

    8 likes

  • Shannon Shen's profile picture

    @shannonzshen

    Agents struggle to effectively allocate shared resources. In another experiment, we give agents a shared pool from which they can request more work sessions. Their behavior differs: gemini-3.8-flash readily requests resources, while opus-5 is more conservative. Yet allocating resources effectively remains difficult: some teams leave much of the pool unused, while others fund work that does not improve the held-out score.

    8 likes

  • Shannon Shen's profile picture

    @shannonzshen

    We built an interactive browser for SWECraft simulations: It replays the agents’ activity step by step and lets you discover where collaboration happens: who opens issues and PRs in whose repository, and what the agents write to each other.

    5 likes

  • Shannon Shen's profile picture

    @shannonzshen

    This work cannot be done without my amazing collaborators and mentors. Huge shout-out to: @yapeichang @catherineliangq @blnewm @jennytliang @RulinShao @ziqiao_ma @oscar_yinn @YungSungChuang @WeijiaShi2 @YejinChoinka @LukeZettlemoyer @PangWeiKoh @MohitIyyer @tongshuangwu @david_sontag

    8 likes

  • Shannon Shen's profile picture

    @shannonzshen

    We have a lot more cool results in our paper, code, and simulations -- check them out! Paper: Code: Website: Simulations:

    12 likes

  • Kris Tan's profile picture

    @memokris

    Three agents rewriting the same 20 lines is such a telling example. A quick 'is this already claimed?' check belongs before the first edit

    3 likes

  • Shannon Shen's profile picture

    @shannonzshen

    Thanks! Check their behavior it's like going back and forth being nice but also wasting a lot of tokens :)

  • Kris Tan's profile picture

    @memokris

    That sounds like a very polite loop. Staying quiet until there's something new to add seems like a useful collaboration skill

  • Dawna Yi's profile picture

    @whensdeyi

    Feels like quite a lot of the results maybe correlated with the stricter resource spending restriction I felt when I used astra and opus/fable. They tend to set upperbounds clearly. Also these frontier models seem to be more conservative than their prev models when planning out their scope of work. And random thought, but I do think the resource allocation part can definitely borrow ideas from econ.

    2 likes

  • Shannon Shen's profile picture

    @shannonzshen

    Yes! I think agents are generally very sensitive to "resources". On agent's behavior side, one interesting finding is that, when we tell the agents they can either be selfish or altruistic, gemini and sol indeed becomes from greedy for resources while opus spend less overall (see sec 6 in the as well). I think it will play an interesting role when when we design different incentive mechanisms for agents to work, and many ideas from econ will certainly be very relevant here!

    2 likes

  • Kye Gomez (swarms)'s profile picture

    @KyeGomezB

    @YapeiChang @catherineliangq @jennytliang @blnewm @RulinShao @tongshuangwu @MohitIyyer @david_sontag @PangWeiKoh @ziqiao_ma Woah incredibly interesting work, congrats 🙌

    2 likes

  • Shannon Shen's profile picture

    @shannonzshen

    @YapeiChang @catherineliangq @jennytliang @blnewm @RulinShao @tongshuangwu @MohitIyyer @david_sontag @PangWeiKoh @ziqiao_ma Thank you!!

  • Matilda Glynn | in SF 10.1-10.30's profile picture

    @MatildaGlynn

    @YapeiChang @catherineliangq @jennytliang @blnewm @RulinShao @tongshuangwu @MohitIyyer @david_sontag @PangWeiKoh @ziqiao_ma This is really cool - would love to chat!

    2 likes

  • Jacob X. Li's profile picture

    @jacobli99

    @YapeiChang @catherineliangq @jennytliang @blnewm @RulinShao @tongshuangwu @MohitIyyer @david_sontag @PangWeiKoh @ziqiao_ma so cool 😮

    1 like

  • Xiangyi Li's profile picture

    @xdotli

    @YapeiChang @catherineliangq @jennytliang @blnewm @RulinShao @tongshuangwu @MohitIyyer @david_sontag @PangWeiKoh @ziqiao_ma sick, peak comms man

    1 like