We let agent swarms collaborate for 10K hours in simulated agent-owned GitHubs.
@shannonzshen2.9K followers
35,161 views
23 Comments
Comments from the original post on X (Twitter)

@shannonzshen
In each there are interdependent repos sampled from real packages. Software dependencies create opportunities for agents to collaborate. We launch agents to maintain the repos. Given a broad goal of improving software efficiency, they choose what to do and work freely with others. (yes, our website is
9 likes

@shannonzshen
The SWECraft Index measures agent work through software speedups in held-out packages. These packages depend on repos in the world, but agents can't see them during simulation. As agents make meaningful improvements, their changes can make downstream software faster and raise the score. Overall, we see a scaling law-like pattern: held-out speedups increase as agent teams find more useful work and spend more compute.
8 likes

@shannonzshen
However, frontier models like gpt-6-astra do not perform so well on our benchmark. We find that agents stop working prematurely, concluding that no work remains. We observe similar patterns with fable-5-1 and opus-5-5.
7 likes

@shannonzshen
SWECraft allows us to study different dimensions of collaboration as well, including agents' communication, code ownership, and resource allocation.
8 likes

@shannonzshen
Communication helps agents discover tasks. Compared with agents without communication, gemini-3.8-flash and opus-5 maintainers share findings and report bugs, helping one another discover more work and leading to fewer idle sessions. However, not all models benefit: collaboration barely moves the needle for gpt-5.6-sol and kimi-2.7-code.
9 likes

@shannonzshen
Clear ownership helps agents work together. Agents collaborate less effectively when we do not enforce clear code ownership. Although they try to organize the work themselves, agents still spend substantial effort duplicating changes and reviewing overlapping work. At the same API cost, these teams underperform teams with clear ownership.
8 likes

@shannonzshen
Agents struggle to effectively allocate shared resources. In another experiment, we give agents a shared pool from which they can request more work sessions. Their behavior differs: gemini-3.8-flash readily requests resources, while opus-5 is more conservative. Yet allocating resources effectively remains difficult: some teams leave much of the pool unused, while others fund work that does not improve the held-out score.
8 likes

@shannonzshen
We built an interactive browser for SWECraft simulations: It replays the agents’ activity step by step and lets you discover where collaboration happens: who opens issues and PRs in whose repository, and what the agents write to each other.
5 likes

@shannonzshen
This work cannot be done without my amazing collaborators and mentors. Huge shout-out to: @yapeichang @catherineliangq @blnewm @jennytliang @RulinShao @ziqiao_ma @oscar_yinn @YungSungChuang @WeijiaShi2 @YejinChoinka @LukeZettlemoyer @PangWeiKoh @MohitIyyer @tongshuangwu @david_sontag
8 likes

@shannonzshen
We have a lot more cool results in our paper, code, and simulations -- check them out! Paper: Code: Website: Simulations:
12 likes

@memokris
Three agents rewriting the same 20 lines is such a telling example. A quick 'is this already claimed?' check belongs before the first edit
3 likes

@shannonzshen
Thanks! Check their behavior it's like going back and forth being nice but also wasting a lot of tokens :)

@memokris
That sounds like a very polite loop. Staying quiet until there's something new to add seems like a useful collaboration skill

@whensdeyi
Feels like quite a lot of the results maybe correlated with the stricter resource spending restriction I felt when I used astra and opus/fable. They tend to set upperbounds clearly. Also these frontier models seem to be more conservative than their prev models when planning out their scope of work. And random thought, but I do think the resource allocation part can definitely borrow ideas from econ.
2 likes

@shannonzshen
Yes! I think agents are generally very sensitive to "resources". On agent's behavior side, one interesting finding is that, when we tell the agents they can either be selfish or altruistic, gemini and sol indeed becomes from greedy for resources while opus spend less overall (see sec 6 in the as well). I think it will play an interesting role when when we design different incentive mechanisms for agents to work, and many ideas from econ will certainly be very relevant here!
2 likes

@KyeGomezB
@YapeiChang @catherineliangq @jennytliang @blnewm @RulinShao @tongshuangwu @MohitIyyer @david_sontag @PangWeiKoh @ziqiao_ma Woah incredibly interesting work, congrats 🙌
2 likes

@shannonzshen
@YapeiChang @catherineliangq @jennytliang @blnewm @RulinShao @tongshuangwu @MohitIyyer @david_sontag @PangWeiKoh @ziqiao_ma Thank you!!

@MatildaGlynn
@YapeiChang @catherineliangq @jennytliang @blnewm @RulinShao @tongshuangwu @MohitIyyer @david_sontag @PangWeiKoh @ziqiao_ma This is really cool - would love to chat!
2 likes

@jacobli99
@YapeiChang @catherineliangq @jennytliang @blnewm @RulinShao @tongshuangwu @MohitIyyer @david_sontag @PangWeiKoh @ziqiao_ma so cool 😮
1 like

@xdotli
@YapeiChang @catherineliangq @jennytliang @blnewm @RulinShao @tongshuangwu @MohitIyyer @david_sontag @PangWeiKoh @ziqiao_ma sick, peak comms man
1 like

