LM agents today primarily aim to automate tasks. Can we turn them into collaborative teammates? Introducing Collaborative Gym (Co-Gym), a framework for enabling & evaluating human-agent collaboration! I now get used to agents proactively seeking confirmation or my deep thinking.

While several HITL systems exist (e.g. OpenAI Canvas, our Collaborative STORM), what makes human-agent collab special? Agents need autonomy to be useful, yet the goal is empowering humans. We start with three tasks: travel planning, surveying related work, and tabular analysis

Excitingly, collaborative agents consistently outperform their fully autonomous counterparts in terms of task performance, achieving win rates of 86% in Travel Planning, 74% in Tabular Analysis, and 66% in Related Work when evaluated by real users.

How does Co-Gym enable collaborative agents? Our infra (1) focuses on environment design and (2) supports async interaction beyond turn-taking. We define primitives for public/private components in the shared env, as well as collaboration actions and notification protocol.

Our vision builds on a long-standing dream in AI: to develop machines that act as teammates, not mere tools. This demands situational intelligence to take initiative, communicate, and adapt. Co-Gym offers an evaluation framework that assesses both collab outcomes and processes.

We built a user simulator and web UI to instantiate Co-Gym in simulated and real settings. Experiments reveal human-like patterns: collaborative inertia, where poor communication hinders delivery; and collaborative advantage, where human-agent teams outperform autonomous agents.

We conducted a detailed error analysis by having authors annotate 300 trajectories. ❗Collaborative agents expose significant limitations in current LMs and agent scaffolding, with communication and situational awareness failures occurring in 65% and 40% of real trajectories.

We are working on making Co-Gym UI accessible to the public. Can’t wait to get more in-the-wild evaluations and observe more dynamics of human-agent collaboration. Stay tuned! Check out our arXiv paper first to learn more:

To me, Co-Gym stems from my SoP on building human-centered agentic systems 2 years ago. I am excited to see how agents could work with us and the demands this poses for advancing model intelligence! Thank you so much @vsamuel2003, @YuchengJiang0, @jyangballin, @Diyi_Yang !!!!!

#AI agents: from tools to teammates. Software that can act, react, and interact with both humans and other agents will transform how businesses operate—and how they should invest in technology. This @EY_US article explains more. #ad

This is really cool! Looking forward to the UI release

Such a nice work and cool Demo video!