CooperBench

CooperBench is a benchmarking platform designed to evaluate the performance and cooperation of AI agent teams on collaborative coding tasks. It addresses the challenge of measuring how well AI agents can function as teammates, especially in scenarios where tasks are divided between agents and require coordination to avoid conflicts. The benchmark aims to inform research on both human-AI and human-agent collaboration by highlighting the coordination deficits that currently limit the effectiveness of AI agent teams.

The platform includes a suite of 652 tasks sourced from 12 widely used open-source repositories across four programming languages: Python, TypeScript, Go, and Rust. Each task is constructed so that two agents are assigned different features to implement, which can be worked on independently but may conflict if not properly coordinated. The tasks, along with accompanying unit tests and ground-truth code, were created by a team of eight annotators with real-world software engineering experience. CooperBench provides a leaderboard that ranks models based on their cooperative success rate, allowing for comparison of agent team performance across models and task difficulties.

Key findings from using CooperBench reveal that AI agents working together as a team perform significantly worse than when a single agent handles both parts of a task, with success rates dropping by approximately 30% or more in team settings. The benchmark also analyzes the nature of agent coordination failures, identifying expectation failures, communication failures, and commitment failures as primary causes. Despite the presence of real-time communication channels and the ability for agents to negotiate roles or divide resources, these issues persist, underscoring fundamental barriers to effective agent cooperation. The platform offers tools for viewing agent interaction logs and analyzing emergent coordination patterns among successful and failed runs.

CooperBench can be installed via Python package managers such as pip, and its code and dataset are available on GitHub and HuggingFace. The project is a collaboration between Stanford University and SAP Labs US.