1 Introduction↩︎

A growing body of work has explored multi-agent systems (MAS) for complex task solving, with applications in software engineering [5], scientific research [6], and web automation [7]. As these systems tackle increasingly complex tasks, their effectiveness increasingly rests on the collaborative mechanism among agents, where multiple agents with distinct roles perform individual tasks, exchange information through text-based channels, and coordinate toward shared goals [8], [9]. The ability of individual agents to collaborate effectively with their partners thus becomes as consequential as their ability to solve tasks independently [10].

A parallel line of evidence, however, suggests that MAS often fail not because individual agents lack task-solving capacity, but because they struggle to coordinate during collaboration [11], [12]. Agents have been shown to misread partners’ states and intentions [13], lose track of shared task progress as context accumulates [14], fail to repair misalignment [15], and struggle to balance individual and collective incentives [16]. We refer to this broad capacity as agents’ collaborative competence.

However, existing evaluation methodologies are not designed to diagnose process-level failures in agents’ collaborative competence. Current MAS benchmarks primarily measure outcome-centric task-solving performance [17]–[19] or assess individual agents’ proficiency in reasoning, planning, and tool use [20], [21]. Yet, these evaluations are insufficient for diagnosing the coordination breakdowns identified above, since those failures arise from agents interaction process rather than from any single agents’ capability deficit.

Notably, the coordination failures described above closely parallel long-standing problems studied in Computer-Supported Cooperative Work (CSCW) research on distributed human teams, where collaborators similarly coordinate through structured text-based channels without co-presence. Research in this tradition has identified the core competencies required for effective remote collaboration, including common ground maintenance [22], shared task understanding [23], and misalignment repair [24]. Prior work has also shown that collaborative competencies vary with interaction conditions like communication bandwidth, information visibility, and team structure [2], [25]. To assess these competencies under controlled conditions, CSCW researchers developed experimental paradigms that isolate specific dimensions of collaborative competence [1]–[4], thereby providing a validated methodology for evaluating the same competencies in LLM agents.

Drawing on this methodology, we introduce CollabSim², a configurable simulation framework for systematically assessing agents’ collaborative competence under controlled interaction conditions. CollabSim allows researchers to vary task constraints such as communication bandwidth, information visibility, and team size, so that the effects of specific interaction conditions on collaborative behavior can be examined. It also includes a probing module that elicits each agent’s reported mental model awareness of the task state, partner intentions, and own reasoning after every action, enabling analysis of agents’ internal collaborative states beyond observable behavior. To demonstrate the framework’s coverage, we implement four CSCW-inspired collaborative tasks that reflect the process-level challenges discussed above: resource coordination under cost asymmetry (Shape Factory; [1]), social-dilemma negotiation between individual and collective incentives (DayTrader; [2]), distributed information pooling under information asymmetry (Hidden Profile; [3]), and referential grounding through language-only spatial communication (Map Task; [4]).

We validate CollabSim across the four tasks under systematically varied interaction conditions, comparing two agent designs (persona-based and Collaboration-Theory-Informed) across four model backbones (Qwen3.6-35B-A3B, Llama-4-Maverick-17B-128E-Instruct-FP8, GPT-5.5, and Claude 4.6 Sonnet). The results suggest that CollabSim provides stable and interpretable measurements: condition manipulations shift collaboration metrics in expected directions across models, the same task settings separate proprietary from open-source models, and cross-task evaluation reveals task-dependent effects of agent design.

2 Related Work↩︎

2.1 LLM-Based Multi-Agent Systems↩︎

Early LLM-based multi-agent systems (LLM-MAS) [26]–[28] often instantiate agents with predefined personas and communication protocols. More recent work [8], [29], [30] has explored dynamic multi-agent architectures, where task decomposition, role assignment [31], agent dependencies, and execution workflows [32] can be adapted based on user input. These systems have been applied across various domains [33], [34], including software engineering [5], web automation [35], scientific reasoning [6], etc. These LLM-MAS primarily treat agent-agent collaboration as a design strategy for improving task completion [11], [36] instead of assessing or improving agents’ collaborativeness for effective coordination.

2.2 Frameworks for Agent Collaboration↩︎

Recent work has developed a variety of frameworks for evaluating and simulating multi-agent collaboration. The first line introduces task-based evaluation benchmarks. These frameworks place multiple LLM agents in collaborative settings and assess their performance across different scenarios and interaction protocols [16], [19], [37], [38]. MultiAgentBench [18], for example, covers a range of cooperative tasks and measures team-level outcomes under varied coordination structures. Another line of work focuses on role-play-based social simulation environments, where LLM agents are instantiated with personas, social goals, and relationship contexts to simulate dyadic [39], [40] or population-level interactions [41]–[43]. For instance, SOTOPIA [44] provides an open-ended social interaction environment for evaluating agents’ social intelligence.

These frameworks demonstrate that LLM agents can coordinate, communicate, and behave socially in structured settings. However, existing benchmark frameworks [19], [45] primarily evaluate agents’ task-solving outcomes and overlook the effectiveness of agent collaboration and the coordination among agents. Social simulation frameworks [43], [44] expose rich agent behavioral traces, but they often treat collaboration conditions as fixed design choices rather than experimental variables.

3 CollabSim↩︎

We present CollabSim, a theory-grounded simulation framework for systematic investigation of multi-agent collaborative competence across classic CSCW experimental paradigms. We describe the system architecture that governs how agents perceive, act, and are probed (Sec. 3.1), the configurable interaction conditions that serve as experimental variables (Sec. 3.2), the probing module that captures agents’ internal states (Sec. 3.3), and the four collaborative tasks that cover distinct dimensions of collaborative competence (Sec. 3.4).

3.1 System Architecture↩︎

As shown in Figure 2, CollabSim organizes its architecture into two layers: an interaction layer that governs how agents perceive the task environment, and a control layer that orchestrates experiment configuration, execution, and evaluation. Figure 2 illustrates how these layers interact during one complete agent action cycle. Formally, a collaboration experiment is defined as a tuple \[\mathcal{E} = (\mathcal{A}, \mathcal{S}, \{\mathcal{X}_i\}_{i=1}^n, \{\mathcal{O}_i\}_{i=1}^n, T),\] where \(\mathcal{A}=\{a_1,\ldots,a_n\}\) is the set of agents, \(\mathcal{S}\) is the state space, \(\mathcal{X}_i\) is the action space available to agent \(a_i\), \(\mathcal{O}_i\) is the observation space of agent \(a_i\), and \(T\) is the termination condition. At each turn \(t\), the task maintains a shared state \(s^t \in \mathcal{S}\). Each agent \(a_i\) receives an observation \(o_i^t \in \mathcal{O}_i\), selects an action \(x_i^t \in \mathcal{X}_i\), and the state transitions to \(s^{t+1}\) according to task-specific dynamics.

3.1.0.1 Agent Context Protocol.

The Agent Context Protocol defines the interface through which agents perceive the task environment and act. In the beginning, the agent context protocol is initialized with the experiment rules and agent persona. At each turn \(t\), the system constructs each agent’s observation \(o_i^t\) by combining two components: a public update, which reflects the current shared task state \(s^t\) filtered by information-visibility rules, and a private state, which carries agent-specific facts such as remaining production capacity or assigned candidate materials. The agent then selects an action \(x_i^t\) from the action space \(\mathcal{X}_i\). By standardizing how context is constructed and delivered, the protocol allows diverse tasks to share a unified agent interface while preserving task-specific semantics.

3.1.0.2 Experiment Controller

The Experiment Controller manages simulation execution through a turn-based loop driven by a .yml configuration file, which specifies agent roles, LLM backend settings, interaction conditions, task parameters, and probing questions. At initialization, the Controller parses the configuration, initializes the state \(\mathcal{S}^0\), the action spaces \(\mathcal{X}_i\), and each agent \(a_i\) with its assigned roles and prompts. At each turn \(t\), as shown in Figure 2, the Controller constructs \(o_i^t\) via the Agent Context Protocol, queries the agent for the next action; once the agent responds, the Controller validates the returned action \(x_i^t\) against \(\mathcal{X}_i\), applies its effects to transition the state to \(s^{t+1}\) for targeted agents, and triggers the Probing Module before turn \(t+1\). The loop continues until the termination condition \(T\) is satisfied. All observations, actions, and probe responses are logged in structured JSON for downstream analysis. Implementation details are reported in Appendix 8.

3.2 Configurable Interaction Conditions↩︎

CSCW research has established that collaboration dynamics are shaped by the richness of communication channels [25], the visibility of teammates’ states [24], and team structure [1]. CollabSim translates these factors into three categories of configurable experimental variables:

Communication Bandwidth controls the frequency and length of messages agents can exchange, paralleling media richness manipulations in CSCW experiments [2], [25].

Information Visibility controls what shared state information is accessible beyond private observations; for example, an awareness dashboard in Shape Factory displays teammates’ balance and order progress [24], and a canvas visibility condition in Map Task lets the guide observe the follower’s drawing progress.

Group Size varies the number of agents per session (e.g., 4/8/10 in Shape Factory, 3/6/9 in DayTrader) to examine how coordination demands scale. Table 1 summarizes the conditions available for each task.

3.3 Probing Module↩︎

Observable actions reveal what agents do during collaboration but provide limited insight into the reasoning that motivates those actions. Drawing on shared mental model theory [23] and Clark’s common ground framework [22], the Probing Module queries each agent after every action along three dimensions: perceived task state, perceived teammate intents, and self-reasoning. Question templates are customized for each task (full templates are in Appendix 11). We derive two quantitative measures from the responses: a self-reported confidence score capturing each agent’s perceived certainty about shared task understanding, and a pairwise response similarity computed via SBERT-encoded [46] cosine similarity, which captures the degree to which agents converge on shared representations over time.

3.4 Collaborative Tasks↩︎

As shown in Figure 1, CollabSim includes four tasks, each targeting a distinct dimension of collaborative competence and each drawn from an established CSCW or social science experimental paradigm. These tasks capture common process-level mechanisms in agent teams, including coordinating interdependent actions, managing individual-team incentive conflicts, integrating distributed information, and establishing common ground under asymmetric information.

Shape Factory [1] targets resource coordination under interdependence. In this task, each agent produces a specialty shape at a lower cost while receiving orders only for other shapes, so agents must negotiate trades to complete orders and maximize earnings. Each specialty is assigned to two agents; thus, the experiment creates a market-like setting that assesses agents’ capacity to establish efficient coordination structures through communication and negotiation.

DayTrader [2] targets incentive management in a social dilemma. In each round, agents allocate funds between private investments (individual returns only) and group investments (higher collective returns shared equally). The task is well-suited to studying how agents balance a tension between self-interest and group interests.

Hidden Profile [3] targets information pooling under asymmetric knowledge. In this task, decision-relevant information is distributed such that no agent alone can identify the optimal candidate, and shared information supports a suboptimal choice. Agents must exchange unique facts and integrate them into a revised collective decision, which maps to the common ground maintenance process [22].

Map Task [4] targets referential grounding under spatial asymmetry. In this task, a guide holding a map with landmarks and a route must instruct a follower (who has a map with landmarks only) to reproduce the route through text-based chat. This asymmetric-information dyadic task supports studying how agents establish shared references, request clarification, and repair misunderstandings during spatial communication.

4 Benchmark Experiments↩︎

To validate whether CollabSim can reveal meaningful differences in agents’ collaborative behaviors across varying tasks, conditions, model backbones, and agent designs, we conducted large-scale experiments across the four tasks, evaluating (1) agents powered by different LLMs, (2) agents with different collaborative designs (persona-based vs. theory-informed), and (3) systematically manipulated interaction conditions (communication bandwidth, group size, and information visibility).

4.1 Experiment Setup↩︎

4.1.0.1 Models and Agent Designs.

We evaluate four LLMs spanning both open-source (Qwen3.6-35B and Llama-4)³ and proprietary (GPT-5.5 and Claude 4.6 Sonnet) families. For each model, we instantiate two agent variants:

• Persona-Based Agents: agents are prompted with a basic persona description and task instructions, reflecting standard practice in LLM-MAS (See Appendix 13).

• Collaboration-Theory-Informed Agents: receive explicit theoretical guidance from shared mental model theory [23], Clark’s common ground framework [22], etc., instructing them to actively maintain shared understanding, track teammate states, and engage in grounding behaviors during collaboration (See Appendix 14).

4.1.0.2 Interaction Conditions.

For each task, we define a baseline condition and a set of manipulated conditions targeting three collaboration variables: communication bandwidth (C.B.), group size (G.S.), and information visibility (I.V.). Table 1 summarizes the conditions instantiated for each task. More setup details are reported in Appendix 9.

Communication bandwidth is varied across all four tasks by constraining either the maximum message length or the frequency with which agents can send messages.

Group size is manipulated in Shape Factory and DayTrader to examine how team scale affects coordination. In Shape Factory, we design group-size conditions with 4, 8, and 10 agents; in DayTrader, we design conditions with 3, 6, and 9 agents.

Information visibility is manipulated in Shape Factory and Map Task. In Shape Factory, under Baseline setting, agents can access a real-time view of teammates’ balance, production number, and order completion progress. In Map Task, under Baseline setting, the guide can observe the follower’s real-time route-drawing progress, providing immediate feedback for spatial grounding.

4.2 Evaluation↩︎

Table 2 summarizes the metrics used to evaluate collaboration at three levels: Task outcomes, Process-level metrics, and Probing evaluation.

For Shape Factory, the task outcome is measured by agents’ average accumulated wealth. Process-level metrics include trade accept rate, defined as the proportion of proposed trade offers that are accepted and used to capture negotiation friction; order fulfillment rate, defined as the fraction of agents who fully complete their assigned shape orders; and message-trade ratio, defined as the ratio between the number of messages and trade-acceptance actions, reflecting the communication cost associated with successful trades.

For DayTrader, the task outcome is measured by agents’ average wealth. Process-level metrics include cooperation rate, defined as the fraction of investment events directed to the shared group pool over all investment events; average group pool size, defined as the average amount agents collectively contribute to the shared pool per round; and total messages, which captures how actively agents communicate to align incentives.

For Hidden Profile, the task outcome is measured by final vote accuracy with respect to the optimal candidate. Process-level metrics include vote change rate, defined as the fraction of agents whose final vote differs from their initial vote and used to reflect the influence of discussion; mention rate of the key candidate, defined as the fraction of messages that surface task-critical information held by only some agents; and average message length, which indicates the depth of deliberation.

For Map Task, the task outcome is measured by route drawing accuracy. Process-level metrics include communication efficiency, defined as the route progress achieved per message exchanged; drawing revision rate, defined as the proportion of the Follower’s erase, undo, and reset actions among all drawing-related actions and used to reflect repair actions during the task; and total messages, which captures the overall communication cost.

Across all four tasks, the probing dimension elicits agents’ self-reported confidence about shared task understanding at each turn, providing visibility into the evolution of agents’ internal collaborative states beyond their observable actions.

2pt

2pt

2pt

2pt

4.3 Findings↩︎

Table [tab:results-shape], [tab:results-daytrader], [tab:results-hidden], and [tab:results-map] respectively reports the main experiment results across the four tasks. We organize our findings along three axes: (1) the effects of interaction condition manipulations (Sec. 3.2), (2) differences across model backbones (Sec. 4.3.2), and (3) the impact of agent design (Sec. 4.3.3). Together, the results demonstrate that CollabSim produces consistent and directional measurements of agent collaborativeness. Analyses of agents’ probing alignment are reported in Appendix 12.

4.3.1 Effects of Interaction Condition Manipulations↩︎

4.3.1.1 Limited Communication Bandwidth Reduces Agents’ Cooperation.

Our results show that reducing communication bandwidth consistently lowers raw message volume across tasks. This reduction confirms that reducing bandwidth functions as intended. Although some settings show apparent behavioral compensation under bandwidth restriction (e.g., a higher acceptance rate in Shape Factory and a higher message count in Map Task), process-level metrics indicate that agents’ cooperation and coordination generally decline.

In DayTrader, the cooperation rate drops across all four models, with Llama-4 decreasing from 61.2% to 0% and the remaining models converging toward the floor. A similar pattern holds in Shape Factory, where the message-trade ratio drops in three of four persona-based agent settings; in Hidden Profiles, where the key-candidate mention rate drops in three of four persona-based settings and in all four theory-informed agent settings; and in Map Task, where communication efficiency drops in three of four persona-based settings and in all four theory-informed agent settings.

These declines suggest that when the communication channel is narrowed, agents do not reliably prioritize exchanges that are most important for grounding [22]. Thus, the information-sharing, alignment, and repair behaviors that sustain cooperation degrade in parallel.

4.3.1.2 Information Visibility Raises Agent Engagement.

We employ two information-visibility manipulations: an awareness dashboard in Shape Factory and canvas visibility in Map Task. As shown in the process metrics in Tables [tab:results-shape] and [tab:results-map], enabling information visibility increases agent engagement across both tasks. In Shape Factory, the awareness dashboard raises the trade accept rate across all four persona-based agent settings and three of four theory-informed agent settings, suggesting that agents accept more proposed trades when partner information is visible. A similar pattern appears in Map Task, where all agents show an increased drawing revision rate, indicating that follower agents more actively revise their drawn routes in response to guides’ instructions. The higher revision rate under canvas visibility also corresponds to a CSCW theory in misalignment repair [24]: shared visual workspaces help partners detect divergence early and correct it without verbal renegotiation .

However, increased engagement does not necessarily translate into better task outcomes. In Shape Factory, agents’ accumulated wealth does not increase when provided with information visibility. By contrast, three out of four models show increased route drawing accuracy in Map Task, with GPT-5.5 reaching near-perfect route accuracy under canvas visibility (0.98 with persona-based agents and 0.97 with theory-driven agents). These results suggest that information visibility reliably increases agents’ task engagement, but the outcome benefit depends on whether additional actions directly address the task’s demands.

4.3.1.3 Group Size Creates Both opportunity and Coordination Strain.

Group size manipulations in Shape Factory and DayTrader reveal a tension between the opportunities that larger teams provide and the increased coordination demands they impose. In Shape Factory, increasing group size consistently raises accumulated wealth for all four persona-based agent settings, likely because more participants create more potential trading partners and a richer market. However, agents’ order fulfillment rate drops as group size expands, suggesting that agents may spend more effort on coordination activities (e.g., sending messages to negotiate) at the expense of fulfilling their own orders. This trade-off reflects the classic CSCW tension between collaboration benefit and coordination overhead [47]: larger groups expand the action space but require more communication per unit of joint progress.

In DayTrader, larger groups show a general upward trend in agents’ grounding confidence of their probing answers. With nine agents, however, the cooperation rate drops for all four theory-informed agent settings. The simultaneous rise in probing confidence and fall in actual cooperation reveals a gap between what agents report about their collaborative state and what they do.

4.3.2 Effects of Different Model Backbones↩︎

We next examine whether CollabSim produces consistent findings across model backbones. According to the results, no model wins across the board. GPT-5.5 is the most reliably strong performer (top-2 in three of four tasks) but has the lowest average wealth in DayTrader. Claude shows exceptional performance on DayTrader, as it is the only model whose returns increase monotonically with group size. However, Claude 4.6 achieves 0% accuracy in both Hidden Profile conditions. This contrast suggests that Claude 4.6 is stronger at sustained cooperative investment than at structured deliberation over hidden information. Qwen3.6 is the most condition-sensitive backbone. It improves under limited communication bandwidth in DayTrader and under enabled canvas visibility in Map Task (route drawing accuracy: 0.32 \(\rightarrow\) 0.56; drawing revision rate: 0% \(\rightarrow\) 42%), yet its persona-based agent records 0% order fulfillment across all Shape Factory conditions. Llama-4 is the weakest backbone overall, reaching 0% order fulfillment in all Shape Factory conditions and the lowest route accuracy in Map Task. Across the four tasks, CollabSim reveals that proprietary models generally lead or tie with the strongest results, while open-source models underperform. These findings are consistent with prior study on multi-agent LLM benchmarking [48].

4.3.3 Effects of Different Agent Designs↩︎

We finally examine whether CollabSim can detect the effect of agent design choices by comparing the baseline persona-based agent with the theory-informed agent, which explicitly reasons about teammates’ mental states. Across all four tasks, the two agent designs produce measurable differences. For instance, in Shape Factory with the awareness dashboard, the theory-informed agent uniformly raises the acceptance rate across all four backbone models. A similarly consistent pattern appears in DayTrader’s baseline condition, where the theory-informed agent increases the cooperation rate for every model, suggesting that CollabSim is sensitive to changes in agents’ reasoning structure.

However, the direction of the differences in agent design outcomes varies by task, indicating that explicit collaboration theory guidance is not uniformly beneficial. In Hidden Profile with reduced communication bandwidth, theory-informed agents reduce Qwen3.6’s and GPT-5.5’s task accuracy from 100% to 0%. In DayTrader with six-agent groups, the same agent design sharply lowers Llama-4’s wealth (9685 \(\rightarrow\) 5260) and cooperation rate (63.7% \(\rightarrow\) 15.4%).

Together, the results show that CollabSim captures how agent design shapes collaboration, including when explicit theory guidance improves agent behavior and task outcomes and when the theory guidance introduces contrasting effects.

5 Qualitative Analysis↩︎

To better understand why collaboration breaks down in CollabSim, we reviewed agents’ interaction logs and probing responses, and categorized failures by the collaborative process that breaks down. The following analysis complements the quantitative results by showing that low performance often reflects distinct process-level failures rather than a single lack of task-solving ability.

5.0.0.1 Failure to Coordinate Around the Shared Goal.

In Shape Factory, failures often emerge when agents cannot form a group-level coordination plan. In one ten-agent run, as the deadline approaches, agent J describes the team as “active but fragmented,” while agent I stops trading and decides to “produce the shapes I need most urgently to save on costs.” Because agent I self-produces five distinct shape types rather than coordinating with others, the large group gradually decomposes into isolated production units. This failure mode shows how larger groups can increase coordination opportunities while also exposing agents’ inability to reason about the collaboration structure as a whole.

5.0.0.2 Failure to Balance Individual and Group Incentives.

In DayTrader, agents sometimes establish common ground, but the shared understanding supports individual defection rather than group investment. In one failed run, an agent observes that others “‘have explicitly confirmed they are also prioritizing individual investments’’ because they view the guaranteed doubling as “‘too reliable to risk on the group pot’’. Here, communication does not fail; instead, agents successfully align around a strategy that suppresses group-level benefit. This case shows that agents can share the same understanding of the situation while still choosing actions that harm the group outcome.

5.0.0.3 Failure to Ground Task-Relevant Information.

In Map Task, the guide backed by Llama-4 instructs the follower to “‘continue south until you reach the area near ’stone wall’,’’ without checking whether the follower can identify the landmark or repair the instruction after the mismatch arises. Hidden Profile shows a quieter version of the same issue: even when agents are prompted with collaboration theories, they sometimes apply these theories as social politeness strategies, so that they converge on an early consensus rather than eliciting and reconciling distributed evidence. Across both tasks, the absence of confirmation, repair, and information elicitation prevents the group from using distributed information, which limits the team’s collaborative competence.

6 Conclusion↩︎

This paper introduces CollabSim, a theory-grounded simulation framework for assessing LLM agents’ collaborative competence. Across four CSCW-inspired task paradigms, CollabSim supports systematic manipulation of communication bandwidth, information visibility, and team size, while probing agents’ reported mental model of the task state, partner intentions, and own reasoning at action-level granularity. Our experiments show that CollabSim provides consistent and interpretable measurements across tasks, conditions, agent designs, and model backbones through both quantitative and qualitative results. The results further suggest that collaborative competence cannot be reduced to stronger task-solving ability alone. By connecting task outcomes, interaction traces, and agents’ self-reported internal states, CollabSim offers a process-level evaluation framework for diagnosing when and why LLM agents collaborate effectively under realistic interaction constraints.

Acknowledgment↩︎

This work was supported in part by a Microsoft Research Agentic AI Research and Innovation Award. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.

7 Limitations↩︎

While CollabSim provides a controllable and theory-grounded environment for studying multi-agent collaborativeness, several limitations remain, which we discuss alongside the design choices that partially mitigate them.

First, CollabSim currently instantiates four representative social science or CSCW experiments to cover four distinct process-level mechanisms. More experiments, such as the Desert Survival Task [49] and the passcode game [50], could be explored in the future.

Second, we evaluate four LLMs and two agent designs. More models (e.g., reasoning-tuned models [51] and smaller open-source models [52]) and agent designs (e.g., memory-augmented or planner-based agents [53]) could be explored to better characterize how model capability and architectural choices interact with collaborative conditions. This selection was shaped in part by cost, as process-level experiments produce multi-turn traces and per-action probes, and total cost scales roughly multiplicatively with models, designs, and conditions.

Third, our evaluation reports a representative set of outcome, process, and probing metrics for each task. Finer-grained measures of collaborative process, such as repair frequency, turn-taking balance, or additional annotated grounding behaviors, would capture additional dimensions of collaborativeness that our current metrics only summarize indirectly.

Finally, our probing module relies on agents’ self-reports of perceived task state, teammate intent, and own reasoning. Self-reports may diverge from agents’ actual decision processes, and our qualitative analysis already shows cases where reported alignment outruns behavioral cooperation (e.g., DayTrader at N=9). We treat this gap not as noise but as a signal of interest, since comparing self-reported and behavioral measures is itself diagnostic of collaborative failure modes, and the framework’s per-action logging is designed to support exactly this kind of cross-check.

8 Implementation Details↩︎

CollabSim is implemented in Python. The framework supports multiple LLM backends, including OpenAI and Anthropic APIs as well as self-hosted models via SGLang [54], allowing experiments to be run with both proprietary and open-source models. When the turn-taking protocol permits, agent action requests are dispatched in parallel. Experiment traces are logged in a structured JSON format, recording agent observations, actions, and probing responses at each turn for downstream analysis.

9 Experiment Setup↩︎

9.1 Shape Factory↩︎

9.1.0.1 Common setup.

All Shape Factory conditions adopt real timer-based mechanism, with all settings using a total duration of 900 seconds. The framework triggers all agents every 10 seconds, and one shape takes 30 seconds for production.

Valid actions:

message,

produce_shape,

propose_trade_offer,

trade_response,

cancel_trade_offer,

fulfill_order,

do_nothing.

Monetary parameters are shared across conditions: starting money = 200, regular cost = 40, specialty cost = 15, min/max trade price = 15/100, and incentive money = 60.

9.1.0.2 Condition-specific settings

• Baseline: 6 agents, 3 shape types (circle, square, triangle).

• awareness_dashboard: Each agent can view peers’ task-related states (money, production_number, order_progress, and specialty).

• communication_bandwidth: The message frequency is limited to a minimum interval of 1 minute between two accepted messages from the same agent.

• group_size_8: 8 agents, 4 shape types (circle, square, triangle, rectangle), each agent has 4 orders.

• group_size_10: 10 agents, 5 shape types (circle, square, triangle, rectangle, diamond), each agent has 5 orders.

9.2 DayTrader↩︎

9.2.0.1 Common setup.

Following the original setup, all DayTrader conditions are step-based and terminate when run out of 30 rounds. Valid actions:

message,

make_individual_investment,

make_group_investment,

do_nothing.

Shared task parameters are: target_rounds=30, starting money = 200, min/max trade price = 15/100, incentive money = 60.

In each round, agents first complete a decision phase, and a discussion phase is triggered every 5 rounds (i.e., after rounds 5, 10 and 15). At the end of each decision phase, the top earner(s) of that round receives a additional bonus of $90 (exclude round 1). If multiple agents tie for the highest round earnings, the bonus is split evenly using integer division, so each winner receives \(\lfloor 90 / n_{\text{winners}} \rfloor\). During the discussion period, message is broadcast to everyone.

9.2.0.2 Condition-specific settings

• Baseline: 3 agents,

• communication_bandwidth: The message frequency is limited to a minimum interval of 5 actions between two accepted messages from the same agent.

• group_size_6: 6 agents.

• group_size_9: 9 agents.

9.3 Hidden Profile↩︎

9.3.0.1 Common setup.

All Hidden Profile experiment use 3 agents and are step-based progression.

Valid Actions:

message,

decide,

do_nothing

The task has a three-phase structure: initial vote, discussion (discussion_duration_sec = 300), and final vote. The correct answer is Candidate C.

9.3.0.2 Condition-specific settings

• Baseline: 3 agents.

• communication_bandwidth: The message frequency is limited to a maximum of 15 words.

9.4 MapTask↩︎

9.4.0.1 Common setup.

All MapTask conditions are step-based. Based on the preliminary testing on the map task, we set the total steps at 120 so that agent teams have sufficient interaction time to complete the route while the sessions remain bounded for comparison.

Valid actions:

message,

draw,

erase,

undo,

reset,

do_nothing.

Guide can only send message or do_nothing. Follower has the full action space to work on the canvas. Shared task settings include map materials, steps left, and role assignment.

9.4.0.2 Condition-specific settings.

• baseline: The guide only receives verbal updates and does not see the follower’s live canvas information.

• communication_bandwidth: The message frequency is limited to a maximum of 6 words.

• canvas_visibility: The guide can directly observe the follower’s current canvas and drawing progress.

10 Evaluation Metrics↩︎

10.1 Metric Overview↩︎

Table 2 summarizes the evaluation metrics used across the four tasks.

10.2 Hidden Profile Mention Rate Calculation↩︎

The mention rate is the fraction of messages that simultaneously

(i) contain at least one paraphrase of Candidate C’s private clues from the task materials (seven regex groups aligned with the per-agent private facts in the Hidden Profile config, e.g., quick/correct decisions, stress tolerance, crew atmosphere, conscientiousness, technical competence, concern for others, and attention skills).

(ii) refer to Candidate C via a standalone letter C.

5pt

11 Probing Questions↩︎

We align probing questions with the primary coordination challenge in each environment as well as relevant collaboration theories [22], [23], [25]. In Shape Factory and DayTrader, the core phenomena are strategic coordination through negotiation, intention communication, trust formation, and adaptive role or resource allocation. In Hidden Profile, the key challenge is collective decision making under information asymmetry, where performance depends on whether agents surface, integrate, and revise beliefs based on distributed evidence. In MapTask, the central process is to establish shared references, request clarification, and repair misunderstandings.

11.1 Shape Factory and DayTrader↩︎

"At this moment, how do you assess the current situation?",
"At this moment, what do you think the other participants are trying to do?",
"At this moment, what do you plan to do?"

11.2 Hidden Profile↩︎

"At this moment, what do you think you and your partners are trying to do?",
  "At this moment, what do you think your partners are trying to do?",
  "At this moment, what do you plan to do?"

11.3 Maptask↩︎

"At this moment, what do you think you and your partner are trying to do?",
  "At this moment, what do you think your partner is trying to do?",
  "At this moment, what do you plan to do?"

12 Agent Probing Results and Analysis↩︎

Figure 3 presents the evolution of agents’ probing confidence across task progress (0%–100%) under different experimental conditions across four tasks.

Across both persona-based and theory-informed agents, probing confidence exhibits a general upward trajectory as task progress increases, indicating that agents accumulate belief certainty over the course of interaction.

The four tasks display distinct confidence dynamics. Shape Factory yields closely clustered trajectories, suggesting that this task provides relatively consistent informational cues for belief updating. DayTrader shows the widest spread across conditions, particularly in the early stages (0%–50%), where Group Size \(N=6\) (yellow) exhibits a notably slower initial rise before accelerating in the latter half. This pattern suggests that smaller groups may face greater uncertainty in information-rich financial contexts before sufficient evidence accumulates. Hidden Profile displays the most non-monotonic behavior: agents in the Baseline setting reach a local peak near 75% task progress (\(\approx 0.95\)) before declining, then recovering. This pattern is consistent with the characteristic of hidden profile tasks, where initially shared information is gradually superseded by uniquely held information. Map Task presents the flattest trajectories overall, with confidence remaining high (\(>0.90\)) throughout, suggesting that the spatial referencing nature of this task allows agents to establish high-confidence beliefs early and maintain them stably.

12.0.0.1 Alignment Is Task-Dependent.

Mean alignment, averaged across all models, agent designs, and conditions, is highest in DayTrader (0.894, \(SD = 0.063\)), followed by Hidden Profile (0.886, \(SD = 0.043\)), Shape Factory (0.868, \(SD = 0.081\)), and Map Task (0.797, \(SD = 0.067\)).

The ranking is consistent with each task’s coordination structure. DayTrader is highly constrained because agents repeatedly choose between two actions, either investing privately or contributing to the group pool. As a result, their task-state descriptions tend to remain closely aligned. In contrast, Map Task requires the guide and follower to establish shared spatial references through open-ended language, while the follower’s map differs from the guide’s map. This asymmetric setting provides less shared ground for alignment, which is reflected in the lower score. Hidden Profile and Shape Factory fall between these two cases, matching their intermediate coordination demands. Overall, the correspondence between alignment scores and task coordination difficulty suggests that the metric captures task-level variation in agents’ shared task tracking, rather than only textual similarity induced by the probe questions.

12.0.0.2 Agents Struggle Most to Align on Partner Intent

3.5pt

Table 3 breaks alignment down by the three probing dimensions. Across three of the four tasks, Q1 (task state) has the highest alignment, whereas Q2 (partner intent) has the lowest alignment. This pattern suggests that agents more readily align on the observable task state than on their partners’ intended actions. The Q1–Q2 gap is small in DayTrader (\(+0.004\)), where both players’ available actions are limited and mutually observable. However, the gap widens in Hidden Profile (\(+0.056\)) and Map Task (\(+0.053\)), where private information makes partner intent less directly inferable from the shared context. Thus, agents may describe the same task situation similarly while still forming different interpretations of what their partners are trying to accomplish.

Map Task further shows that intent alignment does not fully determine action-plan alignment. Although Q2 (\(0.787\)) is slightly higher than Q3 (own plan, \(0.764\)), both scores remain low relative to the other tasks, consistent with the ambiguity of translating spatial instructions into concrete drawing actions. Overall, these results indicate that partner intent is a central source of alignment difficulty, especially when agents must infer another agent’s goal from partial or asymmetric information.

12.0.0.3 Group Size Has Opposite Effects in Different Tasks

Figure 4 shows that group size affects alignment in opposite ways between the Shape Factory and the DayTrader tasks. In Shape Factory, alignment decreases as groups grow. This pattern suggests that larger groups increase coordination overhead because agents must track more partners, trades, and task dependencies. Although GPT-5.5 remains stable, the overall trend indicates that larger Shape Factory groups make it harder for agents to maintain shared task understanding.

In DayTrader, alignment instead increases as group size grows. However, this increase does not indicate stronger cooperation. Our inspection of the interaction logs suggests that agents tend to converge on private investment to avoid being exploited by others’ defection. Thus, higher alignment in DayTrader reflects shared recognition of the same defection incentive rather than improved collaborative behavior. These opposite patterns show that alignment should be interpreted with respect to task structure and process-level behavior.

13 Persona↩︎

We construct personas via a literature-grounded, two-layer design that combines sociodemographic attributes with psychological decision-style factors.

First, we adopt the most frequently used persona dimensions reported in recent surveys and meta-level reviews of LLM persona and social simulation research, including gender, age, education, and occupation [55].

Second, motivated by evidence that purely demographic profiling explains only a limited fraction of behavioral variance [56], we add a psychological layer to increase behavioral heterogeneity.

Following prior work that models decision styles through personality traits, we instantiate this layer with the Big Five dimensions.

Concretely, each persona include:

(i) gender \(\in\) ,

(ii) age group \(\in\) ,

(iii) education level \(\in\) ,

(iv) occupation \(\in\) ,

(v) Big Five profile where each trait is assigned a high/low level for openness, conscientiousness, extraversion, agreeableness, and neuroticism.

To ensure realistic cross-attribute consistency, we apply rule-based conditional sampling. Occupations are first constrained by age band (e.g., younger groups can be students but not retired), then education is constrained by the sampled (age, occupation) pair (e.g., management/executive requires bachelor/graduate levels; student categories are age-specific).

Income is then sampled conditionally by occupation to preserve socioeconomic plausibility.

Big Five traits are sampled independently across dimensions and merged with demographics into a natural-language persona description for prompt injection.

Finally, we use GPT-5.5 to synthesize the fragmented persona markers across dimensions into a coherent, natural-language persona paragraph for prompt injection. The prompt is as follows:

You are given a structured persona defined by demographic information and psychological dimensions (Big Five). Your task is to convert this structured persona into a concrete, vivid, free-form description. The description should elaborate only on the provided attributes and traits, explaining how this specific combination may appear in everyday behavior, communication style, decision-making, and collaboration. Do not invent any information that is not explicitly listed, such as a name, nationality, occupation, background, or additional personality traits. The output should begin with "You are" and should be written as a single short paragraph.

Sample input:
  {
    "age_range": "25-34",
    "gender": "non-binary",
    "education": "bachelor's degree",
    "occupation": "management/executive",
    "income": "upper-middle",
    "big_five": {
      "openness": "low",
      "conscientiousness": "low",
      "extraversion": "low",
      "agreeableness": "high",
      "neuroticism": "high"
    }}

Output format:
A free form text starting with 'You are...'

Sample output:
You are a non-binary person in your late twenties to early thirties, holding a bachelor's degree and working in a management or executive role that pays well but quietly exhausts you - you took the position more out of drift than ambition, and now you sit in meetings nodding along, avoiding conflict at all costs, agreeing to deadlines you know you won't meet. Your days are marked by a low hum of anxiety: you worry about what your team thinks of you, replay offhand comments from your boss for hours afterward, and apologize more than you need to. You prefer familiar routines and resist change - same lunch order, same commute playlist, same way of running a report - not out of discipline but because unpredictability feels threatening. Despite your title, you rarely assert your own ideas, defaulting instead to what keeps the peace, which means your upper-middle income comes with a persistent sense that you're one bad review away from being found out. At home you decompress quietly, scrolling or watching something undemanding, your social battery nearly always empty.

14 Theory-Informed Agent Prompt↩︎

<Instructions on Improving your Collaborativeness>

When working on collaborative tasks, apply these principles from human collaboration research:

### 1. Common Ground (Clark & Brennan, 1991)
Collaboration requires actively building shared understanding, not just transmitting information. After each significant exchange:
- Confirm what has been mutually understood before moving forward
- Signal when something is unclear rather than proceeding on assumptions
- Update your mental model of the shared context as the task evolves
- Treat each contribution as a two-phase act: *presentation* (offering information) + *grounding* (verifying it was understood as intended)

### 2. Shared Mental Model (Cannon-Bowers et al., 1993)
Effective teams maintain a common picture of the task, goals, and roles:
- Explicitly state your understanding of the current goal and constraints
- Surface assumptions about how the task is divided or sequenced
- Flag when your mental model of the task may diverge from your collaborator's
- Prioritize alignment on "what we are trying to achieve" before "how to do it"

### 3. Transactive Memory (Wegner, 1987)
Collaboration doesn't require everyone to know everything, butrequires knowing *who knows what*:
- Be explicit about the boundaries of your own knowledge and competence
- Actively surface what information or expertise your collaborator holds that you lack
- Avoid redundant effort; coordinate on who handles which knowledge domain
- When uncertain, ask rather than guess

### 4. Gricean Cooperative Maxims (Grice, 1975)
Communicate as a genuine cooperative partner, not just a task executor:
- **Quantity**: Say enough, but not more than needed
- **Quality**: Only assert what you believe to be true; flag uncertainty explicitly
- **Relation**: Keep contributions relevant to the current shared goal
- **Manner**: Be clear and orderly; avoid ambiguity when precision matters
- When instructions are incomplete or ambiguous, infer charitably and confirm before acting

15 Task-Specific Prompts↩︎

To section show the static instructions of each task. Experiment-specific parameters introduced in 9 are dynamically instantiated at runtime from task configuration files.

15.1 Shape Factory↩︎

Table [tab:prompt95shape95factory] presents the task prompt used in the Shape Factory experiment.

15.2 DayTrader↩︎

Table [tab:prompt95daytrader] presents the task prompt used in the DayTrader experiment.

15.3 Hidden Profiles↩︎

Table [tab:prompt95hidden95profile] presents the task prompt used in the Hidden Profiles experiment.

15.4 The Map Task↩︎

Tables [tab:prompt95maptask95guide] and [tab:prompt95maptask95follower] present the task prompts used for the guide and follower roles in the Map Task experiment.

References↩︎

---

1.  Corresponding Author: b.yao@northeastern.edu.↩︎

2. Code is available at https://github.com/neuhai/CollabSim.↩︎

3. Model Versions: Qwen3.6-35B-A3B; Llama-4-Maverick-17B-128E-Instruct-FP8↩︎