1 Introduction↩︎

2pt 3pt

Recent multimodal large language models (MLLMs) [10]–[12] have demonstrated strong vision-language understanding and reasoning capabilities, making them promising foundations for embodied agents in interactive environments [13], [14]. While most existing MLLM-based studies focus on single-agent embodied intelligence [15]–[17], many real-world tasks such as household assistance require multiple agents to cooperate [18], as shown in Figure 1. Previous works on pure language-based multi-agent framework [19], [20] have proved that such cooperation could improve efficiency and overcome individual capability limits. However, the potential of multi-agent collaboration under multimodal embodied settings remains an underexplored area.

Answering this is non-trivial, because embodied multi-agent collaboration differs fundamentally from text-based collaboration. Existing multi-agent benchmarks rely on textual task descriptions, shared symbolic states, or pre-processed observations, reducing collaboration to plan coordination or dialogue [2], [7], [9], [21], [22]. By contrast, embodied agents must coordinate from partial, dynamically changing visual observations while exploring environments, resolving perceptual inconsistencies, avoiding spatial conflicts, and acting under physical constraints [23]. This tightly couples perception, exploration, communication, and coordination, leading to failure modes and emergent behaviors that text-based benchmarks cannot capture [24]. Therefore, it is valuable to systematically study MLLM-based multi-agent collaboration in embodied settings, calling for a new benchmark that supports visually grounded interaction and diverse cooperation patterns.

To address this gap, we introduce MECoBench, a multimodal embodied cooperation benchmark together with an evaluation platform for systematically analyzing visually grounded embodied multi-agent cooperation. MECoBench contains diverse embodied tasks covering eight types of common real-world activities, and introduces two cooperation structures to support realistic evaluation. In parallel cooperation, agents can work on different subtasks simultaneously, whereas in space-constrained sequential cooperation, agents must coordinate in a chained manner to complete the task. Beyond task construction, our platform supports controlled variation along multiple dimensions, including collaboration mode, team size and exploration difficulty. This design enables fine-grained analysis of not only whether collaboration helps, but also when, why, and through which mechanisms it improves embodied task solving.

Based on MECoBench, our work aims to answer three research questions progressively: (i) To what extent does multi-agent collaboration benefit MLLMs on embodied tasks, and how does this benefit change as the team size scales? (ii) How do different collaboration modes and communication mechanisms shape collaboration outcomes? (iii) Does collaboration remain effective and robust under challenging information conditions? To answer these questions, we evaluate a wide range of MLLMs across model families and parameter scales, and obtain the following findings:

• Multi-agent collaboration generally improves embodied task completion, but its benefits depend on balancing gains against coordination complexity. Most MLLMs exhibit basic collaborative capability and benefit from a second agent participation. As the team grows further, however, performance saturates and then degrades, forming an inverted-U trend and indicating that moderate team sizes provide the best trade-off between parallelism and coordination overhead. Encouragingly, larger teams remain more robust as task complexity increases.

• Communication is the key driver of collaboration gains, and the best mode is contingent on both team size and model capability. Removing communication consistently degrades performance, with substantially larger drops on sequential tasks and larger teams. Centralized coordination is most effective in small teams but saturates earlier than decentralized coordination as the team scales. Beyond textual messaging, shared memory yields more efficient collaboration and benefits loosely coupled parallel tasks, while vision-augmented leadership improves coordination efficiency—provided the leader model is capable enough to leverage visual input.

• Collaboration further enhances robustness under challenging information conditions. Two-agent collaboration still improves task completion when location priors are removed and agents must explore from scratch, and yields its largest relative gains under noisy priors, showing that multi-agent teams can compensate for misleading information through communication and distributed exploration. Available task information further amplifies the benefits of collaboration.

2 MECoBench↩︎

To investigate MLLM-based multi-agent collaboration under embodied settings, we propose MECoBench, a comprehensive benchmark with an interactive evaluation platform. MECoBench is built upon VirtualHome [25], a realistic household simulator covering a wide range of everyday scenarios, and spans different collaboration structures, variable team sizes, diverse collaboration modes and communication mechanisms. Table 1 compares MECoBench with existing embodied multi-agent collaboration benchmarks.

2.1 Benchmark Construction↩︎

To cover common real-world activities, we define eight semantic task templates (detailed definition in Table 4), including five adapted from WAH [26] and three newly designed. Beyond semantic diversity, we introduce two collaboration structure to capture different forms of spatial accessibility in the real world. In the parallel setting, all agents can freely access all rooms and complete subgoals independently and concurrently, modeling cases where individuals operate in a shared space. In the sequential setting, agents are assigned to disjoint room zones, making inter-agent object transfers necessary and reflecting situations where individuals have access to different regions and must coordinate across spatial boundaries.

The final task is constructed through a two-stage pipeline shown in Figure 2. In the scene grounding stage, we firstly sample subgoals from a task template, and then randomly place the selected goal objects at legal initial locations. We define a broad range of surfaces and containers to enable diverse and realistic object distributions (listed in Table 5). After a task scene is instantiated, we further assign its collaboration structure. Under sequential settings, the apartment is partitioned into disjoint room zones based on the initial locations of task objects and their target goal locations, and different zones are assigned to different agents.

Overall, MECoBench consists of 96 tasks, each evaluated under both parallel and sequential setups, yielding 192 test cases in total. Each task contains 2–7 subgoals. The scenes are uniformly distributed over both the number of subgoals and the eight task templates to ensure balanced task complexity and scenario diversity. Details of construction and statistic are in Appendix 9.

2.2 Evaluation Platform↩︎

The platform supports a flexible number of agents and provides three collaboration modes with four basic protocols to enable systematic evaluation.

2.2.0.1 Collaboration Mode

To analyze how different coordination structures shape multi-agent performance, we design three collaboration modes implemented with four protocols as summarized in Figure 3. The isolated mode disables communication, requiring agents to act independently. The decentralized mode enables peer-level coordination through broadcast or discussion. In broadcast, each agent shares one message with all others before action; in discussion, agents communicate sequentially and can continue for another round when consensus is not reached. The centralized mode assigns one agent as leader to aggregate worker reports and assign actions to the team.

2.2.0.2 Evaluation Pipeline

As illustrated in Figure 4, we formulate the evaluation as an iterative observe–communicate–act workflow. At the beginning of each task, the environment is initialized with the task scene graph and \(N\) agents. Each agent is placed in its assigned initial room with the provided task goal \(G\), the available prior information \(I\), and the shared action space \(A\) (see Appendix 10.3). At each timestep \(t\), each agent \(i\) obtains a panoramic environment observation \(O_t^i\) from its first-person view. To support long-horizon execution over dozens of steps, the agent reasons not only over its current observation \(O_t^i\), but also over its action history \(h_{a,t}^i\), recent dialogue history \(h_{d,t}\), self-maintained memory \(M_t^i\), and initial information \(I\). Depending on the chosen collaboration protocol \(P\), each agent \(i\) first generates the communication message \(c_t^i\) as \[c_t^i = P_i(O_t^i, h_{a,t}^i, h_{d,t}, M_t^i, G, I).\] The messages from all agents form the communication context \(C_t = \{c_t^1, c_t^2, \ldots, c_t^N\}\). After communication, each agent generates its next action conditioned on the current communication context: \[a_t^i = \pi_i(O_t^i, C_t, h_{a,t}^i, M_t, G, I, A),\] where \(\pi_i\) denotes the action policy of agent \(i\). The selected actions are executed in simulator, which returns execution feedback \(F_t\) and resulting environment changes. Finally, the feedback is used to update the history and memory. This workflow continues until the task is completed or the maximum number of steps is reached.

3 Experiment Setup↩︎

3.1 Implementation Details↩︎

For parallel tasks, we compare single-agent and two-agent execution under the same effective 60-step budget, defined as the total number of steps across all agents. Thus, for an n-agent team, each agent is allocated 60/n steps. For sequential tasks, we use a fixed two-agent setup with an 80-step budget. Unless otherwise specified, we use the broadcast collaboration protocol in all multi-agent settings.

By default, we provide the task object appearance and possible location list to agents as the prior information \(I\) at initialization. Since MECoBench focuses on collaboration rather than object search, this prior reduces localization difficulty and better isolates cooperative ability. More detials are in Appendix 10.1.

3.2 Evaluation Metrics↩︎

We evaluate multi-agent collaboration from three aspects: (1) Effectiveness: Success Rate (SR), defined as the proportion of successful cases, and Completion Rate (CR), defined as the average fraction of completed subgoals across cases. (2) Efficiency: Step-CR AUC (AUC), measuring the area under the completion-rate curve over equivalent steps, and Average Token Cost per Step. (3) Collaboration quality: Division of Labor (DOL), Conflict Action Rate (CAR), and Handover Failure Rate (HFR). DOL measures the distribution of subgoal completion across agents, calculated as \[s_i=\frac{\mathrm{count}_i}{\sum_j \mathrm{count}_j}, \quad \mathrm{DOL}=\frac{1-\sum_i s_i^2}{1-\frac{1}{n}}.\] where \(n>1\) is the number of agents, and \(s_i\) denotes the fraction of subgoals completed by agent \(i\) among all completed subgoals. CAR measures the average fraction of conflicting actions between agents, such as grabbing or opening the same object. HFR measures the average fraction of failed handovers in sequential tasks.

3.3 Evaluated Models↩︎

We evaluate a diverse set of both closed-source and open-source models. The closed-source models include state-of-art model GPT-5.4 [11], Gemini-3.1-Pro [12], and GPT-5-mini [27] The open-source models include Qwen3-VL [28], Qwen3.5 [29], Gemma4 [30], InternVL 3.5 [31], Llama4 [32] and GLM 4.6V [33], covering model sizes from 8B to 235B parameters. Appendix 10.2 lists all models evaluated.

4 Will Multi-agent Collaboration Benefit Embodied Task Completion?↩︎

4.1 Do Models Know How To Collaborate?↩︎

4.1.0.1 Can models collaborate when collaboration is necessary?

We compare the model performance under single- and two-agent settings, using parallel single-agent and sequential two-agent results as proxies for individual and collaborative ability, respectively. As demonstrated in Figure 5, single-agent performance on parallel tasks correlates positively with two-agent collaboration performance on sequential tasks, while the sequential success rates are generally lower. This suggests that spatial constraints make collaboration harder than individual execution, while most models still exhibit basic collaborative capability. Among these models, GPT-5.4 and GPT-5-mini achieve the best overall performance, and Gemma4 series performs particularly well on collaboration tasks. In contrast, InternVL3.5 performs well individually but struggles to cooperate, revealing a gap between individual execution and group collaboration.

4.1.0.2 Can models benefit from collaboration when it is optional?

Compared with single agent, Figure 6 shows that most models benefit from the participation of a second agent. The improvement is particularly large for weak and mid-level models, likely because their single-agent baselines leave more room for gains. Notably, InternVL3.5 series exhibits a clear performance drop, further confirming its weakness in multi-agent coordination. Detailed results and analyses are provided in Appendix 12.1.

4.2 Do More Agents Always Help?↩︎

To investigate this, we increase the number of agents from one to five under parallel settings with fixed effective step budget. Results are shown in Figure 7 (a), where collaboration performance generally follows an inverted-U-shaped trend: adding agents initially improves performance, but further scaling leads to saturation and then degradation. Overall, moderate team sizes strike the best balance between collaboration benefits and costs.

We further analyze the effect of task complexity using object count as a proxy. Figure 7 (b) shows that success rates decline with increasing object count, due to longer trajectories and greater planning difficulty (see Appendix 12.6 for full results across all models and task conditions). Nevertheless, multi-agent settings maintain more robust performance under high-complexity tasks, highlighting the value of collaboration in complex embodied scenarios.

4.3 How Do Agents Collaborate?↩︎

We delve deeper into fine-grained collaborative behaviors to understand how agents collaborate in practice. These behaviors are identified from the agents’ communication and action trajectories using keyword-based matching rules, and are organized into three categories, as reported in Table 2. Examples are provided in Appendix 13.1.

4.3.0.1 Information Sharing

Agents share their intended plans (Info-Plan), current locations (Info-Loc), and detected task-relevant objects (Info-Obj). Plan and location sharing are frequent and consistently beneficial, while object discovery reporting shows mixed effects.

4.3.0.2 Task Coordination

Action-level delegation (Act-Del) issues concrete executable requests, such as asking another agent to grab, check, or place, and brings the strongest gains in both parallel and sequential tasks. In contrast, task assignment (Task-Assign) provides coarser responsibility divisions, such as assigning rooms, areas, or subtasks, and offers only marginal benefit in parallel tasks while hurting sequential ones. More specific and actionable communication is easier to translate into successful execution.

4.3.0.3 Alignment and Correction

Coordination query (Coord-Q.) refers to questions that synchronize teammates’ states or locations. It occurs more often in sequential tasks where object transfer requires repeated position checks and serves as an indicator of task and coordination difficulty. Correct. denotes behavioral correction through self-reflection or peer feedback. Despite its negative SR correlation, it consistently improves CR in both settings, suggesting that correction serves as an error-recovery mechanism that improves task progress even when full success is unrecoverable. More analysis provided in Appendix 13.1.

3pt

5 What Makes Multi-agent Collaboration Effective?↩︎

We answer this question along two dimensions: collaboration mode and communication mechanism. We first examine which collaboration mode is most effective. We then study the role of communication by testing whether it is necessary and whether alternative mechanisms further improve collaboration.

5.1 Which Collaboration Mode Works Best?↩︎

5.1.0.1 Two-Agent Collaboration.

We compare different collaboration modes beyond basic broadcast mechanism. As shown in Figure 8 (a), leader-based collaboration achieves the best overall performance for most models, suggesting that centralized collaboration helps small teams allocate tasks and avoid spatial conflicts. However, GPT-5.4 remains competitive with simple broadcast communication, indicating that stronger models may rely less on structured collaboration mode.

5.1.0.2 Scaling to Larger Teams.

We further examine how collaboration modes scale with team size on parallel tasks. Figure 8 (b) illustrates that centralized coordination is effective in small teams but saturates as team size increases, while decentralized coordination becomes more competitive by avoiding the bottleneck of relying on a single leader to aggregate information and assign actions. Moreover, under centralized coordination, model capability matters, as Qwen3-32B-VL benefits from three agents, whereas Qwen3-8B-VL peaks at two agents and then degrades. Therefore, the best collaboration mode depends on both team size and model capability. Full results across all metrics and team sizes are provided in Appendix 12.3.

5.2 Is Communication Necessary?↩︎

We ablate communication from the broadcast protocol under both parallel and sequential settings. As shown in Figure 9, disabling communication consistently degrades performance across models, with a much larger drop on sequential tasks. When the number of agents grows, the gap between communication and no communication widens. These results indicate that communication is essential for effective collaboration, especially when tasks require stronger coordination or involve larger teams. More details in Appendix 12.2.

5.3 Is Textual Communication Enough?↩︎

While explicit textual communication is essential, it may be insufficient for embodied collaboration, where agents must share evolving task states and visually grounded observations. We therefore explore two alternative communication mechanisms: shared memory and vision-augmented leadership.

5.3.0.1 Shared memory leads to efficient and effective collaboration.

In shared memory, agents skip explicit message exchange and coordinate through a shared memory block. As shown in Figure 10, this mechanism improves the performance and greatly reduces token consumption, indicating more efficient collaboration (absolute results in Figure 31). It consistently improves success rate on parallel tasks, but degrades on sequential tasks, where precise timing and spatial alignment may still require explicit communication. Nevertheless, shared memory accelerates early progress, reducing more conflicts than broadcast under smaller step budgets.

5.3.0.2 Vision-augmented leadership helps when the leader is capable.

To give the leader more grounded context, in vision-augmented leader-worker collaboration, we allow workers to send both textual reports and current visual observations to the leader. As shown in Figure 11, additional visual information improves AUC for both models, while its impact on SR and CR varies across models. This difference may stem from models’ different abilities to effectively use visual cues. Models with stronger visual capabilities (Gemma4-31B), benefit from the additional visual input, whereas weaker models (Qwen3-32B-VL) drop.

6 Is Multi-Agent Collaboration Robust?↩︎

We examine robustness from two perspectives: the sensitivity of collaboration to imperfect prior information, and the failure modes that arise under collaboration. Additional prior information ablations and mixed-model team experiments are deferred to Appendix 12.4 and 12.5.0.2.

6.1 Does Collaboration Remain Effective Under Imperfect Information?↩︎

At initialization, we provide all task-relevant object locations in the prior information \(I\), which is hard to obtain in real scenarios. Therefore, we further investigate the collaboration under two realistic settings: (1) Missing location priors: object locations are removed from the prior information, requiring agents to find targets through visual search. (2) Noisy location priors: an additional 30% false object locations are injected into the provided prior information. We extend the effective step budget to 80 steps to allow sufficient search, and apply parallel settings with basic broadcast protocol.

As shown in Figure 12, two-agent collaboration improves the success rate and reduces the conflicts, indicating that collaboration remains useful when task information is incomplete. When the number of agents is further increased, a similar inverted-U-shaped trend emerges, as shown in Figure 13 (a), while performance remains more stable on complex tasks. Moreover, Table 3 demonstrates that collaboration achieves the largest relative performance gains when the information contains noise, suggesting that agents can compensate for misleading information through communication and distributed exploration. These results together validate that multi-agent collaboration is robust to insufficient and noisy information.

3pt

6.2 Where Does Collaboration Break Down?↩︎

We identify two major failure modes in multi-agent collaboration. These failures suggest that robust multi-agent embodied collaboration requires conflict resolution and belief verification mechanisms. Examples and more statistics in Appendix 14.

Multiple agents can introduce conflicts, including duplicate grabs, repeated labor, and redundant exploration. Although communication mitigates these conflicts for most models, InternVL3.5-241B still exhibits the highest duplicate-grab rates among 2-agent setup of 21.9%, partly explaining its performance drop in Figure 6. This also explains why performance drop with large team size: for Qwen3-32B-VL, scaling to five agents increases duplicate grabs to 57.4%.

Hallucinated beliefs can propagate across agents. When one agent falsely believes the task is complete and broadcasts it, the team may stop prematurely. Although rare (1%-1.5%), such hallucinated completion is highly destructive, causing a 73.7% SR drop in parallel tasks and a 40.9% drop in sequential tasks. Object confusion can similarly cause agents to skip unfinished subtasks.

7 Related Work↩︎

7.0.0.1 Multi-Agent System

Inspired by human society, multi-agent systems (MAS) have been studied as a way to improve task efficiency and effectiveness [34]. Recent work has explored LLM-based MAS across diverse domains, including automated research [19], [35], programming [20], [36], medical decision-making [37], [38], and social simulation [39]. Other studies further examine their scalability and emergent collaborative behaviors [40], [41]. Although recent work has begun to explore embodied multi-agent scenarios [42], [43], multimodal embodied collaboration remains underexplored [44].

7.0.0.2 Embodied Agent

Recent studies have explored LLM- or MLLM-based embodied agents for a wide range of tasks, including object manipulation [45], [46], navigation [47], complex perception-interaction tasks [48], and embodied human-AI collaboration [5]. Several benchmarks have also been proposed to evaluate embodied agents across diverse environments and capabilities [15], [49]–[51]. However, these benchmarks predominantly focus on single-agent settings, providing limited support for studying collaboration among multiple embodied agents.

7.0.0.3 Embodied Multi-Agent Collaboration Benchmarks

Several benchmarks have been proposed to evaluate embodied multi-agent collaboration [1], [22], [26], [52], [53]. Many are built on 2D games [7], [9], offering limited support for physical interaction and spatial reasoning. Existing 3D benchmarks often convert visual observations into textual descriptions [2]–[5], [8], decoupling visual grounding and exploration from collaborative decision-making and overlooking behaviors unique to multimodal embodied interaction. Some recent benchmarks support multimodal observations [52], [54], [55], but mainly evaluate visual question answering or offline reasoning rather than closed-loop interaction. TeamCraft [6] takes an initial step toward evaluating embodied MLLM collaboration, but does not systematically study how communication mechanisms, coordination modes, and team size shape collaboration.

8 Conclusion↩︎

We empirically investigate how MLLM-based agents collaborate in embodied environments by constructing MECoBench, a multimodal embodied cooperation benchmark. Through controlled experiments and analyses, our results demonstrate that effective coordination can substantially improve both task performance and efficiency. By systematically examining collaboration modes and communication mechanisms, we reveal how these factors shape collaborative performance. We further evaluate collaboration under noisy and incomplete prior information, demonstrating its robustness in more challenging settings. Beyond aggregate performance, we identify emergent collaborative behaviors and failure patterns, explaining both the benefits of collaboration and the new challenges it introduces. Overall, our study provides a deeper understanding of MLLM-based embodied collaboration and introduces a benchmark for evaluating the collaborative capabilities of future MLLMs.

Limitations↩︎

In this work, we propose MECoBench and conduct systematic experiments to study cooperative behaviors of MLLM-based agents in embodied environments. Our analysis reveals several non-trivial patterns in multi-agent collaboration, but there remain several limitations for future work.

8.0.0.1 Expanding tasks and scenarios.

MECoBench and corresponding evaluation platform is currently built on VirtualHome, an indoor simulator focused on household scenarios. As a result, our tasks are mainly limited to everyday indoor activities and object interactions. Future work can extend the benchmark to more diverse environments, such as open-world or outdoor scenes, and incorporate broader task types such as navigation and long-horizon exploration.

8.0.0.2 Data scale.

MECoBench contains 96 cases from 8 task types under two major task setups. This scale is sufficient for controlled evaluation and for revealing key collaboration patterns, but it remains relatively small compared with the diversity and complexity of real-world embodied tasks. Future work can expand the number of task instances, object categories, scene layouts, and collaboration requirements.

8.0.0.3 Agent scaling.

Due to spatial constraints in the environment and the complexity of task execution, our experiments scale the team size up to 5 agents. Larger teams and more complex tasks are therefore left unexplored.

8pt

9 MECoBench↩︎

This section presents detailed information about MECoBench, covering its task definitions, benchmark statistics, and construction details.

9.1 Task definition↩︎

Table 4 shows the detailed templates of the eight task types designed for MECoBench. Self-caring, work-on-desk, and gaming setup are three newly introduced types, while prepare afternoon tea, wash dishes, prepare food, put food inside the fridge, and setup dinner table are modified from WAH. Figure 14 gives an example of final task goal of gaming setup.

9.2 Benchmark construction details↩︎

The construction of MECoBench tasks follows a two-step pipeline. In step one, task templates are instantiated into concrete scenes with randomized object placement and validated for feasibility.

9.2.0.1 Goal Sampling.

For a given task type, we sample subgoals from predefined object-count ranges, specifying the target objects, required counts, and optional room or surface constraints. To mitigate visibility issues such as occlusion, we add 0–2 redundant instances for each target object category during scene instantiation. For large objects that occupy more surface space, such as keyboards, boardgames, and magazines, the redundancy is limited to 0–1 instance.

9.2.0.2 Scene Instantiation and Object Placement.

Once a template is sampled, the target objects are assigned to legal initial locations based on the predefined legal placement (Table 5). Objects are placed randomly while respecting constraints:

• No object is placed on the same surfaces or containers that are reserved for other goal locations.

• Surface space is checked to ensure sufficient area for the object.

• Containers are verified for available slots using their bounding boxes.

This procedure ensures that the initial scene is diverse yet consistent with the task template.

4pt

9.2.0.3 Validation and Goal Specification.

After placement, the VirtualHome environment is used to verify scene feasibility:

• The environment graph is updated to reflect the actual object positions.

• Target object counts are checked to ensure that all task objects exist in the scene.

• Predicates from the template are instantiated with the real node IDs in the scene.

• Tasks failing these checks (e.g., insufficient instances or invalid placements) are discarded, and a new attempt is generated.

9.3 Benchmark Statistic↩︎

MECoBench involves 25 common everyday objects, which are grouped into six categories: food and drinks, tableware, electronics, hygiene items, entertainment items, and office items. Figure 15 illustrates the correspondence between the eight task types and their associated object categories.

Figure 16 shows the data distribution of MECoBench across different task types and task complexity levels, where task complexity is measured by the number of involved objects.

10 Evaluation↩︎

10.1 Evaluation Setup↩︎

Each observation view is rendered at \(256 \times 256\) pixels, resulting in a four-view concatenated observation of \(256 \times 1024\). The four views are arranged in the order: front, back, left, and right. Both horizontal and vertical fields of view are set to \(90^\circ\). The agent model height is 1.8 m, and the camera is mounted at the same height with a forward offset of 0.15 m relative to the agent’s position. The camera is pitched downward by \(30^\circ\) to capture more of the nearby surfaces and objects. Figure 17 shows an example of the resulting concatenated observation.

10pt

For all evaluated models, we adopt the recommended generation configurations whenever available. For closed-source models, we use the official APIs with their default generation settings. For open-source models, we follow the generation configurations specified in the corresponding model cards or vLLM [56] documentation to ensure strong and reliable performance.

10.2 Model Versions↩︎

Table 6 summarizes the model versions and full identifiers used in our experiments. Proprietary models are accessed through their official APIs, while open-source models are locally deployed using vLLM [56].

10.3 Action Space↩︎

Table 7 lists the action space for agents when evaluation for both parallel and sequential tasks. Parallel tasks include basic navigation and object manipulation actions, while sequential tasks introduce cross-room handover actions.

10.3.0.1 Single-Agent Action Legality

Each agent action must satisfy the following legality constraints:

• Hand occupancy: Each agent can hold at most two objects. When both hands are occupied, the agent cannot execute grab or open. For put_* and handover, the object to be placed or handed over must already be held by the acting agent.

• Visibility: The target objects of walk, grab, and put_* actions must be visible in the agent’s current observation.

• Container state: An agent cannot execute open on a container that is already open.

• Room accessibility: For walk_*, grab, and put_* actions, the target must be located in one of the rooms accessible to the agent.

• Invalid actions: Any action name not listed in Table 7 is considered invalid.

10.3.0.2 Multi-Agent Interaction Constraints

When multiple agents act simultaneously, additional interaction constraints are applied:

• Conflict resolution: The conflict set includes grab and open. If multiple agents attempt the same conflict action on the same object at the same time, one agent is randomly selected as the winner, while the others remain idle.

• Handover distance: A handover action is valid only when the giver and receiver are within 2 meters of each other.

10.4 Action Decoding and Grounding↩︎

Agents output semantic actions based on their observation rather than executable simulator commands. Thus, each generated action is decoded and grounded before execution. Actions that do not require object grounding, such as receive, walk_to_room/door, and handover, are resolved by rule-based mappings. For object-centric actions, including walk, grab, open, put_on, and put_in, semantic object descriptions are grounded to concrete simulator object identifiers.

As shown in Fig. 18, the top panels present the agent’s first-person observation and generated semantic action. The decoder then uses visible object candidates, their bounding boxes, and metadata, together with both the original and bbox-annotated views, to select the target simulator ID and produce the final executable action.

11 Core Prompt Templates↩︎

This section presents the core prompt templates designed for agents in the MECoBench evaluation platform.

11.1 Communicate Phase↩︎

Below we list prompts for broadcast, discuss and leader-based communication protocol (Figure 19, 20, 21, 22, 23).

11.2 Act Phase↩︎

This section presents the prompts used for action decision-making, memory update and action decoding (Figure 24, 25, 26, 27, 28, 29).

12 Experiment Results↩︎

In this section, we provide detailed results for the experiments in Sections 4, 5, and [sec:exp-3], as well as some additional experimental results.

12.1 Performance of Different Models in Parallel and Sequential Tasks↩︎

Table 8 shows the detailed result of figure 5 and 6. Overall, GPT-5.4 achieves the best performance. GPT-5-mini and Gemma4-31B are the second-tier models, showing consistently strong results across the two collaboration structures. In addition, Qwen3.5, Gemma4-26B, and Gemini-3.1-Pro also exhibit relatively strong performance. Qwen3-VL-32B and Qwen3-VL-235B fall into the middle-performance group, with relatively balanced results across parallel and sequential tasks. Qwen3-VL-8B is a relatively weak model, but its performance remains comparatively balanced between the two coordination structures. In contrast, InternVL-3.5 and GLM-4.6V struggle particularly on sequential tasks, where precise collaboration is required. Finally, Llama-4-Scout and GLM-4.6V-Flash perform poorly under both collaboration structures.

12.1.0.1 Model scaling effects.

Overall, except for InternVL-3.5, larger models generally perform better within the same family. However, scaling tends to amplify existing task-specific patterns rather than change them. Some families remain balanced across parallel and sequential tasks, while others are consistently stronger in single-agent or parallel settings, or particularly weak on sequential tasks requiring precise coordination. For example, Gemma4 benefits from scaling especially on sequential tasks, whereas InternVL-3.5 achieves stronger single-agent parallel performance at larger scale but degrades in sequential collaboration. This suggests that stronger individual capability does not necessarily translate into better communication and coordination.

Based on model performance and cost-effectiveness, we select Qwen3-VL-32B and Qwen3-VL-8B as two representative models for studying team-size scaling and related factors in subsequent experiments. For strong models, we conduct most experiments with GPT-5-mini and Gemma4-31B, and include GPT-5.4 when necessary.

4pt

3pt

3pt

12.1.0.2 Team size scaling effects.

Table 9 shows the detailed result for Figure 7 (a) and 9 (b). The results show that: (1) Simply increasing the number of agents does not necessarily improve performance. Without communication, adding more agents can even introduce disorder and redundant actions, leading to lower efficiency and worse task completion. However, when agents are allowed to communicate, multi-agent collaboration becomes more effective in coordinating actions and completing tasks. (2) More agents are not always better. Qwen3-VL-32B benefits from additional agents up to four agents, while Qwen3-VL-8B reaches its best performance with three agents and drops afterward. This suggests that weaker models are more affected by coordination overhead. (3) Collaboration can partially compensate for limited individual model capability, but cannot fully remove the capability gap. With communication, three Qwen3-VL-8B agents approach the single-agent performance of Qwen3-VL-32B, but the final performance remains constrained by the underlying model capability. Table 10 shows the detailed results in figure 7 (b). This indicate that multi-agent collaboration is especially helpful for more complex tasks with more target objects. For 7-object tasks, the SR nearly doubles for both models: from 0.385 to 0.750 for Qwen3-VL-32B, and from 0.077 to 0.154 for Qwen3-VL-8B. A task progress comprison of different team size shown in Figure 30.

12.2 Role of communication↩︎

Table 11 shows the detailed results in figure 9 (a) when ablating communication from collaboration. Removing communication degrades both task performance and collaboration quality. In parallel tasks with a shared workspace, it generally increases CAR, indicating more conflicts actions between agents. In sequential tasks, the impact is more severe, as agents fail to coordinate cross-region handovers, causing HFR to rise substantially.

12.3 Comparison of Different Collaboration Modes↩︎

3pt

Table 12 gives the detailed results of Figure 8 (a). The results show that centralized collaboration generally improves division of labor and task efficiency, reflected by higher DOL and AUC. In contrast, the discuss protocol is more effective at reducing conflicts, leading to lower CAR. For weaker models, centralized collaboration also helps reduce spatial cognition errors in sequential tasks, suggesting that leader-based coordination can provide useful guidance when agents have limited planning or spatial reasoning ability. Figure 32 shows the comparison of task progress under single agent and different communication protocol in two-agent team.

12.3.0.1 Shared Memory.

Figure 31 reports the absolute values of success rate, AUC, and token consumption for broadcast and shared memory, providing a more detailed version of Figure 10.

3pt

3pt

12.4 Mix Team: Strong Model as Leader.↩︎

We further investigate the role of leader in centralized mode. As shown in Figure [fig:strong-leader], using a stronger leader improves the mixed team, but does not make it approach the all-strong setting. The mixed team nearly matches the all-strong team in CR, but its SR is only slightly above the average of the all-weak and all-strong teams, and its AUC is even below this average. This indicates that a strong leader mainly improves task coverage, but efficient and overall success still depends heavily on the worker’s capability.

12.5 Performance Under Imperfect Information.↩︎

12.5.0.1 Missing location priors

Table 14 reports detailed results under the no-prior-location setting on parallel tasks across different models. Figure 33 shows the team size scaling effect under stronger exploration demands, where prior location information is removed. Compared with the full-information setting, Qwen3-32B-VL exhibits a similar but weaker inverted-U-shaped trend, suggesting that moderate team scaling remains helpful. For Qwen3-8B-VL, SR remains low across team sizes, indicating a capability bottleneck. Moreover, when grouping cases by object count, larger teams still show a milder performance drop. More detailed results in Table 15.

4pt

12.5.0.2 Ablation on prior information

To investigate how different types of prior information affect agent performance, we conduct an ablation study by removing object appearance descriptions and prior location information. As shown in figure 34, prior location information is critical for task completion, as it helps agents quickly locate target objects and reduces inefficient exploration. Removing location information leads to a substantial performance drop across all models. In contrast, the effect of appearance descriptions is model-dependent: GPT-5-mini benefits from appearance cues, Gemma4-31B shows only minor changes, and Qwen3-32B-VL performs worse with appearance-only priors than with no prior. A possible reason is that models with weaker fine-grained visual grounding are more susceptible to misleading appearance descriptions and may confuse visually similar objects. Detailed results are reported in Table 13.

4pt

12.5.0.3 Noisy location priors.

Table 16 reports detailed results on parallel tasks where 30% of the prior location information is noisy, comparing single-agent and two-agent settings.

12.6 Additional Performance Analysis↩︎

Figure 35 reports performance by the number of target objects across model groups. As each target object corresponds to a subgoal, a larger number of target objects indicates higher task complexity. Overall, both completion rate and success rate decline as task complexity increases.

13 Emergent collaborative behavior↩︎

By analyzing the communication and action trajectories under the basic broadcast protocol, we identify several emergent collaborative behavior patterns. In addition to the analysis in Section 4.3, this section provides representative examples and further details.

5pt

13.1 Examples↩︎

Table ¿tbl:tab:collab95behavior95examples? presents representative examples of collaborative behaviors, with the relevant text spans highlighted in yellow.

13.2 Detailed Analysis↩︎

3.2pt

3.2pt

13.2.0.1 Coordination Queries Indicate Collaboration Challenges.

In the main text (Table 2), coordination query correlates negatively with SR and CR. To further investigate the underlying cause, we introduce two behavioral indicators of collaboration difficulty: negative report (Neg.), where an agent reports failing to locate a target object at a specific location, and stuck, where an agent reports being stuck in middle. As shown in Table 17, cases with coordination query involve more objects on average and exhibit substantially higher stuck and negative report rate. This pattern holds across both task types, suggesting that coordination query serves as a reactive signal to genuine collaboration obstacles.

2.6pt

6pt

13.2.0.2 Correction Helps Recover from Errors

Section 4.3 shows that correction correlates negatively with SR but positively with CR, suggesting a potential role in error recovery. To examine this effect, we analyze its co-occurrence with wrong grabs (WGs), defined as actions that grasp task-irrelevant objects. Table 18 shows that episodes with correction contain substantially more WGs than those without, indicating that correction is mainly triggered by execution errors. Since WG episodes with correction also exhibit greater error severity, direct comparison may underestimate its recovery benefit. We therefore stratify WG episodes by error count and compute a weighted adjusted effect using shared stratum weights for both groups (Table 19). After adjustment, correction consistently improves both SR and CR across task types, confirming its effectiveness in error recovery. Stratum-level results further show larger gains under higher WG counts, suggesting that correction is especially useful when errors are more severe.

14 Failure Mode Analysis↩︎

In this section, we provide additional examples and a more detailed analysis of the multi-agent collaboration failure modes presented in Section 6.2.

14.1 Examples↩︎

Table 20 presents representative examples of different failure modes.

14.2 Detailed Analysis↩︎

2pt 3pt

Table 21 summarizes the occurrence frequency of different failure modes and their impact on success rate (SR), aggregated over all models and team sizes under the broadcast protocol. Figure 36 compares duplicate-grab rates across models in two-agent teams.

References↩︎