1 Introduction↩︎

Multi-robot coordination underpins a growing range of applications, such as environmental monitoring[1], search and rescue[2], and cooperative manipulation[3], where teams of autonomous robots collectively accomplish objectives beyond individual capabilities. In practical deployments, scalability and communication constraints often necessitate decentralized execution[4], where each agent acts based on its own sensor observations without relying on a central coordinator at runtime. The widely adopted Centralized Training Decentralized Execution (CTDE) paradigm[5]–[7] addresses this by leveraging global information during training, typically through a centralized critic, while constraining actor policies to local observations at deployment.

Yet even with centralized value guidance, actors trained end-to-end from local observations usually converge slowly and settle on suboptimal coordination strategies. A primary cause is partial observability[4], as the same local observation may be consistent with multiple global configurations, each requiring a different cooperative response. This ambiguity is further amplified under communication-free execution, where robots cannot exchange observations to reduce uncertainty. Such constraints are particularly prevalent in marine robotics, where underwater vehicles suffer from severe acoustic latency and packet loss[8], and surface vessels face intermittent, range-limited radio-frequency links[9].

A principled way to mitigate partial observability is privileged policy distillation[10]. Instead of learning coordination directly under limited observations, one first trains an oracle policy with full state access, where the learning problem is substantially easier, and then transfers its behavior to sensor-constrained students via behavioral cloning (BC). This paradigm has achieved notable success in single-robot settings, including vision-based driving distilled from a map-equipped oracle[11] and proprioceptive locomotion distilled from a terrain-aware teacher[12].

Inspired by these successes, we extend privileged distillation to multi-robot coordination. However, this extension faces a fundamental challenge that is largely absent in single-robot tasks. In cooperative multi-agent systems, the oracle action for an individual robot may depend on the joint configuration of all agents and environment states beyond that robot’s sensing range. Consequently, the same local observation can correspond to multiple distinct oracle actions, inducing an inherently multi-modal conditional action distribution (Proposition 1). Standard deterministic policies cannot represent such distributions. Under common regression-based BC objectives such as mean squared error (MSE), they converge to the conditional mean, which may lie between distinct modes and correspond to no valid cooperative action (Corollary 1), leading to hesitant or degenerate coordination.

To address this challenge, we propose CoDiMAD (Cooperative Diffusion-based Multi-Agent Distillation), a three-stage framework that distills a privileged oracle into decentralized diffusion policies[13]. CoDiMAD first trains a MAPPO[14] oracle with global-state access, then collects an offline dataset pairing local observations with oracle actions, and finally distills the oracle behavior into conditional denoising diffusion probabilistic models (DDPMs)[15]. By modeling the conditional action distribution rather than a point estimate, the diffusion student avoids mode averaging and can sample decisive actions from coherent coordination modes. To overcome the high inference latency typical of diffusion models, we further integrate Denoising Diffusion Implicit Models (DDIM)[16] for accelerated, few-step sampling, thereby enabling real-time onboard deployment.

The main contributions of this work are as follows.

• We propose CoDiMAD, a privileged distillation framework that resolves multi-modal ambiguity in partially observable multi-agent coordination through conditional diffusion modeling. To the best of our knowledge, this is the first work to integrate diffusion-based generative policies into multi-robot privileged distillation.

• We formally show that partial observability induces multi-modal action distributions under which regression-based distillation suffers from mode averaging, whereas diffusion-based generation approximates the conditional oracle action distribution.

• We validate CoDiMAD on three cooperative tasks, demonstrating consistent improvements over direct local MARL and deterministic distillation baselines, with analyses confirming its ability to recover multi-modal coordination behaviors.

2 Related Work↩︎

2.0.1 Multi-Agent Coordination↩︎

The CTDE paradigm[5]–[7] enables algorithms to use global information during training while executing with local observations. Many CTDE methods introduce learned communication channels[17]–[20] to exchange task-relevant information at execution time. However, they typically assume reliable communication, which may be unavailable in communication-free or bandwidth-limited robotics deployments. Other approaches pursue implicit coordination through shared conventions[21] or heterogeneous policy architectures[22], but they still learn directly from partial observations and may therefore converge to suboptimal coordination strategies.

2.0.2 Privileged Policy Distillation↩︎

Privileged policy distillation transfers knowledge from a teacher policy with privileged information to a student policy constrained by realistic sensory inputs, and has proven effective in single-agent settings[11], [12]. In multi-agent systems, Zhao et al.[23] introduced CTDS to distill centralized Q-values into decentralized student policies. More recently, Tang et al.[24] investigated resource-aware distillation for resource-constrained agents, while Cho et al.[25] proposed an interactive distillation framework to align student-teacher behaviors. However, these existing methods largely rely on deterministic regression objectives. Such objectives often reduce the multi-modal action distributions induced by partial observability to point estimates, thereby producing averaged, invalid coordination behaviors.

2.0.3 Diffusion Models for Decision-Making↩︎

Diffusion models have recently emerged as a powerful tool for decision-making, including single-agent trajectory planning[26] and visuomotor policy learning[13], demonstrating strong capacity for modeling multi-modal behavior. In multi-agent settings, Negarmehr et al.[27] trained decentralized diffusion policies via imitation learning under a CTDE paradigm to achieve implicit coordination on bimanual manipulation tasks. However, their approach relies heavily on high-quality, expert human demonstrations, which are difficult and expensive to scale to complex, dense coordination scenarios. In contrast, CoDiMAD leverages a privileged reinforcement learning oracle to generate scalable supervision, bypassing the need for human demonstrations while retaining the multi-modal modeling advantages of diffusion-based multi-agent distillation.

3 Problem Formulation↩︎

3.1 Dec-POMDP Formulation↩︎

We model multi-robot coordination as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP)[4], defined by the tuple \(\langle N, \mathcal{S}, \{\mathcal{A}_i\}^N_{i=1}, \{\mathcal{O}_i\}^N_{i=1}, \mathcal{P}, \mathcal{R}, \gamma \rangle\). There are \(N\) agents indexed by \(i\in\{1,\ldots,N\}\), sharing a global state space \(\mathcal{S}\) and a team reward \(\mathcal{R}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}\). Each agent \(i\) selects actions from a continuous individual action space \(\mathcal{A}_i\), which in our robot domains corresponds to a 2D velocity command \([v_x, v_y]\). The joint action space is \(\mathcal{A}=\mathcal{A}_1\times\cdots\times\mathcal{A}_N\). The state transition function \(\mathcal{P}: \mathcal{S} \times \mathcal{A} \to \Delta (\mathcal{S})\) maps a state and a joint action to a probability distribution over the next states. Each agent \(i\) does not observe the global state \(s_t \in \mathcal{S}\) directly, but instead receives only a local observation \(\mathbf{o}_i^t \in \mathcal{O}_i\) and acts without direct communication. The objective is to learn decentralized policies \(\pi_i(\mathbf{a}_i^t\,|\,\mathbf{o}_i^{1:t})\) that maximize the expected cumulative team return, \(J(\pi) = \mathbb{E}\left[\sum_{t=0}^{H} \gamma^t \mathcal{R}(s_t, \mathbf{a}_t)\right]\), where \(H\) is the episode horizon, \(\mathbf{a}_t\) denotes the joint action, and \(\gamma \in [0,1)\) denotes the discount factor.

3.2 Observation Spaces and Information Asymmetry↩︎

In this work, we instantiate the above formulation for a team of homogeneous robots. These agents share identical individual observation and action spaces, allowing us to implement decentralized policies using shared parameters. To represent spatial and task-relevant information in a unified format, all local and global observations are structured as multi-channel bird’s-eye-view (BEV) grid maps with a spatial resolution of \(32 \times 32\), as summarized in Table 1. The privileged oracle policy processes a dual-stream input \((\mathbf{o}_i^{\mathrm{loc}}, \mathbf{o}_i^{\mathrm{glo}})\). The local stream \(\mathbf{o}_i^{\mathrm{loc}} \in \mathbb{R}^{2\times32\times32}\) consists of an egocentric LiDAR occupancy map and a velocity-field map. The privileged global stream \(\mathbf{o}_i^{\mathrm{glo}} \in \mathbb{R}^{3\times32\times32}\) contains three channels that encode arena-scale global information, including the absolute positions of the self-agent, its teammates, and the full task-state map (e.g., global target locations). In contrast, the student policy operates under realistic sensing limitations and observes only a single-stream tensor \(\mathbf{o}_i \in \mathbb{R}^{4\times32\times32}\), where the three global channels are replaced by two locally-restricted channels—a cropped task-state slice and teammate positions clipped to the agent’s limited sensing radius, which yields a 4-channel input combined with the two local channels from \(\mathbf{o}_i^{\mathrm{loc}}\). This information asymmetry is the core source of multi-modal uncertainty: the same \(\mathbf{o}_i^t\) can be consistent with multiple global configurations, each potentially requiring a different coordination action.

4 Methodology↩︎

As illustrated in Fig. 1, CoDiMAD is a three-stage oracle-student distillation framework. In Stage 1, a shared oracle policy \(\pi^O\) is trained via MAPPO with access to both local and privileged global observations. Stage 2 freezes \(\pi^O\) and collects an offline dataset \(\mathcal{D}\) of local-observation and oracle-action pairs from all agents instances and timesteps. In Stage 3, the oracle’s behavior is distilled into a shared diffusion-based student policy \(\pi_\theta\) conditioned solely on local observation history. By adopting a DDPM as the policy class, the student can sample from the conditional oracle-action distribution induced by partial observability, rather than regressing to a single point estimate—thereby circumventing the mode-averaging failure formalized in Corollary 1.

4.1 Stage 1: Oracle Policy Training via MAPPO↩︎

We train the oracle policy \(\pi^O\) following the CTDE paradigm using MAPPO. In our homogeneous robot domains, the oracle actor is shared across agents and is evaluated independently for each agent. The actor receives the dual-stream observation \((\mathbf{o}_i^{\mathrm{loc}}, \mathbf{o}_i^{\mathrm{glo}})\) comprising both local and global map inputs, while the centralized critic receives the joint state for value estimation. During deployment, only the actor is retained for offline data collection, while the critic is discard. As shown in Fig. 1, the actor encodes \((\mathbf{o}_i^{\mathrm{loc}}, \mathbf{o}_i^{\mathrm{glo}})\) through a dual-branch convolutional encoder followed by a GRU[28] for temporal integration, and outputs a 2D velocity command \([v_x, v_y]\) via a diagonal Gaussian action head. The oracle policy is trained following the standard MAPPO configuration[14].

4.2 Stage 2: Offline Trajectory Collection↩︎

Once the oracle policy converges, it is frozen and executed deterministically by taking the mean of its Gaussian action distribution, rather than sampling. For each rollout step, we record \((\mathbf{o}_i^t,\; \mathbf{a}_i^t,\; d_i^t)\) for all agents \(i\) and timesteps \(t\), where \(\mathbf{o}_i^t\) is the student policy’s local observation, \(\mathbf{a}_i^t\) is the deterministic oracle action, and \(d_i^t \in \{0,1\}\) is the episode termination indicator used for hidden-state reset. Importantly, \(\mathbf{o}_i^t\) excludes the oracle’s privileged global stream and contains only the information available to the student at deployment. All samples from different agent instances and timesteps are pooled into a single offline dataset \(\mathcal{D}\) for training the shared student policy. To ensure high-quality supervision, we filter the collected oracle rollouts by discarding episodes that fail to complete the task or contain any collision, and retain only collision-free successful episodes in the offline dataset. Actions are then z-score normalized dimension-wise over the full dataset, with per-dimension statistics \((\boldsymbol{\mu}_a, \boldsymbol{\sigma}_a)\) stored as model buffers for denormalization at inference.

4.3 Stage 3: Diffusion-Based Behavioral Cloning↩︎

4.3.1 Architecture↩︎

The student policy \(\pi_\theta\) is a conditional denoising diffusion model comprising an observation encoder and a noise prediction network \(\hat{\boldsymbol{\epsilon}}_\theta\). Unlike the oracle’s dual-branch encoder, the student encoder employs a single-branch convolutional network followed by a GRU, operating exclusively on the local observation stream. At each timestep, the encoder receives the local observation \(\mathbf{o}_i^t\), while the GRU maintains a recurrent hidden state that summarizes the history \(\mathbf{o}_i^{1:t}\). The hidden state is reinitialized at episode boundaries and produces a conditioning embedding \(\mathbf{c}_i^t\). The noise prediction network receives the noisy action \(\mathbf{a}^{(k)}\in\mathbb{R}^2\), a sinusoidal embedding of the diffusion timestep \(k\), and the conditioning embedding \(\mathbf{c}_i^t\) as inputs. It comprises a cascade of residual MLP blocks, where the fused conditioning signal \((\mathbf{c}_i^t + \mathbf{t}_k)\) is injected into each block to preserve observation-dependent information throughout the network depth. The output is the predicted noise vector \(\hat{\boldsymbol{\epsilon}}_\theta(\mathbf{a}^{(k)}, k, \mathbf{c}) \in \mathbb{R}^2\).

4.3.2 Training Objective↩︎

Let \(\mathbf{a}^{(0)}\) denote the normalized oracle action. For notational simplicity, we omit the agent and time indices in the following diffusion equations. The forward diffusion process corrupts the clean action \(\mathbf{a}^{(0)}\) by systematically adding Gaussian noise over \(T=200\) timesteps according to a cosine noise schedule \[\mathbf{a}^{(k)} = \sqrt{\bar{\alpha}_k}\,\mathbf{a}^{(0)} + \sqrt{1-\bar{\alpha}_k}\,\boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}),\] where \(k\in \{ 1,\dots,T \}\) indexes the diffusion step, and \(\bar{\alpha}_k\) represents the noise schedule coefficient. To train the student policy \(\pi_\theta\) to mimic the oracle’s expertise under partial observability, we optimize the noise prediction network \(\hat{\boldsymbol{\epsilon}}_\theta\) following the DDPM framework [15]. The network is trained to reconstruct the added noise \(\boldsymbol{\epsilon}\) from the perturbed action \(\mathbf{a}^{(k)}\) under the guidance of the conditioning context \(\boldsymbol{c}\). The training loss is formulated with a per-timestep reweighting scheme \[\mathcal{L} = \mathbb{E}_{k,\,\boldsymbol{\epsilon},\,(\mathbf{a}^{(0)},\mathbf{o})\sim\mathcal{D}} \!\left[ (1 - \bar{\alpha}_k) \left\| \boldsymbol{\epsilon} - \hat{\boldsymbol{\epsilon}}_\theta \big(\mathbf{a}^{(k)}, k, \mathbf{c}\big) \right\|^2 \right]. \label{eq:loss95diff}\tag{1}\] Here, \((1-\bar{\alpha}_k)\) downweights nearly clean samples and emphasizes high-noise regimes, where denoising requires stronger dependence on the observation-conditioned prior.

4.3.3 DDIM Inference↩︎

The full DDPM reverse process requires \(T=200\) sequential denoising steps per action sample, which is costly for closed-loop robot control. Song et al. [16] show that a non-Markovian reverse process can preserve the same training marginals as DDPM, yielding the deterministic DDIM sampler with substantially fewer denoising steps. We adopt DDIM with a uniformly sub-sampled schedule of \(K=20\) steps, achieving a \(10\times\) reduction in inference cost without retraining, since the same noise prediction network \(\hat{\boldsymbol{\epsilon}}_\theta\) is used. Let \(\{\tau_j\}_{j=0}^{K}\) denote the sub-sampled timestep sequence, where \(\tau_0=0\) and \(\tau_K=T\). Starting from \(\mathbf{a}^{(\tau_K)}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\), we iteratively compute, for \(j=K,\ldots,1\), \[\begin{align} \mathbf{a}^{(\tau_{j-1})} =\;& \sqrt{\bar{\alpha}_{\tau_{j-1}}}\, \frac{ \mathbf{a}^{(\tau_j)} - \sqrt{1-\bar{\alpha}_{\tau_j}}\, \hat{\boldsymbol{\epsilon}}_\theta \big(\mathbf{a}^{(\tau_j)},\tau_j,\mathbf{c}\big)}{\sqrt{\bar{\alpha}_{\tau_j}}} \\ &+ \sqrt{1-\bar{\alpha}_{\tau_{j-1}}}\, \hat{\boldsymbol{\epsilon}}_\theta \big(\mathbf{a}^{(\tau_j)},\tau_j,\mathbf{c}\big). \end{align}\] Although the DDIM update is deterministic conditioned on the initial noise, different initial noise samples can yield different action modes. The final normalized action \(\mathbf{a}^{(0)}\) is denormalized using the stored statistics \((\boldsymbol{\mu}_a,\boldsymbol{\sigma}_a)\) to produce the velocity command. Algorithm 2 summarizes the complete three-stage training procedure.

4.4 Theoretical Analysis↩︎

The key difficulty in privileged multi-agent distillation is that the oracle conditions on global information, whereas the student only observes a local projection. Consequently, a single local observation may correspond to multiple global configurations and thus to multiple valid oracle actions. This turns oracle-to-student distillation into a conditional density estimation problem rather than a simple regression problem.

5 Experiments↩︎

5.1 Evaluation Environments↩︎

We evaluate CoDiMAD on three cooperative multi-robot tasks, as shown in Fig. 3. All tasks are implemented in a \(200\!\times\!200\) continuous arena with \(N\!=\!3\) agents, randomized obstacles, and 200-step episodes. The observation encoding is summarized in Table 1.

• Cooperative Coverage: Agents explore the arena to maximize coverage of the occupancy grid. The reward encourages discovering new cells, penalizes collisions, and grants a bonus upon completing the coverage objective.

• Pursuit-Evasion: Pursuers cooperate to capture a single evader controlled by an Artificial Potential Field (APF) policy. The evader has a maximum speed twice that of the pursuers. The reward combines a capture bonus, a distance-shaping term, and a collision penalty.

• Box Pushing: Agents cooperatively push a heavy box toward a randomly placed goal zone. The reward encourages box displacement toward the goal and simultaneous multi-agent contact, while penalizing collisions.

5.2 Experimental Setup↩︎

To isolate the individual contributions of privileged distillation, temporal history encoding, and diffusion-based action generation, we compare CoDiMAD against four comparison methods, including one privileged upper bound:

• MAPPO-Oracle (Upper Bound): The privileged policy trained with full global state via MAPPO. While non-deployable in practical scenarios, it establishes the performance ceiling.

• MAPPO-Local: MAPPO trained exclusively on local observations, representing the performance baseline of communication-free, model-free MARL.

• BC-RNN: A deterministic distillation baseline sharing the same recurrent observation encoder as CoDiMAD, but replacing the diffusion head with an MLP trained via MSE regression. This variant isolates the effect of diffusion-based action generation.

• CoDiMAD w/o RNN: An ablation that removes the GRU and conditions solely on single-frame observations, isolating the contribution of temporal history encoding.

All methods are implemented in PyTorch and trained on an NVIDIA RTX 4090 GPU. The oracle policy is trained for \(5\) M environment steps. For the student, the noise prediction network consists of three residual MLP blocks with hidden dimension \(128\), and the observation encoder uses a GRU layer with the same hidden size. The forward diffusion process employs a cosine noise schedule (\(s\!=\!0.008\), \(\beta \in [10^{-4},\, 0.02]\)) with \(T\!=\!200\) training steps, while DDIM inference uses \(K\!=\!20\) denoising steps. The student network is optimized with AdamW (\(\text{lr}\!=\!10^{-4}\)) and exponential moving average (with a decay of \(0.99\)) for \(50\) epochs with a batch size \(128\).

5.3 Comparative Results↩︎

We evaluate each method across three random seeds, with each seed evaluated over 200 episodes. We report task-specific success metrics, including coverage rate, capture rate, and box-pushing success rate, together with a collision metric. The collision metric measures the average number of total collisions per episode, including both inter-agent and obstacle collisions. Table 2 summarizes the quantitative results. Bold entries denote the best-performing deployable method for each metric. We organize the analysis into four pairwise comparisons.

CoDiMAD vs.MAPPO-Oracle. Despite relying only on local observations and using no inter-agent communication, CoDiMAD closely approaches the privileged oracle on two of the three tasks. On Coverage, CoDiMAD achieves \(95.7\%\) coverage rate compared with the oracle’s \(97.6\%\), retaining approximately \(98\%\) of oracle-level performance while maintaining a comparably low collision count. On Pursuit-Evasion, CoDiMAD reaches a \(90.6\%\) capture rate compared with the oracle’s \(99.1\%\), with similar collision frequency. On Box Pushing, CoDiMAD achieves \(72.2\%\) success compared with the oracle’s \(98.2\%\), which is the largest gap among the three tasks. This gap reflects the tight force-coordination requirements of cooperative manipulation, where small deviations in pushing timing or angle can lead to failure. These results indicate that diffusion-based distillation transfers a substantial portion of the globally-informed cooperative behavior to a sensor-constrained decentralized student.

Diffusion vs.regression (CoDiMAD vs.BC-RNN). CoDiMAD consistently outperforms BC-RNN, supporting the motivation developed in Section 4.4. On Coverage, CoDiMAD achieves \(95.7\%\) coverage rate compared with BC-RNN’s \(80.9\%\), corresponding to a \(14.8\) percentage-point improvement. At the same time, it reduces collisions from \(10.37\) to \(0.55\) per episode, an approximately \(19{\times}\) reduction. This large collision reduction is consistent with the mode-averaging failure of deterministic regression. For example, when the oracle action distribution contains two valid modes, such as bypassing an obstacle from the left or from the right, BC-RNN may regress to their conditional mean, producing an intermediate action that drives the agent approximately straight toward the obstacle. Such averaged actions do not correspond to any valid coordination mode and therefore substantially increase collision risk (cf.Corollary 1).

On Pursuit-Evasion, the two methods achieve comparable capture rates (\(90.6\%\) vs.\(90.3\%\)), but CoDiMAD reduces collisions from \(3.41\) to \(0.80\), a \(4.3{\times}\) improvement. This suggests that BC-RNN can often produce actions sufficient for eventual capture, yet its averaged trajectories are less coordinated and more collision-prone. On Box Pushing, the advantage of CoDiMAD becomes more pronounced: it achieves \(72.2\%\) success rate compared with BC-RNN’s \(6.5\%\). This result suggests that the gap between CoDiMAD and deterministic regression may become larger as task difficulty increases, particularly when success requires precise and sustained multi-agent coordination.

Distillation vs.direct local RL (CoDiMAD vs.MAPPO-Local). CoDiMAD substantially outperforms MAPPO trained end-to-end from local observations, validating the privileged distillation paradigm. The gap is especially large on Pursuit-Evasion, where MAPPO-Local achieves only \(14.2\%\) capture rate compared with CoDiMAD’s \(90.6\%\). Pursuit requires coordinated behaviors such as flanking and encirclement, which are difficult to discover from local observations and sparse task rewards alone. On Box Pushing, a similar pattern emerges: MAPPO-Local achieves \(18.0\%\) success rate compared with CoDiMAD’s \(72.2\%\), as cooperative manipulation requires agents to discover pushing configurations that are highly sensitive to relative positioning. On Coverage, MAPPO-Local performs comparatively better, yet CoDiMAD still outperforms it by \(12.0\) percentage points with fewer collisions. Across all tasks, the distillation pipeline mitigates the exploration bottleneck by first learning coordinated behavior in the fully-observable setting and then transferring that behavior to a sensor-constrained decentralized student.

Temporal history (CoDiMAD vs.CoDiMAD w/o RNN). Removing the GRU leads to a moderate but consistent performance drop across all three tasks. The effect is most visible on Pursuit-Evasion, where the capture rate decreases from \(90.6\%\) to \(80.4\%\) and collisions increase from \(0.80\) to \(1.41\) per episode. On Coverage and Box-Pushing, the gaps are smaller, but still indicate a positive contribution from temporal history. These results suggest that the GRU provides useful contextual information for disambiguating single-frame observations, especially in sequential coordination tasks.

5.4 Analysis of Multi-Modal Behavior↩︎

The theoretical analysis in Section 4.4 suggests that partial observability can induce multi-modal conditional action distributions, that deterministic regression yields invalid mode-averaged outputs, and that diffusion sampling recovers individual modes. We empirically examine these claims using action-space visualizations and rollout-level trajectory comparisons.

5.4.1 Action Distribution Visualization↩︎

To visualize action multi-modality, we select a local observation \(\mathbf{o}_i\) from the Pursuit-Evasion dataset whose nearest neighbors in observation space exhibit bimodal oracle actions. This indicate that similar local observations may correspond to different global configurations requiring different cooperative responses. Fig. 4 (a) shows the oracle actions of the 200 nearest neighbors in the 2D action space, where two clear clusters emerge. Fig. 4 (b) shows that CoDiMAD, sampled 200 times from the same observation embedding with independent initial diffusion noise, generates actions covering both oracle modes. In contrast, Fig. 4 (c) shows that BC-RNN produces a single deterministic output located between the two clusters, consistent with the mode-averaging behavior described in Corollary 1.

5.4.2 Trajectory Diversity↩︎

We further examine whether action-level multi-modality leads to distinct coordination strategies over time. Starting from identical initial states, we execute five independent rollouts for both CoDiMAD and BC-RNN. As shown in Fig. 5, CoDiMAD produces diverse but coherent trajectories across rollouts: agents may cover different regions in Coverage or adopt different approach paths in Pursuit-Evasion. By contrast, BC-RNN produces the same deterministic trajectory across rollouts, reflecting its lack of sampling diversity. These results show that CoDiMAD’s stochastic diffusion policy preserves multi-modal behavior over extended horizons, enabling different valid coordination strategies from the same initial condition.

6 Conclusion↩︎

We presented CoDiMAD, a three-stage learning framework for communication-free multi-robot coordination. CoDiMAD first trains a privileged MAPPO oracle policy with access to the full global state, and then distills its behavior into deployable student policies parameterized as conditional denoising diffusion models that act solely on local observations. Unlike regression-based distillation, which predicts a single conditional mean action, CoDiMAD parameterizes each student policy as a conditional denoising diffusion model and therefore learns the full conditional action distribution induced by partial observability. This allows each agent to sample actions from coherent coordination modes rather than averaging across multiple valid strategies. Experiments on three multi-agent tasks show that CoDiMAD consistently outperforms both direct local MARL and deterministic distillation baselines, while approaching the performance of the privileged oracle under communication-free decentralized execution. Action-distribution and trajectory visualizations further support our theoretical analysis: local observations can correspond to multi-modal oracle action distributions, CoDiMAD recovers distinct modes through diffusion sampling, and deterministic distillation method tends to produce intermediate mode-averaged actions.

Several limitations suggest directions for future work. First, as an offline distillation method, CoDiMAD may struggle to recover from states outside the oracle’s trajectory distribution. Incorporating online fine-tuning with diffusion-compatible RL objectives [31] could improve robustness under such distribution shifts. Second, scalability beyond \(N\!=\!3\) agents remains to be evaluated. As the number of agents grows, the coordination space may become increasingly complex, motivating more expressive architectures such as attention-based denoisers. Finally, deployment on physical marine robots is an ongoing direction of our work, aiming to validate its applicability in real-world communication-constrained environments.

References↩︎