1 Introduction↩︎

Cooperative multi-agent reinforcement learning (MARL) studies how a team of decentralized agents can jointly maximize a shared return when each agent observes only part of the environment. Compared with single-agent RL, the central challenge in cooperative MARL arises from multi-agency under partial observability: each agent must make decisions based on its own local observation, while the information needed for effective coordination may be distributed across teammates. Communication provides a natural mechanism to mitigate this bottleneck. By exchanging previously observed information, agents can avoid inferring hidden state solely from teammates’ actions, reduce redundant exploration of the joint state space, and learn coordinated policies with fewer environment interactions [1]–[5].

In this work, we focus on cooperative environments whose observations exhibit rich hierarchical structure. A representative example is cyber defense, as instantiated by the CAGE Challenge 4 environment [6], [7]. In such settings, hosts are organized into subnets, agents may observe only their assigned hosts or subnets, and effective coordination is required to track activity across the system. At enterprise scale, with thousands of assets and rapidly evolving events, such coordination must be both efficient and scalable. These constraints apply equally to coordinated attackers (red teams), which operate under similar partial observability and distributed control; consequently, effective red-team evaluation depends on communication mechanisms that capture these limitations while enabling coherent multi-agent behavior. However, designing communication protocols that efficiently exploit these hierarchical observation structures remains challenging.

Existing communication methods largely treat messages as flat objects. A sender typically encodes its local observation into a fixed-width vector, and the receiver consumes this vector without explicitly indexing the structure of the underlying observation. A related line of hierarchical communication has explored hierarchy in other parts of the system: feudal architectures stratify the agent population into managers and workers [8], [9], while clustered or order-based protocols impose hierarchy on the communication graph or the message-passing schedule [10], [11]. However, these approaches do not organize the message body according to the hierarchy of the sender’s observation. As a result, a readily available source of structure in many cooperative environments remains unused.

We address this gap with HiComm, a communication module for cooperative MARL that grounds each message in the sender’s hierarchical observation. The module is receiver-driven: a receiver first issues a query, and the hierarchy is resolved through a three-stage decoding process. Specifically, the receiver first selects a group, then a sender, and the addressed sender finally selects an entity within that group on its own observation, returning the corresponding feature slice as the message. In this way, HiComm turns communication from transmitting an unstructured dense vector into retrieving structured information from the sender’s observation hierarchy. Our contributions are listed as follows:

• We propose HiComm, a plug-in communication module for cooperative MARL settings whose observations admit a natural group-entity hierarchy. Instead of transmitting an unstructured dense vector, HiComm lets the receiver query a group-entity location in a selected sender’s observation hierarchy, and the sender returns the corresponding feature slice as the message.

• We instantiate this hierarchical query-response mechanism as a differentiable and lightweight module. HiComm uses Straight-Through Gumbel-Softmax for discrete group, sender, and entity selection, shares a single projection head across hierarchical encoders, and keeps the receiver query compact, allowing it to attach to standard MARL training pipelines without changing the underlying policy or critic architecture.

• We empirically validate HiComm across cooperative MARL tasks with different observation structures and coordination demands. As an add-on communication module, HiComm matches or outperforms representative learned communication baselines while reducing communication volume by up to \(23\times\) per receiver per episode.

2 Related Work↩︎

Cooperative MARL under partial observability has long relied on interagent communication [1], [2], with prior work refining the practice along three axes. On message content, NDQ [12], VBC [13], and TMC [14] regularise messages to be succinct, while MAIC [15] and CACOM [16] condition each sender’s projection on a model of the intended receiver. On scheduling, IC3Net [17] and ATOC [3] let agents gate themselves silent, SchedNet [18] and SeqComm [19] prioritise or sequence communicators across the team, and Who2com [20] together with When2com [21] use a multi-stage handshake in which the receiver requests, candidate senders return a brief match score, and the receiver finally connects to the best-matching peer. On message integration, TarMAC [4] and SARNet [22] apply receiver-side soft attention over incoming messages, MASIA [23] aggregates teammate embeddings with permutation-invariant self-supervised objectives, T2MAC [24] weighs evidence under uncertainty, and CoDe [25] aligns intent with timeliness for delay-robust collaboration. A parallel line learns the routing graph itself: DGN [26] convolves over a multi-head attention graph, DICG [27] infers an implicit adjacency via self-attention, CASEC [28] sparsifies the coordination graph using payoff-variance scores, MAGIC [29], CommFormer [5], and TGCNet [30] learn graph attention, differentiable adjacency, or a dynamic directed graph, and G-Designer [31] learns a task-adaptive topology via a variational graph auto-encoder. Two cross-cutting properties hold across this entire body of work. First, the protocols are sender-driven by default: the sender unilaterally decides what to encode, and even MAIC and CACOM adjust the sender’s projection toward an intended receiver rather than letting the receiver actively query. Second, the channel payload is a learned, and therefore lossy, projection of the sender’s observation into a fixed-width vector, decoupled from any structure intrinsic to that observation; even the Who2com and When2com handshakes select whom to ask without specifying which part of the sender’s observation to return. A complementary efficiency line attacks bandwidth from outside the message form: NDQ, VBC, and TMC compress the payload, CASEC and message-pruning approaches sparsify the graph, and graph-learning methods like CommFormer and G-Designer thin out edges, but none of them changes the form of what is sent.

A subset of MARL work labels its mechanism as hierarchical, but the hierarchy invariably lives outside the message body. Feudal Multiagent Hierarchies [8] stratify the agent pool into managers and workers via subgoal signals, and GMAH [32] keeps the same vertical decomposition inside each agent’s policy stack with a QMIX-style goal-mixing network in lieu of any explicit channel; HiMPo [9] marries this feudal layout with peer-to-peer GNN exchange of flat hidden states. LSC [10] hierarchises the routing graph by electing group leaders and propagating learned embeddings within and between groups, and HAMA [33] hierarchises the attention itself by stacking inter-agent and inter-group attention layers over a fixed agent partition. DHCG [11] hierarchises the message-passing order via a learned DAG with edges carrying flat action and trajectory embeddings. In every case the hierarchy sits on agents, attention, routing, order, or the policy stack, and the transmitted content is either an abstract subgoal token or a flat hidden-state embedding, never indexed against the observation hierarchy itself. That last dimension is where HiComm introduces structure: the receiver actively issues a query that selects a coordinate in the sender’s intrinsic group-then-entity partition, and the response is delivered verbatim as the raw feature vector at that coordinate, exchanging the encoder’s lossy projection for an in-distribution observation entry (the precise payload form is given in Sec. 4).

3 Preliminaries↩︎

3.0.0.1 Cooperative MARL with communication.

We study cooperative multi-agent tasks with partial observability and learnable communication, formalised as a tuple \(\mathcal{M} = \langle S, U, P, Z, O, N, r, \gamma \rangle\). Here \(S\) is the global state space, \(U\) is the per-agent action space, and \(N \in \mathbb{N}\) is the number of agents. The transition kernel \(P(s' \mid s, \mathbf{u})\) governs the dynamics under joint actions \(\mathbf{u} \in U^N\), the function \(r : S \times U^N \to \mathbb{R}\) is the team-shared reward, and \(\gamma \in [0,1)\) is the discount factor. Each agent perceives the environment through the per-agent observation space \(Z\) via the observation function \(O : S \times \{1, \dots, N\} \to Z\).

At every step, agent \(i\) receives a local observation \(z^i=O(s,i)\), exchanges a fixed-width message \(m^i\in\mathbb{R}^{d_m}\) with its peers, and selects an action according to \(\pi^i(u^i\mid z^i,m^{1:N})\), where \(m^{1:N}=(m^1,\dots,m^N)\) collects the per-agent message slots. When self-loops are excluded, we set \(m^i=\mathbf{0}\) on the self-slot by convention, and \(d_m\in\mathbb{N}\) denotes the message bandwidth. We call an observation flat if it is represented as a single ungrouped list of entity slots, with all entities placed at the same address level. In a flat observation, any notion of which entities form a meaningful unit must be inferred from features during training.

3.0.0.2 Hierarchical observations.

We consider hierarchical observation \(z^i\) in this work. We assume that each agent’s observation is a three-axis hierarchical tensor \(z^i \in {\mathcal{G}\times\mathcal{E}\times\mathcal{F}}\), where \(|\mathcal{G}|,|\mathcal{E}|,|\mathcal{F}|\in\mathbb{N}_{>0}\), and write \([K]:=\{1,\dots,K\}\) for an axis of size \(K\). The hierarchy consists of the following components:

• \(\mathcal{G}\) is the coarse group axis, indexing \(\mathcal{G}\) groups.

• \(\mathcal{E}\) is the finer entity axis, indexing \(\mathcal{E}\) entities within each group.

• \(\mathcal{F}\) is the per-entity feature axis, storing a feature vector for each group–entity pair.

Thus, an entry \(z^i[g,\ell,:]\) is addressed by a group index \(g\in[\mathcal{G}]\) and an entity index \(\ell\in[\mathcal{E}]\), and returns the feature vector \(F:=z^i[g,\ell,:]\in\mathbb{R}^{|\mathcal{F}|}\). Here \(g\) and \(\ell\) are discrete coordinates, while \(F\) is the continuous feature vector stored at the coordinate \((g,\ell)\).

The partition \([\mathcal{G}]\times[\mathcal{E}]\) is supplied by the environment rather than invented by the model. It may arise in either of two ways:

• as a native hierarchy, where the observation tensor already has separate group and entity axes;

• as a semantic hierarchy, where an observable per-entity attribute induces the group label.

This hierarchy frames the communication problem studied in this paper: given \(N\) agents whose observations are jointly indexed by \([\mathcal{G}]\times[\mathcal{E}]\), can each per-round message \(m^i\in\mathbb{R}^{d_m}\) be constructed from the environment-provided partition, namely from a selected coordinate \((g,\ell)\) and the feature vector \(z^i[g,\ell,:]\), rather than from a flattened sender-side code that ignores this structure?

3.0.0.3 Communication on hierarchical observations.

We call a communication protocol observation-space hierarchical if every transmitted message is an address–value pair \(\big((g,\ell),\,F\big)\) whose value \(F = z[g,\ell,:]\in\mathbb{R}^{\mathcal{F}}\) is the verbatim feature vector at the coordinate \((g,\ell)\in[\mathcal{G}]\!\times\![\mathcal{E}]\), with the address drawn from a factored discrete distribution \(p(g)\,p(\ell\mid g)\) over the same partition that organises the observation, rather than the message being a flat sender-side encoding of the whole tensor or the address being drawn from a single flat \(\mathcal{G}\!\cdot\!\mathcal{E}\)-way categorical. The protocol then addresses information at the same granularity at which the observation is already organised, and the open design problem reduces to specifying \(p(g)\,p(\ell\mid g)\) jointly with the policy. In \(N\)-agent settings the same hierarchy can be reused to choose whom to query: drawing a peer \(j\in[N]\setminus\{i\}\) conditional on the topic \(g\) via \(p(j\mid g)\) extends the descent to \(p(g)\,p(j\mid g)\,p(\ell\mid g,j)\), factoring sender selection through the same partition rather than along an orthogonal axis.

4 Methodology↩︎

We introduce HiComm, a communication protocol that turns the hierarchical structure of the observation space introduced in §3 into the addressing space of every transmitted message. HiComm is a learned, stateless transform parameterised by \(\theta_{\text{comm}}\) that, at every environment step, ingests the joint observation \(\{z^{i}_t, M^{i}_t\}_{i=1}^{N}\), where \(M^{i}_t \in \{0,1\}^{|\mathcal{G}|\times|\mathcal{E}|}\) is agent \(i\)’s observability mask over the \([\mathcal{G}]\!\times\![\mathcal{E}]\) partition, and returns the joint augmented observation \(\{\tilde{z}^{i}_t\}_{i=1}^{N}\) that the decentralised policy then consumes. Because the transform leaves every other component of \(\mathcal{M}\) unchanged, HiComm is independent of the cooperative MARL algorithm wrapped around it: its parameters ride on the surrounding actor-critic gradient, complemented by a scalar information-gain shaping prior added to the actor loss. §4.1 fixes this outer loop and shows where HiComm plugs in (Algorithm 1); §4.2 specifies the three-phase HiComm step that builds messages from the \([\mathcal{G}]\!\times\![\mathcal{E}]\) partition (Algorithm 2). Throughout, \(\mathrm{score}(q,k)=\langle q,k\rangle\) refers to the dot-product scoring, and ST-GS refers to the Straight-Through Gumbel-Softmax estimator [34], which returns a one-hot sample on the forward pass and a temperature-\(\tau\) soft surrogate on the backward pass.

4.1 MARL Framework↩︎

4.1.0.1 Cooperative MARL backbone.

We build on the standard cooperative MARL rollout-and-update framework, which we briefly review for completeness. At each training iteration, the agents jointly execute a horizon-\(T\) rollout in the environment \(\mathcal{M}\) under the current policy \(\pi_{\theta_{\mathrm{pol}}}\), and store the resulting transitions in a replay buffer \(\mathcal{D}\). A surrogate objective \(\mathcal{L}\) is then constructed from \(\mathcal{D}\) and optimized using a standard cooperative MARL algorithm, such as IPPO [35] or MAPPO [36]. This backbone determines how the policy and critic are updated, while leaving the communication mechanism modular.

4.1.0.2 Where HiComm plugs in.

HiComm preserves this rollout-and-update loop and inserts a differentiable communication transform between the joint observation and the per-agent action sampling step: \[\underbrace{\big\{\tilde{z}^{i}_t\big\}_{i=1}^{N} \leftarrow \mathrm{\small HiComm}\Big(\big\{z^{i}_t,\,M^{i}_t\big\}_{i=1}^{N};\,\theta_{\mathrm{comm}}\Big)}_{\text{Algorithm~\ref{alg:hicomm}}} \;\longrightarrow\; \underbrace{ u^{i}_t \sim \pi_{\theta_{\mathrm{pol}}}\!\big(\cdot \mid \tilde{z}^{i}_t\big) \quad (\forall i) }_{\text{decentralized action sampling}} .\] Here \(\tilde{z}^{i}_t\) denotes the augmented observation used by agent \(i\) for action selection. It concatenates the local observation \(z^i_t\) with fixed-size message slots of width \(d_m\), one slot per peer as in §3. Under our sender-selection design, only the slot corresponding to the selected sender \(j^*\) is populated, while the remaining peer slots are set to zero; the construction of \(j^*\) and the message \(m_{j^*\to i}\) is given in §4.2. We bundle the communication parameters with the actor parameters as \(\Theta=(\theta_{\mathrm{pol}},\theta_{\mathrm{comm}})\), so that a single backward pass on \(\mathcal{L}(\Theta,\phi)\) propagates task gradients through both the policy and the HiComm module, together with the critic parameters \(\phi\) when required by the chosen actor-critic backbone. Consequently, replacing \(\mathcal{L}\) swaps the underlying MARL algorithm, such as IPPO or MAPPO, without changing the HiComm module. Algorithm 1 summarizes the full procedure.

4.2 Communication via Hierarchical Observation Structure↩︎

We now specify the three-phase HiComm step (Algorithm 2). The protocol rests on a single structural principle: the \([\mathcal{G}]\!\times\![\mathcal{E}]\) hierarchy is the addressing scheme not just for what the message carries, but also for whom to ask. The receiver does not pick a peer out of thin air and then walk the hierarchy; it descends the hierarchy and the peer identity emerges along the way. Concretely, each round commits to three discrete picks made in order: an integer topic-group index \(g^{*}\!\in\![|\mathcal{G}|]\) that names what the receiver wants to ask about; a sender \(j^{*}\!\in\![N]\!\setminus\!\{i\}\) whose learnable affinity for \(g^{*}\) best matches the query; and an integer entity index \(\ell^{*}\!\in\![|\mathcal{E}|]\) within group \(g^{*}\) on the chosen sender’s observation. In brief, Phase 1 generates the receiver’s query \(q_i\); Phase 2 performs the two-stage hierarchical sender selection (group \(g^{*}\), then agent \(j^{*}\!\mid\!g^{*}\)) entirely on the receiver side; Phase 3 transmits \((q_i, g^{*})\) to \(j^{*}\), who locally selects the entity \(\ell^{*}\) on its own observation and returns the addressed feature vector \(F = z^{j^{*}}[g^{*},\ell^{*},:]\in\mathbb{R}^{|\mathcal{F}|}\). The receiver initiates and steers each round, while the sender executes only the final entity stage on its private observation; we therefore call HiComm query-driven rather than fully receiver-resolved.

This descent has three structural consequences. First, the hierarchy is not relegated to a retrieval-only role: it drives sender selection itself, since \(j^{*}\) is chosen conditional on \(g^{*}\) rather than orthogonally to it. Second, the total selector-head output dimension is \(|\mathcal{G}|+(N{-}1)+|\mathcal{E}|\), summed across the three stage-wise picks, rather than the \(|\mathcal{G}|\!\cdot\!|\mathcal{E}|\!\cdot\!(N{-}1)\) of a flat joint pick over (group, entity, sender) (self-loops masked in both cases), reducing the discrete decision space the policy gradient must traverse and giving each stage a dedicated parameter set into which to localise credit. Third, the transmitted feature vector \(F = z^{j^{*}}[g^{*},\ell^{*},:]\) is in-distribution and verbatim: it is one of the sender’s own observation entries, never a learned mixture across cells. Empirical bandwidth comparisons against alternative protocols are reported in §5.

4.2.0.1 Phase 1: query generation.

Let \(M^i \in \{0,1\}^{|\mathcal{G}|\times|\mathcal{E}|}\) be receiver \(i\)’s observability mask, with \(M^i[g,\ell]=1\) iff entity \((g,\ell)\) is observed in \(z^i\). Note that \(M^i\) is a deterministic function of \(z^i\), the indicator of which \((g,\ell)\) slots carry observed values rather than an unobserved sentinel, and we expose it explicitly only for notational convenience. Surfacing the mask alongside \(z^i\) lets \(q_i\) treat the receiver’s epistemic footprint as a first-class signal: when evidence is sparse, the query can deliberately steer toward blind spots; when local support is too uncertain to commit to a confident pick, it can soften its retrieval rather than commit to a request over an unreliable basis. The receiver summarises its observation context together with its observability footprint into a query, \[h_i = f_\text{gap}(z^i, M^i), \qquad q_i = \tanh\!\big(f_\text{q}(h_i)\big) \in [-1,1]^{d_q},\] where \(f_\text{gap}\) and \(f_\text{q}\) are learnable MLPs and \(h_i\) is the GapNet-encoded query feature (kept distinct from the action symbol \(u^i\) of §3). The query \(q_i\) thus represents a prior over what to ask given the receiver’s current state, and is reused without recomputation across all subsequent stages so that whom-to-ask, what-group, and what-entity decisions are anchored in a single information-need representation.

4.2.0.2 Phase 2: hierarchical sender selection (receiver-side, two stages).

The receiver descends the hierarchy in two ST-GS draws, both executed on its local device.

Stage 1, group selection. The receiver picks the topic group \(g^{*}\) by scoring its query against \(|\mathcal{G}|\) learnable group-key embeddings \(\kappa_g\in\mathbb{R}^{d_q}\) (one per group, parameter-shared across receivers and senders), \[g^{*} = \arg\max_{g}\,\mathrm{ST\text{-}GS}\!\Big(\mathrm{score}\big(q_i,\,\kappa_g\big)\Big).\] The keys \(\{\kappa_g\}_{g=1}^{|\mathcal{G}|}\) belong to \(\theta_{\text{comm}}\) and are updated by the same task gradient as the rest of HiComm. Stage 1 commits the receiver to a topic before naming any peer.

Stage 2, agent selection conditional on \(g^{*}\). The receiver scores the same \(q_i\) against a context-conditioned agent-group affinity \(\Phi(j, g^{*};\, h_i)\in\mathbb{R}^{d_q}\) based on the receiver’s own GapNet output \(h_i\) from Phase 1 (re-used without recomputation): \[s_{ij\mid g^{*}} = \mathrm{score}\!\big(q_i,\,\Phi(j, g^{*};\, h_i)\big), j^{*} = \arg\max_{j \neq i}\,\mathrm{ST\text{-}GS}\!\big(s_{i,:\mid g^{*}}\big).\] The hard top-\(1\) is structural rather than approximate: the address \((g^{*},\ell^{*})\) produced in Phase 3 indexes a specific sender’s tensor, so a soft mixture over senders would leave the address ill-defined.

4.2.0.3 Phase 3: entity retrieval (executed at sender \(j^{*}\)).

The receiver transmits the pair \((q_i, g^{*})\) to \(j^{*}\). The sender then resolves the entity \(\ell^{*}\) within group \(g^{*}\) on its own device, masking unobserved entries with its own mask: \[M^{(\text{ent})}_{j^{*}}(g^{*})[\ell] = \begin{cases} 0 & \text{if } M^{j^{*}}[g^{*},\ell] = 1, \\ -\infty & \text{otherwise}, \end{cases}\] \[\ell^{*}\mid g^{*}, j^{*} = \arg\max_{\ell}\,\mathrm{ST\text{-}GS}\!\Big(\mathrm{score}\big(q_i,\,\psi(z^{j^{*}}[g^{*},\ell,:])\big) + M^{(\text{ent})}_{j^{*}}(g^{*})[\ell]\Big),\] where \(\psi:\mathbb{R}^{|\mathcal{F}|}\to\mathbb{R}^{d_q}\) is a sender-side MLP that projects each entity feature vector \(F = z^{j^{*}}[g^{*},\ell,:]\in\mathbb{R}^{|\mathcal{F}|}\) into the scoring space. Because the entity keys \(\psi(z^{j^{*}}[g^{*},\ell,:])\) and the mask \(M^{j^{*}}\) are derived only from \(j^{*}\)’s private observation, no part of \(z^{j^{*}}\) ever leaves the sender; only \((q_i, g^{*})\) enter, and the address-value pair \[m_{j^{*}\to i} = \big((g^{*},\ell^{*}),\;F\big),\qquad F := z^{j^{*}}[g^{*},\ell^{*},:]\in\mathbb{R}^{|\mathcal{F}|},\] returns to the receiver. The transmitted feature vector \(F\) is in-distribution and verbatim: it equals one of the sender’s own observation entries, never a learned mixture across cells (during training the soft surrogate of ST-GS converges to one-hot at \(\tau\to\tau_{\min}\)). All three key types, namely the group keys \(\kappa_g\), the context-conditioned affinity \(\Phi(j, g;\, h_i)\), and the entity projections \(\psi(\cdot)\), live in the common \(\mathbb{R}^{d_q}\) scoring geometry, so \(q_i\) uses the same dot-product across all three stages without re-projection.

4.2.0.4 Training Objective

We update \(\Theta=(\theta_{\text{pol}},\theta_{\text{comm}})\) in a single actor backward pass on \[\mathcal{L}_{\text{actor}}(\Theta) \,=\, \mathcal{L}^{\text{PPO}}(\Theta) \,-\, c_H\,\mathcal{H}\!\big[\pi_{\theta_{\text{pol}}}\big] \,+\, \lambda_{\text{ig}}(t)\,\mathcal{L}_{\text{ig}}(\theta_{\text{comm}}),\] where \(\mathcal{L}^{\text{PPO}}\) supplies task gradient through every ST-GS selector and \(\mathcal{L}_{\text{ig}}\) cross-entropies the Stage 1, Stage 2 and Phase 3 pre-ST-GS logits against soft labels obtained by softmax-normalising the per-coordinate information gain \(\Delta_{ij}[g,\ell,:] = \big[z^{j}[g,\ell,:] - z^{i}[g,\ell,:]\big]_{+}\) along the matching axis, plus a small batch-marginal entropy term that discourages sender collapse. As joint coverage saturates, \(\Delta_{ij}\!\to\!0\) and the soft labels degenerate to uniform, so \(\mathcal{L}_{\text{ig}}\) vanishes structurally; the schedule \(\lambda_{\text{ig}}(t)\) only accelerates this fade. The critic update on \(\phi\) is unchanged, leaving HiComm’s training-side footprint at one scalar added to the actor loss.

5 Experiments↩︎

The empirical study answers three questions. (RQ1) Performance: does HiComm close the gap between the no-communication PartialObs floor and the FullObs ceiling, and does it match or beat the strongest learned-communication baselines, CACOM [16] and T2MAC [24]? (RQ2) Bandwidth: at matched performance, how much information does HiComm actually ship per receiver per episode, relative to those baselines and to FullObs? (RQ3) Generality: do RQ1 and RQ2 hold across IPPO [35] and MAPPO [36], and across varied benchmarks?

We evaluate on three partially observable benchmarks chosen to span qualitatively different regimes: CAGE Challenge 4 [6], [7], an adversarial cyber-defense task with accumulative reconnaissance; SMACv2 [37], cooperative micromanagement under instantaneous sight-range fog of war; and Google Research Football (GRF) [38], set-piece control under a bounded sensing radius. All three instantiate the group-entity hierarchy of §3, with opponents (where present) collapsed into a single group.

5.1 Setup↩︎

5.1.0.1 Methods.

We compare five settings under two cooperative-MARL backbones: IPPO [35] (decentralised critic) and MAPPO [36] (centralised critic). PartialObs keeps each receiver on its own local view \(z^{i}\) with all \(N{-}1\) message slots zeroed (the no-communication floor); FullObs concatenates every teammate’s local view \(\{z^{j}\}_{j\neq i}\) verbatim into \(\tilde{z}^{i}\) (the ceiling on team reward, at the cost of shipping every full local view at every step). The three communication settings share a fixed message width \(d_m\). CACOM [16] first lets each sender broadcast a short summary so everyone knows roughly what is going on, then tailors a separate message to each receiver via attention. T2MAC [24] only talks to selected teammates instead of broadcasting, and on the receiver side it weighs each incoming message by how informative it looks before mixing them. Both, however, treat the observation as a flat vector and ignore its \(\mathcal{G}\!\times\!\mathcal{E}\) group-entity structure. HiComm (ours) is the three-phase receiver-driven protocol of §4: the receiver picks a topic group \(g^{*}\) and a sender \(j^{*}\) in two ST-GS stages, the chosen sender picks an entity \(\ell^{*}\!\in\!g^{*}\) on its own observation, and returns the address-value pair \(\big((g^{*},\ell^{*}),\,F\big)\), where \(F = z^{j^{*}}[g^{*},\ell^{*},:]\in\mathbb{R}^{\mathcal{F}}\) is the feature vector at the addressed coordinate.

5.1.0.2 Metrics.

We report team reward and per-receiver communication cost in KB per episode (per-step payload \(\times\) episode length). Across all benchmarks the receiver-side group pick (Phase 2 Stage 1) carries no data-dependent mask; the sender-side entity stage masks slots in \(g^{*}\) that sender \(j^{*}\) cannot currently observe, derived from \(j^{*}\)’s own mask \(M^{j^{*}}\). Per-benchmark simulator, wrapper, and baseline-port details are deferred to App. 7.1-7.3.

5.1.0.3 CAGE Challenge 4 (CC4).

CC4 is a procedurally generated enterprise-network cyber-defense benchmark in which five blue defenders face six red attackers over \(500\)-step episodes across nine subnets organized into five defended zones. The official release ships CC4 as a blue-side training problem with red implemented as a scripted finite-state machine; we instead lift red to a learned cooperative MARL task, freezing blue at the pretrained IPPO checkpoint of [39]. Each red agent is assigned to a single subnet and is initially inactive, with the sole exception of the agent in the Contractor Network; the remaining agents activate only after another red agent establishes a session within their subnet. Partial observability is accumulative: at each step a red agent learns a few new bits about hosts it has reached (an IP, a service, a privilege) and that knowledge persists. CC4 exposes a native subnet\(\,\supset\,\)host containment that aligns directly with HiComm, with \(\mathcal{G}{=}9\) subnets and \(\mathcal{E}{=}16\) hosts.

5.1.0.4 StarCraft Multi-Agent Challenge (SMACv2).

SMACv2 is a cooperative StarCraft II micromanagement benchmark in which each unit is operated by an independent agent under instantaneous sight-range fog of war and randomised start configurations. We adopt the standard \(5\)v\(5\) setup across the three races (protoss, terran, zerg); per-unit observations restricted to entities within sight, the discrete action set, and the damage-shaped reward with terminal win bonus all follow SMACv2 defaults. Because no native observation hierarchy exists, we induce one by indexing the group axis with ally unit class (a one-hot tag already exposed by the per-unit observation), with \(\mathcal{G}\) the active ally classes plus a single opponent group. HiComm unicasts every step to track rapid shifts in visibility.

5.1.0.5 Google Research Football (GRF).

GRF is a 3D physics-based football simulator with a discrete high-level action set, widely used as a partially observable cooperative-control benchmark. We adopt academy_counterattack_hard (\(4\)v\(3\)): a fast break in which \(N_a{=}3\) controlled attackers face two defenders plus a goalkeeper across \(400\) steps, with the controlled side’s keeper scripted following [40]. Each agent observes the simple115v2 feature vector restricted to entities within its sensing radius (\(\mathcal{F}{=}7\)); the \(19\)-action discrete control set and the Dense (scoring,checkpoints) reward of [40] are GRF defaults. Partial observability is again instantaneous. We index the group axis with each ally’s on-pitch field-position role (\(\{\text{GK},\text{DEF},\text{MID},\text{FWD}\}\), drawn from the engine) plus a single opponent group. This is the third group indexing under test, complementing the spatial (CC4) and unit-class (SMACv2) cases.

5.2 Results↩︎

Table 1 reports the IPPO and MAPPO grid on a representative scenario per benchmark, together with per-receiver per-episode communication cost (in kb).

5.2.0.1 Performance (RQ1).

HiComm closes a substantial fraction of the partial-to-full gap and outperforms both flat-message baselines in \(9\) of \(10\) algorithm-scenario cells. CC4 makes the algorithmic contrast especially clean. Under IPPO the no-communication PartialObs floor already reaches \(1{,}436\), roughly \(81\%\) of the \(1{,}763\) FullObs ceiling, so the headroom available to any communication protocol is narrow on this row; CACOM, T2MAC, and HiComm accordingly land within a tight band (\(1{,}517\), \(1{,}623\), \(1{,}527\)) and CC4-IPPO does not strongly differentiate the three communication methods on reward alone. Under MAPPO the picture changes sharply: the PartialObs floor collapses to \(990\), the partial-to-full gap more than triples, and HiComm delivers a markedly larger lift than the flat-message baselines, rising to \(1{,}941\) to recover \(\approx\!85\%\) of the gap to the \(2{,}110\) ceiling and beating CACOM by \(+249\) and T2MAC by \(+387\). The MAPPO row is therefore the clearest demonstration of HiComm’s advantage on this benchmark: it is precisely when the no-communication baseline leaves real headroom that the protocol’s gain becomes visible. On SMACv2, HiComm closes \(69\)-\(72\%\) of the gap (vs.\(15\)-\(46\%\) for the baselines); on GRF it closes \(67\%\) under IPPO and \(70\%\) under MAPPO. Querying coordinates in the group-entity grid lets the receiver retrieve precisely the parts of teammate state it cannot reconstruct from its own view, which a flat receiver-conditioned projection (CACOM) or a flat sender broadcast with evidence weighting (T2MAC) cannot match at the same channel width.

5.2.0.2 Bandwidth (RQ2).

HiComm attains these gains while shipping one to three orders of magnitude less data than every other communication setting: \(5\)-\(17\times\) less than CACOM/T2MAC on CC4, \(4\)-\(23\times\) less on SMACv2, and \(\approx\!10\times\) less than T2MAC on GRF (with CACOM’s already-narrow GRF channel only \(1.2\)-\(1.5\times\) wider than HiComm’s); relative to FullObs, the saving exceeds \(1000\times\) on CC4. Mechanically, HiComm writes one \((g^{*},\ell^{*})\) address plus the \(|\mathcal{F}|\)-dim feature vector \(F\) into a single message slot and zeros the rest, whereas the flat baselines populate every slot at every step. Because the address-value pair carries the feature vector verbatim rather than a lossy projection, the bandwidth saving comes without a fidelity penalty.

5.2.0.3 Generality (RQ3).

The performance-bandwidth pattern survives both critic styles and all three benchmarks’ group-axis instantiations: HiComm ranks first or tied-first on reward in \(9\) of \(10\) cells and is strictly cheapest in bandwidth in all \(10\), even though the underlying group axis is read off from a router map (CC4, native), a unit-type one-hot (SMACv2, semantic), or a coarse positional class (GRF, semantic). Switching between IPPO and MAPPO shifts absolute reward levels but does not perturb the relative ordering of the four communication settings, indicating that the gain attributable to HiComm lives in the message protocol rather than in the critic.

6 Conclusion and Limitations↩︎

In all, we presented HiComm, a receiver-driven communication protocol that grounds every message in the sender’s observation hierarchy. A query selects a topic group, a sender, and an entity via three Straight-Through Gumbel-Softmax draws, and the sender returns the raw feature vector at the addressed coordinate. Across CC4, SMACv2, and GRF under both IPPO and MAPPO, HiComm matches or beats learned-communication baselines in \(9\) of \(10\) algorithm-scenario cells while shipping up to \(23\times\) less data per receiver per episode. The principal limitation of this work is the absence of a formal theoretical analysis of HiComm’s communication efficiency: our bandwidth advantage rests on empirical comparison rather than an information-theoretic bound that explains why hierarchical addressing is provably more efficient than flat encoding at matched task performance. Establishing such a theory is the primary direction for follow-up work.

7 Additional Details for Experiments↩︎

All experiments were conducted on a single node equipped with 4 NVIDIA L40S GPUs. Representative single-run wall-clock times are approximately 24 hours for CC4 (5M env steps), 48 hours for SMACv2 5v5 (10M env steps, averaged across protoss/terran/zerg and across IPPO/MAPPO/QMIX), and 48 hours for GRF academy_counterattack_hard (5M env steps).

Sections 7.1 through 7.3 cover, for each benchmark, (i) simulator setup (wrappers, reward, baseline ports), (ii) the instantiation of \(\mathcal{G},\mathcal{E},\mathcal{F}\) that HiComm retrieves over, (iii) full tables extending Table 1. Sec. 7.4 ablates HiComm along three axes (payload form, retrieval factorisation, gradient pathway).

7.0.0.1 Common conventions across all three benchmarks.

(i) Grouping. CC4 uses the simulator’s native subnet \(\supset\) host containment; SMACv2 and GRF have no native containment, so we use a semantic grouping with one group per ally class plus one collapsed opponent group (communication is between teammates only and class-level addressing of an enemy is never queried). The class tag (unit type on SMACv2, on-pitch position on GRF) is read directly from the per-agent observation, so no additional supervision is required to instantiate \(\mathcal{G},\mathcal{E}\). (ii) Stationary opponent. Each benchmark’s opposing side is frozen or scripted (CC4: pretrained IPPO/MAPPO blue checkpoint; SMACv2: SC2 built-in heuristic AI at difficulty 7; GRF: built-in scripted AI), so from the controlled team’s perspective the environment is stationary throughout training and evaluation, and the only non-stationarity comes from per-episode randomisation (unit composition / start positions on SMACv2; player start positions on GRF). (iii) Observation segments. On SMACv2 and GRF the per-agent observation is a fixed-length flat vector formed by concatenating semantically distinct segments in a canonical order; we expose this structure to HiComm as a list of \((\text{slots},\,\text{features per slot})\) pairs called obs_segs, with per-benchmark segment lists given in the respective subsections. (iv) Baseline ports. T2MAC [24] and CACOM [16] were originally evaluated on SMACv1 / Hallway / MPE-style benchmarks under QMIX; our SMACv2 and GRF ports share three structural adjustments: (a) feed the per-agent observation into each baseline’s original observation encoder unchanged but hand it the canonical obs_segs layout, so its per-entity attention head sees ally and enemy rows as separate tokens (the same layout HiComm uses, ensuring all three protocols index the same per-agent state); (b) additionally train both baselines under IPPO and MAPPO with the same actor/critic architectures, PPO hyperparameters, entropy floor, and gradient clip values as our other policy-gradient baselines. (v) Mask scope. Per Sec. 4.2, the only data-dependent mask consumed by HiComm is the Phase 3 entity-stage mask \(M^{(\text{ent})}_{j^{*}}(g^{*})\), derived deterministically from sender \(j^{*}\)’s own observability mask \(M^{j^{*}}\). We therefore only specify when \(M^j[g,\ell]=1\) holds for each benchmark and let the Phase 3 mask follow mechanically. The HiComm modules of Sec. 4.2 are unchanged across the three benchmarks; only the partition \(\mathcal{G},\mathcal{E}\), the feature space \(\mathcal{F}\), and the semantics of \(M^j\) differ.

7.1 CAGE Challenge 4 (CC4)↩︎

This subsection covers (i) the wrapper stack that turns CC4 into a red-side MARL task, (ii) the native subnet \(\supset\) host hierarchy used by HiComm on this benchmark, and (iii) the full results table extending Table 1.

7.1.1 Red-Side MARL Reformulation↩︎

The original CC4 release [6], [7] distributes the environment as a blue-side only training problem. Concretely, the official PettingZoo interface exposes the five blue defenders as policy-gradient agents with per-step observations, an action mask, and a shared team reward, while the red team is hard-wired to a scripted finite-state machine (FiniteStateRedAgent) that selects actions from internal heuristics; red has no observation tensor, no reward channel, and no policy-callable action interface. Any attempt to train red with a standard RL algorithm against this stock release would therefore require either rewriting the simulator or replaying the FSM as a fixed environmental dynamic. Because this paper contributes a communication protocol for the attacker side, we reformulate red as a first-class MARL problem, keeping the simulator itself untouched and instead adding a thin wrapper stack around it that turns red into a standard PettingZoo multi-agent task. The result is drop-in compatible with off-the-shelf IPPO, MAPPO, and QMIX runners and is what every red-side training and evaluation number in this paper is run against.

7.1.1.1 Observation and action spaces.

A RedFlatWrapper exposes, for each of the six red agents, a \(722\)-dimensional observation that concatenates the \(9{\times}16{\times}5{=}720\) host matrix (subnet \(\to\) host, five per-host features: access level, recon level, neighbours_discovered, is_reachable, deception_known) with two global scalars (current mission phase, previous-action success flag). The action space is factorised as a pair \((a_\text{type}, (a_\text{subnet}, a_\text{host}))\) over ten primitive actions (DiscoverRemoteSystems, StealthServiceDiscovery, AggressiveServiceDiscovery, ExploitRemoteService, DegradeServices, DiscoverDeception, Withdraw, PrivilegeEscalate, Impact, Sleep) and all \(9{\times}16\) target coordinates, filtered by a per-step validity mask so the policy never commits to a syntactically illegal action.

7.1.1.2 Attack chain into Impact.

Impact is not directly invocable on an arbitrary host: red must first discover the subnet, scan the host’s services, exploit a service for a user-level session, and locally escalate to root. Table 2 names the five canonical stages with their advancing actions and per-host preconditions. Because Impact is the terminal stage of every successful trajectory, the count of successful Impacts is exactly the number of hosts on which red completed the entire five-stage chain in an episode; we report it alongside reward in Table 4 as a behavioural sanity check, since a method that inflates reward by spamming intermediate-stage actions (e.g.DegradeServices) would leave Successful #Impact unchanged.

4pt

7.1.1.3 Reward function.

On top of the stationary-opponent setup of Sec. 7 (here a HybridFixedBlueWrapper plugs in a fixed pretrained IPPO or MAPPO blue policy, the standard CC4 blue baseline), CC4’s official release [6], [7] bills three blue-side event types per step: Local Work Fails (a green agent’s local work action fails, e.g.while a blue agent is restoring its host), Access Service Fails (a green agent cannot connect to a service it relies on), and Red impact/access (red has run Impact on, or otherwise compromised, a host that a green agent occupies). Penalties are zone-specific and phase-dependent. We report all CC4 rewards in the red-reward convention \(r^{r}=-r^{b}\) to keep “higher is better” uniform across benchmarks; the simulator is unchanged. Table 3 consolidates CC4 Tables 4A-4C into a single zone \(\times\) phase grid in this convention (OZA spikes in Phase 2a, OZB in Phase 2b, CON only in Phase 1, INT never contributes). On each step our wrapper restricts the blue rewards \(r^{b}_{1:5}\) to the subset \(\mathcal{B}_t\) of defenders co-occurring with red under the current phase’s connectivity policy and assigns every red agent the shared scalar \[\label{eq:cc4-red-reward} r^{r}_t \;=\; -\frac{1}{|\mathcal{B}_t|}\sum_{b\in\mathcal{B}_t} r^{b}_{t},\tag{1}\] which is zero-sum on the co-occurring slice and is the only modification to CC4’s reward semantics.

4pt

7.1.2 Hierarchy Instantiation↩︎

CC4’s native subnet \(\supset\) host containment is used directly: \(\mathcal{G}{=}\{\text{RZA, OZA, RZB, OZB, CON, PAZ, ADM, OFF, INT}\}\) (the nine enterprise subnets), \(|\mathcal{E}|{=}16\) hosts per subnet, \(|\mathcal{F}|{=}5\) per-host features (access level, recon level, neighbours discovered, reachable, deception known). Observability mask: \(M^j[g,\ell]=1\) iff host \(\ell\) in subnet \(g\) has recon_level \(\ge 1\) in red agent \(j\)’s belief state.

7.1.3 Detailed Results↩︎

Table 4 extends the CC4 columns of Table 1 with (i) the NaiveCom reference (the canonical broadcast-autoencoder protocol, run to expose the encoder/reward decoupling pathology that motivates HiComm’s observation-entry payload design over a learned reconstruction objective; Sec. 4.2); (ii) the FSM scripted opponent as a non-RL reference; and (iii) Successful/Total Impact counts as a behavioural sanity check. The NaiveCom pathology is visible directly: total Impact attempts inflate while successful ones collapse below PartialObs under both algorithms, because reconstruction preserves what minimises MSE rather than what helps teammates succeed. HiComm’s observation-entry response avoids this failure mode, keeping successful Impact at or above PartialObs while substantially improving reward.

7.2 SMACv2↩︎

This subsection covers (i) the SMACv2 simulator setup we use, including the per agent observation and action spaces, the reward function in detail, and the ports of the T2MAC and CACOM baselines from SMACv1 to SMACv2 needed to run them on this benchmark, (ii) the semantic (unit type) hierarchy instantiation HiComm runs on this benchmark, and (iii) the full \(3{\times}5\) grid extending Table 1.

7.2.1 Setup and Baseline Ports↩︎

The original SMACv2 release [37] ships the StarCraft II micromanagement benchmark of [41] with two changes that defeat the memorisation shortcuts of SMACv1: per episode unit type compositions are sampled stochastically from a race specific distribution (weighted_teams on stalker/zealot/colossus for protoss, on marine/marauder/medivac for terran, and on zergling/hydralisk/baneling for zerg), and start positions are sampled from a surrounded_and_reflect distribution on a \(32{\times}32\) map rather than fixed across episodes. The reward schedule, per agent observation construction, and primitive action set are inherited from SMACv1 unchanged. We adopt the released configs verbatim for the \(5\)v\(5\) grid (n_units=5, n_enemies=5, step_mul=8, difficulty 7) and turn on capability_config.team_gen.observe=True so the per agent observation includes the sampled unit type one hot, which is what makes the role / position indexed hierarchy of HiComm addressable from the observation alone.

7.2.1.1 Observation and action spaces.

The per-agent observation segments (canonical order, exposed to HiComm as obs_segs; cf.Sec. 7) are:

1. Move availability, \((1, 4)\) — one bit per cardinal direction (N/S/E/W) indicating whether the unit can step that way under map collision.

2. Enemy entities, \((N_e, |\mathcal{F}|)\) — one row per enemy slot.

3. Ally entities (excluding self), \((N_a{-}1, |\mathcal{F}|)\) — one row per teammate slot, in the same per entity format as the enemy rows.

4. Own state, \((1, o_a)\) — current health, current shield (protoss only), and a unit class one hot of length \(3\).

5. Last action one hot, \((1, 11)\) — the agent’s previous primitive action; the \(5\)v\(5\) action space has \(N_e + 6 = 11\) entries (see action space description below).

6. Agent identity one hot, \((1, 5)\) — the agent’s slot index inside the controlled team of size \(N_a = 5\).

The per entity feature vector \(F\in\mathbb{R}^{|\mathcal{F}|}\) shared by enemy and ally rows carries the same fields in the same order: visibility bit, relative distance, relative \(x\), relative \(y\), current health, current shield (protoss only), and a \(3\)-way unit class one hot. This gives \(|\mathcal{F}|{=}9\) on protoss and \(|\mathcal{F}|{=}8\) on terran/zerg; the own state width \(o_a\) tracks the same shield distinction, with \(o_a{=}5\) on protoss and \(o_a{=}4\) on terran/zerg. Absent slots — a sampled team smaller than the per race upper bound, dead units, or units outside sight range — have their entire row zeroed and their visibility bit set to \(0\). On the \(5\)v\(5\) scenarios this resolves \((N_e, N_a{-}1)\) to \((5, 4)\), so the concrete obs_segs we hand to HiComm is \[\big[(1, 4),\;(5, |\mathcal{F}|),\;(4, |\mathcal{F}|),\;(1, o_a),\;(1, 11),\;(1, 5)\big],\] with \(|\mathcal{F}|\in\{8, 9\}\) and \(o_a\in\{4, 5\}\) chosen by race as above. The action space is a discrete head of size \(N_e + 6\) (\(N_e\) unit targeted attacks plus no-op, stop, and the four cardinal moves), with a per step validity mask that zeroes out attacks on out of sight enemies, moves blocked by terrain or map borders, and no-op unless the unit is dead. Sight range and shoot range are inherited from the SMACv1 unit table and are unit class specific (e.g.Stalker sight \(10\), Marine sight \(9\)); we do not modify these.

7.2.1.2 Reward function.

SMACv2 inherits the dense battle reward of SMACv1 [41] unchanged; SMACv2’s only deviations from v1 are the stochastic team composition and start positions, not the reward. With reward_only_positive=True (default), per-step team reward is \(r_t = \Delta H_{t}^{\text{enemy}} + \rho_{\text{death}} \cdot \Delta D_{t}^{\text{enemy}}\), where \(\Delta H\) is health\(+\)shield damage dealt and \(\Delta D\) is enemy units killed; on the terminal step a win bonus \(R_{\text{win}}\) is added if the enemy team is fully eliminated. The unscaled cumulative return is divided by a normaliser \(Z\) so an idealised maximum episode return is approximately \(20\): \[\label{eq:smacv2-scale} R \;=\; \frac{1}{Z}\!\Big(\!\sum_{t=1}^{T} r_t + R_{\text{win}}\!\cdot\! \mathbb{1}[\text{win}]\Big), \qquad Z \;=\; \frac{H_{\text{enemy,total}} + \rho_{\text{death}}\cdot N_{\text{enemy}} + R_{\text{win}}}{S},\tag{2}\] with \(S{=}\texttt{reward\_scale\_rate}\), \(H_{\text{enemy,total}}\) the sum of initial health\(+\)shield over enemy units, and \(N_{\text{enemy}}\) the enemy team size. Table 5 lists the coefficients used (matched to the official SMACv2 release). The shaping is identical across FullObs, PartialObs, CACOM, T2MAC, and HiComm, so the only mechanism by which a communication method can improve return is by changing what each agent observes.

7.2.1.3 Baseline ports from SMACv1 to SMACv2.

On top of the common port adjustments of Sec. 7, SMACv2 introduces one additional concern: SMACv1 has a fixed unit composition per scenario, so per-teammate output heads in the original baselines were sized to a fixed \(N_a\), whereas SMACv2 keeps the team size fixed but resamples which unit class fills each slot per episode. We rely on the unit-type one-hot inside the per-entity row to disambiguate slot identity rather than baking it into the architecture, so the per-teammate head dimensions of T2MAC and CACOM are unchanged across episodes.

Per-baseline specifics on SMACv2. For T2MAC, the evidence encoder emits an \(n\!\cdot\! K\)-dimensional Dirichlet concentration per teammate with \(K{=}N_e+6{=}11\) on \(5\)v\(5\), and the selector network, uncertainty-delta pseudo-labels, DST combination rule, and auxiliary BCE loss are taken from the published code without modification. For CACOM, latent dim \(=8\), attention dim \(=32\), and the discrete payload uses \(2\) bits per channel, matching the released SMACv1 configs; the two-round broadcast / gated-reply schedule, mutual-information loss weight (\(10^{-3}\)), entropy regulariser (\(10^{-2}\)), gating threshold, and gate training schedule are all unchanged. We disclose these ports so readers do not conflate our SMACv2 numbers with the SMACv1 numbers in the original papers.

7.2.1.4 Training budget.

Every (algorithm, method, scenario) is trained for \(10\)M environment steps, identically across all five communication settings (FullObs, PartialObs, CACOM, T2MAC, HiComm) and all three backbones (QMIX, IPPO, MAPPO). The budget is held fixed across the \(3{\times}5\) grid so that any cross method or cross algorithm differences in win rate and scaled return reflect the protocol or the optimiser, not the wall clock allowance.

7.2.2 Hierarchy Instantiation↩︎

SMACv2 has no native containment, so per the common convention (Sec. 7) we use semantic grouping with ally unit type as the group axis. Concretely, \(|\mathcal{G}|{=}|\mathcal{C}_{\text{ally}}|+1\) (one group per active ally type, plus one collapsed opponent group); per-race taxonomies are protoss \(=\{\)Zealot, Stalker, opponent\(\}\), terran \(=\{\)Marine, Marauder, Medivac, opponent\(\}\), zerg \(=\{\)Zergling, Hydralisk, Baneling, opponent\(\}\). Each group’s \(\mathcal{E}\) holds the unit slots padded to per-group capacity. The per-unit feature vector matches the native per-unit feature count: \(|\mathcal{F}|{=}9\) on protoss, \(|\mathcal{F}|{=}8\) on terran/zerg. Observability mask: \(M^j[g,\ell]=1\) iff sender \(j\) currently sees the unit at slot \(\ell\) of group \(g\) (alive, within sight range, non-padding).

7.2.3 Detailed Results↩︎

Table 6 extends Table 1 with the QMIX backbone, reporting reward alongside per-episode communication cost; Table 7 consolidates the underlying win rate across all three backbones (QMIX, IPPO, MAPPO). Bandwidth for IPPO and MAPPO is reported in the main paper and not duplicated here.

7.2.3.1 Win rate definition and computation.

Following [37], an evaluation episode is a win iff every enemy unit dies strictly before the horizon and at least one ally is alive at that moment; horizon timeouts and ally wipes both count as losses (no draws). Per (algorithm, method, scenario, seed) tuple we report the mean win rate over the last \(K{=}10\) evaluation checkpoints (each running \(32\) episodes with greedy/argmax action selection).

The QMIX bandwidth signature is consistent with the IPPO and MAPPO ones in the main paper: under QMIX, HiComm is the best communication method on all three \(5\)v\(5\) scenarios (\(+2.9\), \(+1.6\), \(+1.4\) reward points over CACOM and \(+3.3\), \(+2.0\), \(+1.6\) over T2MAC, equivalent to \(+25\), \(+12\), \(+10\) and \(+28\), \(+15\), \(+12\) win rate points respectively), closing most of the gap to FullObs, and does so at roughly \(4{\times}\) lower bandwidth than CACOM and roughly \(18{\times}\) lower than T2MAC (\(10.3\) vs.\(39.1\) vs.\(185.1\) KB per episode on protoss_5v5).

7.3 Google Research Football (GRF)↩︎

This subsection covers (i) the GRF simulator setup we use, including the per agent observation and action spaces, the reward function in detail, and the ports of the T2MAC and CACOM baselines to GRF, (ii) the semantic (player position) hierarchy instantiation HiComm runs on this benchmark, and (iii) the full results table extending the GRF column of Table 1 with the underlying mean win rate (%), the GRF community convention metric, on academy_counterattack_hard, the sole GRF scenario in our setup (Section 5.1.0.5).

7.3.1 Setup and Baseline Ports↩︎

The original GRF release [38] ships a continuous control football simulator with a discrete primitive action set and a small set of academy scenarios designed to isolate specific tactical motifs. We adopt academy_counterattack_hard (\(4\)v\(3\) with keeper) as the sole GRF scenario in our setup: a short fast break in which the controlled side has \(N_{\text{left}}{=}4\) left team players (three controllable attackers and one AI keeper) racing against \(N_{\text{right}}{=}3\) right team players (\(2\) defenders plus a goalkeeper) before the defenders can reorganise. The goalkeeper on the controlled side is handed off to GRF’s built in scripted policy following the academy scenario protocol of [40], leaving \(N_a{=}3\) controllable attackers; the right team is fully scripted. Episodes are capped at \(400\) steps and we use the representation="raw" observation channel so the per agent feature encoder has access to ball state, sticky actions, role indices, and team positions, matching the encoder of [38], [40] that subsequent learned communication work also builds on.

7.3.1.1 Observation and action spaces.

The per-agent observation segments (canonical order, exposed to HiComm as obs_segs; cf.Sec. 7) are:

1. Own / player, \((1, 29)\) — the agent’s position, direction, speed, a role one hot of length \(10\), and four state flags (ball far, tiredness, dribbling, sprinting).

2. Ball, \((1, 18)\) — ball position, a ball zone one hot of length \(6\), ball relative position, ball direction, ball speed, ball distance to the agent, ball ownership flag, and an “owned by us” flag.

3. Enemy entities, \((N_e, 7)\) — one row per enemy slot, with scaled position, direction, speed, distance to agent, and tiredness.

4. Ally entities (excluding self), \((N_{\text{left}}{-}1, 7)\) — one row per left-team teammate slot (i.e.the other policy-controlled attackers and the scripted goalkeeper, with \(N_{\text{left}}{=}4\)), in the same per-entity format as the enemy rows.

5. Closest left teammate summary, \((1, 7)\) — a single per entity row for the closest left team player to the agent.

6. Closest right opponent summary, \((1, 7)\) — a single per entity row for the closest right team player to the agent.

Crucially, the per entity feature vector \(F\in\mathbb{R}^{|\mathcal{F}|}\) is identical for the ally and enemy segments with \(|\mathcal{F}|{=}7\), which is the requirement under which a single HiComm response head can return either a teammate or an opponent row. On academy_counterattack_hard the entity counts instantiate as \((N_e, N_{\text{left}}{-}1) = (3, 3)\) (note the ally segment counts \(N_{\text{left}}{-}1\) teammates including the scripted keeper, not \(N_a{-}1\) controllable attackers), so the concrete obs_segs we hand to HiComm is \[\big[(1, 29),\;(1, 18),\;(3, 7),\;(3, 7),\;(1, 7),\;(1, 7)\big],\] giving a per agent observation size of \(29 + 18 + 21 + 21 + 7 + 7 = 103\) dims. The FullObs upper bound concatenates the per agent observation across all \(N_a{=}3\) controllable attackers (each re centred around its own active index by the feature encoder), giving an actor input of \(3 \times 103 = 309\) dims. The action space is the GRF default \(19\) dim discrete head: idle, eight directional moves (left, top left, top, top right, right, bottom right, bottom, bottom left), long_pass, high_pass, short_pass, shot, sprint, release_move, release_sprint, slide, dribble, and release_dribble; a per step validity mask zeroes out passes / shots when the agent does not own the ball or is too far from it, slides when the team owns the ball, set piece restrictions during goal kicks / corner kicks / penalty kicks, and the release actions whenever the matching sticky action is not currently engaged.

7.3.1.2 Reward function.

GRF natively exposes two reward signals, SCORING and CHECKPOINT, defined in the original release [38] and re-formalised as the Sparse and Dense variants by [40]. SCORING returns \[\label{eq:grf-scoring} r_t^{\text{score}} \;=\; +1 \cdot \mathbb{1}[\text{ally goal at }t] \;-\; 1 \cdot \mathbb{1}[\text{enemy goal at }t],\tag{3}\] i.e.a \(\{-1, 0, +1\}\) sparse signal that fires only on goal events. CHECKPOINT partitions the half pitch in front of the opponent goal into ten concentric checkpoint regions \(\{C_1, \ldots, C_{10}\}\) ordered by decreasing distance to goal, and grants \[\label{eq:grf-checkpoint} r_t^{\text{cp}} \;=\; \sum_{k=1}^{10} 0.1 \cdot \mathbb{1}\!\big[\text{first crossing of } C_k \text{ by ally ball-carrier at }t\big],\tag{4}\] which sums to at most \(1.0\) per episode (each region is rewarded at most once) and resets on possession exchange. The per step team reward we optimise is the Dense sum \[\label{eq:grf-dense} r_t \;=\; r_t^{\text{score}} \;+\; r_t^{\text{cp}},\tag{5}\] shared across all \(N_a\) controlled players. We instantiate eq. 5 via the GRF flag rewards="scoring,checkpoints", matching the Dense setting that [40] report as the strictly better variant on academy scenarios; Table 8 lists the per-step contributions. Eq. 5 is used identically across FullObs, PartialObs, CACOM, T2MAC, and HiComm (cf.SMACv2). The reported win rate depends only on the SCORING component and is therefore invariant to the CHECKPOINT shaping coefficient.

7.3.1.3 Baseline ports to GRF.

On top of the common port adjustments of Sec. 7, the only GRF-specific change is that the action head is the \(19\)-way GRF discrete set rather than \(K{=}11\) (SMACv2 \(5\)v\(5\)); both T2MAC’s Dirichlet/Subjective-Logic/DST machinery and CACOM’s gated-reply/mutual-information stack are defined over a discrete action space, so the port requires only the action-size change. T2MAC’s belief mass and uncertainty follow Eq. (2) of [24] with \(K{=}19\); CACOM reuses the SMACv2 port’s latent dim \(=8\), attention dim \(=32\), \(2\)-bit channel, MI weight \(10^{-3}\), and entropy regulariser \(10^{-2}\) verbatim, with the only environment-specific quantity being the observation encoder input width (\(103\) on academy_counterattack_hard).

7.3.1.4 Training budget.

Every (algorithm, method) configuration in Tables 1 and 9 is trained for \(5\)M environment steps on academy_counterattack_hard, allocated equally across the five communication settings (FullObs, PartialObs, CACOM, T2MAC, HiComm) under each of IPPO and MAPPO. The same \(5\)M-step budget is used for the baseline ports above so that any reward / win rate gap is attributable to the message protocol rather than to a difference in optimisation horizon. The reported win rate is averaged over \(32\) evaluation episodes against the same scripted opponent at the end of training.

7.3.2 Hierarchy Instantiation↩︎

Per the common convention (Sec. 7), GRF uses semantic grouping with each ally’s on-pitch position class as the group axis (drawn from \(\{\text{GK},\text{DEF},\text{MID},\text{FWD}\}\) via the role one-hot of segment (1)) plus one collapsed opponent group. academy_counterattack_hard (\(4\)v\(3\)) instantiates \(\{\)ally MID, ally FWD, opponent\(\}\) (the opponent group collapses enemy DEF and GK). Each group’s \(\mathcal{E}\) holds the player slots padded to per-group capacity; \(|\mathcal{F}|{=}7\) matches segments (3)-(6) above (scaled position, direction, speed, distance to agent, tiredness). Observability mask: \(M^j[g,\ell]=1\) iff the player at slot \(\ell\) of group \(g\) is within sender \(j\)’s sensing radius and the slot is non-padding.

7.3.3 Detailed Results↩︎

Table 9 extends the GRF column of Table 1 by reporting the underlying mean win rate (%), the GRF community convention metric, across the same IPPO and MAPPO grid on academy_counterattack_hard, after the \(5\)M-step training budget specified in Sec. 7.3 above.

7.4 Ablation Studies↩︎

We run a single ablation grid on CC4 against the same fixed pre-trained blue opponent and under the same training budget as the main results in Table 1. Every variant differs from HiComm along exactly one axis, holding the rest of the protocol (state encoder, hierarchy \(|\mathcal{G}|{=}9\) subnets \(\to |\mathcal{E}|{=}16\) hosts \(\to |\mathcal{F}|{=}5\) features, per-agent observability mask \(M^i\) as defined in Sec. 7.1 together with the Phase 3 entity-stage mask \(M^{(\text{ent})}_{j^{*}}(g^{*})\) of Sec. 4.2, unicast \(k{=}1\), PPO hyper-parameters, blue checkpoint) exactly as in Section 5.1.0.3. The variants partition into three axis groups (payload form, retrieval factorisation, gradient pathway) so that the marginal contribution of each design choice can be read off the resulting table. Each variant is trained on both IPPO and MAPPO and evaluated under the same protocol as Table 1.

7.4.0.1 Module naming.

Throughout this section we use the implementation names QueryNet, CommScheduler, DecodeNet, and host_selector for the four HiComm sub-modules of Sec. 4.2. Concretely, QueryNet produces the receiver query \(q_i = \tanh(f_\text{q}(h_i))\) (Phase 1); CommScheduler produces the Stage 2 sender-selection logits \(s_{ij\mid g^{*}}=\mathrm{score}(q_i,\Phi(j,g^{*};h_i))\); DecodeNet jointly hosts the Stage 1 group keys \(\{\kappa_g\}_{g=1}^{|\mathcal{G}|}\) (used at the receiver to pick \(g^{*}\)) and the Phase 3 entity projection \(\psi(\cdot)\) (used at sender \(j^{*}\) to pick \(\ell^{*}\)); and host_selector is the parameter-free gather \(z^{j^{*}}[g^{*},\ell^{*},:]\) that returns the addressed feature vector. Because \(\kappa_g\) and \(\psi\) live in DecodeNet, detaching “DecodeNet’s subnet/host Gumbel logits” below means detaching both the Stage 1 receiver-side group logits and the Phase 3 sender-side entity logits.

7.4.0.2 Payload form (observation entry vs.encoded).

Two variants probe the claim of Section 4.2 that the message should be the entry of the sender’s observation tensor at the retrieved coordinate, by replacing it with reconstruction-style encoders. HiComm-encoded replaces the raw \(|\mathcal{F}|{=}5\)-dim feature-vector lookup at the queried coordinate with a learned message encoder over the sender’s full host matrix (autoencoder-style, comparable in capacity to CACOM’s message head), so the receiver consumes a compressed summary rather than the feature vector itself; everything else is identical to HiComm. HiComm-encoded-cell keeps the two-stage hierarchical retrieval and unicast scheduler untouched but appends a small Linear(\(5{\to}8\))\(\to\)ReLU\(\to\)Linear(\(8{\to}5\)) MLP encoder after host_selector’s \(5\)-dim raw cell output, so the receiver consumes an encoded version of the same selected cell rather than the cell itself.

7.4.0.3 Retrieval factorisation (hierarchical vs.flat).

A single variant probes the contribution of the two-stage subnet\(\to\)host decode that gives HiComm its name. HiComm-Flat collapses the receiver-side Stage 1 group pick (against \(\{\kappa_g\}_{g=1}^{|\mathcal{G}|}\), Sec. 4.2) and the sender-side Phase 3 entity pick (against \(\psi(z^{j^{*}}[g,\ell,:])\)) into a single \(144\)-way Gumbel-Softmax over the flattened \(9{\times}16=144\) (subnet, host) coordinate set, executed at sender \(j^{*}\) once \(j^{*}\) has been selected by Stage 2; the receiver no longer transmits a topic group \(g^{*}\) and the sender alone produces the joint coordinate \((g^{*},\ell^{*})\). Phase 1 query, Phase 2 Stage 2 sender selection, the unicast bandwidth budget, and the raw-cell payload are all unchanged, so the only quantity that changes is whether the (group, entity) retrieval is factored hierarchically across the receiver/sender boundary or modelled jointly at the sender.

7.4.0.4 Gradient pathway (which modules receive PPO task-reward gradient).

The remaining variants stress-test the task-gradient pathway emphasised in Section 4.2 by selectively detaching each HiComm module’s output before the actor head, during the differentiable forward. Recall that the actor objective in Sec. 4.2 is \(\mathcal{L}_\text{actor}=\mathcal{L}^{\text{PPO}}-c_H\mathcal{H}+\lambda_{\text{ig}}(t)\,\mathcal{L}_\text{ig}\), where \(\mathcal{L}_\text{ig}\) is the information-gain shaping prior cross-entropying the Stage 1, Stage 2, and Phase 3 pre-ST-GS logits against soft labels (plus a small batch-marginal entropy term that discourages sender collapse). For the ablations only, we expose its three constituent terms separately so that the contribution of each pathway is readable: \(\mathcal{L}_{\text{decode}}\) is the cross-entropy on the Stage 1 subnet and Phase 3 host logits; \(\mathcal{L}_{\text{target}}\) is the cross-entropy on the Stage 2 sender logits; \(\mathcal{L}_{\text{explore}}\) is the batch-marginal entropy on sender selection. In the base HiComm all three run together as \(\mathcal{L}_\text{ig}\) alongside the PPO surrogate; the experiment isolates, for each module, how much of its learning is carried by the task-reward gradient once the corresponding sub-prior becomes the sole signal.

HiComm-NaiveComm .detach()s the outputs of CommScheduler, DecodeNet, and QueryNet simultaneously inside HiCommR_Actor.forward, so all three communication modules are trained only by the (decomposed) info-gain prior and the policy receives the resulting messages without back-propagating into the protocol. HiComm-NoCS .detach()s only CommScheduler’s output, leaving \(\mathcal{L}_{\text{target}}+\mathcal{L}_{\text{explore}}\) as the sole training signals for target selection while the query and decode paths still receive PPO gradient. HiComm-NoDec .detach()s DecodeNet’s subnet/host Gumbel logits (i.e.both Stage 1 and Phase 3) and trains the two-stage retrieval purely from \(\mathcal{L}_{\text{decode}}\). HiComm-NoQuery .detach()s QueryNet’s output \(q\); since none of the three sub-priors is attached to QueryNet directly, this leaves it without any training signal and the query becomes a fixed random projection of the receiver’s hidden state.

7.4.1 Results↩︎

Table 10 reports the ablation grid on CC4 over \(100\) evaluation episodes against the same fixed pre-trained blue opponent (IPPO blue policy) used in Table 4. Each variant is trained independently under both IPPO and MAPPO with the training budget and hyper-parameters of Section 7.1; only the axis listed in the variant description differs from the HiComm reference. We report mean episode reward, together with the number of Impact attempts and successes summed across the \(100\) episodes. HiComm (top row of each block) is the reference configuration; the ordering inside each block follows the payload \(\to\) retrieval \(\to\) gradient axis grouping of the variant descriptions above.

Across both algorithm blocks, HiComm (ref.) is the unique row that is strictly best on both reward and Success #Impact, and every single-axis variant loses on both metrics simultaneously. The Total #Impact column further shows that the reward gap is not driven by attempt count: variants that lower Total #Impact still lose Success #Impact, so the regression reflects worse strategic choices rather than fewer attempts. We read the three axis groups in turn.

7.4.1.1 Payload form.

Both encoded variants degrade on both algorithms, and Success #Impact tracks the reward drop. The cell-level encoder is the milder of the two but still costly, even though it is only a small MLP appended to host_selector’s raw cell output: a learned bottleneck whose reconstruction objective is decoupled from policy reward is enough to disrupt the raw lookup. The full-host autoencoder of HiComm-encoded is worse still, consistent with the encoder being dominated by the mostly-zero entries of an unexplored host matrix in CC4. This confirms the design choice of Section 4.2: with the receiver issuing a coordinate-level query, the natural payload is the raw observation entry at that coordinate, and inserting any encoder degrades the protocol.

7.4.1.2 Retrieval factorisation.

HiComm-Flat replaces the two-stage subnet\(\to\)host Gumbel-Softmax decode with a single Gumbel-Softmax over the flattened entity set. Reward drops modestly under IPPO but collapses under MAPPO, the largest single-axis regression in the table, with Success #Impact also dropping sharply. The hierarchical decomposition is therefore not a parametric shortcut: it injects CC4’s native subnet \(\supset\) host containment as an inductive bias that turns retrieval into two short categoricals, and the centralised MAPPO critic does not recover the lost optimisation tractability on its own.

7.4.1.3 Gradient pathway.

Detaching all three communication modules simultaneously (HiComm-NaiveComm, only the decomposed info-gain prior remains) hurts both algorithms, and substantially more under MAPPO: the prior partially compensates under IPPO but cannot recover the task-gradient signal that carries credit assignment under a centralised critic. The per-module breakdown isolates which of the three modules is most reliant on PPO gradient. HiComm-NoQuery (no signal at all, since none of \(\mathcal{L}_{\text{decode}}\), \(\mathcal{L}_{\text{target}}\), \(\mathcal{L}_{\text{explore}}\) attaches to QueryNet) suffers the most catastrophic regression on MAPPO: without any signal, the query collapses to a fixed random projection of the receiver hidden state and neither the retrieved coordinate nor the consumed payload can be aligned with what the policy actually needs. HiComm-NoDec (only \(\mathcal{L}_{\text{decode}}\), info-gain over \((g,\ell)\)) is also clearly worse than the reference, showing that an info-gain prior is a useful but insufficient substitute for task gradient through the two-stage retrieval, since informative entries are not always reward-relevant. HiComm-NoCS (only \(\mathcal{L}_{\text{target}}+\mathcal{L}_{\text{explore}}\)) is the mildest perturbation: the per-target novelty proxy is enough to keep target selection close to the reference, indicating that which sender to query is the easiest of the three sub-problems to learn from the prior alone. Together, the gradient-pathway block establishes that the task-reward gradient is essential precisely where the prior is least informative (QueryNet, then DecodeNet), and motivates running the full HiComm stack inside the actor’s differentiable forward as in Section 4.2.

7.4.1.4 Takeaway.

Across the results, we found that removing the raw observation-entry payload, the hierarchical retrieval factorisation, or the task-reward gradient pathway each induces a distinct failure mode, and no alternative training signal we tested recovers the lost performance. Across all six axis perturbations and both PPO surrogates, HiComm (ref.) is strictly best on reward and Success #Impact, supporting the claim that each of the three design choices in Section 4.2 is necessary for the protocol as a whole.

References↩︎

---

1. Corresponding author: Ankit Shah (ankit@iu.edu).↩︎