1 Introduction↩︎

Many AI scenarios, such as autonomous driving [1], smart grids [2], and IoT applications [3], can be modeled as self-interested and online learning multi-agent systems (MAS), where conflicts arise due to opposing goals or shared resources [4]. To maximize social welfare, cooperation is essential in self-interested MAS, requiring agents to avoid selfish behavior for the collective good. This tension between individual and collective rationality is typically modeled as a social dilemma (SD) [5], [6]. Designing distributed mechanisms to incentivize cooperation in SDs remains a key challenge because it directly affects global efficiency (e.g., aggregated social welfare as defined in Sec. [subsec:problem95formulation]) and sustainability (e.g., long-run resource availability and fairness across agents) [6]–[9]. Multi-agent reinforcement learning (MARL) is a widely used framework to train rational agents in SDs and temporally extended sequential SDs (SSD), where each agent maximizes its own reward [9]–[12].

However, non-cooperative game theory and empirical studies show that naive MARL approaches often fail to sustain cooperation due to independent adaptation leading to mutual defection [6], [13]. Further, policy optimization can be affected by inconsistencies in the reward function, summarized as changing rewards. Such variations are common when moving from simulation to reality, where abstract reward proxies are replaced by physical signals. Noisy sensors, hardware degradation, shifting specifications, or changing market demands can likewise alter reward scales during training [14]. In human-in-the-loop settings, evaluators may also adapt feedback online, leading to abrupt or irregular shifts [15]. In these cases, the underlying SSD structure (i.e., greed and fear inequalities) is preserved, but the numerical values fluctuate. Methods that are brittle to such changes struggle to sustain cooperation beyond narrow, hand-tuned conditions.
Peer incentivization (PI) enables agents to reward or penalize each other to foster cooperation in SDs [12], [16]. It has gained traction for its success in complex SSDs and links to biology, economics, and social science [17]–[19]. Yet, existing PI methods rely on fixed or domain-tuned incentive values and fail under changing rewards, even if cooperation conditions remain unchanged [20], [21]. As Fig. 1 illustrates, prior PI methods might succeed in a given instance of the Prisoner’s Dilemma [11], [12], [22], but fail when payoffs shift or scale, despite satisfying the same greed and fear inequalities [6], [23]. Notably, popular approaches such as Learning to Incentivize Other learning agents (LIO) [12] or inequity aversion (IA) [24] appear superficially capable of adapting to such variation, since they rely on learned or relative incentives. Yet, IA requires precise hyperparameter tuning to be effective and fails to promote cooperation in both the original (unscaled) payoff matrix of Fig. 1 (left) and the uniformly scaled variant (right). LIO achieves cooperation under the original payoffs but degrades under simple scaling, as its learned incentive magnitudes are tied to the absolute reward scale. As a result, these methods either lose robustness under reward shifts or suppress valuable heterogeneity in agent performance. Addressing such shifts typically requires hyperparameter retuning, which is impractical for ad-hoc- or online learning scenarios.
To this end, we propose Dynamic Reward Incentives for Variable Exchange (DRIVE), an adaptive PI framework for SDs with changing rewards. By adapting incentives directly to reward differences DRIVE sustains cooperation without retuning and without conflating legitimate variance with exploitative behavior. We summarize our contributions as follows:

• We introduce a reciprocal exchange mechanism to incentivize cooperation via reward differences.

• We prove that DRIVE aligns incentives toward mutual cooperation in generalized Prisoner’s Dilemma games, remains invariant to reward shifts and scaling.

• We empirically demonstrate DRIVE’s robustness and superior cooperation performance in SSDs with changing rewards.

2 Background↩︎

Problem Formulation. In this work, we focus on partially observable Markov games \(M = \langle \mathbb{D},\mathbb{S},\mathbb{A},\mathbb{P},\mathbb{U},\mathbb{Z},\Omega \rangle\), where \(\mathbb{D} = \{1,...,n\}\) is a set of agents \(i\), \(\mathbb{S}\) is a set of states \(s_{t}\) at time step \(t\), \(\mathbb{A} = \langle \mathbb{A}_{1}, ..., \mathbb{A}_{n} \rangle = \langle \mathbb{A}_{i} \rangle_{i \in \mathbb{D}}\) is the set of joint actions \(a_{t} = \langle a_{t,i} \rangle_{i \in \mathbb{D}}\), \(\mathbb{P}(s_{t+1}|s_{t}, a_{t})\) is the transition probability, \(\langle u_{t,i} \rangle_{i \in \mathbb{D}} = \mathbb{U}(s_{t},a_{t}) \in \mathbb{R}\) is the joint reward, \(\mathbb{Z}\) is a set of local observations \(z_{t,i}\) for each agent \(i\), and \(\Omega(s_{t+1}) = z_{t+1} = \langle z_{t+1,i} \rangle_{i \in \mathbb{D}} \in \mathbb{Z}^{n}\) is the subsequent joint observation. Each agent \(i\) maintains a local history \(\tau_{t,i} \in (\mathbb{Z} \times \mathbb{A}_{i})^{t}\). \(\pi_{i}(a_{t,i}|\tau_{t,i})\) is the action selection probability based on the individual policy of agent \(i\). In addition, we assume each agent \(i\) to have a neighborhood \(\mathcal{N}_{t,i} \subseteq \mathbb{D} - \{i\}\) of other agents at every time step \(t\), which is domain-dependent, as suggested in [16], [25]. \(\pi_{i}\) is evaluated with a value function \(V_{i}^{\pi}(s_{t}) = \mathbb{E}_{\pi}[G_{t,i}|s_{t}]\) for all \(s_{t} \in \mathbb{S}\), where \(G_{t,i} = \sum_{k=0}^{\infty} \gamma^{k} u_{t+k,i}\) is the individual and discounted return of agent \(i \in \mathbb{D}\) with discount factor \(\gamma \in [0,1)\) and \(\pi = \langle \pi_{j} \rangle_{j \in \mathbb{D}}\) is the joint policy of the MAS. The goal of agent \(i\) is to find a best response \(\pi_{i}^{*}\) with \(V_{i}^{*} = max_{\pi_{i}}V_{i}^{\langle \pi_{i}, \pi_{-i} \rangle}\) for all \(s_{t} \in \mathbb{S}\), where \(\pi_{-i}\) is the joint policy without agent \(i\). In practice, the global state \(s_{t}\) is not directly observable for any agent \(i\) s.t. \(V_{i}^{\pi}\) is approximated with local information, i.e., \(\tau_{t,i}\) instead [10], [26], [27]. To measure cooperation in the MAS, we define the social welfare or utilitarian metric \(U = \sum_{i \in \mathbb{D}}\sum_{t=0}^{H-1} u_{t,i}\) as undiscounted sum of rewards.

Social Dilemmas. Social dilemmas (SD) are games where independently optimized policies \(\pi_{i}\) fail to achieve globally optimal outcomes that maximize collective welfare. SDs are commonly studied in 2-player matrix games with two actions: \(C\) (cooperate) and \(D\) (defect), producing four possible payoffs (Fig. 2): \(R\) for mutual cooperation, \(P\) for mutual defection, \(T\) for exploiting the other, and \(S\) for being exploited. A matrix game is a Prisoner’s Dilemma (PD) if the payoffs satisfy [6], [23]: \[\label{eq:PD95inequalities} T > R > P > S\tag{1}\] Here, \(T > R\) represents greed, and \(P > S\) fear. In Iterated PDs (IPD), an additional condition holds: \(2R > T + S\) [6], [23]. Fig. [fig:PD95instance] shows an instance where \(D\) is individually rational despite \(C\) being socially optimal. As long as the inequalities are satisfied, the game’s strategic nature remains invariant to exact payoffs [5], [6]. PDs and IPDs are particularly important SDs, as greed and fear often drive agents away from mutual cooperation despite its collective benefit, a phenomenon observed in both nature and human society [5], [6], [28].
Sequential social dilemmas (SSD) extend SDs by introducing temporal structure, modeled as stochastic games [9], [10]. SSDs allow more realistic scenarios where behavior is captured by policies, not atomic actions. These can still be mapped to matrix games by classifying policies as \(C\) or \(D\) and evaluating empirical payoffs [10], making core SD concepts applicable to SSDs.

Multi-Agent Reinforcement Learning. We consider decentralized (independent) learning, where each agent \(i\) optimizes its policy \(\pi_{i}\) using local data like \(\tau_{t,i}\), \(a_{t,i}\), \(u_{t,i}\), and \(z_{t+1,i}\) via reinforcement learning (RL) [11], [25], [29]. For methods with peer incentives, the shaped reward simply replaces \(u_{t,i}\) in this trajectory. Policy gradient RL is a common method to approximate best responses \(\pi_{i}^{*}\) [11], [12], [30]. A function approximator \(\hat{\pi}_{i,\phi_{i}} \approx \pi_{i}^{*}\) is trained using gradient ascent on an estimate of \(J = \mathbb{E}_{\pi}[G_{0,i}]\) [31], where the policy gradient is approximated as [32]: \[\label{eq:policy95gradients} g = (G_{t,i} - b_{i}(s_{t}))\nabla_{\phi{i}} \textit{log} \hat{\pi}_{i,\phi_{i}}(a_{t,i}|\tau_{t,i})\tag{2}\] Here, \(b_{i}(s_{t})\) is a state-dependent baseline, typically approximated by a learned value function \(\hat{V}_{i,\omega_{i}}(\tau_{t,i}) \approx V_{i}^{\hat{\pi}}(s_{t})\) [11]. For simplicity, we omit parameters and write \(\hat{\pi}_{i}\), \(\hat{V}_{i}\). Modern actor-critic methods add such variance-reduction baselines, entropy bonuses, or centralized critics, but still only optimize each agent’s own return, which remains misaligned with social welfare in social dilemmas. Also, independent learning introduces non-stationarity as agents adapt simultaneously [33], [34], often driving overly greedy behavior and mutual defection unless rewards or incentives are modified.

3 Dynamic Reward Incentivization↩︎

We assume a decentralized MARL setting as formulated in Algorithm 3, where at every time step \(t\) each agent \(i\) with history \(\tau_{t,i}\), policy \(\hat{\pi}_{i}\), and value function \(\hat{V}_{i}\) observes its neighborhood \(\mathcal{N}_{t,i}\) through its local observation \(z_{t,i}\) and executes an action \(a_{t,i} \sim\pi_{i}(\cdot|\tau_{t,i})\). DRIVE uses \(\mathcal{N}_{t,i}\) only for the incentive exchange; the underlying policy does not receive additional messages beyond the environment observation. The environment then transitions to a new state \(s_{t+1} \sim \mathbb{P}(\cdot|s_{t}, a_{t})\) which is observed by each agent \(i\) through an observation \(z_{t+1,i}\) and a reward \(\hat{u}_{t,i}\). This is obtained by passing the environmental reward \(u_{t,i}\) through an external and possibly unknown reward‑change function \(f_{\textit{mod}}\) (Alg. 3, l. [l:r95mod]), which simulates modified specifications, varying supply and demand, or sensor degradation [20], [21]. In Sec. 5 we analyze an affine special case of \(f_{\textit{mod}}\), applying a shared map varying by epoch. All agents collect their respective experience tuple \(e_{t,i} = \langle \tau_{t,i}, a_{t,i}, \hat{u}_{t,i}, z_{t+1,i} \rangle\) for PI exchange and independent updates to \(\hat{\pi}_{i}\) and \(\hat{V}_{i}\) [12]. We assume each agent \(i\) has a domain‑dependent neighborhood \(\mathcal{N}_{t,i}\subseteq D \setminus \{i\}\) at each time step, as introduced in Sec. [subsec:problem95formulation]. Lines 13–20 in Alg. 3 correspond to a short parallel communication phase in which neighboring agents exchange DRIVE requests and responses.

DRIVE Token Exchange. DRIVE uses a reciprocal incentive scheme, inspired by [7] and illustrated in Fig. 4, to exchange dynamic incentives for distributed reward shaping. In the request phase (Fig. 4 (a)), each agent \(i\) checks its advantage or temporal difference residual \(\textit{TD}_{i}(\hat{u}_{t,i})\) [35]: \[\label{eq:mi95td} \textit{TD}_{i}(\hat{u}_{t,i}) = \hat{u}_{t,i} + \gamma \hat{V}_{i}(\tau_{t+1,i}) - \hat{V}_{i}(\tau_{t,i})\tag{3}\] If the advantage is non-negative, agent \(i\) sends its reward \(\hat{u}_{t,i}\) as a request to all other agents \(j \in \mathcal{N}_{t,i}\). In the response phase (Fig. 4 (b)), each request‑receiving neighbor \(j \in \mathcal{N}_{t,i}\) compares the request \(\hat{u}_{t,i}\) to its epoch‑average reward \(\overline{u}_j\) and computes \(\Delta_{t,1,2} = \overline{u}_2 - \hat{u}_{t,1}\), which is then sent back as a response. After this exchange, the DRIVE reward \(u^{\textit{DRIVE}}_{t,i}\) is computed for each agent \(i\) as follows: \[\label{eq:DRIVE95reward} u^{\textit{DRIVE}}_{t,i} = \hat{u}_{t,i} - \textit{min}\{\langle \Delta_{t,j,i} \rangle_{j \in \mathcal{N}_{t,i}}\} + \textit{min}\{\langle \Delta_{t,i,j} \rangle_{j \in \mathcal{N}_{t,i}}\}\tag{4}\] Using the responder’s epoch‑average \(\overline{u}_j\) rather than its instantaneous reward makes DRIVE sensitive to systematic exploitation instead of single noisy outcomes: only agents whose recent average return is consistently lower than a neighbor’s request generate strong negative \(\Delta\) terms. Other combinations (instant–instant, instant–average, average–average) are conceivable and may yield different trade‑offs, which is an interesting direction for future work. Appendix 9.1 provides an initial comparison.

Distributed Reward Shaping. The DRIVE reward \(u^{\textit{DRIVE}}_{t,i}\) is used to update the policies of the corresponding agents using any RL algorithm, e.g., policy gradient methods, according to Eq. 2 . Formally, whenever DRIVE is enabled we obtain returns \(G_{t,i}\) by replacing \(u_{t,i}\) with \(u^{\text{DRIVE}}_{t,i}\) in Eq. 2 (cf., Alg. 3, l. 17). Sec. [subsec:problem95formulation] therefore describes the environment-level returns, while DRIVE defines how these are transformed into the shaped rewards that the RL updates optimize. The non-negativity condition of advantage \(\textit{TD}_{i}(\hat{u}_{t,i})\) in the DRIVE request is needed to expose defecting agents that typically have a greater advantage than the exploited agents in SDs [6]. If there is unilateral defective behavior, the defective agent \(i\) will be penalized by the most exploited neighbor agent \(j \in \mathcal{N}_{t,i}\), since \(\Delta_{t,i,j} < 0\), which is ensured by the \(\textit{min}\) aggregation terms¹ in Eq. 4 . However, if all agents act equally cooperative, then all \(\Delta_{t,i,j} = 0\), and there is no additional reward or penalization. The complete formulation of DRIVE at time step \(t\) for any agent \(i\) is given in Algorithm 5. \(\hat{V}_{i}\) is the approximated value function to calculate \(\textit{TD}_{i}(\hat{u}_{t,i})\), according to Eq. 3 , \(\mathcal{N}_{t,i}\) is the current local neighborhood, \(e_{t,i}\) is the experience tuple, and \(\overline{u}_{i}\) is the current average reward of agent \(i\).

4 Related Work↩︎

MARL in Social Dilemmas. MARL has achieved substantial progress across a range of domains [4], [29], [36], [37]. In decentralized SDs and SSDs, a key challenge is resolving misaligned incentives without centralized control. Recent work addresses this through local, peer-induced, or socially conditioned reward-shaping mechanisms [12], [22], [24], [38], which operate in mixed-motive environments with observation-limited agents and thus align with our setting. By contrast, much of the broader MARL literature — methods for mitigating non-stationarity in cooperative settings [39]–[41], globally informed reward shaping [10], [42], opponent-shaping requiring access to opponents’ parameters [11], [43]–[45], and centralized incentive design [46], [47] — assumes global information or centralized coordination, and therefore does not extend to decentralized SDs.
Peer Incentivization. Many PI methods have been introduced to promote mutual cooperation in a distributed fashion via reward exchange [48]. Gifting extends the action space of each agent \(i\) with a reward action to incentivize other agents \(j \in \mathcal{N}_{t,i}\) [16]. In contrast, DRIVE is built upon reciprocal reward shaping and does not require an extended action space. [18] proposes market-based PI where agents can establish bilateral agreements. A public sanctioning approach, where agents can reward or penalize each other based on known group behavior patterns, has been proposed in [17]. [24] defines an inequity aversion (IA) scheme based on non-negative reward differences weighted by two domain-dependent coefficients. Learning to Incentivize Other learning agents (LIO) automatically learns an incentive function for each agent \(i\) under full observability, based on the joint action of all other agents \(j \neq i\) [12]. [49] propose Mutual Acknowledgment Token Exchange (MATE), a two‑phase protocol where agents exchange fixed tokens x: cooperating agents issue tokens while exploited agents receive them, yielding the modified payoff matrix shown in Fig. 6. Most PI methods are sensitive to the reward values of an environment due to relying on fixed incentive values or domain-dependent parameters, thus fail to cooperate when the reward values change (Fig. 1). Considering the PD as an example, Fig. 6 shows the modified payoff matrices of DRIVE, MATE, LIO, and IA. According to these matrices, DRIVE is the only method that does not depend on any particular hyperparameter or explicitly learned value. IA only modifies unilateral cooperation and defection but depends on two domain-dependent coefficients \(\alpha\) and \(\beta\) [24]. LIO learns an incentive function conditioning on the actions of the other agents, which requires sufficient experience and time to adapt accordingly. In case of MATE, the cooperation depends on an appropriate global token \(\mathbf{x} \geq \textit{max}\{P-S, \frac{T-R}{3}\}\) that can be derived from the modified payoff matrix. In the example instance in Fig. [fig:PD95instance], \(\mathbf{x}\) should be at least 1, which is the original token value used for MATE in [22]. Interestingly, this value is used for other PI methods as well without further analysis or questioning [16], [18].
Teaming. Beyond independent learners, recent work studies how social preferences, teams, or coalition structures shape incentives in mixed-motive settings, e.g., heterogeneous SVO-based reward shaping, team-based reward sharing, and adaptive reward mixing via price-of-anarchy minimization [50]–[53]. These methods typically assume fixed or emergent group structures or modify each agent’s global utility function. In contrast, DRIVE introduces no explicit coalitions or shared rewards: incentives arise solely through local, bilateral reward exchanges, enabling decentralized alignment without predefined teams or population-level reward mixing.

5 Theoretical Analysis↩︎

In the following, we analyze the incentive alignment properties of DRIVE in the general PD and its invariance to changing rewards. We emphasize that the mechanism does not guarantee convergence of learning dynamics, but rather reshapes incentives such that cooperation becomes the individually rational choice in repeated interactions. Although our formal discussion focuses on the canonical PD, many well-known sequential social dilemmas (SSDs), such as Coin and Harvest, instantiate PD-like incentive structures, as shown empirically in [10] and explained in the Appendix. In Section 7, we further show that DRIVE also yields promising results in more complex SSDs with more than two agents.

5.1 DRIVE in the General Prisoner’s Dilemma↩︎

In social dilemmas with a payoff table, as shown in Fig. 2, and inequalities \(T > R > P > S\) of Eq. 1 , the DRIVE incentives do not change the payoffs \(R\) for mutual cooperation and \(P\) for mutual defection in the long run because the reward difference \(\Delta_{t,i,j}\) would be zero for both agents. With \(R > P\), this will still favor mutual cooperation over mutual defection. However, if there were (repeated) unilateral defection, the defective agent \(i\) would send a request \(T\) to the exploited agent \(j\), which responds with a difference of \(\Delta_{t,i,j} = S - T < 0\) in the worst case, where the reward of the defective agent \(i\) would change to \(T + \Delta_{t,i,j} = T + S - T = S\), while the reward of the exploited agent \(j\) would change to \(S - \Delta_{t,i,j} = S - (S - T) = T\), according to Eq. 4 . As shown in Fig. 6, the DRIVE payoff switch of \(T\) and \(S\) enables both agents to overcome greed and fear, therefore incentivizing cooperation.

Overall, DRIVE does not rely on fixed incentive values or domain-dependent parameters. Thus, it can cope with reward changes that still satisfy the PD inequalities w.r.t. greed and fear, as in the example of Fig. 1. For clarity, Appendix 9.1 provides an extended example that concretely demonstrates the shaping mechanism in a single PD exchange step. Note that Eq. 4 effectively turns the stage game into a coordination game with \((C,C)\) as the unique Nash equilibrium. This reflects standard reward-design practice: instead of relying on learners to infer long-term externalities, we adjust instantaneous payoffs so that cooperation is individually rational while leaving the environment dynamics unchanged.

5.2 Invariance to Changing Rewards↩︎

We now analyze DRIVE and MATE w.r.t.their invariance to changing rewards in the PD, i.e., where the original payoffs \(R\), \(P\), \(T\), and \(S\) are dynamically altered by some external change function \(f_{\textit{mod}}\).

Note that Theorem 2 can be generalized to any other PI method that uses fixed peer incentive values. One could in principle re‑tune \(x\) after each change in \(f_{\textit{mod}}\), but this requires additional global knowledge or meta‑optimization, whereas DRIVE adapts automatically through local reward differences.

We now extend the invariance analysis from the PD to general SSDs using policy gradient methods with return normalization, as detailed in Appendix 9.6.

Consider sequential social dilemmas with returns normalized per epoch to zero mean and unit variance. If rewards are transformed by a positive affine map per epoch then both (i) standard policy gradient learning and (ii) DRIVE’s shaped rewards remain invariant with respect to the normalized return \(\overline{G}_{t,i}\). Consequently, the incentive alignment induced by DRIVE persists under per-epoch reward scaling and shifting within this class of transformations. Overall, under per-epoch normalization, both the base policy-gradient updates and DRIVE’s shaped rewards are invariant to shared per-epoch positive affine reward transformations (Lemma 2 in Appendix 9.6).

5.3 Effects of Non-Adherence to the Protocol↩︎

The preceding analysis assumes that all agents truthfully and synchronously follow the DRIVE protocol, i.e., each agent shares its epoch-average reward, responds to requests, and applies the shaping rule consistently. In practice, however, communication failures, delays, or strategic misreporting may occur. Here we analyze the impact of such non-adherence compared to other PI approaches; for a detailed example, see Appendix 9.2.
Full compliance. When all agents follow the protocol truthfully, the results of Theorem 1 and Theorems 3–4 apply. Payoffs \(T\) and \(S\) are swapped under unilateral defection, ensuring incentive alignment toward cooperation. This reasoning can be generalized to SSDs beyond the PD, provided the characteristic inequalities for greed and fear are preserved.
No requests sent (despite TD \(\geq 0\)). An agent that withholds requests can still receive requests and respond truthfully. For its neighbors, this means one fewer reciprocal request. Due to the \(\min\) aggregation, their penalization levels may increase slightly, but are not substantially affected as long as other agents send requests. For the non-requesting agent itself, \(\hat{u}_{\textit{res}}=0\), so it forfeits the opportunity for reciprocal improvement and is indirectly penalized. If the agent defects, it may temporarily avoid penalization, but overall, it loses the benefits of mutual shaping and does not gain a stable advantage.
No responses sent. If an agent does not respond to requests (due to communication loss or intentional withholding), the effect depends on the context: If the agent is defecting, withholding its negative \(\Delta\) weakens the penalty on its exploited neighbor. However, it also loses the chance to benefit from reciprocity, so deliberate non-response is not rational. If the agent is cooperative, the missing (typically positive or small) \(\Delta\) has limited effect because the \(\min\) aggregation dampens single missing contributions. Thus, non-response is mostly neutral or slightly beneficial for defectors but costly overall.
False requests or responses. Two types of misbehavior are possible: Sending requests with TD \(<0\): This can occur due to noise or early training misestimation. Such requests typically yield small or positive differences that are filtered out by the \(\min\) operator and thus have little impact. Sending false responses: An agent could deliberately misreport \(\Delta\) to distort shaping. In the worst case, the \(\min\) aggregation could impose unjustified penalties on requesters, potentially undermining cooperation. This depicts a fragility and limitation of DRIVE.
No compliance at all. If all agents withhold or communication fails globally, then all shaping terms vanish. The system collapses to plain MARL without incentives, and mutual defection again becomes a possible equilibrium. This is similar to fixed-token methods like MATE, which also break down if their coordination mechanism is unavailable.

5.4 Conceptual Discussion↩︎

DRIVE offers a simple, adaptive PI mechanism for fostering cooperation in SDs with changing rewards. It operates through local peer interactions, requiring no central controller, global knowledge, or all-to-all communication. Similar to [12], [22], [24], Algorithm 5 scales linearly with \(\mathcal{O}(4(n-1))\) in the worst case with respect to incentive exchanges. At the same time, DRIVE relies on truthful peer communication to function correctly. Partial compliance leads to graceful degradation: as long as at least one honest responder provides feedback, defectors are still penalized; if all agents fail to comply, DRIVE reduces to baseline MARL dynamics. Compared to fixed-token methods, DRIVE is at least as robust and often more adaptive, but its reliance on strict adherence to the exchange protocol introduces a new potential fragility. Designing protocol variants that are robust to partial compliance – for example through redundant aggregation rules, stochastic auditing, or alternative shaping operators — remains an important avenue for future work.
A key modeling assumption is the neighborhood \(N_{t,i}\) that restricts which agents can exchange incentives. In our experiments, this neighborhood is tied to the environment as introduced in the subsequent section. Crucially, DRIVE’s min-aggregation means a defector’s shaped reward depends only on its most exploited neighbor, so, unlike simple reward-sharing schemes, penalties do not dilute with larger populations. Appendix 9.2 extends the previous compliance analysis to general N-agent systems and arbitrary communication graphs, and formalizes the resulting robustness: defector penalization holds exactly when compliant agents form a dominating set. Thus cooperation incentives degrade smoothly rather than collapsing abruptly under partial compliance.
As shown in the modified PD payoff tables in Fig. 6, DRIVE only depends on the environmental rewards without requiring fixed incentive values \(x\) as MATE, domain-dependent coefficients \(\mathbf{\alpha}, \mathbf{\beta}\) as IA, or time-consuming incentive learning of \(x^i_C, x^i_D\) as LIO. Note that given the characteristics of greed and fear exhibited by general PDs, the above theoretical analysis generally applies to various SDs. Thus, DRIVE can be used in any SD, where rewards change dynamically through an external and unknown change function \(f_{\textit{mod}}\). If the conditions for cooperation, i.e., characteristic inequalities of the SD, remain the same, DRIVE can adapt and maintain cooperation, as shown in Fig. 1 and later demonstrated.

6 Experimental Setup↩︎

We use three well-known SSDs based on [11], [26]. At every time step, the order of agent actions is randomized to resolve conflicts, e.g., when multiple agents step on a coin or tag each other simultaneously. All SSDs represent PD instances, as empirically shown in [10] and Appendix 10. Since we modify the rewards by \(f_{\textit{mod}}\), we assess the cooperation with domain-specific measures.
Iterated Prisoner’s Dilemma. For IPD, we use the payoff matrix shown in Fig. [fig:PD95instance]. Both agents observe the previous joint action \(z_{t,i} = a_{t-1}\) at every time step \(t\), which is the zero vector at the start state \(s_{0}\). The Nash equilibrium is to always defect (DD). An episode consists of \(H = 150\) iterations, and we set \(\gamma=0.95\). The neighborhood \(\mathcal{N}_{t,i} = \{j\}\) is defined by the other agent \(j \neq i\). We measure the Cooperation Rate as the cooperation (CC) count per episode divided by \(H\).
Coin. Coin-2 and Coin-4 are SSDs and consist of \(n \in \{2, 4\}\) agents with different colors, which start at random positions and have to collect a coin with a random color and position [11], [54]. If an agent collects a coin, it receives a reward of +1. However, if the coin has a different color than the collecting agent, another agent with the actual matching color is penalized with -2. After being collected, the coin respawns randomly with a new color. All agents can observe the whole field and are able to move north, south, west, and east. Each agent can only determine if a coin has the same color as itself or not. An episode terminates after \(H = 150\) time steps, and we set \(\gamma=0.95\). The neighborhood \(\mathcal{N}_{t,i} = \mathbb{D} - \{i\}\) is defined by all other agents \(j \neq i\). We measure the “own coin" rate \(P(\textit{own coin}) = \frac{\textit{\# collected coins with same color}}{\textit{\# all collected coins}}\) based on the coins collected by each agent.
Harvest. Harvest-12 is an SSD and consists of \(n = 12\) agents, starting at random positions and having to collect apples. The apple regrowth rate depends on the number of surrounding apples [26]. If all apples are harvested, then no apple will grow anymore until the episode terminates. At every time step, all agents receive a time penalty of -0.01. For each collected apple, an agent receives a reward of +1. All agents have a \(7 \times 7\) field of view and are able to do nothing, move north, south, west, east, and tag other agents within their view with a tag beam of width 5 pointed to a specific cardinal direction. If an agent is tagged, it is unable to act for 25 time steps. Tagging does not yield any rewards. An episode terminates after \(H = 250\) time steps, and we set \(\gamma=0.99\). The neighborhood \(\mathcal{N}_{t,i}\) is defined by all other agents \(j \neq i\) being in the field of view of agent \(i\). We measure the Sustainability, defined by the average number of time steps at which apples are collected.
MARL Algorithms. To isolate the impact of reward changes on PI mechanisms rather than the base RL algorithm, we implement all methods including DRIVE based on the standard policy gradient algorithm (Eq. 2 ) with normalized returns (Appendix 9.6), as suggested in [55], which also serves as Naive Learning baseline without any PI-based reward shaping [11]. We use LIO, MATE with \(x = 1\) as suggested in [22], and IA with \(\alpha = 5\) and \(\beta = 0.05\) as state-of-the-art PI baselines [12], [22], [24]. For IPD and Coin-2, we directly include the performance of the opponent shaping techniques LOLA-PG and POLA-DiCE, as reported in [11], [45], due to the high computational demand of the second-order derivative calculation for deep neural networks.

7 Experimental Results↩︎

For each experiment, all respective algorithms were run 20 times over \(E = 4,000\) epochs of 10 episodes to report the average progress and the 95% confidence interval.

7.1 Standard Setting – Without Reward Changes↩︎

Setting. We compare DRIVE with the baselines in IPD, Coin-2, Coin-4, and Harvest-12 without any reward change, i.e., \(f_{\textit{mod}}(u,m) = u\) (Algorithm 3, Line 15), in all epochs \(m\).
Results. The results are shown in Fig. 7. DRIVE achieves competitive and stable cooperation in IPD compared with all baselines. In Coin-2, DRIVE and MATE achieve the highest “own coin" rate, significantly outperforming POLA-DiCE, which is more cooperative than LIO, IA, and LOLA-PG. In both larger SSDs, namely Coin-4 and Harvest-12, DRIVE and MATE achieve the highest level of cooperation, where DRIVE is slightly more cooperative in Coin-4, while MATE is slightly more cooperative in Harvest-12.

7.2 Dynamic Reward Changes↩︎

Setting. Next, we evaluate DRIVE and the baselines with different reward change functions \(f_{\textit{mod}}\), as visualized in Fig. 8:

1. Linear increase: \(\> \> \> \> \> \> \> \> f^{I}_{\textit{mod}}(u,m) = u(\eta m + 1)\)

2. Exponential decay: \(\> \>f^{II}_{\textit{mod}}(u,m) = ue^{-\eta m}\)

3. Stepwise increase: \(\> \> \> f^{III}_{\textit{mod}}(u,m) = u(\floor{\eta m} + \chi)\)

4. Damped cos.: \(f^{IV}_{\textit{mod}}(u,m) = \eta + u(1 - \frac{m}{E})\textit{cos}^2(2\eta m)\)

We set \(\eta = 0.001\) and \(\chi = 10\). We choose these functions to present different scenarios where the reward scale or shift either increases monotonically or converges to zero in the limit without altering the underlying inequalities of the SDs. (1) could exemplify an external factor like increasing demands on the overall system. (2) could, e.g., simulate an internal factor like sensory wear or overall hardware decay over time. (3), on the other hand, demonstrates larger steps of change, as typically observed for shifts between simulations or simulation and reality. Finally, (4) could be interpreted as the irregularities caused by human feedback or specifications being modified online.

Results. The results are shown in Fig. 9. The learning curve of DRIVE is not significantly affected by training with the functions \(f^{I}_{\textit{mod}}\), \(f^{II}_{\textit{mod}}\), and \(f^{III}_{\textit{mod}}\). Function \(f^{IV}_{\textit{mod}}\) causes occasional dips in the learning curve of DRIVE, which quickly recovers to its original level afterward. The occasional dips are caused by the minima of the damped cosine function, where it is close to zero, which briefly nullifies the PD inequalities of Eq. 1 . However, this is a numeric issue rather than a conceptual one, as DRIVE quickly recovers afterward. LIO is robust against any change function in Coin-4, but its cooperation level significantly deteriorates in other domains. Despite automatically learning an incentive function, LIO is especially sensitive to \(f^{IV}_{\textit{mod}}\). IA is affected in Coin-2 and Coin-4, where it never outperforms Naive Learning. MATE is only able to resist reward changes in Harvest-12 when training with \(f^{I}_{\textit{mod}}\). In all other cases, the cooperation level of MATE significantly deteriorates, which is even worse than Naive Learning in some cases. The learned behavior is least stable with \(f^{IV}\), where MATE is not able to recover from dips in the learning curve, unlike DRIVE. DRIVE is the only PI method that outperforms Naive Learning, LOLA-PG, and POLA-DiCE in IPD and Coin-2. Appendix 11 provides further cooperation metrics (social welfare, equality, sustainability, and peace) in Harvest-12.

8 Discussion↩︎

We presented DRIVE, an adaptive PI approach to achieve and maintain cooperation in SDs with changing rewards. DRIVE agents reciprocally exchange reward differences to incentivize mutual cooperation in a fully decentralized manner.
Our theoretical analysis shows how DRIVE achieves incentive alignment toward mutual cooperation in the general Prisoner’s Dilemma by overcoming greed and fear while remaining invariant to per-epoch reward shifts and scaling. This contrasts with prior PI methods that rely on fixed incentive values, domain-dependent coefficients, or time-consuming incentive learning. Consequently, DRIVE is better aligned with game-theoretic assumptions and safer to deploy in environments with drifting rewards.
Our experiments further demonstrate that DRIVE significantly outperforms state-of-the-art methods such as LIO, MATE, and IA, as well as opponent-shaping baselines like POLA-DiCE and LOLA-PG, across multiple SSDs with dynamic reward changes. The results confirm the sensitivity of existing PI approaches to reward magnitudes and highlight DRIVE’s ability to sustain cooperation under such transformations, provided the underlying strategic inequalities remain intact.
Despite these promising results, DRIVE relies on truthful and synchronous peer communication. Partial non-compliance leads to graceful degradation in performance. Moreover, the current theoretical analysis assumes homogeneous populations and complete or well-connected communication graphs. These assumptions may not hold in more realistic settings with strategic manipulation, communication noise, or sparse interaction structures.
Several directions for future work follow from these observations. One avenue is to develop mechanisms that are robust to partial or strategic non-compliance, for example, through redundant aggregation, stochastic auditing, or consensus systems [38]. Extending the theoretical analysis to heterogeneous populations and more general network topologies represents another important step toward real-world applicability. A natural next step is to study learned and dynamic neighborhoods, and to combine DRIVE with models of evolving social structure and coalitions, as suggested by recent work on reward sharing in teams and coalitions. Finally, incorporating agent identification could enable targeted bilateral responses, e.g., against adversarial peers.

9 Extended Theoretical Results↩︎

In this section, we provide additional worked-out examples and detailed arguments to complement the theoretical analysis in the main body. These results give a more concrete and intuitive understanding of how DRIVE operates in simple settings, how its behavior changes under different assumptions of protocol adherence, and how the microscopic request/response mechanism underpins the macroscopic equilibrium results, and provides a simple extension to larger multi-agent settings.
Preliminaries. We recall the standard two-player Prisoner’s Dilemma (PD) with actions \(C\) (cooperate) and \(D\) (defect) and payoffs \((R,R)\), \((P,P)\), \((T,S)\), and \((S,T)\) for profile \((C,C)\), \((D,D)\), \((D,C)\), and \((C,D)\), respectively (Fig. 2). The game is a PD if \(T > R > P > S\) (cf. Eq. 1 ), where \(T>R\) encodes greed and \(P>S\) encodes fear [6]. We denote the environmental reward of agent \(i\) at time \(t\) by \(u_{t,i}\), the externally modified reward in epoch \(m\) by \(\hat{u}_{t,i}=f_{\textit{mod}}(u_{t,i}, m)\), and the running average of \(\hat{u}_{t,i}\) within epoch \(m\) by \(\bar{u}_i\). The DRIVE shaping rule for agent \(i\) at time \(t\) is \[u^{\mathrm{DRIVE}}_{t,i} = \hat{u}_{t,i} - \min_{j \in N_{t,i}}\{\Delta_{t,j,i}\} + \min_{j \in N_{t,i}}\{\Delta_{t,i,j}\}, \label{eq:DRIVE95reward95app}\tag{5}\] where \(\Delta_{t,i,j} = \bar{u}_j - \hat{u}_{t,i}\) is the response sent from \(j\) to \(i\) when \(i\) issues a request and the temporal difference gate TD\(_i(\hat{u}_{t,i})\) in Eq. 3 is non-negative. In the 2-agent PD we have \(N_{t,1}=\{2\}\) and \(N_{t,2}=\{1\}\) so the minima reduce to single terms.

9.1 Single Exchange Mechanism Underlying Incentive Alignment in a 2-Player Prisoner’s Dilemma↩︎

We give a concrete, single-step derivation showing how the DRIVE request/response protocol (Alg. 5) together with the TD gate in Eq. (3) and the shaping rule in Eq. (4) yields the payoff swap \(T \leftrightarrow S\) under unilateral defection. This turns the PD into a coordination game with \((C,C)\) as the unique Nash equilibrium. Unless stated otherwise, we assume full protocol compliance.
Setup. Consider a single time step \(t\) in a 2-agent PD with payoffs \(T>R>P>S\) and neighborhoods \(N_{t,1}=\{2\}\), \(N_{t,2}=\{1\}\). Rewards may already be transformed by \(f_{\textit{mod}}\) in epoch \(m\); for notational convenience we write \[\hat{T} := f_{\textit{mod}}(T,m),\quad \hat{R} := f_{\textit{mod}}(R,m),\quad \hat{P} := f_{\textit{mod}}(P,m),\quad \hat{S} := f_{\textit{mod}}(S,m).\] We assume that, in the long run under stationary play, epoch averages coincide with the corresponding instantaneous payoffs for each action profile (e.g., under \((C,C)\) both agents have \(\bar{u}_i = \hat{R}\)).
Mutual cooperation and mutual defection. If both agents cooperate at time \(t\), each receives \(\hat{u}_{t,1}=\hat{u}_{t,2}=\hat{R}\) and, in steady state, \(\bar{u}_1=\bar{u}_2=\hat{R}\). Hence \(\Delta_{t,1,2} = \bar{u}_2 - \hat{u}_{t,1} = 0\) and \(\Delta_{t,2,1}=0\), so Eq. 5 yields \[u^{\mathrm{DRIVE}}_{t,1}=\hat{R},\quad u^{\mathrm{DRIVE}}_{t,2}=\hat{R}.\] A similar argument shows that under mutual defection both agents retain payoff \(\hat{P}\). Thus DRIVE leaves \(R\) and \(P\) unchanged in the long run.
Unilateral defection. Now consider a state where agent 1 defects (\(D\)) and agent 2 cooperates (\(C\)), so the instantaneous modified rewards are \(\hat{u}_{t,1} = \hat{T}\), \(\hat{u}_{t,2} = \hat{S}\). Under the usual PD ordering, the defector benefits from a higher reward than the cooperator. We assume that this yields a non-negative TD advantage for the defector and a non-positive one for the cooperator, so that \[\mathrm{TD}_1(\hat{u}_{t,1}) \ge 0 \;\Rightarrow\; \text{agent 1 sends a request},\qquad \mathrm{TD}_2(\hat{u}_{t,2}) < 0 \;\Rightarrow\; \text{agent 2 does not send}.\] This corresponds to the intended operating regime of DRIVE and matches the empirical behavior observed in our experiments. Agent 1 sends its instantaneous reward \(\hat{u}_{t,1}=\hat{T}\) to agent 2 as a request. Upon receiving this, agent 2 computes the (epoch-\(m\)) difference \(\Delta_{t,1,2} \;=\; \overline{u}_2 - \hat{u}_{t,1}\). In the PD step considered, the worst-case consistent choice is \(\overline{u}_2 = \hat{S}\), yielding \(\Delta_{t,1,2} \;=\; \hat{S} - \hat{T} \;<\; 0\). Agent 2 returns this value to agent 1. Because only agent 1 sent a request, the reverse difference \(\Delta_{t,2,1}\) is undefined in this step; by Algorithm 5 it defaults to \(0\) for aggregation when no responses arrive. Using the responder’s epoch-average \(\overline{u}_j\) rather than its instantaneous reward makes DRIVE sensitive to systematic exploitation instead of single noisy outcomes. Only agents whose recent average return is consistently lower than a neighbor’s request generate strong negative \(\Delta\) terms. If both sides used instantaneous rewards, a single lucky or unlucky step could trigger large transfers, making incentives noisy and easy to game via short-term risk-taking. Using instantaneous rewards against averages would instead turn DRIVE into a risk-sharing mechanism, while comparing averages on both sides would be very stable but slow to react. The chosen average–instant design therefore balances robustness to noise with responsiveness to emerging exploitation. In the 2-player case, the minima over neighbors reduce to single terms. The per-step shaped rewards are \[\begin{align} \hat{u}^{\mathrm{DRIVE}}_{t,1} &= \hat{u}_{t,1} \;-\; \underbrace{\min\{\Delta_{t,2,1}\}}_{=\,0} \;+\; \underbrace{\min\{\Delta_{t,1,2}\}}_{=\,\hat{S}-\hat{T}} \;=\; \hat{T} + (\hat{S}-\hat{T}) \;=\; \hat{S}, \tag{6}\\[4pt] \hat{u}^{\mathrm{DRIVE}}_{t,2} &= \hat{u}_{t,2} \;-\; \underbrace{\min\{\Delta_{t,1,2}\}}_{=\,\hat{S}-\hat{T}} \;+\; \underbrace{\min\{\Delta_{t,2,1}\}}_{=\,0} \;=\; \hat{S} - (\hat{S}-\hat{T}) \;=\; \hat{T}. \tag{7} \end{align}\] Thus, the unilateral defection step \((\hat{T},\hat{S})\) is reshaped to \((\hat{S},\hat{T})\): the defector receives the sucker payoff and the cooperator the temptation payoff. Eqs. (6 –7 ) explicitly demonstrate how the TD gate together with the DRIVE shaping rule realizes the \(T\leftrightarrow S\) swap in a single exchange step. By symmetry, the profile \((C,D)\) is reshaped from \((\hat{S},\hat{T})\) to \((\hat{T},\hat{S})\).
Resulting incentives and equilibrium interpretation. Collecting all action profiles, DRIVE induces the modified payoff matrix

with \(\hat{T}>\hat{R}>\hat{P}>\hat{S}\). Hence, if the opponent plays \(C\), responding with \(C\) yields \(\hat{R}>\hat{S}\), and if the opponent plays \(D\), responding with \(C\) yields \(\hat{T}>\hat{P}\). Cooperation is therefore a strict best response to both actions, so \((C,C)\) is the unique Nash equilibrium.
Takeaway. Under the TD gate (Eq. 3 ) and shaping rule (Eq. 4 ), a single request/response exchange in a unilateral defection state deterministically maps \((\hat{T},\hat{S})\mapsto(\hat{S},\hat{T})\), eliminating greed and fear incentives in the transformed PD for that step. This microscopic mechanism directly underpins the equilibrium-level result that DRIVE turns the Prisoner’s Dilemma into a coordination game with cooperation as the unique Nash equilibrium, providing a worked-out complement to the proof of Theorem 1.

9.2 Analysis of Protocol Compliance Cases↩︎

To complement the discussion in Section 5.3, we present a worked example for a two-player Prisoner’s Dilemma under DRIVE with different levels of protocol compliance. We consider generic PD payoffs \(T>R>P>S\) as in Eq. (1); for concreteness one may think of the canonical values \((T,R,P,S)=(5,3,1,0)\), but none of the arguments below depend on these particular numbers. In each case, agent \(i\) defects while agent \(j\) cooperates, and DRIVE applies reward shaping based on request/response behavior.

(1) Full compliance. If all agents truthfully follow the protocol, the results of Theorem 1 apply. Unilateral defection \((T,S)\) is reshaped into \((S,T)\): the defector receives \(S\) and the cooperator \(T\), eliminating the incentive to defect.
(2) Defector does not send a request. If the defector refrains from sending a request despite positive TD, it avoids penalization from neighbors but also loses the chance of mutual improvement. Its payoff remains \(T\), while the cooperator remains at \(S\). This strategy indirectly penalizes the non-requesting agent, which cannot gain from reciprocal shaping.
(3) Cooperator withholds a response. If the cooperator fails to respond (intentionally or due to loss), the defector avoids the intended penalty. The payoffs revert to \((T,S)\). Because of the min-aggregation across neighbors, the absence of a single response only matters if it is the unique or strongest penalty.
(4) Misreporting. Misreporting can occur in two forms: (i) agents send requests despite having no improvement signal (TD\(<0\)), or (ii) agents respond with incorrect values of \(\Delta\) (due to error or intent). In the first case, the resulting differences are typically small or positive, which the min operator filters out, making the effect negligible. In the second case, however, false responses can distort the aggregation: a misreporting neighbor may reduce or nullify the penalty for a defector, or unjustly penalize a compliant requester. While this can destabilize cooperation in the worst case, it is not worse than the vulnerabilities faced by other PI methods. Overall, misreporting represents a robustness limitation but does not fundamentally break the mechanism as long as the majority of responses remain truthful.
(5) No compliance at all. If no requests or responses are exchanged, all shaping terms vanish. DRIVE collapses to plain MARL with payoffs \((T,S)\) under unilateral defection, and cooperation incentives are lost.
Extension to larger scenarios. The two-player example above presents the limiting case where any single non-compliant agent corresponds to \(50\%\) protocol adherence, which strongly magnifies the impact of individual deviations. In larger populations, however, the effect of partial non-compliance is more gradual: as long as a sufficient fraction of agents continue to respond truthfully, the min-aggregation still enforces penalization of defectors.
Three-player example. Consider a group of three agents, where one defects (\(i\)) and two cooperate (\(j,k\)). Each cooperator truthfully responds with \(\Delta_{i,j}, \Delta_{i,k} < 0\). The defector’s shaped reward is \[u^{\text{DRIVE}}_i = T + \min\{\Delta_{i,j}, \Delta_{i,k}\},\] so as long as at least one cooperator responds truthfully, \(u^{\text{DRIVE}}_i\) is penalized. If one cooperator withholds, the other still enforces the penalty, though possibly weaker. Thus, unlike in the 2-player case, cooperation incentives do not collapse entirely when a single agent is non-compliant.

9.3 Extension to a Simple N-Agent PD Setting↩︎

The main theorem focuses on the 2-agent Prisoner’s Dilemma (PD). Here we first provide a clean extension to an \(N\)-agent graphical PD under full protocol compliance, where payoffs decompose into symmetric 2-player PD interactions, and then progressively relax structural and behavioral assumptions.

To isolate the effect of incentive alignment, we assume that each pairwise interaction is shaped independently using the 2-agent DRIVE protocol, and that all agents comply fully with the request and response rules. That is, in every interaction with neighbor \(j\), agent \(i\) behaves exactly as in the 2-agent analysis of Appendix 9.1.

This proposition formalizes how the 2-player incentive alignment result extends to a simple class of \(N\)-agent games with additive pairwise PD interactions. Our SSD experiments instantiate more complex interaction patterns, but the graphical PD result provides a clean theoretical illustration of how DRIVE scales beyond two agents.
Pairwise PD interactions across neighborhoods. In many multi-agent systems, agents repeatedly engage in pairwise social dilemmas with dynamically changing or randomly matched partners (an interaction graph), while incentive exchanges are restricted to a possibly different and fixed communication graph defined by neighborhoods \(N_{t,i}\). In such settings, the DRIVE-shaped reward of a defector \(i\) still takes the form \(u^{\mathrm{DRIVE}}_i = T + \min_{j \in N_{t,i}} \Delta_{i,j}\). Whenever the exploited cooperator in a given PD round is also a neighbor of \(i\), the exact \(T \leftrightarrow S\) payoff swap from the 2-agent case is recovered. When interaction partners lie outside \(N_{t,i}\), the penalty is instead delivered by whichever compliant neighbors suffer a sustained long-run disadvantage from \(i\)’s behavior (e.g., through shared resources or future interactions). Thus, the interaction graph determines which behaviors create negative externalities, while the communication graph determines who can hold whom accountable. As long as each defector has at least one compliant neighbor whose long-run return is reduced when it defects, the sign of the incentive remains cooperative, even if the magnitude differs from the fully local 2-agent PD.
Beyond graphical PDs. Many benchmark social dilemma environments, such as Coin Game or Harvest, do not admit an explicit decomposition into pairwise PD payoffs. Instead, incentives arise indirectly through shared resources, delayed consequences, and aggregate system dynamics. Nevertheless, the same principle applies: agents whose behavior systematically reduces their neighbors’ long-run returns generate negative differences \(\Delta_{i,j}\) and are penalized by DRIVE. This perspective explains why DRIVE empirically romotes cooperation in such environments, even though no explicit PD structure is present.
General \(N\)-agent compliance and robustness. The preceding results assume full protocol compliance. We now consider a general \(N\)-agent setting in which only \(M\) agents truthfully follow the DRIVE protocol, while \(K=N-M\) agents may behave adversarially by withholding requests or responses or by misreporting. For any game-theoretic defector \(i\), effective penalization requires two conditions: (i) \(i\) itself must comply with the request gate (sending a request when \(\mathrm{TD}\ge 0\)), and (ii) at least one neighbor \(j\in N_i\) must respond truthfully. Under these conditions, the shaped reward of \(i\) is \[u^{\mathrm{DRIVE}}_i = T + \min\{\Delta_{i,j}\}_{j\in N_i,\; j\;\mathrm{compliant}}.\] As long as a compliant responder exists, defection is penalized; the severity depends on the most negative difference among the responders, but the penalty does not collapse entirely. In fully connected populations, this reduces to the simple requirement that at least two agents comply (the requester and one responder), i.e.\(M/N \geq 2/N\). In general communication topologies, the compliant set must form a dominating set, ensuring that each requester has at least one compliant neighbor. Thus, DRIVE exhibits a graceful degradation: as compliance \(M/N\) decreases, cooperation incentives weaken smoothly but only collapse completely if no responder is available.
Connectivity requirements. In fully connected graphs, penalization is ensured as soon as at least two agents comply (the requester and one responder), corresponding to a compliance rate \(c \ge 2/N\). In general communication topologies, the compliant agents must form a dominating set: every requester must have at least one compliant neighbor. Formally, for graph \(G=(V,E)\) with compliant set \(C\subseteq V\), penalization holds iff \(\forall i\in V:\; N(i)\cap C\neq\varnothing\). The minimal compliant set size is the domination number \(\gamma(G)\), yielding a topology-aware threshold \(|C|\ge \gamma(G)\).
Adversarial robustness. The above threshold can be interpreted directly in terms of \(K\) adversarial agents. If \(K\) agents are adversarial (never providing truthful responses), DRIVE continues to penalize every protocol-compliant defector as long as the remaining \(M=N-K\) compliant agents contain a dominating set. Equivalently, the system tolerates up to \(K\le N-\gamma(G)\) adversarial agents. However, agents that are adversarial both in their actions (always defecting) and their protocol behavior (never sending requests) cannot be punished by any PI scheme based on self-reported signals, including DRIVE. To compliant neighbors these agents simply appear as players whose defections generate no negative differences; this reduces local cooperative pressure and gradually pushes the system toward the underlying non-PI MARL dynamics (Case 5), but does not cause an abrupt collapse.
Overall, these results highlight DRIVE’s property of graceful degradation: full compliance recovers strong incentive-alignment guarantees, while partial compliance weakens but does not immediately destroy cooperative incentives, provided the communication topology satisfies minimal coverage conditions. In two-agent systems, a single non-compliant agent already reduces compliance to \(50\%\), eliminating penalization. By contrast, in larger populations (\(N>2\)), even if several agents defect from the protocol, the remaining majority can still enforce penalization through the min-aggregation. Hence, DRIVE remains effective as long as the compliance rate stays above the topology-specific threshold, often close to a simple majority in practice. Nevertheless, systematic misreporting or universal non-compliance collapses DRIVE to baseline MARL, emphasizing the importance of designing variants that are robust to missing or adversarial responses in future work.

9.4 Affine Reward Transformations: Scope and Edge Cases↩︎

Here, we expand on the affine reward-change model used in our analysis. In each epoch \(m\), the environmental reward is transformed by a shared positive affine map \(r' = c_m\, r + b_m\), applied uniformly to all agents and timesteps. The schedules \(c_m\) and \(b_m\) may vary arbitrarily across epochs, covering all reward dynamics used in Sec. 7.2 (linear, exponential, stepwise, and damped–cosine).
Example: damped–cosine schedule. For illustration, in the damped–cosine setting we use \(c_m = (1 - m/E)\cos^2(2\eta m)\) and \(b_m = \eta\), so the effective reward scale oscillates while decaying over time.
Degenerate epochs where \(c_m \approx 0\). Isolated epochs may satisfy \(c_m \approx 0\), yielding \(r' \approx b_m\) and temporarily collapsing the Prisoner’s Dilemma inequalities (\(T, R, P, S\) become indistinguishable). The stage game becomes nearly payoff-indifferent in these epochs. Our results treat such cases explicitly by applying the PD analysis only to epochs with \(c_m > 0\).
Transformations not covered. This affine class excludes genuinely non-linear reward changes such as clipping, saturation, sign-dependent remapping, or state-dependent transformations that cannot be written as a shared affine map. These represent a broader family of sim-to-real shifts that we leave to future work.

9.5 Reward Changes and Invariance in the PD↩︎

We first formalize how affine reward transformations affect the PD inequalities.

We can now restate Theorems 2 and 3 more formally.

9.6 Training with Normalized Returns↩︎

In our paper, \(\hat{\pi}_{i}\) and \(\hat{V}_{i}\) are trained with normalized returns \(\overline{G}_{t,i} = \frac{1}{\sigma_{i}}(G_{t,i} - \mu_{i})\), where \(\mu_{i}\) and \(\sigma_{i}\) are the mean and standard deviation of all returns \(G_{t,i}\) of agent \(i\) in an epoch \(m\), to improve training stability [56]. The normalization makes standard policy gradient RL (without PI mechanism) invariant to reward changes, where rewards are scaled by a factor \(c_m > 0\) or shifted by a scalar \(b_m \in \mathbb{R}\), according to Lemma 2.

According to Lemma 2, the non-PI baselines Naive Learning, LOLA-PG, and POLA-DiCE are not affected by the dynamic reward changes in Section 7.2 and Fig. 8 (but are still less cooperative than DRIVE). However, most state-of-the-art PI methods like LIO and MATE cannot accommodate such changes (by the generally unknown function \(f_{\textit{mod}}\)) in their payoff modifications (Fig. 6) and thus fail to adapt, in contrast to DRIVE, as shown in Figs. 1 and 8.

9.7 Invariance in SSDs Under Normalized Policy Gradients↩︎

We can restate and prove Theorem 4 in more detail using the normalized return defined in 2 \[\bar{G}_{t,i} = \frac{G_{t,i} - \mu_i}{\sigma_i},\] where \(\mu_i\) and \(\sigma_i\) are the mean and standard deviation of all returns \(G_{t,i}\) for agent \(i\) in epoch \(m\).

We now combine this with the structure of DRIVE.

10 Technical Details and Hyperparameters↩︎

10.1 Hyperparameters↩︎

All common hyperparameters used by all MARL approaches in the experiments, as reported in Section 7 in the paper, are listed in Table 3. The final values were chosen based on a coarse grid search which finds a tradeoff between performance and computation w.r.t. DRIVE, LIO, IA, and MATE in IPD and Coin-2 without dynamically changing rewards, i.e., \(f_{\textit{mod}}(u,m) = u\). We directly adopted the final values in Table 3 for all other domains. All training and test runs were performed in parallel on a computing cluster of 15 x86_64 GNU/Linux (Ubuntu 18.04.5 LTS) machines with i7-8700 @ 3.2GHz CPU (8 cores) and 64 GB RAM. We did not use any GPU in our experiments.

For LIO, we set the cost weight for incentive function learning to 0.001 and the maximum incentive value \(R_{\textit{max}}\) to the highest absolute penalty per domain (3 in IPD, 2 in Coin-2 and Coin-4, and 0.25 in Harvest-12), as suggested in [12]. For IA, we set \(\alpha = 5\) and \(\beta = 0.05\), as suggested in [24].

10.2 Neural Network Architectures↩︎

We coarsely tuned the neural network architectures in the paper w.r.t. performance and computation by varying the number of units per hidden layer {\(32\), \(64\), \(128\)} for \(\hat{\pi}_{i}\) and \(\hat{V}_{i}\). The number of hidden layers was varied between 1, 2, and 3, but significantly deeper architectures led to deteriorating performance (possibly due to vanishing gradients). Using ELU or ReLU activation did not make any significant difference for any neural network. Thus, we stick to ELU throughout the experiments. DRIVE, MATE, and Naive Learning only use \(\hat{\pi}_{i}\) and \(\hat{V}_{i}\) as neural networks. The incentive function network of LIO has the same hidden layer architecture as \(\hat{\pi}_{i}\) and \(\hat{V}_{i}\). In addition, the joint action of the \(n-1\) other agents is concatenated to the flattened observations before being input into the incentive function, which outputs an \(n-1\) dimensional vector. The output vector is passed through a sigmoid function and multiplied with \(R_{\textit{max}}\) (Section 10.1) afterward.

10.3 Coin-2 and Coin-4↩︎

We adopted the setup of [11] in Coin-2, as shown in Fig. [fig:coin-environment1], with the same rules and reward functions. The order of executed actions is randomized such that situations where two agents simultaneously step on a blue coin lead to an expected payoff of +1 for the red agent and -1 for the blue agent (Fig. [fig:coin-environment1] left), and vice versa for a red coin. In addition, we extended the domain to 4 agents in Coin-4 (Fig. [fig:coin-environment1] right). All agents are able to move freely, and grid cell positions can be occupied by multiple agents. Any attempt to move out of bounds is treated as “do nothing" action.
Prisoner’s Dilemma Connection Assuming that an agent always picks up a coin at every time step in Coin-2, the expected rewards per time step represent a PD instance, according to Section [subsec:social95dilemmas] and Fig. 2:

• If both agents cooperate, i.e., only collect their matching coins, each agent gets \(R = 0.5(1 + 0) = 0.5\) on average.

• If both agents defect, i.e., collect any coin regardless of their color, each agent gets \(P = 0.25(1 + 1 - 2 + 0) = 0\) on average.

• If one agent defects, the defecting agent gets \(T = 0.5(1 + 0) + 1 = 1.5\) on average (its own coin 100% of the time and the other’s coin 50% of the time). The exploited agent gets \(S = 0.5(1 - 2) = -0.5\)

The expected payoffs \(R\), \(P\), \(T\), and \(S\) satisfy the characteristic PD inequalities w.r.t. greed and fear, namely \(T > R > P > S\). Thus, DRIVE also works well in Coin-2 and Coin-4 (which extends the SD to 4 agents), even with changing rewards, as shown in Fig. 9.

10.4 Harvest-12↩︎

We adopted the setup of [26] in Harvest-12, as shown in Fig. [fig:harvest-environment1], with the same dynamics and apple regrowth rates. Harvest-12 always starts with the initial apple configuration in Fig. [fig:harvest-environment1] with randomly positioned agents. The agents are able to observe the environment around their \(7 \times 7\) area and have no specific orientation. Thus, they have 4 separate actions to tag all neighbor agents, which are either north, south, west, or east of them. All agents are able to move freely, and grid cell positions can be occupied by multiple agents. Any attempt to move out of bounds is treated as “do nothing" action.
Prisoner’s Dilemma Connection We empirically tested a 2-agent instance of Harvest to determine the payoff table, similar to [10]. The table is shown in Fig. [fig:harvest-payoffs] and represents a PD instance, according to Section [subsec:social95dilemmas] and Fig. 2:

• If both agents cooperate, i.e., no stunning and occasional waiting for apple regrowth, each agent gets \(R \approx 0.3\) on average.

• If both agents defect, i.e., stunning and complete harvesting that prevents apple regrowth, each agent gets \(P \approx 0.06\) on average.

• If one agent defects, i.e., stunning and harvesting, the defecting agent gets \(T \approx 0.9\), and the exploited agent gets \(S \approx -0.01\) on average (due to being stunned most of the time with a 0.01 time penalty).

The expected payoffs \(R\), \(P\), \(T\), and \(S\) satisfy the characteristic PD inequalities w.r.t. greed and fear, namely \(T > R > P > S\). Thus, DRIVE also works well in Harvest-12 (which extends the SD to 12 agents), even with changing rewards, as shown in Fig.  9 and Fig . 12. This also confirms the empirical result of [10], stating that Harvest can represent a PD instance w.r.t. greed and fear, depending on the availability and regrowth rate of apples.

11 Additional Results↩︎

We provide additional results regarding different cooperation measures for Harvest-12, namely Social Welfare or Efficiency (U) (according to Section [subsec:problem95formulation]), Equality (E) (1 minus the Gini coefficient), Sustainability (S) (the average number of time steps at which apples are collected), and Peace (P) (the average number of untagged agents at any time step). All measures are based on the original environmental rewards \(u_{t,i}\) instead of \(\hat{u}_{t,i} = f_{\textit{mod}}(u_{t,i}, m)\) (Algorithm 3, Line 15) to reliably assess the stability of cooperation. They are defined by [9]:

\[\begin{align} E &= 1 - \frac{\sum_{i \in \mathcal{D}}\sum_{j \in \mathcal{D}}|\sum_{t=0}^{H-1} (u_{t,i}-u_{t,j})|}{2n\sum_{i \in \mathcal{D}}\sum_{t=0}^{H-1} u_{t,i}}, \\ S &= \frac{1}{n}\sum_{i \in \mathcal{D}}\chi_{i} \text{, where } \chi_i = \mathbb{E}[t| u_{t,i} > 0], \\ P &= n - \frac{1}{H}\sum_{i \in \mathcal{D}}\sum_{t = 1}^{H}\mathbf{1}[\text{agent timed-out on time step }t] \end{align}\]

Fig. 12 shows the progress of the alternative cooperation measures in Harvest-12 with the different reward change functions \(f_{\textit{mod}}\) (I, II, III, and IV) from Section 7.2. While DRIVE is robust w.r.t. all measures, MATE and LIO significantly change under most settings, except for equality regarding reward change functions \(f^{I}_{\textit{mod}}\) and \(f^{III}_{\textit{mod}}\). IA is only affected by \(f^{IV}_{\textit{mod}}\) regarding all cooperation measures. Apart from \(f^{III}_{\textit{mod}}\), MATE always exhibits more peace than DRIVE, although the peace level significantly varies depending on the reward change function. LIO only exhibits more peace than DRIVE under \(f^{II}_{\textit{mod}}\) and \(f^{IV}_{\textit{mod}}\), which both converge to zero over time.

References↩︎

---

1. The \(\textit{min}\) aggregations could be replaced by a sum or mean, similar to [24]. However, this would weaken the influence of the most exploited agent relative to others, thus tolerating individual dissatisfaction, which could spread in later epochs.↩︎