Supplementary Video↩︎

This paper is accompanied by videos of the agents trained with our algorithm, : go.ea.com/RAID

1 Introduction↩︎

With game worlds becoming vaster and game systems becoming more complex, the effort to test games grows likewise. To reduce repetitive aspects of game testing, prior work has investigated automating parts of it through the use of autonomous agents playing a game and reporting bugs [1]–[3]. While Reinforcement Learning (RL) agents can find individual exploits within a game [4], they tend to converge to one single “best” solution of solving a task. In this paper, we use EA SPORTS NHL 26 (short NHL) as our test-case. In particular, we aim to test the goalie AI of the game, training a forward agent to find high chance scoring strategies, which could represent potential exploits. Our results indicate that standard RL algorithms tend to collapse to a small set of high-reward behaviors, rather than exploring a larger, more diverse set. Thus, exploit discovery becomes sequential: developers must fix one issue before retraining the agent to find others. Since this slows down the exploit finding process, we aim to develop an algorithm that can find multiple potential exploits without human intervention.

Prior work on RL for game testing has proposed methods to increase coverage of a game environment [5], [6]. However, the added bias towards exploration comes at the cost of a loss in optimality with respect to the original goal of the agent. For our use case, we are explicitly looking for high-performing scoring strategies and we therefore require a method that achieves high performance with respect to the scoring chance, while also finding multiple diverse solutions without human intervention.

The field of quality diversity investigates how to find multiple diverse solution that are high-performing [7]. While they have reported impressive results, existing solutions are often brittle in complex domains, one reason being a moving diversity target learned in parallel to the policies [8]. We aim for playtesters to independently use our tool and therefore want the tool to be as simple as possible to require little RL expertise.

Following these requirements, we propose Reward-Adaptive Iterative Discovery (), an approach to find a set of high-quality, diverse solutions by iteratively training a population of agents. Figure 1 shows a brief overview of the results of the approach. We enforce diversity through not rewarding agents for strategies that are similar to those found by any previously trained agent. By measuring diversity over sequentially trained agents, leaving the diversity measure fixed during an individual agent’s training time, we increase robustness compared to previous work. We define diversity in an intuitive, domain-specific way, such that the diversity measure can be interpreted and adjusted by non-RL practitioners, facilitating the use by playtesters to extend their toolset.

2 Related Work↩︎

We present related work in two research directions connected to our approach: (1) automated playtesting, specifically approaches based on reinforcement learning, and (2) the field of behavior diversity, which motivates our approach.

2.1 Automated Playtesting↩︎

Contrary to recent work that apply reinforcement learning to train agents to master a game and surpass human players [9], [10], research on automated playtesting is generally less concerned with maximizing the same success metrics as human players, and more focused on having agents play a game in various ways that reveal insights on potential issues in the game. One direction for this is to maximize the coverage of states an agent visits in a game. A notable example is the work by [5], that uses a count-based exploration reward scheme to promote visitation of less seen states in a game map. Another direction aims to train agents that play games with behavior similar to human players, for example to visualize the interactions a game designer can expect from users or to balance the effectiveness of different strategies within a game [2], [11], [12]. [6] combine both directions by exploring strategies similar to those of human players with an additional intrinsic reward for exploring new states.

While prior work achieves good results in terms of game state coverage and player-like behavior, we aim to investigate exploits that are closely tied to high performance with respect to the original human success metric of a game, in our case scoring goals with a high percentage in a hockey game. We therefore aim to develop an approach to increase the diversity in behaviors of the trained agents while preserving the performance with respect to the game’s success metric as much as possible.

2.2 Diversity↩︎

The field of quality diversity aims to generate diverse populations of high-performing solutions through evolutionary algorithms [13]–[15]. More recent work brings the concept of diversity into the field of reinforcement learning through intrinsic rewards. The work by [16], for example, introduces an intrinsic reward based on optimizing the mutual information between options – i.e. closed-loop policies for taking action over a period of time – and the options’ final states, using entropy optimization to maximize diversity across options and distinction of individual options. Similarly [17] optimize diversity over all visited states and show that this form of unsupervised discovery can serve as an effective pretraining mechanism for reinforcement learning. DOMiNO defines diversity in the space of state-action occupancy [7], represented by successor features [18]. It combines the intrinsic diversity reward and an extrinsic reward specific to the environment with Lagrange multipliers to control the trade off between diversity and optimality with respect to the extrinsic reward. While DOMiNO can train multiple diverse agents in parallel this way, [8] highlight that the approach fails to learn diverse behaviors in complex domains and can suffer from instability due to the approach learning the successor feature representations in parallel to the agents’ policies.

Due to this instability and our aim for a stable algorithm that can be used by non-RL practitioners, we train a group of agents in a simple, sequential fashion, by masking strategies similar to those found by a prior agent from the reward function of all subsequent agents. The diversity measurement introduced by this method of reward masking is static for a specific agent within a single training iteration and not any more complex in practice than simple static reward shaping in standard reinforcement learning [19]. Similar to prior work [16], we define diversity in terms of the final state of an agent. Rather than maximizing the distance between states, we however introduce diversity via hard constraints on state difference, enforced through reward masking of prior behaviors. This allows users of  to easily update the algorithm to their desired level of diversity, for example in the later introduced diversity criterion for NHL by increasing the area used for the shot similarity criterion to a 4 m radius. In comparison, the diversity hyperparameters of prior work, for example relative to the maximum achievable reward in [7], are significantly harder to interpret, especially for non-RL practitioners.

3 Methodology↩︎

Our goal is to create an RL-based approach to autonomously find multiple diverse, high chance scoring strategies for the NHL environment without any human intervention in between iterations. We first outline the base RL setup to find a single exploit within the NHL environment. We then describe , a novel approach to satisfy the posed requirements. For the foundations of reinforcement learning, such as the concepts of reward, policy, and Q-value function, we refer the reader to [20].

3.1 Base RL Setup↩︎

  We aim to train a hockey forward player agent that learns high chance scoring strategies in a one-versus-goalie setting, similar to those that a human player could find by exploiting weaknesses in the goalie AI system. To make sure that the found strategies are executable by a human, we provide the agent with a selection of the same actions available to human players on a game controller:

• Discrete actions: the \(4\) face buttons and the left trigger button that execute so-called dekes, i.e., decoy movements to draw the goalie out of position.

• Continuous actions: the two sticks for player skating and hockey stick control, each represented by continuous \(x\) and \(y\) axes.

Through the use of those actions, the agent can perform different kinds of shot types – backhand, wrist shot, etc. – which we will later consider for defining shot diversity. Furthermore, we only let the agent act every \(5\) frames and smooth the stick actions using an exponential moving average with a smoothing factor of \(0.2\), to make sure that the agent does not leverage any super-human movement capability or reaction time.

  We provide the agent with the normalized position, velocity, and orientation of puck, net, and goalie relative to the agent, as well as a stack of the \(8\) last performed actions.

  We reward the agent for every goal. To simplify learning, we additionally include shaping rewards for the puck getting closer to the goal, as well as for the scoring chance of every shot taken, based on an NHL-internal calculation.

  While from an interface perspective all RL algorithms that use a reward function are technically compatible with , ’s effectiveness depends on the base algorithm’s ability to explore various strategies – even when previously found strategies are no longer rewarded – before converging to the best one. We therefore use Soft Actor-Critic [21], since it features a robust exploration mechanism through entropy maximization, exploring more in less known states. Furthermore, Soft Actor-Critic has a high sample efficiency, which is crucial for environments with slow data generation, such as full-scale games. To support both discrete and continuous actions, we use the architecture described by [22] for \(2\) Q-value functions, as well as a mixed-action policy, each with \(5\) hidden layers with \(512\) units, allowing for a maximum of one discrete action at a time. When we use the term agent, we refer to both the policy and Q-value functions used for training and inference. For efficient training, we stop training once an agent either reaches a \(90\)% scoring chance or does not improve its performance for \(50\)k training steps.

3.2 ↩︎

We outline the general, domain-agnostic  method in Algorithm ¿fig:alg:RAID?. The algorithm extends a standard reinforcement learning setup by training multiple agents sequentially, aiming for each agent to come up with a new strategy for maximizing the reward of the environment. For NHL we define a strategy \(z\) as the shot type and shot position at the end of an episode, the shot position meaning the last position of the puck before the puck is shot or, if the puck is not shot, the last puck position before a goal. We start by training a first agent with a standard reinforcement learning setup. After training of the first agent converges, we evaluate the agent to capture the strategy of its highest performing checkpoint. For the NHL game, we evaluate the agent until it scores \(100\) goals and describe the strategy \(z\) as the average shot position and the most common shot type over those \(100\) goals. We store this strategy in a list of previous strategies \(Z_\text{prev}\). We then continue sequentially, training each next agent the same way, but for each new iteration modify the reward function to not reward strategies similar to those found by any previous agent. Based on feedback from the NHL development team, we define similar(\(z\), \(Z_\text{prev}\)) as the shot position of a strategy being within a 2 m radius of a previous strategy that used the same shot type, i.e. a strategy \(z\) is considered novel or sufficiently diverse with respect to a set of existing strategies \(Z_\text{prev}\) if: \[\forall z' \in Z_\text{prev}\quad z_\text{shottype} \neq z'_\text{shottype} \vee l^2(z_\text{shotpos},z'_\text{shotpos}) >\text{\SI{2}{\meter}}.\]

For NHL, we skip adding strategies with a scoring chance of less than \(10\)% after convergence, since we are interested in high-performing strategies and convergence at this low performance potentially indicates that the algorithm got stuck in a local optimum, meaning that future iterations starting with a different random seed could potentially find more performant solutions for the same shot position and type. For NHL, we use the details from Section 3.1 for the base reward function, training algorithm, and convergence criterion.

While the definition of strategy similarity is highly domain specific, we argue that this is exactly what makes the method suitable for game development. While previous work defines diversity in a more general way that can be used across domains, for example via successor features [7], we argue this makes it significantly harder for a playtester without RL knowledge to tune diversity to a level fitting their domain, given that [7], for example, requires the diversity level to be expressed with respect to the maximum achievable reward. Furthermore, having an explicit list of previous strategies allows playtesters to warm start training using a list of manually defined strategies, instead of an empty set, for example using high scoring strategies that are known and desired by the game, or exploits that are known but the game studio could not fix yet.

4 Experiments↩︎

We compare  to a naive baseline to demonstrate the diversity in strategies found by the approach. This is most notably supported by  finding \(6\) exploit strategies that were previously also found by human playtesters, but had not been fixed on the outdated version of the game that we conducted the experiments on. We furthermore test the limitations of the algorithm with respect to the number of high chance scoring strategies it finds before failing to find any new strategies.

4.1 Experimental Setup↩︎

We run all experiments on a single PC with an NVIDIA GeForce RTX 4090 GPU and an AMD Ryzen Threadripper PRO 7975WX CPU. We use a pre-release development version of EA SPORTS NHL 26, additionally reducing rendering to a minimum to speed up the simulation. The simulation includes a single goalie, controlled by the game’s AI we want to test, as well as a forward player, controlled by RL. For both training and evaluation we slightly vary the initial position of the forward around the centerline to make the approach more robust to variations in the setup and to help with exploration.

4.2 Naive Baseline↩︎

As a baseline, we run the base RL setup from Section 3.1 for \(20\) independent iterations, each with a different random seed, changing the initialization of the policy and value networks, as well as the random states of the game environment. Each iteration uses the full reward function, assigning rewards for all goals, independent of shot position or type, without masking previous strategies. The experiments take between 16 and 152 to converge. RL algorithms aim to find the best solution to a problem with respect to a reward function. In environments with a set of clearly superior strategies, returning a higher reward, an optimal algorithm therefore always converges to this same set of solutions, if not otherwise incentivized. The left of Figure 2 shows how all of the \(20\) independent iterations end up with either one of two patterns, a snapshot from the left side of the goal or a backhand shot from the right side.

Assuming that a playtester classifies the found strategies as unwanted exploits, they have to improve the goalie AI behavior to mitigate those exploits before the baseline approach is likely to find any additional ones. The naive RL approach therefore adds considerable overhead to the exploit finding process by requiring human intervention between iterations. If on the contrary the playtester considers the found strategies to be a desired way of scoring in the game, they would need to find a way for the reward function to distinguish between exploits and desired scoring strategies, such that the agent ignores the desired strategies and finds novel exploits. Exploits are highly unique in their appearance and therefore hard to define before first seeing them.

4.3 ↩︎

We run  starting with two different initial random seeds, influencing the random initialization of the agents of all iterations, for \(N=30\) and \(N=10\) iterations respectively. The \(30\) iteration experiment takes 48 h and the \(10\) iteration experiment takes 14 h for all iterations to converge. Figure 2 shows how the strategies found by adhere to the diversity criteria we set for NHL: while the average shot positions of strategies with the same shot type are all at least 2 m away from each other, for example the three backhand shot strategies in green for seed \(1\), the algorithm still finds strategies with close shot position but different shot types, e.g. strategies \(8\) and \(2\) for seed \(1\), sharing almost the same position. This way the algorithm finds a set of diverse strategies without any human intervention in between iterations to fix prior exploits, as would be required for the naive baseline.

As an illustrative example of the usefulness of  for playtesting, when we ran for the first time and presented the results to the development team, \(6\) of the found strategies matched with exploits previously found by human playtesters. Since we did however use an outdated version of the game for our experiments, those exploits were still available for the agents to find. This shows that the agents are able to find exploits qualitatively similar to those found by human playtesters, while only requiring human supervision after the algorithm has found a set of candidates for a human expert to review. To not reveal any information that allows players to further exploit game mechanisms, we do not reveal which of the found strategies match with those deemed exploits.

  Figure 3 shows \(10\) trajectories each for a selection of iterations of seed \(1\) of . While we only define diversity in terms of shot position and type, the visualized trajectories demonstrate how this diversity definition implicitly generates diversity in states before the shot as well. This shows that the agents take into account player direction, as well as orientation and stick position, which are visible in the supplementary videos, when learning scoring strategies. While trajectories found by iterations \(12\) and \(13\), for example, have similar shot positions, they require significantly different skating trajectories to score with their respective shot types, backhand and snapshot.

  To our surprise, the agent also finds strategies to score without shooting, marked in dark blue in Figure 2 and visible in the supplementary video. This initially led to a bug in our setup, since previous shot strategies were only recorded when the agent actually shot the puck. This further underlines the difficulty of defining exploits before first seeing them, outlined in Section 4.2.

  While most of the strategies found in seed \(1\) and seed \(2\) resemble each other, they appear in different order and do not match completely – for instance, strategies found in iteration \(8\) of seed \(1\) and iteration \(8\) of seed \(2\) are not found by the other seed. RL algorithms have a high variance in their exploration process and therefore converge to different solutions given the existence of multiple similarly performant solutions, as we also demonstrate with the two dominant solutions the naive baseline finds. This however means that if a playtester attempts to fix an exploit found by a first run of , the same exploit not appearing in a second run of  after the fix is not a sufficient guarantee that the fix was successful. Similarly, the policies trained in the first run can not be used for this purpose, since RL is highly overfitting to the exact dynamics of the environment and can therefore be easily thrown off by slight changes in the goalie’s behavior, without proving that the exploit is fixed. Playtesters therefore still have to validate that an exploit is indeed fixed by imitating the strategy found by the approach before the fix.

  Figure 4 shows the training progress of the \(30\) iterations of seed \(1\). The first \(10\) iterations all reach performances above the \(10\)% goal rate cut-off criterion before triggering any of the early stopping criteria. The longer the experiment goes on, the more iterations do not make the cut, with iterations \(11\), \(15\), \(17\), \(20\), \(21\), \(23-25\) and \(27-29\) all converging at goal rates below the threshold and therefore not being added to the previous strategies list (\(Z_\text{prev}\)). We manually stop the search after \(30\) iterations, since we only find \(3\) valid strategies in the last \(11\) iterations, reducing the likelihood of discovering further interesting strategies in practical time. More generally, the iteration limit allows for a trade-off between run time and the expected number of diverse solutions found, with the optimal choice being highly domain-specific and dependent on the diversity criterion. In NHL for example, if we did not differentiate the shot type and only used the position as discriminative feature, we would expect to find less solutions over all, reducing the optimal search time.

  Figure 5 shows the shot position of \(100\) goals after training converged for the \(8\)th iteration of seed \(2\). Around one third of all shot positions lie outside of the 2 m radius around the average of all shot positions, meaning that in succeeding training iterations of  the exact same shots will still be rewarded. This highlights a potentially bigger problem around the introduced diversity criterion for NHL: in a more extreme hypothetical case, a single agent can learn to shoot from two positions that are more than \(4\) meters apart, going for either of the two shot positions at a rate of \(50\)% depending on where the agent is spawned. This way, most of the shots of the agent could be outside of the 2 m radius around the average of the shots’ positions, meaning that future iterations could still learn the exact same behavior without it being excluded. This motivates further investigations into more elaborate representations of a policy’s strategy, for example through clustering methods such as k-means [23].

5 Conclusion, Limitations, and Future Work↩︎

We introduce , a method to find multiple diverse high chance scoring strategies without human intervention in between iterations. The algorithm extends a standard reinforcement learning setup by training multiple agents sequentially, aiming for each agent to come up with a new strategy by not rewarding agents for strategies similar to those of previous iterations. As a domain expert’s review of the learned strategies shows,  finds exploits similar to those identified by human playtesters. We furthermore argue that defining diversity with respect to NHL allows game testers to better follow and adjust the method, compared to prior work that requires RL expertise to interpret the hyperparameters for tuning diversity.

  While our method finds potential exploits autonomously, it still requires human supervision post training, since telling high chance scoring strategies intended by game designers and actual exploits apart is not covered by the method. Furthermore, due to the high variance in RL’s exploration process, our method is not sufficient to determine whether a found exploit has been fixed successfully by an engineer, relying on playtesters for this verification task.

Our method’s runtime grows (at least) linearly with the number of strategies we aim to find. While previous work trains multiple diverse behaviors in parallel [7], we deliberately forego this option to keep our method simple and therefore facilitate its use by playtesters not familiar with RL.

  We design parts of the  method specifically for the use on NHL, though the base idea outlined in Algorithm ¿fig:alg:RAID? could be applicable to other domains as well. While diversity is somewhat intuitive to define for the domain of scoring goals in NHL, future work could research how diversity can be defined in more complex domains, including multiplayer scenarios or even agents outputting text as actions. While previous work that defines diversity in a domain-agnostic way often requires RL knowledge to interpret and adjust hyperparameters, it would be interesting to research definitions of diversity that are domain-agnostic but still interpretable by non-RL practitioners, for example by defining the minimum distance of trajectories used in  in a more general way.

References↩︎