1 Introduction↩︎

Mastering real-time strategy (RTS) games such as StarCraft II has long stood as a grand challenge for artificial intelligence, one only partially addressed by recent advances in reinforcement learning (RL) [1], [2]. The difficulty stems from the cognitive demands these games place on their players: precise control of units, careful management of economies, and continuous decision-making in adversarial settings where even momentary lapses can prove decisive. Success thus hinges on the interplay of multitasking, strategic foresight, and rapid reaction [3], making RTS an especially rich testbed for studying intelligent behavior. These high-frequency interactions make RTS games especially well-suited for large-scale behavioral study through open-source replay parsers and direct game-engine access [4], [5]. A growing body of work leverages data to surface game information for player decision support, both through digital interfaces and physical prototypes [6]. In StarCraft II, community tools such as sc2replaystats [7] and replayman [8] have emerged to support replay analysis, alongside real-time dashboards that contextualize gameplay for spectators and post-match review [9]. Strategic summaries and encounter-level analysis are highly valued by players across genres [10]. Game state retrieval by similarity to estimate win probabilities on demand is a possibility [11]. In parallel, AI methods have become deeply embedded in game research and development, powering procedural content generation [12], voice-driven agents that deepen immersion [13], human-like behavior modeling [14], and automated quality assurance [15]. However, most existing analysis tools remain oriented toward broadcast and streaming audiences rather than the players themselves [16].

This player-facing gap is not unique to gaming. In robotics, efficient simulators have driven dramatic breakthroughs [17], [18], producing systems that now rival or exceed human performance in domains as varied as drone racing [19], [20], badminton [21], [22], and table tennis [23]. Such interdisciplinary efforts are increasingly recognized as accelerators of research progress [24]. However, despite agents and robotic systems consistently surpassing average human ability, few of these works offer mechanisms to translate the resulting expertise back to human practitioners seeking to improve. The skill translation problem is well understood in the sport sciences, where the championship model describes how athletes shorten the path to performance gains by adopting the training methods, techniques, and tactics of successful peers [25], [26]. In domains where the competitive space is naturally digitized, this dynamic increasingly extends to AI. In chess, AI has reshaped human learning by serving as a scalable training partner. Analyses show that elite human play has steadily improved across the engine era [27]–[29]. AlphaZero’s games further illustrate how superhuman agents can surface novel strategic ideas for human study [30]. Similar patterns have emerged in Go following the rise of superhuman agents [31], [32].

RTS games share the same computational substrate, but, to our knowledge, no comparable bridge exists between agent expertise, representational learning, and human improvement. In this work, we introduce a latent-space feedback system that learns compressed representations of quantitative gameplay features from StarCraft II replays and provides manifold-aware improvement guidance to players. Our contribution is a framework for training representational models that recover counterfactual improvement trajectories and reconstruct them back into the original feature space. Given a well-trained model, points sampled along an “improvement trajectory” in latent space can be decoded and compared with the player’s input vector, yielding immediate feedback on which gameplay features need to change to improve performance, as determined by the model. Our work builds directly on “SC2EGSet”, rather than asking “who will win”, we ask “what the player should do differently”.

2 Related Work↩︎

Our work sits at the intersection of four research threads: StarCraft II as a machine learning domain; variational autoencoders and disentangled representation learning; latent space traversal; and counterfactual explanations as actionable feedback. We discuss each in turn and position our contribution relative to prior art.

2.0.0.1 StarCraft II as a Machine Learning Domain:

StarCraft II has become a canonical benchmark for sequential decision-making under partial observability. [33] demonstrated that a combination of imitation learning, multi-agent self-play, and a latent conditioning variable for strategy style can produce grandmaster-level play, establishing that large-scale replay data contains rich, learnable structure. However, AlphaStar is an autonomous agent; it optimizes for winning, not for explaining to human players how to improve. On the other hand, the simplistic nature of benchmarks geared primarily towards multi-agent solutions does not fit well in the context of providing feedback to players [34], [35]. Work on modeling player skill from replays predates AlphaStar. [36] showed that even simple classifiers trained on APM and economy features can predict a player’s league with meaningful accuracy. Subsequent work demonstrated that macro-level economic measures, such as the Spending Quotient introduced by [37], are among the strongest predictors of both skill and match outcomes.

2.0.0.2 Variational Autoencoders and Disentangled Representations:

Naturally, the Variational Autoencoder (VAE) [38] acts as the backbone for our work. By learning a probabilistic encoder and decoder jointly with a Kullback-Leibler (KL) divergence regulariser, the VAE produces a smooth, continuous latent space from which new samples can be reconstructed. As an extension, [39] introduced \(\beta\)-VAE, and addressed the interpretability by up-weighting the KL term. The \(\beta\) factor was applied to force the model to trade reconstruction fidelity for statistical independence between latent dimensions. Guided VAE addresses this in a direct manner [40]. It attaches a supervised classifier to designated latent dimensions, and uses an adversarial excitation-inhibition mechanism to concentrate the target factor in that dimension while preventing it from leaking into the remaining dimensions. [41] presented “SAIL”, which learns persistent skill embeddings from naturalistic behavioral data using expert-novice basis blending and counterfactual subskill swaps applied to motor tasks such as driving and baseball batting. We draw inspiration and intuition from these works.

2.0.0.3 Latent Space Traversal:

Generating semantically meaningful paths through a learned latent space is a non-trivial problem. Naive linear interpolation between two latent codes can pass through low-density regions of the prior, leading to decoded samples that lie outside the data manifold. [42] formalize this distribution mismatch and show that optimal transport (OT) maps can correct linear trajectories so that all intermediate points remain consistent with the prior distribution while minimally deviating from a straight line. [43] propose a more general framework that models latent structures as learned dynamic potential landscapes, deriving traversal trajectories as the gradient flow of a partial differential equation (PDE)-based potential field. [44] take a related approach in the explainability domain with partial focus on StarCraft II by leveraging a fixed “SC2 Assault” scenario. They generate counterfactuals from a jointly trained generative latent space where the traversal is guided toward a target outcome during decoding.

2.0.0.4 Counterfactual Explanations as Actionable Feedback:

Counterfactual explanations answer the question: “what is the minimal change to the input that would change the model’s prediction?” In a performance-improvement context, this is equivalent to algorithmic recourse. Providing a ranked list of feature changes that move a player from their current state to a more desirable one. [45] proposed “CEILS”, generating counterfactuals as interventions in the latent space of a trained VAE [45]. Work beyond passive explanation towards actionable coaching was shown by [46], leveraging counterfactual explanations in racing scenarios with language-based guidance. [47] extend latent-space counterfactual generation by equipping the VAE latent space with a Riemannian metric pulled back through both the decoder and the classifier.

3 Material and Methods↩︎

3.0.0.1 Replay Preprocessing and Feature Extraction:

To fulfill our goal of providing feedback based on learned representations, we have decided to use a dataset consisting of professional StarCraft II games named “SC2EGSet” [4] licensed under CC-BY 4.0. Please refer to Appendix 8 for a simplified game description. At the time our work was prepared, the dataset consisted of 23,476 files containing game-state information sourced from 71 “replaypacks”. Before training, each replay is converted into a tensor containing information about both players. For every player, we extract a 196-dimensional feature vector \(\mathbb{R}^{196}\). Selected features primarily include economical game progression statistics that inform various aspects of the game. Additionally, for more expressive modeling insights, we split the array of player statistics into three windows, each spanning one-third of the total game duration. These features are named as “early”, “middle”, and “late game” windows (\(3 \cdot 39\) features). Tracker statistics averaged within each window. Finally, we include the final economy state (\(39\) features) and the economy difference features (\(39\) features), computed as late-game economy minus early-game economy. Besides the economy, we include a scalar value, “supply capped percent”, denoting the percentage of the game duration during which the player was unable to build additional units due to insufficient in-game infrastructure. The final replay tensor has shape \(\mathbb{R}^{2 \times 196}\), where the first dimension corresponds to the player. The label used for disentanglement is the match result from player 0’s perspective, represented as a binary outcome \(y \in {0, 1}\), where \(y = 1\) indicates that player 0 won. Prior to training, all features are standardized using per-feature z-score normalization. The normalization statistics (mean and standard deviation) are computed exclusively on the training set and subsequently applied to the validation and test sets, preventing any data leakage. A small epsilon (\(\epsilon = 10^{-8}\)) is added to each standard deviation to avoid division by zero for constant features.

The initial split between training, validation, and test sets was random (80%/10%/10%). Replays with missing player statistics, player information, or undecided/draw outcomes were skipped. Finally, to assure that the training, and validation sets were indeed (80%/10%), the missing samples were taken out of the test set without replacement. After excluding samples, the splits consisted of the following numbers of samples in the training set (\(n_{training} = 18780\)), the validation set (\(n_{validation} = 2347\)), and the test set (\(n_{test} = 2178\)). To confirm the efficacy of our method on out-of-distribution (OOD) data, we have randomly sampled an additional (\(n_{ood} = 2178\)) samples from an unreleased dataset of players who submitted their replays to the sc2replaystats [7] in 2016-2020. Access to all of the pre-processed data and code is available. For more information please see Appendix 14.

3.0.0.2 Modelling

To ensure the possibility of transitioning between learned latent space representations and reconstruction of the player features, we have decided to use a model adapted from the original Guided VAE [40] architecture. We guide the latent space separation by the game outcome. Therefore, a part of the latent representation is encouraged to contain information about features that are significant for predicting winning or losing outcomes. The structure of our model consists of a symmetrical encoder-decoder multilayer perceptron (MLP) network with ReLU activations, with an additional supervised classifier attached to the latent space. In our modified Guided VAE [40], a selected number of dimensions are set to be supervised. An adversarial classifier is trained on the remaining latent dimensions, excluding the supervised dimensions. In training, the supervised classifier receives concatenated supervised latent dimensions of both players, \([z^{(0)}{1:k} | z^{(1)}{1:k}]\), and is trained to predict \(y\) directly. At inference-time, when generating improvement paths for the player in seat 1, the input order is swapped, and the output probability is complemented, so that the score always represents the win probability of the player whose path is being improved. Please refer to Appendix 9 for more implementation details. We use AdamW optimizers [48], [49], implemented in PyTorch. The first updates the VAE and its guided classifier jointly. The second trains an auxiliary adversarial classifier on the non-guided (free) latent dimensions. The third updates the VAE parameters adversarially, penalising the encoder for producing free-dimension representations that are predictive of match outcome. Validation is monitored using the VAE loss, and early stopping is used to avoid overfitting. We conducted a hyperparameter search to find the best-performing model, see Appendix 11. All of the experiments were run on consumer hardware, see Appendix 10.

3.0.0.3 Latent Space Traversal and Feedback Generation:

After training, a replay is encoded into a latent vector. We construct a path from the current game state to a winning region in the supervised dimensions of the latent space. Each point interpolated along the path in the latent space can be reconstructed back to the feature space, where the difference between the current feature values and the reconstructed feature values informs the player about the improvement “trajectory”. We compute feedback in three ways. First, we measure the raw change from the start of the path to the end. Second, we compute a minimum viable change that stops at the first waypoint where the predicted win probability crosses 0.5, if such a crossing exists. Third, we compute a win-probability-weighted change, where each step’s feature change is weighted by the corresponding increase in predicted win probability; steps where win probability decreases contribute zero weight. The final output is a ranked list of feature changes. These changes act as rough suggestions, and should be interpreted as model-generated hypotheses. Example generated feedback reports are available in Appendix 13.

4 Standardized Path Representation↩︎

To ensure compatibility with downstream generation and visualization tasks, the output of the path charting pipeline is strictly standardized, regardless of the specific employed strategy. The generated path \(\mathcal{P}\) is defined as an ordered sequence of \(n\) discrete waypoints in the \(d\)-dimensional latent space. This sequence is structured as a matrix \(\mathbf{P} \in \mathbb{R}^{n \times d}\), where each row \(\mathbf{p}_j \in \mathbb{R}^d\) corresponds to a specific waypoint along as seen in 1 .

\[\mathbf{P} = \begin{bmatrix} \text{---} & \mathbf{p}_0 & \text{---} \\ \text{---} & \mathbf{p}_1 & \text{---} \\ \vdots & \vdots & \vdots \\ \text{---} & \mathbf{p}_{n-1} & \text{---} \end{bmatrix} = \begin{bmatrix} \text{---} & \mathbf{z}_{start} & \text{---} \\ & \vdots & \\ \text{---} & \mathbf{z}_{end} & \text{---} \end{bmatrix} \label{eq:path95representation}\tag{1}\] By definition across all strategies, the first waypoint \(\mathbf{p}_0\) corresponds exactly to the initial latent vector \(\mathbf{z}_{start}\). The final waypoint \(\mathbf{p}_{n-1}\) corresponds to the terminal state of the strategy, denoted generally as \(\mathbf{z}_{end}\) (which may represent a predefined target \(\mathbf{z}_{target}\), a successfully converged optimization state, or the integrated endpoint of a velocity field). Traversal methods have their own set of tunable parameters fully described in Appendix 11.

4.1 Linear Strategy↩︎

The linear strategy serves as the foundational baseline for counterfactual generation. It implements a simple Linear Interpolation (LERP) to navigate the high-dimensional latent space, constructing a straight-line trajectory between the initial losing latent vector and a specified winning target. Target Selection Strategies: Because the "winning" state is represented by a distribution of points rather than a single vector, the algorithm first collapses the winning latents \(\mathcal{Z}_{win} = \{\mathbf{w}_1, \dots, \mathbf{w}_M\}\) into a single target vector \(\mathbf{z}_{target} \in \mathbb{R}^d\) using one of two selectable methods. Centroid Strategy (method="centroid"): The target is defined globally as the unweighted mean position (centroid) of all known winning latent vectors. This provides a robust, generalized direction toward the center of the winning class, see 1. Nearest Neighbors Strategy (method="nearest"): To preserve local manifold structure and find the "closest" way to win, the target is derived locally. It calculates the mean of only the \(k\) winning latents that are closest to the starting sample \(\mathbf{z}_{start}\) (where \(k\) is defined by k_neighbours), see 2.

Linear Interpolation (LERP) Formula: Once the target \(\mathbf{z}_{target}\) is established, the path is generated as a sequence of \(n\) waypoints \(\{ \mathbf{p}_0, \mathbf{p}_1, \dots, \mathbf{p}_{n-1} \}\). The interpolation coefficient \(\alpha_j\) is defined by an evenly spaced linear progression from \(0.0\) to \(1.0\), see 3. Each intermediate point \(\mathbf{p}_j\) along the trajectory is calculated using a convex combination of the start and target vectors, moving progressively closer to the target as \(\alpha\) increases, see 4. This straight-line interpolation assumes a globally Euclidean latent space, transitioning semantic features at a constant velocity without explicit regard for the underlying data density.

4.2 Iterative Optimal Transport Strategy↩︎

The iterative optimal transport (OT) strategy simulates a vector field flow toward the winning distribution. Instead of interpolating toward a single static target (such as a centroid), the algorithm dynamically recalculates a local barycentric target at each step by finding the optimal mass transport plan between the current position and the target distribution. Distance Matrix and Transport Plan: At each step \(t\), the current latent position \(\mathbf{z}^{(t)}\) is treated as a point mass with weight \(\mathbf{a} = 1\). The set of \(N\) winning latents \(\mathcal{Z}_{win} = \{\mathbf{w}_1, \dots, \mathbf{w}_N\}\) is treated as a uniform target distribution with weights \(\mathbf{b} = \frac{1}{N} \mathbf{1}\). First, the squared Euclidean distance matrix \(\mathbf{M}^{(t)} \in \mathbb{R}^{1 \times N}\) is computed. To prevent numerical underflow during exponentiation in the regularized transport step, the distance matrix is normalized by its maximum value as seen in 5. Next, the optimal transport plan \(\mathbf{T}^{(t)}\) is computed. If entropic regularization \(\lambda > 0\) (ot_reg) is provided, the mathematically stable log-domain Sinkhorn algorithm is utilized as seen in 6, where \(H(\mathbf{T})\) is the entropy of the coupling matrix. If \(\lambda = 0\), the exact Earth Mover’s Distance (EMD) is computed. Local Barycentric Target: The resulting transport plan \(\mathbf{T}^{(t)}\) dictates the optimal distribution of mass from the current position to the winning points. These transport weights are normalized to form a localized probability distribution as seen in 7. The local target \(\mathbf{z}_{target}^{(t)}\) is then defined as the barycenter (weighted average) of the winning points, pulled specifically according to the optimal transport plan presented as 8. Euler Step (Vector Flow): Rather than jumping directly to the target, the algorithm treats the vector \((\mathbf{z}_{target}^{(t)} - \mathbf{z}^{(t)})\) as a local velocity field. It takes a small Euler step of size \(\eta\) (step_size) toward the local barycenter, as seen in 2 . This process is repeated iteratively to trace a smooth trajectory. Because the transport plan dynamically updates at each spatial step, the resulting path closely mimics a continuous vector flow into the densest regions of the winning distribution.

\[\mathbf{z}^{(t+1)} = \mathbf{z}^{(t)} + \eta \left( \mathbf{z}_{target}^{(t)} - \mathbf{z}^{(t)} \right) \label{eq:vector95flow}\tag{2}\]

4.3 Gradient Ascent Strategy↩︎

In the context of a Guided VAE, the gradient ascent strategy actively searches for a counterfactual path. Starting from a starting latent representation, the goal is to discover the minimal feature changes required to transition into a winning state. To ensure the generated counterfactuals remain realistic and do not exploit adversarial blind spots in the classifier, the trajectory is explicitly regularized by the data manifold. Objective Formulation: The optimization seeks to iteratively adjust the latent vector \(\mathbf{z}\) to maximize an opponent-aware classification score \(S(\mathbf{z})\) (e.g., the logit of winning against a specific opponent \(\mathbf{z}_{opp}\)), constrained by a manifold density penalty \(D(\mathbf{z})\). The density is modeled using a fully differentiable Gaussian Kernel Density Estimate (KDE) evaluated over the reference dataset of known winning latents \(\mathcal{Z}_{win} = \{\mathbf{w}_1, \dots, \mathbf{w}_N\}\) with bandwidth \(h\), as seen in 9. The total combined gradient at step \(t\) merges the direction that increases the likelihood of winning with the direction that points toward denser, realistic regions of the latent space, as shown in 10, where \(\lambda\) represents the weighting of the density prior (density_weight). Momentum-Based Optimization Update: To traverse the disentangled latent space smoothly and avoid local minima, the latent vector is updated using gradient ascent with momentum. Let \(\alpha\) be the learning rate and \(\beta\) be the momentum factor. The velocity \(\mathbf{v}\) and position \(\mathbf{z}\) are updated as seen in 11, and 12. Convergence and Resampling: This iterative process continues until the predicted probability of the winning class exceeds a specified convergence_threshold \(\tau\) as in 3 . Because the number of optimization steps \(T\) required to reach this threshold is variable, the resulting sequence \(\{ \mathbf{z}^{(0)}, \mathbf{z}^{(1)}, \dots, \mathbf{z}^{(T)} \}\) is evenly resampled to extract exactly \(n\) waypoints. This final trajectory is then output as the standardized path matrix \(\mathbf{P} \in \mathbb{R}^{n \times d}\), representing a smooth, realistic counterfactual feature transition.

\[P(\text{win} \mid \mathbf{z}^{(t)}, \mathbf{z}_{opp}) \ge \tau \label{eq:convergence95condition}\tag{3}\]

4.4 Neural Flow Strategy↩︎

The neural flow strategy utilizes continuous normalizing flows via an Optimal Transport (OT) Flow Matching framework. Rather than relying on simple geometric interpolations or local gradient steps, this approach trains a neural network to learn a global velocity field. This field models the continuous optimal transport of probability mass from the "losing" to the "winning" latent distribution. Velocity Field Training and OT Pairing: A Multi-Layer Perceptron (MLP) acts as a time-conditioned velocity field \(\mathbf{v}_\theta(\mathbf{z}, t)\), parameterized by weights \(\theta\). During training, mini-batches of losing latents \(\mathbf{Z}_0\) and winning latents \(\mathbf{Z}_1\) are extracted. To ensure the network learns the most efficient, non-crossing paths between these distributions, the samples are dynamically paired using exact Earth Mover’s Distance (EMD) based on squared Euclidean distance. For each \(\mathbf{z}_0\), an optimal \(\mathbf{z}_{1, \text{paired}}\) is identified. At a uniformly sampled time \(t \in [0, 1]\), the intermediate state is defined by linear interpolation as shown in 13. The network is then trained to predict the constant-velocity vector between paired samples by minimizing the Mean Squared Error (MSE), as shown in 14. Trajectory Integration (Euler Method): To generate a counterfactual path during inference, a starting (losing) latent \(\mathbf{z}_{start}\) is integrated through the learned velocity field from \(t=0\) to \(t_{max} = 1.0\). Using a discrete number of integration steps \(N_{steps}\), the time increment is \(\Delta t = \frac{1.0}{N_{steps}}\). The latent position is updated iteratively using the Euler method seen in 15, where the initial condition is \(\mathbf{z}^{(0)} = \mathbf{z}_{start}\). This produces a smooth flow along the learned data manifold. Classifier Guidance (Optional): To explicitly steer the trajectory toward regions with a higher probability of winning against a specific opponent, optional classifier guidance can be injected into the Euler integration. At each step, the gradient of the probability score \(S(\mathbf{z})\) is computed. To ensure the guidance scale remains a consistent fraction of the flow step size, regardless of the raw gradient’s magnitude, the gradient is normalized to a unit vector. The position is updated as shown in 16, where \(\gamma\) (guidance_scale) dictates how strongly the path is pulled toward the classifier’s optimal regions, and \(\epsilon\) prevents division by zero.

5 Experiments↩︎

Model Performance: To evaluate the quality of the trained Guided VAE, we assess both its reconstruction capabilities and the effectiveness of the latent space separation. The model’s performance on the held-out test set is summarized using several key metrics as seen in 1.

Evaluation of the GuidedVAE demonstrates strong generative and reconstruction fidelity, with the MSE and KL divergence confirming accurate game-state reconstruction from a well-regularised latent space. Furthermore, evaluation of the predictive guidance imposed on the first latent dimension—measured via test accuracy, ROC-AUC, and Brier score—indicates robust classification performance and highly calibrated win probabilities. Overall, the model successfully balances precise feature reconstruction with meaningful latent disentanglement, establishing a reliable foundation for generating counterfactual improvement trajectories. Conterfactual Paths: To evaluate the performance of the final model against our main goal of latent space traversal generating counterfactual “improvement trajectory”, we conduct a comparative assessment of all path generation strategies in 2. Where the success rate is the fraction of samples for which the path reaches \(P(\text{win}) \geq 0.5\) at any waypoint along the path. Crossover \(\alpha\) is the position along the path at which \(P(\text{win})\) first crosses 0.5, where \(\alpha = 0\) is the start and \(\alpha = 1\) is the end. Reported only for successful runs, \(\boldsymbol{\Delta}\)P(win) is the absolute gain in predicted win probability from path start to path end, i.e.\(P(\text{win})_{\text{end}} - P(\text{win})_{\text{start}}\)., AUC is the area under the \(P(\text{win})\) curve over \(\alpha \in [0,1]\). Monotonicity is the fraction of consecutive waypoint pairs for which \(P(\text{win})\) is non-decreasing. A value of 1.0 means \(P(\text{win})\) increases or stays flat at every step. Lower values indicate oscillation or regression along the path. Finally, the nearest-win distance is the Euclidean distance in the supervised latent subspace between the path endpoint and the closest winning latent vector in the training set. While Gradient Ascent achieves a nominally perfect success rate (\(1.000\)), further inspection of the secondary metrics suggests this performance is largely driven by adversarial off-manifold drift. Compared to geometrically grounded strategies such as Optimal Transport, Gradient Ascent tends to explore low-density regions of the latent space. This is directly evidenced by a heavily inflated maximum latent norm (\(\max \|\mathbf{z}\| = 3.95 \pm 1.74\), compared to just \(2.03 \pm 0.58\) for Optimal Transport) and a severely degraded path KDE density (\(-4.63 \pm 1.40\) vs. \(-2.80 \pm 0.50\)). Furthermore, its significantly reduced monotonicity (\(0.727 \pm 0.227\) vs. \(0.998 \pm 0.041\)) and higher nearest-win distance (\(0.15 \pm 0.12\) vs. \(0.06 \pm 0.04\)) indicate erratic, unconstrained traversal rather than smooth semantic interpolation. Consequently, the representations generated by this strategy are likely to exploit classifier blind spots and warrant much closer examination, see Appendix 12. We hypothesize that these failure modes could be addressed in future work by imposing stricter regularization constraints to firmly anchor the trajectory to the learned data prior.

The experimental results reveal several key insights regarding the trade-offs between path reliability and quality:

The substantial variance in \(\Delta P(\text{win})\) across the linear baselines further suggests that simple interpolation is insufficient to capture the model’s complex decision boundaries. In contrast, neural and transport-based methods provide more consistent counterfactual evidence, with a mean \(P(\text{win})\) along the counterfactual path shown in 17, and directly showcase some of the aforementioned trade-offs on OOD data. Further model interpretability is covered in Appendix 12.

6 Limitations and Future Research↩︎

Limitations: Despite the promising results, our approach has several limitations that should be acknowledged. First, we train our model on a dataset of games spanning multiple years. We do not explicitly account for the game updates. Additionally, we do not encode the players’ in-game race information in any way. By design, our model is incapable of providing feedback directed towards specific in-game actions and environment configurations. The model feedback is additionally constrained by the dataset choice; tournament gameplay samples can be seen as a very specific subset of all of the games. Our model does not directly convey a more granular approach of jumping between leagues when leveraging its feedback. Finally, the model and method parameters were not verified with human participants, and aside from expert input from known professional players, we were unable to set up a human-in-the-loop type of experiment. Future Research: We hope that by extending research efforts in representational learning geared towards providing feedback, we can inspire others to prepare end-to-end feedback generation. Automating ways to improve humans based on deep generative solutions and other computational means. In the future, we wish to address most of the concerns raised above. Moving towards optimizing human performance jointly with actions against an environment sounds incredibly exciting, with the potential to uncover environment configurations that promote positive training or learning outcomes. Given the recent advancements and rapid adoption of AI systems, creating models that provide data-driven, actionable feedback is crucial.

7 Conclusion and Summary↩︎

We have demonstrated the computational feasibility and the potential for developing many practical solutions for providing feedback based on learned representations. Additionally, simplistic methods, while appealing, have drawbacks that become more evident when dealing with a more advanced nonlinear model. Finally, we have accomplished our goal of bridging the gap between representational learning and generating feedback by extracting actionable, counterfactual improvement trajectories from the latent space, effectively shifting the analytical paradigm from predicting game outcomes to providing players with tangible guidance on what they should do differently. Based on our ongoing discussions with esports professionals, our solution is proving to be a real asset for future feedback systems.

8 StarCraft II Game Description↩︎

Before diving deeper it is important to state the rules governing StarCraft II competitive gameplay. The game contains three main “races” of choice for the players. Each race is differentiated by unit mechanics therefore forcing certain playstyles. In most cases the game is played in the format of player versus player (PvP), one versus one (1 vs 1). The ultimate goal for competitors is to destroy all of the opponents’ structures, or to force their counterpart to resign. In many cases tournaments have varying stages, such as the initial group stage, and a subsequent knockout bracket stage. Depending on the tournament format in most cases the group stages are played out as best of one (Bo1), best of two (Bo2), or best of three (Bo3) matches. Finally, the knockout bracket features Bo3, best of five (Bo5), and best of seven (Bo7) matches.

9 Model Details↩︎

9.0.0.1 Training Objective:

The VAE is trained with the standard reconstruction and KL divergence objective. The reconstruction term is mean squared error between the original feature vector and the decoded feature vector. The KL term regularizes the posterior distribution toward a unit Gaussian prior. The VAE loss is shown in 18 where \(\mathcal{L}_{\mathrm{MSE}}\) is the summed mean squared error between the input feature vector and its reconstruction, and \(\mathcal{L}_{\mathrm{KL}}\) regularizes the approximate posterior toward a unit Gaussian prior. To guide the latent space, we add a binary cross-entropy loss on the prediction produced from the first latent dimension. The main Guided VAE update is seen on 19, where \(\lambda_{\mathrm{cls}}\) controls the strength of supervision and encourages outcome-relevant information to be concentrated in the guided latent dimension.

Aside from the typical structure of a Guided VAE model we clamp the log variance output of the encoder to the interval \([-20, 2]\) before the reparameterization step. This hard bound constrains the effective standard deviation to the range \([\approx 4.5 \times 10^{-5},\;\approx 2.7]\), preventing two sources of numerical instability: a near-zero variance, which causes the KL divergence term in the evidence lower bound (ELBO) to diverge and produces exploding gradients; and an excessively large variance, which overwhelms the mean and injects too much noise into the decoder input, destabilizing reconstruction. These dimensions are used to predict the game outcome. The point of this is to force at least one direction of the latent space to be directly related to winning and losing.

10 Hardware↩︎

10.1 Hardware and Computational Requirements↩︎

All experiments were conducted on a high-performance workstation using consumer-grade hardware. The specific configuration of the system components is detailed in Table 3.

The model architectures and optimization strategies were designed for high efficiency. Consequently, the majority of individual experimental runs, including those within the Guided-VAE hyperparameter sweep and path generation evaluations, were completed in under 5 minutes.

11 Hyperparameter Search↩︎

11.1 Guided-VAE Hyperparameter Search↩︎

11.1.1 Search Space↩︎

The search space for the Guided-VAE is defined by a set of hierarchical constraints to ensure a valid bottleneck architecture. Let \(\mathcal{W} = \{w_0, w_1, \dots, w_m\}\) denote the set of available layer widths in ascending order.

The encoder configuration is sampled via two primary parameters: the number of layers \(n \in \{2, 3, 4\}\) and a categorical start width \(w_{start} \in \mathcal{W}\). To guarantee that the resulting hidden dimensions \(\mathbf{H}_{enc}\) are strictly decreasing, we calculate the actual starting index \(i\) as: \[i = \max(n - 1, \text{index}(w_{start}))\] The sequence of hidden dimensions is then defined as: \[\mathbf{H}_{enc} = \{ w_{i-j} \}_{j=0}^{n-1}\]

The latent dimensionality \(n_z\) is coupled to the final encoder width \(h_{last} \in \mathbf{H}_{enc}\) through a fraction \(f_z \in \{0.25, 0.5, 1.0\}\), constrained by a minimum floor: \[n_z = \max(8, \lfloor h_{last} \cdot f_z \rfloor)\]

The remaining parameters are sampled according to the following distributions:

• Learning Rates: \(\eta_{vae}, \eta_{cls} \sim \text{LogUniform}(10^{-5}, 10^{-3})\)

• Weight Decays: \(\lambda_{vae}, \lambda_{cls} \sim \text{LogUniform}(10^{-6}, 10^{-3})\)

• Classification Weight: \(\alpha \sim \text{LogUniform}(1, 250)\)

• Supervised Dimensions: \(n_s \in \{1, 2, 4\}\)

11.1.2 Hyperparameter Run Configuration↩︎

To find the best hyperparameters for our training, we ran the search using Ray (https://www.ray.io/) and Optuna (https://optuna.org/). Upon execution, we have decided on 150 total runs. The objective function, \(\mathcal{O}_{HPO}\), is defined as a weighted scalar sum of validation metrics logged during the training of the Guided VAE model. Formally, the minimization objective is expressed as:

\[\min_{\theta} \mathcal{O}_{HPO} = \sum_{i \in \mathcal{M}} w_i \cdot \mathcal{L}_i(\theta)\]

where \(\mathcal{M}\) denotes the set of validation metrics, \(\mathcal{L}_i\) is the value of the \(i\)-th metric, and \(w_i\) is the user-defined weight for that metric. In the configuration utilized for this sweep, the objective was set to equally weight the reconstruction and classification components:

\[\mathcal{O}_{HPO} = w_{\text{vae}} \mathcal{L}_{\text{val\_vae}} + w_{\text{cls}} \mathcal{L}_{\text{val\_cls}}\]

Given our configuration parameters \(w_{\text{vae}} = 0.5\) and \(w_{\text{cls}} = 0.5\), the final objective function simplifies to:

\[\mathcal{O}_{HPO} = 0.5(\mathcal{L}_{\text{val\_vae}}) + 0.5(\mathcal{L}_{\text{val\_cls}})\]

11.1.3 Guided VAE Final Hyperparameters↩︎

4 contains the model we have selected for a best performing model.

11.2 Path Generation Strategies: Hyperparameter Search↩︎

11.2.1 Search Space↩︎

The search space for the latent space traversal is structured hierarchically, where the subset of active hyperparameters is conditioned on the chosen strategy \(S\).

11.2.1.1 Linear Strategy

The linear interpolation strategy relies on neighborhood density constraints:

• Nearest Neighbors (\(k_{neighbors}\)): \(k_{nb} \sim \text{DiscreteUniform}(3, 15)\)

• Opponent Constraints (\(k_{opponents}\)): \(k_{opp} \sim \text{DiscreteUniform}(10, 200)\)

11.2.1.2 Gradient Ascent Strategy

This strategy utilizes a density-based optimization approach with fixed steps \(T=2000\) and a convergence threshold \(\tau=0.95\):

• Learning Rate (\(\eta\)): \(\eta \sim \text{LogUniform}(10^{-4}, 0.1)\)

• Momentum (\(\mu\)): \(\mu \sim \text{Uniform}(0.0, 0.95)\)

• Density Weight (\(w_{\rho}\)): \(w_{\rho} \sim \text{Uniform}(0.0, 1.0)\)

• KDE Bandwidth (\(h\)): \(h \sim \text{Uniform}(0.1, 2.0)\)

11.2.1.3 Optimal Transport Strategy

The optimal transport strategy balances regularization and geometric constraints:

• Regularization (\(\epsilon\)): \(\epsilon \sim \text{LogUniform}(0.01, 0.5)\)

• Step Size (\(\gamma\)): \(\gamma \sim \text{Uniform}(0.05, 0.5)\)

• Opponent Constraints (\(k_{opponents}\)): \(k_{opp} \sim \text{DiscreteUniform}(10, 200)\)

11.2.1.4 Neural Flow Strategy

The neural flow strategy utilizes a fixed guidance scale for its transformation:

• Guidance Scale (\(s\)): \(s = 1.0\) (fixed)

11.2.2 Hyperparameter Run Configuration↩︎

The optimization of hyperparameters for the latent space traversal strategies was performed using the Optuna framework. For each of the strategy we executed \(N=100\) independent trials. In each trial, the performance was evaluated by generating \(n_{samples} = 1000\) latent paths.

Unlike the model training phase, the objective for path charting is a maximization task. The objective function \(\mathcal{J}\) is defined as the mean performance of a specified evaluation metric \(\mathcal{M}\): AUC across all generated samples:

\[\max_{\phi} \mathcal{J}(\phi) = \mathbb{E}_{s \sim \mathcal{S}(\phi)} [ \mathcal{M}(s) ]\]

where \(\phi\) represents the set of strategy-specific hyperparameters sampled from the search space, and \(\mathcal{S}(\phi)\) denotes the distribution of paths generated under those parameters.

11.2.3 Path Generation Strategies: Final Hyperparameters↩︎

5 contains the specific hyperparameters used for the final runs of our path generation strategies.

12 Additional Results↩︎

12.1 Model Interpretability↩︎

To verify that the Guided VAE’s outcome classifier grounds its predictions in strategically meaningful features, we analyze its decision process using SHapley Additive exPlanations (SHAP) as shown in 20 and 21. SHAP values provide a unified measure of feature importance by attributing the log-odds of the predicted outcome to the individual input features. The mean absolute SHAP values highlight the top global contributors to the win-probability prediction, confirming that the model relies on core economic and macroscopic indicators rather than spurious correlations. Furthermore, the SHAP beeswarm plot reveals the distribution of these impacts across the test. It visualizes both the magnitude and the direction of the feature effects, illustrating how higher or lower values of specific features correlate with the predicted likelihood of winning. This transparency is crucial, as it ensures that the counterfactual paths generated by traversing the latent space correspond to interpretable, domain-consistent shifts in player behavior.

13 Example Feedback Reports↩︎

22 and 23 are examples of the output, informing the user about the counteractual latent space path, and which parameters they should focus on.

14 Data and Code Repositories↩︎

Anonymized version of our code and the pre-processed tensor data, as well as the model checkpoint are available at: https://anonymous.4open.science/r/SC2_LatentTrainer-1E5B/. The original data repository is: https://huggingface.co/datasets/Kaszanas/SC2EGSet

References↩︎