Introduction↩︎

With artificial intelligence (AI) entering the mainstream, there is a growing focus on enabling users to meaningfully align the system’s behavior to the user’s preferences. While users are now familiar with requesting generative AI tools [11], [12] to follow style requests ("generate an impressionist painting of a cat playing table tennis."), such control has so far been unavailable for agents operating in real-time control settings. Concurrently, reinforcement learning (RL) has become a leading method for achieving state-of-the-art performance across a wide range of real-time control domains [3], [13], [14]. These RL agents typically master a single task and perform it in a single, near-optimal way. In practice, in addition to what an agent accomplishes, users often care about how it accomplishes the task, which may depend on the context at run-time. For example, we might prefer our cleaning robot to clean up the room quietly because the baby is sleeping, or to prioritize speed because guests are about to arrive. These preferences do not change the underlying task itself—cleaning the room—but they do express valuable, context-dependent preferences over the agent’s behavior.

Coachable agents are trained to perform a task in a variety of different styles, and to apply those styles on-demand at execution time. This problem framing differs from multi-task[15], [16], multi-objective[17], [18] or goal-conditioned learning [19] in which the problem statement is focused on teaching an agent to accomplish multiple different tasks. To understand the difference, consider the robot cleaning scenario: classifying “cleaning quietly” and “cleaning quickly” as two different tasks downplays their commonality and obscures the fact that there is a spectrum of quiet and quickly that the user may want to select from. With styles—similar to how an adverb modifies a verb—we keep the primary task constant but control the secondary performance characteristics.

As when coaching human athletes, the majority of the effort in coaching agents comes in creating the incentives and practice scenarios that allow the agents to learn the skills and style variations they will need in their particular domains. In this study, we explore these topics in three domains, each with a unique set of challenges. The simplest environment is the humanoid model from the DeepMind Control Suite (DMC)[20], a standard RL test environment. The other two domains are AAA video games: the racing game Gran Turismo 7 (GT7) and the open-world adventure game Horizon Forbidden West (HFW). Modern commercial video games present richly interactive worlds with well-defined rules and clear success metrics, making them excellent proving grounds for advances in AI [21], [22]. Moreover, video games are often designed to allow players to play with a variety of different playstyles [23], making them ideal test environments. In this work, we describe the innovations necessary to scale to the complexity of modern video games, and suggest best practices for training coachable agents.

Approach↩︎

The foundation of our framework for coaching agents is reinforcement learning, the framework in which software agents learn a task through trial and error [24]. The field gained broad public attention around 2016 when RL, combined with deep neural networks, was fundamental to several high-profile AI milestones [1], [25]. RL algorithms usually make use of a policy, \(\pi(s): s \rightarrow a\), and/or a critic, \(Q(s,a): s, a \rightarrow V\), where \(V\) approximates the expected cumulative (usually discounted) future reward of taking action \(a\) in state \(s\) and following \(\pi\) thereafter. To make RL work in practice, designers must choose among a variety of algorithms, environment state representations, reward formulations, and training paradigms to produce an agent that can achieve the desired outcome [26]. The standard application of RL often leads to agents that perform their tasks well but offer little run-time flexibility as to how those tasks are carried out.

To train coachable agents we build off prior work on universal value function approximators (UVFAs)[27], [28], which extend the standard value function to condition on “goal states”. Unlike the original UVFA formulation, our approach addresses scenarios in which the core objective remains fixed, but multiple valid behaviors exist for accomplishing it. We refer to this variation as style-conditioned UVFA. As for related work, de Woillemont [29], [30] showed how carefully-designed reward terms could be used to encourage agents to learn five policies that could implement several playstyles on a small, turn-based video game. And recently, Nauman et al. [31] used reward-conditioning as well. However, they used training data generated under just the task reward and off-policy learning to train on additional reward functions. Consequently, without finetuning, the resulting policy performs poorly when evaluated on one of these additional reward functions. And they applied their approach to relatively small domains such as OpenAI Gym and DMC Vision. Our work elevates the UVFA concept to high-dimensional control domains—a substantially more challenging setting that requires further algorithmic innovations.

In our version of UVFA, the reward function is split into two parts: a part that is fixed across episodes and captures the task objective, and a part that pushes the agent towards certain behavioral styles: \(r(s,a; \theta) = r_{task}(s,a) + r_{style}(s,a,\boldsymbol{\theta})\). The term \(r_{style}(s,a,\boldsymbol{\theta})\) combines different style-reward functions through a multi-dimensional style-vector \(\boldsymbol{\theta}\): typical style parameters include weights to linearly combine rewards, as well as thresholds (for gating/ramping), and targets (setpoints) that penalize deviation from desired ranges. The scale of \(r_{style}(s,a,\boldsymbol{\theta})\) relative to that of \(r_{task}(s,a)\) determines how the agent balances different style objectives versus its task objective.

As in standard RL, we must choose the state features, rewards, and action representations for the agent to have the capacity to succeed in each domain. For training agents that can exhibit styles, we have further found that scenario training[32]–[34], data augmentation [35], and special-purpose replay buffers[36] help speed up and stabilize learning. Figure 1 illustrates the major components of the coaching framework that lead to an agent that can play HFW with a wide variety of styles. To make the first coachable agent for a AAA open-world game—where the action space, reward structure, and behavioral variance are all substantially more complex than our other test domains—we developed a new member of the Soft Actor-Critic (SAC) family of algorithms[37], which we call Categorical Regularized Actor-Critic (Cat-RAC). The extensions to Soft Actor-Critic to create Cat-RACare detailed in the appendix sections.

The test domains↩︎

Our prior work on GT Sophy [3] led to the agent being successfully incorporated into GT7. Our initial motivation for exploring styles was to enable the designers at Polyphony Digital, Inc. (PDI) to have more control over GT Sophy’s in-game behavior without sacrificing the agent’s ability to control the car. The styles we present for GT Sophy reflect real-world driving behaviors, such as the strategic decision to sacrifice lap time to delay a pit stop.

Because the motifs of the game HFW drive our style choices, we must provide the reader with enough background to understand the results. In this open-world game, the heroine, Aloy, battles a variety of animal-inspired robots across a post-apocalyptic version of the American West. Aloy always carries a spear for melee combat, and, through a weapon wheel, the player has quick access to six projectile weapons, each having two or three ammo types. Aloy can also lay proximity traps and lure machines into them, and she can consume healing berries to recover health. All of this flexibility requires extensive use of the 14 boolean buttons, two 2D joysticks, and the two continuous triggers on the PlayStation 5 controller. The enemy machines are complex non-player characters (NPCs) with dozens of animated attack types, many of them unique to each machine type. They typically have components—including weapons, shields, and canisters—that can be targeted and removed. Aloy’s ammo and traps, often carry an elemental type which can build up and put the target into an elemental state. For example, if Aloy hits a machine with several fire arrows, the machine may reach a burning state in which it takes continuous fire damage for a period of time. Table 1 in Appendix D shows details of the machines in the training set, and we refer interested readers to any number of online resources with extensive information on the game. The styles our agent exhibits in HFW reflect the breadth of combat choices the developers created for players, allowing them to choose their favorite weapons, play it safe with traps, or leverage the powerful elemental systems of the game.

To establish the generality of our approach beyond games, we train an agent for the DeepMind Control Suite [20] humanoid model. The humanoid is an unstable dynamic system with a 21 dimensional action space whose core objective is to walk across an infinite plane. On top of the base task we layer styles to control discrete arm poses that can be combined with continuous gait lengths, as shown in Figure 2e. From this third domain, one can see how the approach could extend into many robotics applications. Table 4 in Appendix D shows which tools from the coaching toolbox we use in each of the three test domains.

Results↩︎

We first show the use of UVFA to influence aspects of GT Sophy’s driving style in a pair of experiments to control tire wear and fuel usage (Figure 2a-d). By adjusting these UVFA weights, the user can trade off pace against fuel range or tire life, enabling control over the key strategic decision about when to take a pit stop. In contrast, we can create a style that encourages GT Sophy to drift (for a video, see Appendix E). In Appendix C, we provide an additional example in which a style parameter is used as a threshold to set a variable-sized penalty region around opponents which can be used to modulate GT Sophy’s apparent aggressiveness (see Figure 6, Appendix D).

For experiments in HFW, we configure the agent as a late-stage player in order to give us access to the widest variety of skills, weapons, and challenging opponents. Thus, Aloy has maximum health (700 HP), the fully unlocked skill tree, upgraded armor, and six fully upgraded weapons. Table 2 in Appendix D shows Aloy’s weapons, and more details on Aloy’s configuration and the agent’s action space are available in Appendix C.

To enable styles consistent with the combat themes of the game, we implemented rewards to encourage melee, traps, the six weapons configured in the weapon wheel, 7elemental styles, and 3part-removal styles. When none of these extra rewards are supplied, the agent has the flexibility to use any combination of these behaviors to defeat the enemies, a setting we refer to as the default style. To highlight the flexibility of the approach, we include both positive and negative rewards for the use of one of the more dominant in-game weapons (named the SkyKiller), which allows us to request a style that encourages the agent to avoid this weapon.

We ran five randomly seeded trainings to a fixed 4.5 million gradient steps and evaluated the resulting five policies across 20styles against 19enemies in three different map locations (see Figure 1). For each combination of machine-style-location-seed, we collected ten samples, resulting in data from 57,000 total battles. For each battle, we recorded whether the agent or the machine won (win rate), what types of damage the agent inflicted on the machine (damage type), and the fraction of damage done that counts towards the requested style (style score).

Figure 3a shows the types of damage inflicted when different styles (in columns) are requested. The key result is that the diagonal elements dominate, showing a strong correspondence between the requested style and the damage type it is rewarded for. We also see that the agent’s default style (column 1) favors the SkyKiller weapon, but the agent also respects the request discouraging its use in the “No SkyKiller” (column 2). The timeline in Figure 3b shows the details of an individual battle against one of the game’s iconic enemies: the Apex Thunderjaw. The first style request asks the agent to use the Ancestors Return weapon, but leaves the choice of ammo up to the agent. From the three ammo choices for that weapon, the agent chooses the acid ammo and quickly puts the machine into a corroding state. Because this weapon does a lot of damage per second, the agent continues to use it until a second style request is made. In contrast, for the fourth style, frost elemental, we see the agent chooses the Sunscourge—a relatively weak weapon but the only one with frost ammo—and then quickly switches to the Skykiller once the machine enters the brittle state to maximize the damage reward. This suggests that the agent is not just simply obeying the style requests, but contextualizing its choices.

Because elemental styles are subtle, they allow for more variation in how they are satisfied. Figure 3c shows that different training runs (seeds) lead to fairly distinct weapon choices for inflicting damage in elemental states. On the one hand, when the shocked elemental style is requested, all five seeds inflict damage with the Ancestors Return. In contrast, for the other six elemental styles, the five seeds can have quite different preferences for which weapon to use. We interpret this result as evidence that the game designers achieved their goal of balancing the weapons so as to create several near-equal options for players to choose from.

Part removal styles are scored by the number of machine components removed, rather than damage inflicted, so we separate those results into Figure 3d. When requested to, the agent removes about twice as many parts as all other styles. However, because the agent was rewarded for “removing” the parts, regardless of whether they were detached or destroyed, the agent learned to use the SkyKiller, with its explosive ammo, a crude strategy which often removes multiple parts at the same time.

We also care about how well the agent performs the base task: winning the fight. Figure 4a shows the fraction of battles won for each style against each type of machine. This graph makes clear that most of the styles are very effective against most of the machines, but a few outliers stand out. In particular, the agent survives fights against the Thunderjaw and the Slaughterspine less than half the time when asked to use melee, the SunScourge, or the IceStorm boltblaster styles. Interestingly, this graph also shows that the agent sometimes struggles against the weaker enemies when asked to remove parts. A deeper investigation into this outcome showed that, because the part-removal reward is relatively high compared to the base rewards, some training seeds result in a policy that chases these weaker machines while trying to line up a good shot targeting parts until the episode times out (see Table 3 in Appendix D for complete win-loss records). To find out whether the agent had overfit to the training scenarios, we tested it in five out-of-distribution (OOD) situations. Figure 4a shows that the agent continues to perform well when it faces machines that it has never seen before, and whose attacks and behaviors (like flying) are outside the training set.

A significant benefit of style-conditioning is that it gives the user the ability to fine-tune agent behavior after training is complete. Figure 4b shows the Pareto curves trading off the style performance versus the win rate for six styles. By decreasing the style weight, the user gives the agent more flexibility, making it more likely to win the fight. By choosing the right style weight, the user can decide where they want to sit on this curve.

Some of the styles in HFW can be combined. We present one such example in Figure 3a, column group D, which shows results from combining one weapon style with three different elemental styles. The agent is able to successfully execute the combined styles even though it has seen each of those specific requests less than once every million training scenarios.

The humanoid testbed provides a more direct example of composable styles. The agent is able to successfully compose the discrete arm poses with the continuous gait control even though the arm poses impact the humanoid’s center of mass, and therefore its balance. Moreover, the gait length is an example where the style parameter \(\boldsymbol{\theta}\) is used as a setpoint. See Appendix E for a video showing the agent walking with these styles being changed in real-time and the supplied source code to reproduce the results.

Conclusions↩︎

Coachable agents have immediate uses in gaming, robotics, and other fields where RL is used to solve real-time control problems. Style-conditioned UFVA, with the appropriate learning algorithms, scenario training, and sampling strategies, can produce agents that can succeed at complex tasks—like walking, driving, or fighting in a video game–while being able to modify their style on command. AI systems trained in this manner have broad practical applications because, rather than learning a single behavior, they encode a family of behaviors.

The technology can have an immediate impact on the $300B video game industry by improving QA processes or enhancing in-game NPCs. It can also lead to novel accessibility features to enable a broader audience to enjoy video games by letting agents take over when the game is too difficult. The advanced interactivity these agents provide can open up new gaming experiences for players and new design options for game developers.

Outside of games, as more general purpose robots make their way out of labs, it is inevitable that their human users will have varying preferences over how the machines perform their tasks. The approaches described here offer a powerful mechanism to enable users to shape a robot’s behaviors with goal modifiers. Perhaps soon we will have robots that can not only clean for us, they will be able to do so with flair.

Acknowledgements↩︎

We thank our collaborators at Guerrilla Games and Polyphony Digital who allow us to use their games for research. We also thank our colleagues in the platform team at Sony AI who build and maintain the infrastructure that supports our experimentation.

Author contributions statement↩︎

R.C. and H.v.S. managed the project and led the research and development efforts across domains. N.B., J.D., D.M., D.O., and F.R. supervised parts of the research and development efforts for the HFW playstyles. I.D., S.G., R.K., E.A., and A.L. participated in the research and development of the HFW agent playstyles. A.S., K.N.F., J.G., V.R.K., J.A.M., and K.W. participated in the research and development of the HFW agent. T.J.W. led the research and development in DMC Humanoid. N.D.B, D.H., and S.M. participated in the research and development of the DMC Humanoid agent. S.J. curated the open source release of the algorithm and the DMC Humanoid experiments. K.S. led the research and development for GT Sophy. A.D., Y.D., K.K., P.M., C.S., F.F., T.S. participated in the research and development of GT Sophy. F.D., G.Z.H., S.B., M.G., M.S., M.D.T., and J.B.T. provided engineering support and built the libraries and APIs that connected to the different videogames. P.S. and M.S. provided executive support and technical and research advice. P.W. oversaw the project, provided executive support, resources, and technical advice and managed stakeholders.

Appendices Overview↩︎

• Appendix A: Style-Conditioned UVFA

• Appendix B: Training Algorithms

• Appendix C: Domain Details

• Appendix D: Extended Data

  • Table 1 - Horizon Forbidden West machine types reference.

  • Table 2 - configured weapons reference.

  • Table 3 - win-loss records.

  • Table 4 - per domain training configurations.

  • Figure 5 - Damage by style, numbers.

  • Figure 6 - aggressiveness control in GT.

  • Figure 7 - HFW ablations.

  • Table 5 - description of HFW rewards.

  • Table 6 - base task rewards for HFW.

  • Table 5 - style rewards for HFW.

  • Table 8 - HFW policy head outputs.

  • Table  [tbl:tab:feat95rankings95shap] and  [tbl:tab:feat95rankings95saliency] - Feature importance analysis.

• Appendix E: Videos and source code

Appendix A: Style-Conditioned UVFA↩︎

We train our policies using universal value function approximation (UVFA) [28] within an actor-critic framework. Classical UVFA conditions the policy, value, and reward functions on both the state and a goal \(g\), producing goal-dependent behavior. Our style-conditioned UVFA instead fixes the core task (e.g., defeat the enemy) and exposes runtime style parameters \(\Theta\) that specify how the task is pursued (e.g., weapon preference). Both policy and critic are conditioned on \(\Theta\), letting a single policy express multiple behavioral modes without retraining. Below, we formalize the style-parameter interface \(\Theta\), its encoding, and the parameter-sampling strategies used during training.

0.0.0.1 Formalization.

We encode the active style via \(f(\theta_t)\) and form the augmented state \(\tilde{s}_t\mathrel{\vcenter{:}}=(s_t,f(\theta_t))\). The policy and critic then condition on \(\tilde{s}_t\) as \(\pi(a_t\mid\tilde{s}_t)\) and \(Q(\tilde{s}_t,a_t)\). With discount \(\gamma\in[0,1)\) and horizon \(T\), the per-step reward is \[r_t=r_{task}(s_t,a_t)+ r_{style}(s_t,a_t;\theta_t),\] where \(r^{task}\) is the task reward (fixed across time/episodes), and \(r^{style}\) is the style reward that depend on \(\theta_t\in\Theta\). Typical style parameters include weights, thresholds (for gating/ramping), and targets (setpoints) that penalize deviation from desired ranges.

In the general case, these style parameters may evolve within an episode as a sequence \(\theta_{0:T-1}\) with \[p_{\Theta}(\theta_{0:T-1})=p_{\mathrm{init}}(\theta_0)\,\prod_{t=0}^{T-2}\kappa(\theta_{t+1}\mid \mathcal{C}_t),\] where \(p_{\mathrm{init}}\) is the initial distribution and \(\kappa\) is a (generally unknown) transition rule that may depend on context \(\mathcal{C}_t\) (e.g., user input, schedules, or other signals). This style-parameter dynamics induces the augmented transition \(P(s_{t+1}\mid s_t,a_t)\kappa(\theta_{t+1}\mid\mathcal{C}_t)\). However, since \(\kappa\) is unknown at training time, we fix \(\theta_t\) within each training episode, thus avoiding credit assignment through \(\kappa\). In our studies, this approach empirically accelerated learning while preserving test-time controllability via \(\theta\). Given \(\theta\) sampled at episode start, the return and objective are \[G_{\theta}\;=\;\sum_{t=0}^{T-1}\gamma^{~t}\!\Big[r_{task}(s_t,a_t)+ r_{style}(s_t,a_t;\theta_t)\Big],\qquad J(\phi)\;=\;\mathbb{E}_{\substack{s_0\sim \rho,\, \theta\sim p_{\Theta} \\ \tau \sim \pi_\phi}}\!\big[\,G_{\theta}\,\big],\] where \(\rho\) is the initial-state distribution, \(p_{\Theta}\) is the episode-level sampling distribution over styles, and \(\tau\) is the trajectory induced by \(\pi_{\phi}\) under \(\theta\).

0.0.0.2 Implications.

Conditioning the policy on style parameters \(\Theta\) while keeping the core objective \(r_{task}\) fixed allows one policy to express many behaviors without retraining. This methodology replaces multiple specialized policies, lowers training cost, and enables systematic performance-vs-style trade-off sweeps over \(\theta\in\Theta\). Encoding styles on distinct parameters enables a compositional space of preferences: they can be turned on independently, combined, or varied continuously. When a request conflicts with the core task, the resulting behavior reflects the relative weighting of the task and style terms: the agent trades off task return against style adherence according to the configured reward scales. Styles encode designer preferences, so their rewards must be carefully designed. Agents may learn well for one \(\theta\) yet generalize poorly to \(\theta'\). Because we condition the policy and value function on \(\theta\in\Theta\), stable learning benefits from a well-behaved dependence on such parameters: nearby style settings should induce comparable return targets, which we encourage by bounding/normalizing style-reward magnitudes. Accordingly, data sampling should cover the style settings expected at evaluation time or deployment. At execution, \(\Theta\) enables real-time behavioral modulation, reliable only when style axes are interpretable and parameters stay within trained regions. Within these limits, the agent responds smoothly to changes in \(\theta\). Evaluation must therefore cover representative regions of \(\Theta\) to verify coherent, responsive stylistic control.

0.0.0.3 Parameter Sampling.

As defined above, the reward decomposes into a task term and a style term. Previous work explored related ideas in small, turn-based games via configurable reward shaping [29], [30], typically using linear weight vectors \(w\). Building on that formulation, we apply style-conditioned rewards in complex, high-dimensional, real-time control domains by decomposing the style reward into \(m\) distinct style terms: \[r_{style}(s_t,a_t;\theta) \;=\; \sum_{i=1}^m w_{i}\; g_i(s_t,a_t;\xi_{i}),\qquad \theta=(w_0, \dots, w_m, \xi_{0}, \dots, \xi_{m})\] where \(w_{i}\in\mathbb{R}\) is a scalar weight, \(g_i\) a style reward, and \(\xi_{i}\) a (possibly zero-dimensional) style parameter vector. Both \(w_{1:m}\) and \(\xi_{1:m}\) are part of the observable style vector \(\theta\) (hence in \(f(\theta)\)). By convention, \(w_i>0\) encourages style \(i\), \(w_i<0\) discourages it, and \(w_i=0\) expresses no preference.

During training we sample each \(w_i\) from the parameter’s domain. In some environments, it is helpful to extend the training domain beyond the range used at evaluation time. For example, in GT we expand \([0,k]\) ranges to train on \([-i,0,j]\), where \(i\geq 0\) and \(j\geq k\), to encourage the agent to learn a spectrum of behaviors, rather than collapsing to an “average” policy that partially satisfies both 0 and positive values. Using 0 to indicate “not encouraged” (as opposed to negative values meaning “discouraged”) also enables more coherent behavior when multiple, overlapping styles are combined.

For instance, encouraging part targeting without pushing the agent away from general weapon use is straightforward if \(w_{\text{part}}>0\) while \(w_{\text{weapon}}=0\), whereas forcing only \(\{\text{positive},\text{negative}\}\) on both axes induces undesirable trade-offs.

Domains may expose discrete or continuous style knobs: categorical choices (e.g., a selector) are naturally discrete, while weights are continuous. In practice, we found that continuous weights appear to be as easy to learn as discretized sets (e.g., \(\{a,0,b\}\)) and preserve post-training reward tuning without retraining.

Appendix B: Training Algorithms↩︎

Because our experimental domains differ in complexity, dynamics, and action spaces, we use different RL algorithms per environment (Table 4, Appendix D) while keeping a common off-policy actor-critic backbone with replay. For all domains, we use many asynchronous rollout workers that pull the latest policy networks and sample UVFA weights at episode start (fixed within an episode), collect data and stream transitions to a shared replay buffer. A single learning procedure updates the policy/critic, details of which are discussed below. Episodes end on success/failure (e.g., all opponents defeated) or on a fixed time limit. Style conditioning follows the UVFA formulation above. Scenario training[32]–[34] is used in all domains to expose the agent to diverse initializations and task variants. Stratified sampling, in the form of multi-tables[3] or event tables[36] enable replay by scenario/style to balance mini-batches, ensuring that multiple situations are represented in each network update. An adaptation of task impersonation [35] augments each sampled mini-batch by relabeling transitions under additional UVFA parameter vectors drawn from the same distribution, creating \((n{+}1)\) labeled versions of each sampled transition at update time without writing duplicates to replay.

Next, we summarize the different training algorithms that we use: SAC for the Humanoid domain, QR-SAC for Gran Turismo and (Cat-)RAC for Horizon, followed by a description of Task Impersonation and Stratefied Sampling. Unless noted otherwise, we optimize with Adam (TensorFlow defaults; learning rates and other hyperparameters are reported with the experiments) and use target networks for stability.

SAC↩︎

For the Humanoid domain, we use Soft Actor–Critic (SAC) [37], that maximizes an entropy-regularized (“soft”) return by learning a stochastic policy and twin critics. The critic targets use TD(0) bootstrapping with a minimum over two Q-networks to reduce overestimation bias, and slowly updated target networks (Polyak averaging). The policy update is based on the critic’s gradient; for continuous actions we use the reparameterization trick for \(\tanh\)-Gaussian heads so gradients flow from \(Q\) through the sampled action to the policy parameters. Using this approach, a sample, \(X\), from a \(\mathcal{N}(\mu, \sigma^2)\), is computed as \(X = \mu+ \sigma\cdot Z\), where \(Z \sim \mathcal{N}(0, 1)\). In our setup, we follow the canonical SAC formulation and perform simultaneous actor/critic updates each step using batches from the replay buffer, while the entropy temperature is fixed.

QR-SAC↩︎

For Gran Turismo 7 we adopt the quantile-regression variant of SAC (QR-SAC) from Wurman et al.. [3], replacing scalar Q with a distributional critic trained by quantile regression while keeping the SAC structure (stochastic policy, replay, target networks). In this distributional setting we update the policy before the critic each step, which we found to stabilize the moving value distribution.

RAC and Cat-RAC↩︎

Horizon’s PlayStation 5 control mixes discrete button combinations with continuous sticks/triggers. To support mixed action spaces of this kind we generalize SAC to a regularized actor–critic (RAC) objective that replaces the entropy term with a generic regularizer \(\mathcal{R}(a\mid s,f(\theta))\). This modification lets us apply different regularizers per action subspace and sum them in a single objective while retaining the SAC training loop. Specifically, we apply exact categorical entropy to the discrete policy heads and an n-sample empirical mean of the negative log density from the \(\tanh\)-Gaussian policy heads. In our experiments, we estimate the negative entropy by averaging \(n=10^3\) samples per component, instead of the conventional single sample. This approach is substantially more accurate than the single-sample surrogate commonly used in SAC, without a meaningful slowdown in practice. Using exact categorical entropy is necessary because gradients cannot be propagated through categorical samples.

As in SAC, we use the reparameterization trick for the Gaussian policy heads. A reparameterization trick also exists for a (\(d\)-dimensional) categorical distribution, \(\pi \in \triangle^d\): the Gumbel-max trick[38] enables the computation of a sample \(Y \sim \pi\) as \(Y = \mathop{\mathrm{arg\,max}}_{i} \log \pi_i + Z_i\), where \(Z = \set{Z_i \sim \mathrm{Gumbel}(0, 1)}_{i = 1}^d\). However, the presence of the argmax operation prevents this reparameterization from being differentiable. For this reason, we employ a Gumbel–Softmax continuous relaxation [39], [40] where we query the critic with relaxed actions \(\exp\!\big(\log \pi + Z\big)\,\big/\,\sum_{i=1}^d \exp\!\big(\log \pi_i + Z_i\big)\) to compute critic gradients w.r.t.categorical policy heads during policy updates, while the environment executes “hard” action samples¹. A naive “softmax over logits” can be numerically brittle in our setup (temperature sensitivity, overflow errors). We therefore normalize the positive part of the network outputs \(x\in\mathbb{R}^d\): \[\pi_i \;=\; \frac{\max\{0,x_i\}+\varepsilon}{\sum_j \big(\max\{0,x_j\}+\varepsilon\big)}, \qquad \varepsilon=10^{-8}/d,\] which stabilizes entropy and avoids pathological scaling.

To backpropagate through categorical heads, we use a regression regret-matching [42] update. Let \(g=\nabla_{x}\mathcal{L}\in\mathbb{R}^d\) be the \(d\)-dimensional gradient of the policy loss w.r.t.pre-normalized outputs, and define the instantaneous regret \[\rho_i \;=\; \sum_{j} g_j\,\pi_j \;-\; g_i,\qquad i=1,\ldots,d.\] We pass the modified gradient \(g'\) to the optimizer, composed as \[g'_i \;=\; \begin{cases} 0, & x_i<0 \;\text{and}\;\rho_i<0,\\ -\rho_i, & \text{otherwise.} \end{cases}\] All other backpropagation is unchanged. Intuitively, this modified gradient shifts probability toward actions with positive regret while avoiding updates that merely push already-negative \(x_i\) further negative, as only positive parts affect \(\pi\). The modified gradient is very similar to the pseudoregret change in regret matching\(^+\) [43], [44] and, in our setting, yields stable, scale-robust gradients for categorical components without introducing temperature hyperparameters. Regret matching [45]–[51] provides theoretical guarantees about the optimality of its decisions that degrade gracefully as the capacity of its function approximator decreases [52]–[54].

For representing values, we use a categorical distributional encoding of the critic on top of RAC, yielding Categorical RAC (Cat-RAC), inspired by Farebrother et al..’s use of categorical value functions[55]. The critic predicts a discrete value distribution; a fixed decoder (expectation under the support) produces scalars where needed. We train the critic with cross-entropy against HL-Gauss [56] encodings of TD(0) targets (std/width ratio \(0.75\), 255 bins, value range \((-600,1000)\)). As with QR-SAC, we use a policy-first, then-critic update order to reduce non-stationarity in distributional targets.

Task Impersonation↩︎

Task impersonation [35] is a relabeling technique that improves data efficiency by reusing the same transitions under alternative “task” descriptors. We adopt this approach in HFW, and in our setting there are no discrete tasks; instead, each UVFA parameter vector \(\theta\) specifies a distinct style-reward configuration. We therefore treat sampled UVFA settings as impersonated tasks: a single transition can be evaluated under multiple \(\theta\)’s without altering the underlying dynamics.

As discussed above, we sample style parameters from a certain distribution. During updates, however, we increase coverage over \(\Theta\) by drawing additional parameter vectors i.i.d.from the same distribution (i.e., \(\hat{\theta}^{(j)}\!\sim\!p_{\mathrm{\theta}}\)). In effect, task impersonation provides a more accurate Monte Carlo approximation of our objective’s expectation over \(\theta\), but without modifying the replay buffer: every transition sampled from the replay buffer is paired, on each step, with several freshly sampled UVFA configurations, expanding the style space coverage.

Specifically, each update proceeds by sampling a mini-batch of transitions \((s_t,a_t,s_{t+1})\) from the replay buffer, retaining the label defined by the rollout parameters \(\theta^{(0)}\), and generating \(n\) auxiliary labels by recomputing the style rewards under \(\hat{\theta}^{(1)},\ldots,\hat{\theta}^{(n)}\). When forming the new update targets, we condition the networks on the corresponding encodings \(f(\theta)\) . New relabeled data is created at the start of each step and discarded at the end, and is never written to the replay buffer. Because each update processes \((n{+}1)\) labeled versions of every transition, throughput decreases with \(n\); empirically we found \(n=4\) to be a good trade-off in wall-clock time.

A design choice is worth noting. We neither filter nor re-balance impersonated labels: styles act through rewards and conditioning features rather than the environment dynamics, so most transitions remain informative—positively or negatively—for most UVFA configurations. Averaging contributions uniformly across the \((n{+}1)\) labels aligned well with the objective function’s expectation over \(\theta\) and worked reliably in our experiments.

Stratified Sampling↩︎

In GT and HFW, we additionally use stratified sampling via multi-tables[3] and event tables [36], respectively. Multi-tables partition the replay buffer into separate tables that capture different portions of trajectories, grouped by the features and scenarios they represent, enabling balanced learning across the situations the agent encounters. Event tables organize replay data by style configuration and promote learning across style settings. For each style dimension \(i \in {1,\dots,m}\), trajectories generated under a style vector \(\theta=(w_{1:m},\xi_{1:m})\) are inserted into the event table associated with that style configuration. Transitions are stored only when a style-specific event is triggered (e.g., damage attributable to that style or part removal), with a maximum event horizon \(H_{\max}\) representing the number of steps stored per event.

For both multi- and event tables, each table has a fixed capacity and is updated in a first-in, first-out manner. During training, mini-batches are sampled in a stratified manner across these tables, ensuring balanced coverage of scenarios or contrasting style settings, and preventing over-representation of dominant regimes. Unlike task impersonation, which relabels transitions with alternative UVFA parameters to expand style coverage, these approaches operate at the data organization level, preserving the original labels while promoting balanced sampling across scenarios and styles.

Appendix C: Domain Details↩︎

As described in the main paper, we apply our approach to three domains that reflect our goals and history: Horizon Forbidden West (HFW), Gran Turismo 7 (GT), and DMC Humanoid from the DeepMind Control Suite.

Horizon Forbidden West↩︎

Large environments, complex interactions, and highly non-linear dynamics- together with long-horizon objectives and sparse/delayed rewards - make HFW a demanding RL testbed. A practical complication in such games is a cascade, or “butterfly” effect: small timing/state perturbations can cause two nearby trajectories to diverge within tens of frames, yielding different game-event graphs (e.g., failing to evade an attack, inaccurate weapon selection) and locally discontinuous returns. We condition styles episodically, report aggregates over multiple runs and scenarios, and train with Cat-RAC’s distributional critic and task impersonation – supporting multi-modal, high-variance targets without suppressing the game’s natural dynamics. As mentioned in the previous section, we also validate these design choices through ablations and summarize the results in Figure  7 in Appendix D.

We use two execution modes of the game: a headless build, which runs the full simulation without rendering (no video output), and a graphical build, which renders frames and supports video capture. All data reported in this paper is generated by agents trained and evaluated using the headless build. The graphical build is used to visually inspect and interpret the learned behaviors, and all supplemental videos are recorded from this rendered version. Consequently, due to small dynamics discrepancies between the two builds, these videos are slightly out of distribution relative to the headless training/evaluation setting.

Actions.

We interface with a PlayStation 5 gamepad comprising 14 binary buttons, two 2D analog sticks, and two analog triggers. To reflect human ergonomics (thumbs on sticks, D-pad, face buttons) and avoid impractical combinations, we impose virtual hand constraints that (i) prevent simultaneous presses with no in-game effect and (ii) mirror the trade-offs of a casual player. Concretely, at most one face button and one D-pad direction may be pressed at a time; L1 and L2 are mutually exclusive, as are R1 and R2; when the right stick is active, face buttons are disabled for that step; symmetrically, when the left stick is active, L3 may be pressed but D-pad inputs are disabled. We additionally exclude the triangle button and R3, treat the triggers as binary inputs (a press corresponds to a full pull), and expose only valid trigger–button combinations (booleans) as the support of categorical heads. Continuous stick values are outputted in \([-1,1]^2\). Even under these constraints, the action space remains large: 315 discrete combinations plus four continuous dimensions (the two sticks). The policy acts every six frames (Horizon runs near 30 fps in our setup), so inputs are held for a 6-frame window (\(\approx\)​5 Hz control).

Features.

The policy and value network inputs are the current observation and a representation of the style-request. Observations are based on data coming from the game engine (not pixels) and include features of the current step plus short histories of actions and features from previous steps. Because the raw game state is immense and often irrelevant (e.g., mesh vertices, textures), we extract a focused subset. However, some entities (e.g., transient projectiles) lack a consistent and concise representation exposed from the game engine, so a fair amount of partial observability remains.

Features cover the player, enemies, and environment. Player features include health, stamina, inventory counts, and weapon loadout. Enemy features include position/orientation, health, active attack ID (if available), and body-part descriptors (types, per-part health, current part positions)—crucial in HFW because parts affect combat mechanics (e.g., enabling abilities and attacks, exposing weak points and elemental vulnerabilities). All network inputs are scaled to lie roughly in \([-1,1]\). To aid generalization, we use agent-centric coordinates: each 3D position is expressed relative to the player and aligned to camera axes. The resulting relative 3D position vector \(\langle x,y,z\rangle\) is encoded as \(\{x/d, y/d, z/d, \sin(d), \sin(0.1 \, d), \sin(0.01 \cdot d)\}\), where \(d=|\langle x, y, z \rangle|\), describing direction (unit vector) plus a multi-scale radial encoding of the distance. The style request is encoded as \(f(\theta_t)\) by normalizing each parameter \(\theta_{i,t}\) to \(\bar{\theta}_{i,t} \mathrel{\vcenter{:}}= \theta_{i,t}/\max\{|a_i|,|b_i|\}\) over its domain \([a_i,b_i]\), and then concatenating \([\bar{\theta}_{1,t},\ldots,\bar{\theta}_{m,t}]\). The augmented state \(\tilde{s}_t=(s_t,f(\theta_t))\) exposes these normalized style requests to policy and critic.

Rewards.

A brief text description of every reward term is provided in Table 5 in Appendix D. Task rewards, which are constant across styles, define the core objective. Table 6 in Appendix D lists each task reward component with its unscaled range and weight \(w\). Style rewards are modulated by style parameters (as above) and are described in Table 7 in Appendix D.

Style Parameters.

In HFW, we have \(m=18\) style weights and no \(\xi_i\) parameters, supporting a total of 20 style behaviors (including the ‘default’ and no-Skykiller styles). Each weight \(w_i\) lies in a continuous range, with \(a_i < 0 < b_i\). Weights are sampled episodically from a mixture: (i) with probability \(0.03\), set \(w_i=0\) for all \(i\) (creating the default style); (ii) with probability \(0.10\), sample each \(w_i \sim \mathcal{U}([a_i,b_i])\) independently; (iii) with probability \(0.87\), draw \(k \sim \mathcal{U}(\{1,\dots,m\})\), sample \(w_k \sim \mathcal{U}([a_k,b_k])\), and set \(w_i=0\) for all \(i\neq k\) (continuous one-hot). In Figure  7 (c) in Appendix D, we compare this style-sampling scheme against variants of component (iii) of the mixture.

Scenarios.

We train for 1-vs-1 combat across multiple enemy types and locations. To simplify the effect of terrain and avoid, we use three mostly flat sites—Ice-lake, Shining-wastes (desert land), and the in-game combat Arena, which is walled. We select 19 reliably spawning enemies from the 43 machine types in the game (Table 1, Appendix D). Aerial/aquatic or unstable enemies are excluded. The agent starts each fight at level 50 with a full skill tree, full stamina, 14 healing berries (+105 HP each), and a loadout comprising 10 traps of each type (acid, purgewater, shock, blast) plus 6 weapons spanning major classes and elemental ammo (Table 2, Appendix D). Valor surge is disabled and no coils or weaves are applied. For each training task we uniformly sample a location and enemy. To ensure sufficient variation, the agent and enemy spawn with random orientations within 25 m of a manually annotated center. Episodes terminate on enemy defeat, agent defeat, stepping outside the Dynamic Anchor Bound (DAB), or reaching 4500 steps. The DAB captures the spatial relationship between the agent and nearby hostile entities by defining a bounded region whose center is either the agent’s current position or a recent past position: when the agent moves toward the nearest hostile entity, DAB recenters on the agent’s new position; when the agent moves away, the center remains fixed. A style request is sampled as described earlier and held fixed for the episode.

Models.

Policy and critic are 4-layer MLPs with widths \((4096,\,2048,\,1024,\,512)\): each block is Linear \(\rightarrow\) LayerNorm \(\rightarrow\) ReLU with a padded residual connection. We alternate the padding location to avoid identity paths, i.e. we use pad-at-end for layers of size \(4096\) and \(1024\), and pad-at-beginning for \(2048\) and \(512\). Initialization uses Glorot–Uniform weights and zero biases; networks are trained with Cat-RAC.

Policy. Actions for analog sticks use tanh-Gaussian distributions, with mean \(\mu\in[-1,1]\) (after tanh) and standard deviations \(\sigma\) obtained by squashing the network output to \([-2,2]\) with \(2\tanh(\cdot)\) and then exponentiating. The discrete actions necessary to enforce the virtual hand constraints use Categorical distributions, learned via the Gumbel-max regret matching trick mentioned in the Training Algorithm appendix. To obtain the entropy regularization term that’s necessary for Cat-RAC, empirical estimates of Squashed Gaussian actions’ entropy is summed with the exact entropy values of the Categorical distribution actions; 1000 samples are used to obtain the empirical estimates. Table 8 in Appendix D reports the detailed policy head layout.

Critic. Categorical distributional \(Q\) with \(255\) bins on \((-600,1000)\), trained by cross-entropy against HL-Gauss TD(0) targets (\(0.75\) std/width ratio).

Model explanations. Feature-attribution analyses provide evidence that the learned policy is sensitive to the UVFA parameters. The Saliency[57] (gradient-based) method consistently ranks UVFA style inputs among the most influential features (Table [tbl:tab:feat95rankings95saliency], Appendix D), while SHAP[58] primarily attributes action selection to observation features capturing player and enemy state (Table [tbl:tab:feat95rankings95shap]). Although attribution methods do not establish causality, the consistent separation between sensitivity to conditioning variables and explanatory contributions of state features is consistent with the agent behavior modulation through UVFA parameters.

0.0.0.4 Computing resources.

The trainer runs as a single process with 10.5 vCPU, 55 GiB RAM, and one NVIDIA H100 GPU, writing checkpoints every \(\sim\)​10 minutes. We train our Horizon runs for 1500 epochs, corresponding to approximately 3 days (average training time is 470 epochs/day). Unless otherwise stated, all reported results use 100 rollout workers, although most exploratory experiments were run with 50 rollout workers and showed only marginal differences.

Gran Turismo↩︎

We study style control for GT Sophy in a variety of ways: drifting with 360-degree loops, strategic control with tire and fuel saving, and modulating aggressiveness by controlling the longitudinal and lateral distance to opponent cars. In each case, the designer is given control over the style to show a wide range of capabilities. Unless noted, details of features, actions, training approach, and rewards remain the same as in Wurman et al.. [3].

The drifting style was showcased on the Trial Mountain track with a Lamborghini Countach. The tire and fuel saving style control was trained on the Autodrome Lago Maggiore track with the Red Bull X2019 Competition car. Aggressiveness control was studied on the 24 Heures du Mans track, while learning to drive over 500 cars and 9 tire compounds.

Style reward and parameters.

The drifting style vector is composed of \(m=4\) style weights \(w_i\) (no \(\xi_i\)), controlling the lateral slip angle, longitudinal slip ratio for the front and rear car tires. Each \(w_i\) is sampled independently from a three-way mixture: \(w_i\sim0.5\delta_0+0.25\mathcal{U}([-0.4, 0.0])+0.25\mathcal{U}([0.0, 5.0])\), where \(\delta_0\) is a Dirac point mass at \(0\).

For tire and fuel saving, we use \(m=4\) style weights \(w_i\) (no \(\xi_i\)), to control overtaking, defending, tire saving, and fuel saving. We sample \(w_{\text{overtake}},w_{\text{defend}}\sim\mathcal{U}([0, 2])\), \(w_{\text{tire}}\sim 0.10\delta_{0} + 0.45\,\mathcal{U}([-60,0]) + 0.45\mathcal{U}([0,12000])\), and \(w_{\text{fuel}}\sim0.10\delta_{0} + 0.45\mathcal{U}([-3,0]) + 0.45\mathcal{U}([0,105])\). To make these styles learnable, the observation includes tire temperature/grip and powertrain signals (RPM, torque, gear).

Finally, for aggressiveness modulation, the style vector is \((\xi_{\mathrm{lon}},\xi_{\mathrm{lat}})\) (meters), i.e. longitudinal \(\xi_{\mathrm{lon}} \sim \mathcal{U}([0,2])\) and lateral \(\xi_{\mathrm{lat}} \sim \mathcal{U}([0,1])\) car-body distance parameters and no weight. The sampled parameters are applied to all cars for an episode and are used to compute a dense penalty as our style reward: \(P=(d_{\mathrm{lon}}-\xi_{\mathrm{lon}})_{-}+(d_{\mathrm{lat}}-\xi_{\mathrm{lat}})_{-}\) with \((x)_{-}\mathrel{\vcenter{:}}=\min(0,x)\), summed over nearby opponents, where \(d_{\mathrm{lon}},d_{\mathrm{lat}}\) are the shortest longitudinal/lateral body distances (with cars approximated by 4-vertex polygons). Here smaller distance margins encourage more assertive passing maneuvers while larger margins produce cautious driving. Results are shown in Figure 6 in Appendix D.

Scenarios.

To train the drifting style, we use similar scenarios to the ones in Wurman et al.. [3]. Additionally, at the start of every episode we sample style weights, and cars are launched at random positions on track with random speeds and orientations. To learn tire- and fuel-saving behavior, the agent must experience trajectories spanning a wide range of tire-wear levels—across 3 racing tire compounds—as well as different fuel loads. In our training scenarios, the initial tire wear conditions are sampled as \(X\sim\mathcal{U}([0, 0.75])\), and for each tire we then draw the wear from \(\sim N(X, 0.05)\). The initial fuel level is instead sampled from \(\mathcal{U}([10.0, 100.0])\).

Training the aggressiveness style uses cars drawn from a pool of \(530\) vehicles, with adaptive sampling that up-weights cars prone to spinning out or losing control. Tires are sampled uniformly from \(9\) compounds (comfort/sport/race \(\times\) hard/medium/soft). During evaluation we focus on the GT3-class of cars and report averages over multiple races for each of the \(50\) Gr.3 cars. Episodes use rolling starts with \(5\) cars of the same model, over \(4\) laps, using a combination of fixed and sampled distance parameters to analyze the learned behaviors. The evaluations match the approach taken in Werner et al..[59] with agents starting in leading and pursuing positions. For each race, we log the win-rate both as a leader and pursuer, overtakes, and collision events.

DMC Humanoid↩︎

We apply style-conditioned UVFA in the Humanoid task from the DeepMind Control Suite (DMC) [60], [61], where the agent controls a simulated humanoid on an infinite plane. The core task is forward walking; styles target combinations of arm poses and gait.

Actions.

The walk task requires the humanoid to stand up and then move at a minimum speed of 1 over episodes of 1000 time steps. The 21-dimensional continuous action space \(A \coloneq [-1, 1]^{21}\) specifies control inputs for each joint actuator.

Features.

Observations are proprioceptive and include angle and velocity of the joints, as well as velocity of the center of mass. Additionally, we include features to support the gait style: one-hot of the foot that should strike next; per-step min/max knee angles (both legs); normalized pelvis orientation; and relative foot positions \((x,y)\) recorded at the start of the current step and stride.

Rewards:

The total reward decomposes into a core movement term \(r_{task} = r_{\mathrm{move}}\) and two style terms (\(r^{\sigma_i}\)): \[r = r_{\mathrm{move}} + \lambda_{\mathrm{pose}}\, r_{\mathrm{pose}} + \lambda_{\mathrm{gait}}\, r_{\mathrm{gait}},\] with \(\lambda\) fixed multipliers (not style parameters \(\theta\)). At every step, \(r_{\mathrm{move}}\) incentivizes forward motion only, while penalizing large torques and decaying as head height drops. Both style terms use DMC tolerance() with long_tail decay (dense gradients, bounded in \([0,1]\) before shifting/scaling). At every step, the arm pose reward (\(r_{\mathrm{pose}}\)) keeps six arm joints (ELBOW, SHOULDER1, SHOULDER2 per arm) within pose-specific angles for \(\xi_{\mathrm{pose}}\)—with margin \(\approx 30^\circ\) around bounds. The gait reward (\(r_{\mathrm{gait}}\)) instead targets a step length \(\xi_{\mathrm{gait}} = \ell\) on alternating steps using margin \(m=0.05\). Here we use additional shaping terms: (i) stride contribution bounds \([1/3,\,2/3]\) per foot to avoid “vibrating” gaits; (ii) knee range-of-movement limits (min/max knee angles within \([-145^\circ,0^\circ]\)); (iii) timeout penalty \(-v_m\) if no alternation for \(t_{\text{step}}>t_{\text{step\_max}}\).

Style Parameters.

Humanoid conditions on a style vector \(\theta=(\xi_{\mathrm{pose}},\xi_{\mathrm{gait}})\). These style-specific parameters are independently sampled at each episode and encode the style geometry and constraints: \(\xi_{\mathrm{pose}}=\text{onehot(pose\_id)}\), where \(\text{pose\_id}\in\{\text{arms-in-front},\text{arms-up},\text{T-pose}\}\) corresponds to specific arm joint-angle bounds with \(\sim30^\circ\) margins; \(\xi_{\mathrm{gait}}=\ell\), where \(\ell\) is the target step length. Therefore, arm poses and gait constraints are trained jointly.

Scenarios.

We modify initialization/termination to speed early learning on walk: the humanoid starts upright (non-arm joints at \(0^\circ\), arm joints randomized). An episode terminates when head height \(<1.1\) (i.e., a fall) or at \(t=1000\). A style request is sampled as described earlier.

Models.

Policy and critic are 3\(\times\)​512 residual MLPs with LayerNorm, trained using SAC with Adam (lr \(10^{-4}\)), discount \(0.99\), and fixed entropy weight \(\alpha=0.01\). Policy: \(\tanh\)-Gaussian over the 21-D torque actions. Critic: twin scalar \(Q\)-nets with Polyak-averaged targets.

Computing resources.

Humanoid experiments ran on a single NVIDIA H100 GPU with 10 CPU rollout workers and experience replay (buffer size \(7.5\times10^{5}\) transitions). Training was throttled until 15000 steps were collected and further constrained to 60 env-steps/s.

Correspondence and requests for materials should be addressed to P.W. or M.S.

Appendix D: Extended Data↩︎

4pt

3pt

6pt

4pt

Appendix E: Videos and source code↩︎

In a Google Drive folder (link), the following can be found:

• Source code for the complete DMC humanoid style experiment.

• Various videos:

  • Overview video of the project

  • Video of GT Sophy drifting

  • Video of humanoid walking and changing gaits and arm poses

  • Videos of default style versus every enemy

  • Videos of every style versus selected enemies

  • OOD videos

References↩︎

---

1. See Huijben et al.. [41] for a survey of Gumbel-max/Gumbel-Softmax methods.↩︎