1 Introduction↩︎

The widespread adoption of machine learning in sports analytics, driven by the increasing availability of event and tracking data, has transformed tactical analysis and player evaluation [1]. In association football, modern datasets have expanded the analytical lens from traditional video analysis to encompass machine learning-based, multi-agent evaluation methods [2], [3]. Foundational models such as Expected Goals [4], [5], Pitch Control [6], [7] and Expected Possession Value (EPV) [8]–[12], are immensely valuable for quantifying spatial ownership and the value of dynamic game states.

While highly effective at characterising and diagnosing historical player and coach decisions, their applicability does not extend to answering counterfactual queries about what would have happened had a different decision been taken. Recent works have therefore adopted Markov Decision Process (MDP) formulations of football to explicitly reason about the value of alternative player actions or tactical decisions [13]–[15]. However, these approaches remain primarily tied to the historical state-action distribution. While they support counterfactual evaluation, they do not enable reward-driven discovery of novel configurations that may never have appeared in the data. Additionally, these works study open play, which is subject to large variations in tactics and player actions.

In contrast to open play, corner kicks provide an ideal environment for automated tactical exploration. Unlike the fluid phase shifts of open play, a corner kick is a highly structured, static restart with definitive start states. Furthermore, corners are heavily scripted, rely on repeatable tactical schemes, and account for a significant proportion of total goals (approximately 16% in the 2025/26 English Premier League season). The current state-of-the-art in automated set-piece tactic generation, TacticAI [16], leverages geometric deep learning to produce realistic, expert-preferred spatial adjustments. Yet, a critical gap remains: such architectures optimise for reconstruction loss, conditioning outputs to imitate historically observed player trajectories. While this imitation ensures generative realism, it inherently restricts the discovery of novel, physically plausible tactical innovations to the boundaries of the historical training distribution.

In this work, we shift the objective of automated set-piece design from the imitation of historical routines to true reward-driven optimisation. This captures the dilemma faced by coaches between exploiting reliable, well-rehearsed routines versus exploring novel variations that opposition defences have yet to encounter, naturally paralleling one of the main challenges in Reinforcement Learning (RL) [17]. Our core contributions are:

1. Problem Formulation: We formalise corner kick optimisation as an MDP. A central decision-maker is tasked with adjusting attacking player positions and velocities based on the starting state when the corner is taken. To quantify success and provide rewards, we propose an objective function (Expected First Contact Shot probability, \(xFCS\)), based on a predictive Graph Neural Network (GNN) model trained on Premier League data.

2. Computational Methodology: We contribute a Graph Reinforcement Learning (Graph RL) [18] method for learning a general policy to adjust attacking player positions in arbitrary corner states. Concretely, we integrate GNN embeddings with deep RL, evaluating both SAC [19] and PPO [20].

3. Empirical Evaluation: We present an evaluation on a dataset of over 3,000 Premier League corners. Results show that our method yields significantly higher-reward tactical variations compared to traditional optimisation techniques (Random Search and Simulated Annealing [21]) with matched inference computational budgets. Furthermore, our approach maintains competitive performance even when the metaheuristics are granted vastly disproportionate search budgets equal to the RL agent’s upfront training cost.

2 Methods↩︎

2.1 Problem Formulation↩︎

We formulate the optimisation of corner kick routines as a Markov Decision Process comprising the tuple \((\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma)\). This framing allows an RL agent to discover configurations that directly maximise expected tactical yield.

To provide an intuitive overview, Figure 2 illustrates a single agent-environment interaction step. The environment initialises a historical corner and passes the graph state \(s_t \in \mathcal{S}\) to the agent. The policy \(\pi(s)\) outputs an action specifying the required positional and velocity adjustments \(a_t \in \mathcal{A}\) for an attacker. The deterministic transition function \(\mathcal{P}\) updates the pitch to the new state \(s_{t+1}\). The reward function \(\mathcal{R}\) calculates the objective function change against the frozen defence to return the reward \(r_t\). The discount factor \(\gamma \in [0,1]\) balances immediate spatial improvements with long-term expected returns. We next formally detail the states, actions, and rewards.

States \(\mathcal{S}\). To capture the spatial dynamics and permutation invariance of the 22 players on the pitch, we define the state \(s \in \mathcal{S}\) as a fully-connected homogeneous graph \(\mathcal{G} = (\mathcal{V}, \mathcal{E})\). The node set \(\mathcal{V}\) contains the \(11\) attacking and \(11\) defending players. Each node \(v_i\) carries a 12-dimensional feature vector encompassing scaled spatial coordinates (\(x, y\)), velocity vectors (\(u_x, u_y\)), physical attributes (height, weight), binary role identifiers (corner taker, defending goalkeeper, delivery type, attacking team), and a binary feature indicating which player’s position is to be adjusted in this state. In terms of the edges \(\mathcal{E}\), the graph is fully connected, and edges \(e_{ij}\) carry a single feature consisting of a binary flag indicating whether the source and target nodes belong to the same team.

Actions \(\mathcal{A}\). Optimising the spatial configuration occurs via a sequential update process across the attacking players. At each timestep \(t\), the policy outputs a continuous action vector \(a_t\) that maps to two physical components: a spatial displacement \(\Delta p_t = [\Delta x, \Delta y]\) and a velocity delta \(\Delta u_t = [\Delta u_x, \Delta u_y]\). Within the environment, these continuous outputs are bounded to enforce realistic movement limits, restricting maximum per-step displacement to \(\pm 2\text{m}\) and velocity changes to \(\pm 2\text{m/s}\). Following the update step, if an attacker’s resulting velocity vector magnitude \(\|u_{\text{adj}}\|_2\) exceeds \(5.2\text{m/s}\) (the \(95^{\text{th}}\) percentile of all observed player speeds within the tracking data) the vector is proportionally scaled down to the safety boundary. This ensures that the policy cannot reach artificially high-reward states by recommending physiologically impossible sprints.

Rewards \({\mathcal{R}}\). The reward function \(\mathcal{R}(s)\) provides a dense signal quantifying the value of a given spatial configuration. We calculate this using a frozen, pre-trained GNN that maps a scene to the aforementioned \(xFCS\) metric. The model decomposes the probability of a shot into two conditional factors, mirroring the causal sequence of a set piece: the delivery, the first contact, and the subsequent shot. The probability of a shot occurring given a specific scene is formulated as: \[xFCS(s) = \sum_{i \in \text{attackers}} P(\text{contact} = i \mid s) \times P(\text{shot} \mid \text{contact} = i, s).\] This factorisation is modelled by two separate GNNs: a reception probability model which performs corner recipient node classification, and a conditional shot model given attacker \(i\) wins first contact. Extensive details regarding the architecture, training pipeline, and validation of this baseline reward model are provided in [22]. Within the RL environment, the step reward \(r_t\) is defined as the incremental change in the value of the spatial configuration following the kinematic adjustments of the attacking players: \[r_t = xFCS(s_{t}) - xFCS(s_{t-1}).\] This dense, delta-based formulation inherently penalises the policy for moving players into sub-optimal configurations and rewards continuous spatial optimisation towards configurations that maximise gains in \(xFCS\).

2.2 Architecture and Implementation↩︎

Parallelised JAX Environment. We implement the above problem formulation as an RL environment and simulator. We opt for JAX [23] to enable significant hardware acceleration. This is particularly relevant if set-piece optimisation is to be deployed in a time-sensitive analysis or live match context, as exploring many configurations benefits from parallelised vectorisation. Our simulator is built upon a 2D physics engine [24] that supports continuous-time dynamics, which can be exploited in future work.

Deep RL Algorithms. As RL algorithms, we consider Proximal Policy Optimization (PPO) [20] and Soft Actor-Critic (SAC) [19]. The former is a widely used on-policy actor-critic algorithm with a clipped surrogate objective for training stability. The latter uses the objective \(\mathcal{J}(\pi) = \mathbb{E}[\sum R(s,a) + \alpha \mathcal{H}(\pi(\cdot|s))]\) that combines expected returns and a maximum-entropy component. This forces the agent to explore a diverse array of valid tactical adjustments rather than collapsing prematurely to a single local maximum.

Graph Neural Network Representations. The graph state \(s_t\) is passed through a Graph Attention Network v2 (GATv2) [25], [26] to capture non-local, relational player interactions. The resulting embedding \(\phi(s_t)\) of the state is the input that is fed to the actor and critic networks. Specifically, the actor and critic networks each consist of two GATv2 layers with separate, unshared parameters. GATv2 layers are also used in the reward model \(\mathcal{R}\). We note that the reward model for predicting shot probabilities is trained on historical data and frozen; while the parameters of the GATv2 layers for obtaining state representations \(\phi(s_t)\) evolve throughout the RL training process.

3 Experiments and Results↩︎

3.1 Evaluation Protocol and Training Process↩︎

To evaluate the generalisability of our Graph RL method, we utilise a dataset of 3,223 corner kicks from the English Premier League. Rather than a randomised cross-validation split, we enforce a strict 80/20 sequential temporal split. Agents are trained exclusively on the chronologically earliest 80% corners of the season and evaluated zero-shot on the final 20% of unseen test corners. This constraint prevents temporal data leakage and mimics the operational reality of a football season, showing how the learned policy behaves when applied to late-season structural shifts and evolutionary tactical adjustments adopted by teams.

During the training phase, the RL agents interact with the JAX environment to maximise the expected \(\Delta xFCS\). As illustrated in Figure 3, we train both the SAC and PPO agents for 2.5 million environment steps. To ensure robustness and account for training variance, we train 8 independent random seeds for each algorithm. Evaluating the policies on the held-out test set at regular intervals reveals that test-set reward plateaus near the 2.5 million step mark, with SAC converging to a peak mean improvement of approximately \(+0.068\;xFCS\), slightly outperforming PPO which reaches \(+0.063\;xFCS\). This convergence point informs our stopping criterion and forms the basis for our total computational budget calculations.

3.2 Baselines and Evaluation Budgets↩︎

We benchmark the learned RL policies against two traditional, gradient-free metaheuristic baselines:

1. Random Search (RS): Uniformly samples continuous kinematic vectors within the defined bounding limits, acting as a brute-force spatial exploration baseline. For evaluation, we compute the mean performance across the 8 RS seeds for each individual corner; this RS mean serves as the baseline score against which all other approaches are normalised.

2. Simulated Annealing (SA): Iteratively perturbs the spatial coordinates of individual attacking players in sequence, utilizing a decaying geometric temperature schedule to probabilistically accept sub-optimal adjustments and escape local minima.

Comparing a fully trained neural network against metaheuristic search algorithms requires establishing fair computational budgets. Therefore, we evaluate the baselines under two distinct regimes. Under Inference Compute Matching, the metaheuristics have a search budget that matches the computational cost of a single episode executed by the RL agent, reflecting an operational time-sensitive scenario. In Total Training Compute Matching, the metaheuristics are granted an immense search budget equivalent to the entire 2.5 million environment step cost required to train the RL agent, prorated across the test set. This serves as an approximate upper bound on the performance of the RL agent.

3.3 Performance Profiles↩︎

We use the rliable library [27] to construct performance profiles (Figure 4) that visualise the fraction of corners where a method achieves a relative performance score greater than or equal to a threshold \(\tau\). To ensure a fair aggregation across the dataset, we must account for varying task difficulty; some defensive configurations natively possess more exploitable spatial flaws than others. Therefore, we define \(\tau\) as the ratio of a method’s \(\Delta xFCS\) to the RS baseline mean for that specific corner. This standardises performance regardless of the raw reward scale: \(\tau = 1.0\) indicates parity with RS, \(\tau < 1.0\) indicates underperformance, and \(\tau = 2.0\) indicates the algorithm generated twice the danger of the RS baseline. To ensure numerical stability and prevent \(\tau\) from exploding in scenarios where RS discovers negligible improvements, we enforce a minimum denominator of \(0.001\). The tight shaded regions in the plot represent 95% stratified bootstrap confidence intervals.

Figure 4 (a) evaluates the architectures under Inference Compute Matching, i.e., realistic operational constraints where the upfront RL training cost is amortised. In a production environment, generating a tactic via a forward pass through the SAC/PPO policy is relatively computationally inexpensive. Under these strict latency constraints, the generalised RL policies dominate. Because the SAC and PPO agents apply a policy informed by prior training rather than searching from scratch, they outperform RS in 94.2% and 92.6% of evaluated scenarios, respectively. Conversely, SA collapses under the restricted budget, outperforming RS in only 6.6% of cases. This confirms that Graph RL is uniquely equipped to provide high-quality tactical adjustments in real-time.

Figure 4 (b) shifts the lens to evaluate the methods under Total Training Compute Matching. Under these conditions, SA treats each corner as an isolated optimisation puzzle. Given 970 reward calls per individual corner, SA acts as a powerful per-scenario optimiser, outperforming RS in 76.8% of cases, compared to 69.4% for the generalised SAC policy and 63.3% for the PPO policy. While SA can outperform RL when granted a disproportionate, non-real-time compute budget, this approach scales poorly. The metaheuristic algorithm learns no overarching tactical principles and must restart its search from the beginning for every new scenario.

3.4 Quantitative Performance Evaluation↩︎

Beyond comparative search efficiency, we evaluate the tactical value generated across the 645 unseen test corners (two examples were visualised in Figure 1). The generalised RL policies demonstrate strong reliability: both SAC and PPO successfully identify advantageous kinematic adjustments (\(\Delta xFCS > 0\)) in over 99% of the evaluated scenarios. In absolute terms, the SAC policy yields a mean improvement of \(+0.068\;xFCS\) (median \(+0.046\)) per corner. Given the low-scoring nature of professional football, systematically increasing the expected number of shots from set pieces by this margin could represent a substantial competitive advantage. Under operational latency constraints (the inference budget), the SAC policy achieves a mean normalised score of \(2.40\) (median \(2.05\)). This indicates that, on average, a single forward pass of the policy generates more than double the tactical value of a compute-matched RS. Even when the metaheuristic baselines are granted the disproportionate total training compute budget, SA only achieves a mean normalised score of \(1.22\) against the expanded RS baseline compared to SAC and PPO scoring \(1.11\) and \(1.08\), respectively.

4 Discussion and Future Work↩︎

Qualitative inspection of the generated routines reveals several consistent tactical preferences exhibited by the SAC agent. Most notably, the policy strongly favours dynamic routines. Rather than maintaining attackers in static, high-value zones (such as the centre of the six-yard box), the agent frequently displaces attackers away from their markers only to have them attack the vacated space at high velocities. This aligns with the physical reality of set pieces, where arriving in space at speed generates higher-quality shot opportunities than contested, stationary headers.

However, evaluating the generated tactics also highlights the potential limits of a single-frame reward model. We hypothesise that while elite cooperative routines, such as deliberate “pinning" or”pick-and-roll" blocking schemes [28]–[30], exist within the historical training set, they are inherently temporal events that unfold over a sequence of timesteps. Because the \(xFCS\) model evaluates a single spatial snapshot composed of positions and velocities, it may struggle to definitively distinguish a coordinated blocking manoeuvre from two players simply being in close proximity. Consequently, we suspect the reward model learns a smoothed statistical expectation over these ambiguous spatial configurations. Potentially lacking a sharp, definitive signal to reward intricate blocking schemes, the RL agent may naturally gravitate toward exploiting unambiguous tactical signals that the model can confidently measure, such as generating high-velocity arrivals into undisputed space near the goal.

While metaheuristic methods such as SA are competitive in raw performance within our current formulation, this parity is largely a function of the environment’s constrained dimensionality. The present optimisation task was restricted to single-team kinematic adjustments against a static defence in order to provide a proof-of-concept validation. If the environment is expanded to a reactive, multi-timestep domain, such as an adversarial Multi-Agent Reinforcement Learning (MARL) [31] setting where defenders dynamically adjust to counteract attacking movements, the growth of the search space would render memoryless metaheuristics computationally intractable. Conversely, our Graph RL method is better equipped to scale to these highly relational, reactive environments. Future work will move from static optimisation to adversarial MARL paradigms.

5 Conclusion↩︎

In this work, we aimed to transition the computational analysis of football set pieces from historical evaluation and imitation toward active, reward-driven tactical discovery. We formalised corner kick optimisation as a kinematically constrained MDP and contributed a Graph RL method integrating GNN state embeddings with continuous control policies (SAC and PPO). Our evaluation on more than 3,000 Premier League corners demonstrates that a generalised policy can effectively optimise attacking spatial configurations without relying on exhaustive, per-scenario metaheuristic search. Our approach yields significant improvements in Expected First Contact Shot Probability, strongly outperforming traditional optimisation baselines under realistic operational inference constraints. While the current methodology could provide immediate value for optimising set-piece design, future work will seek to expand the scope of tactical generation to adversarial, reactive environments.

References↩︎