1 Introduction↩︎

Intercepting agile aerial targets with autonomous drones is a key challenge in robotics and security. As Unmanned Aerial Vehicle (UAV) incursions into restricted airspace become more common, applications such as airspace protection, infrastructure security, and event safety demand solutions that can neutralize unauthorized drones with minimal collateral damage [@Park2021]. Fleets of interceptor drones with capture-nets are a promising option, but their success depends on advanced control and coordination against evasive targets.

Classical interception methods rely on accurate models, preplanned strategies, or predictable target behaviour [@Yanushevsky2018]. However, modern quad-rotor drones can perform highly dynamic manoeuvres, and will actively evade capture, making classical methods ineffective [@Chung2011].

Recent advances in deep reinforcement learning (RL) have shown that drones can learn agile flight behaviours directly from interaction with the environment. RL-trained policies achieved superhuman performance in drone racing, where agents navigate challenging courses with dynamic manoeuvres [@Kaufmann2023champion]. However, drone racing typically involves static or predictable gates, whereas interception requires agents to respond to adversarial targets that actively evade capture. At the same time, Multi-Agent RL (MARL) has demonstrated remarkable success in competitive environments, particularly in complex games [@Silver2017; @OpenAI2019].

In this work, we formulate agile drone interception between a team of net-carrying pursuers and an evasive target as a competitive MARL task. The main contributions of this paper with respect to the literature are:

• A competitive MARL framework for cooperative interception of an agile evader by multiple pursuer drones using low-level control.

• A Prioritized Fictitious Self-Play training scheme for both the pursuers and the evader to learn robust policies against varied opponents.

• Extensive simulation results demonstrating stronger performance than standard baselines.

The remainder of the paper is organized as follows. Section 2 reviews related work on interception using teams of drones. Section 3 provides background on reinforcement learning and competitive self-play. Section 4 details our training methodology. Section 5 presents experimental results and comparisons. Section 6 concludes and discusses future directions.

2 Related Work↩︎

Agile flights in multi-rotors drones is typically characterized by the ability to perform large-angle manoeuvres, sustain high linear and angular accelerations, maintain precise control near dynamic limits and do so reliably in real-time, often in complex and cluttered environments. Traditional guidance laws for interception, such as Proportional Navigation (PN), are computationally simple and widely used in missile guidance. But, they typically assume predictable target manoeuvres and are less effective against highly agile or adversarial evaders. PN was recently adapted for catching manoeuvring quadrotors [@pliska2024towards], yet still rely on predictable target models. This issue can be mitigated by limiting the evader’s escape options using multiple pursuers, but this requires additional mechanisms such as target-pursuer allocation [@asadi2021optimal], inter-pursuers collision avoidance [@zhu2017distributed] or a geometric coordination layer [@zhou2016cooperative] to ensure proper deconfliction and coverage of the evader’s escape routes.

Learning-based methods, particularly MARL, offer an alternative for adversarial multi-UAV interception. In [@Xiao2024], the authors introduces a co-evolution framework that uses RL to train both the pursuers and the evader in the pursuit-evasion task. However, like most RL approaches [@Zhang2023; @zhang2023dacoop], they rely on high-level control inputs, in this case velocity commands, and simplified quadrotor dynamics, which limits the agility of the learned behaviours. In contrast, [@Chen2024] uses RL for low-level control of agile pursuers, however they assume fixed evader behaviours and only consider the pursuit aspect. Recently, [@roncero2025learned] and [@gavinagile] both combined a competitive RL framework and agile control inputs, but they limit their study to the 1v1 pursuit-evasion case. Overall, while RL has shown promise in the multi-pursuer interception task, current research either overlooks the adversarial element of a learning evader or typically lacks highly dynamic flight capabilities in both the pursuers and the evader.

3 Background↩︎

3.1 Multi-Agent Reinforcement Learning↩︎

Reinforcement Learning is a type of machine learning for sequential decision-making. In a rollout phase, an agent interacts with an uncertain environment which provides it with partial observations of its state, takes a series of actions following a policy and receives a scalar feedback in the form of rewards. These sequences of observe-act-reward, repeated over time, form the rollouts. The collected rollouts are then used to update the policy in a learning phase, which will then be employed in the rollout phase of the next training iteration. The goal of the agent is to learn a policy that maximizes the expected cumulative reward over time.

Multi-Agent Reinforcement Learning (MARL) extends RL to scenarios with multiple agents interacting in a shared environment. MARL suffers from the curse of dimensionality and non-stationarity, as the environment dynamics change as other agents learn and adapt their policies. Recent works in MARL adopted centralized training with decentralized execution (CTDE) [@Lowe2017], where agents have access to global information during training but operate based on local observations during execution. In competitive settings, this alleviates non-stationarity by allowing the agents to access the state and actions of their opponents during training.

3.2 Self-Play↩︎

Self-Play (SP) is a training method where an agent plays against oneself without any direct supervision. It yields excellent performance in challenging tasks such as Go, Poker [@Silver2017], and competitive video games like Dota 2 and StarCraft II [@OpenAI2019; @vinyals2019grandmaster]. In practice, each agent periodically freezes its policy while the other agent adapt to that frozen policy. Then the roles swap. But, it famously suffers from the problem of overfitting to a single opponent and forgetting how to win against past versions. Fictitious Self-Play is one solution to solve this problem. Each agent learns the best response to the empirical frequency distribution of the opponent’s past strategies. However, using FSP, the agents waste time playing against past versions of themselves that are too weak or too strong to provide a meaningful learning signal. Prioritized Fictitious Self-Play (PFSP) [@vinyals2019grandmaster; @OpenAI2019] addresses this issue by improving the diversity of the opponents’ policies the agents face. By sampling opponent policies from a pool according to a performance-based distribution, it ensures continual exposure to past strategies while prioritizing more challenging opponents.

4 Methodology↩︎

4.1 Problem Formulation↩︎

We study a pursuit–evasion scenario with quadrotors, a team of \(N\) pursuers and a single evader with identical manoeuvring capabilities, operating in an obstacle-free rectangular arena. The pursuers seek to catch the evader quickly, while the evader tries to avoid capture. At the beginning of each episode, the agents’ initial positions are drawn uniformly at random inside the arena. Capture occurs when the evader’s centre comes within a capture distance of any pursuer’s rigid, circular net of radius \(R\), which is mounted below each pursuer and aligned with its body frame. In such settings, the evader quickly learns to exploit the pursuers’ tendency to avoid boundary violations by flying close to the arena walls. To discourage this behaviour and promote agile evasive flight in the central region, a buffer zone is added along each wall. Only the evader is penalized for entering this zone, which restricts their movement to a smaller volume in the centre of the arena.

Target detection, state estimation, and trajectory prediction are not addressed in this work. We acknowledge that, together with the obstacle-free arena, and the constrained area for the evader, these are strong assumptions that affect the behaviour of the agents and the practical relevance of the results, but we reserve the relaxation of these assumptions for future work.

4.2 Observation and Action Space↩︎

We denote \(\mathcal{N_{\mathrm P}} = {1, \ldots, N}\) the set of \(N>1\) pursuers and \(e\) the single evader. We denote the position, velocity, and rotation matrix of agent \(i\in \mathcal{N_{\mathrm P}} \cup \{e\}\), in world coordinates as \(\mathbf{p}_i\), \(\mathbf{v}_i\), and \(\mathbf{R}_i\) respectively. For an agent \(j\in \mathcal{N_{\mathrm P}} \cup \{e\}\), we denote its relative position and velocity in agent \(i\)’s body frame as \(\mathbf{p}^{i}_j = \mathbf{R}_i^\top(\mathbf{p}_j - \mathbf{p}_i)\) and \(\mathbf{v}^{i}_j = \mathbf{R}_i^\top(\mathbf{v}_j - \mathbf{v}_i)\).

For each agent \(i \in \mathcal{N_{\mathrm P}} \cup \{e\}\), its observation \(\mathbf{o}_i\) is composed of the self-state observation \(\mathbf{o}^{\mathrm self}_i\), the observation of the other agents \(\mathbf{o}^{\mathrm others}_i\) (including the evader in the case of pursuers), and the observation of the arena bounds and the ground \(\mathbf{o}^{\mathrm env}_i\). The self-state observation is \(\mathbf{o}^{\mathrm self}_i=[\mathbf{R}_i^\top\mathbf{v}_i, \text{vec}(\mathbf{R}_i)]\) containing the agent’s linear velocity in body frame \(\mathbf{R}_i^\top\mathbf{v}_i\), and its rotation matrix \(\mathbf{R}_i\), with \(\text{vec}(\cdot)\) being the flattening function. The observation of the other agents is \(\mathbf{o}^{\mathrm others}_i=[\mathbf{p}^i_j, \mathbf{v}^i_j]_{j\in {\mathcal{N_{\mathrm P}}\cup\{e\}\backslash\{i\}}}\) containing the position and velocity of each agent \(j\) relative to the agent expressed in the agent body frame. Finally, the observation of the arena bounds and the ground is \(\mathbf{o}^{\mathrm env}_i=\{z_i\}\cup\{\mathbf{d}_m\}_{m=1}^M\) and is composed of the altitude of the agent \(z_i\) and a set of distance to arena bounds along \(M\) rays uniformly distributed in the horizontal plane. We normalize the observations before passing them to the neural network.

The control policies are trained using Multi-Agent Proximal Policy Optimization (PPO) [@Yu2022]. This Actor-Critic method uses two neural networks for each team of agents: a policy network and a value network and follows the CTDE paradigm. Each team has a single shared policy network that is executed independently by each agent using only its local observation, generating, for each agent, an action \(\mathbf{a}_i\), composed of a collective thrust \(\mathbf{a}^{th}_i\) and body rates \(\mathbf{a}^\omega_i\). And a single centralized value network used only during training that receives privileged global information about the opponent team’s state, which is not available to the policy network. This alleviates the non-stationarity of the environment due to the simultaneous learning of all agents [@Lowe2017]. The input of the value network of each team is the concatenation of the exact position, velocity, and rotation matrix of each agent, as well as the actions taken by the opponent(s) at this time step. This input is normalized before being fed to the neural network.

4.3 Rewards↩︎

The reward of the pursuers \(r^{\text{P}}_p\) (\(p \in \mathcal{N_{\mathrm P}}\)) and the evader \(r^{\text{E}}\) are given by: \[\begin{align} r^{\mathrm{P}}_p & = \phantom{-}r^{\mathrm{catch}} - r^{\mathrm{step}} - r^{\mathrm{coll}}_{p,\mathrm{P}} - r^{\mathrm{coll}}_{p,\mathrm{E}} - r^{\mathrm{fail}}_p - r^{\mathrm{cmd}}_p - r^{\mathrm{dist}}_p, \\ r^{\mathrm{E}} & = -r^{\mathrm{catch}} + r^{\mathrm{step}} - r^{\mathrm{coll}}_{e,\mathrm{P}} - r^{\mathrm{fail}}_e - r^{\mathrm{cmd}}_e - r^{\mathrm{bnd}}. \end{align}\] in which \(r^{\mathrm{catch}}\) rewards all the pursuers when one catches the evader, \(r^{\mathrm{fail}}_i\) penalizes any agent for crashing or going out of bounds, \(r^{\mathrm{coll}}_{p,\mathrm{P}}\) penalizes any pursuer for colliding with the body of another pursuer, \(r^{\mathrm{coll}}_{p,\mathrm{E}}\) penalizes collisions between any pursuer body and the evader body, \(r^{\mathrm{step}}\) rewards the evader for surviving each time step, and \(r^{\mathrm{cmd}}\) discourages dynamically infeasible commands. Instead of terminating the episode upon collision between a pursuer body and the evader, we apply a soft continuous penalty \(r^{\mathrm{coll}}_{p,\mathrm{E}} = r^{\mathrm{coll}}_{e,\mathrm{P}}\) to both agents, allowing for gradual learning of collision avoidance while maintaining focus on the primary task of catching with the net. However, for inter-pursuer collisions, our experiments showed that soft penalties slow down the learning of effective spreading tactics. So we terminate the episode for inter-pursuer collisions and apply an important penalty \(r^{\mathrm{coll}}_{p,\mathrm{P}}\). We still terminate the episode if any agent goes out of bounds or crashes on the ground and apply a hard penalty \(r^{\mathrm{crash}}_i\). Yet, neither the evader nor the pursuers receive a reward when the opponent reaches a failure state, encouraging actual pursuit–evasion behaviour rather than strategies that force the opponent to crash. Additionally, the reward shaping terms \(r^{\mathrm{dist}}_p\) penalizes the pursuer for being far from the evader to encourage pursuit, and \(r^{\mathrm{bnd}}\) penalizes the evader when approaching the arena bounds to discourage exploiting the boundaries.

Specifically, the reward terms are: \[\begin{align} r^{\rm catch} &= \lambda_{\rm catch} \cdot \mathbf{1}_{\rm catch}, & r^{\mathrm step} &= \lambda_{\mathrm step}, \\ r^{\mathrm{coll}}_{p,\mathrm{E}} &= \lambda_{{\mathrm{coll}}{\mathrm{PE}}} \cdot \mathbf{1}_{\mathrm{contact}}^{p,e}, & r^{\mathrm{coll}}_{p,\mathrm{P}} &= \lambda_{{\mathrm{coll}}{\mathrm{PP}}} \cdot \mathbf{1}_{\mathrm{contact}}^{p,\mathrm{P}}, \\ r^{\mathrm{fail}}_i &= \lambda_{\mathrm{fail}} \cdot \mathbf{1}_{\mathrm{fail}}^{i}, & r^{\mathrm{cmd}}_i &= \lambda_{\mathrm{cmd}} \cdot \lVert\mathbf{a}_i^\omega\rVert. \\ r^{\rm dist}_p &= \lambda_{\rm dist}\cdot \bigl\lVert {\mathbf{p}}_e - {\mathbf{c}}^{\rm net}_p\bigr\rVert_2, & r^{\mathrm{bnd}} &= \phi_{\mathrm{bnd}}(d^{\mathrm{bnd}}), \end{align}\] in which the indicator functions return \(1\) when their condition is met : \(\mathbf{1}_{\rm catch}\) when catching the evader, \(\mathbf{1}_{\mathrm{contact}}^{p,e}\) in case of a contact between a pursuer \(p\) and the evader, \(\mathbf{1}_{\mathrm{contact}}^{p,\mathrm{P}}\) in case of a contact between a pursuer \(p\) and any other pursuer, and \(\mathbf{1}_{\mathrm{fail}}^{i}\) when an agent \(i\) reaches a failure state because of a ground crash or leaving the arena bounds. \(c^{\mathrm net}_p\) is the centre of the catching net of pursuer \(p\), and \(\mathbf{a}_i^\omega\) are the commanded body rates of agent \(i\). \(\phi_{\mathrm bnd}\) is a function that penalizes the evader for approaching the arena bounds, triggering under a set threshold and growing exponentially the shorter the distance to the arena bounds \(d^{\mathrm bnd}\). \(\lambda_{\mathrm catch}\), \(\lambda_{\mathrm dist}\), \(\lambda_{\mathrm step}\), \(\lambda_{{\mathrm col}\mathrm{PE}}\), \(\lambda_{{\mathrm col}\mathrm{PP}}\), \(\lambda_{\mathrm fail}\), \(\lambda_{\mathrm cmd}\) are positive hyperparameters that balance the different reward terms and have been tuned to obtain the desired behaviour and listed in Table 1.

6pt

4.4 Training Method↩︎

To avoid overfitting to the current opponent and becoming less effective against previously seen strategies, we use Prioritized Fictitious Self-Play [@vinyals2019grandmaster; @OpenAI2019]. The pursuers and the evader train in separate loops and maintain a growing pool of past frozen policies. During each training episode, each team’s latest policy trains against a set of frozen opponent policies from the opponent’s pool, sampled according to a performance–weighted distribution. When the performance score of an agent against the full current pool of opponent strategies is past a certain threshold, a snapshot of the team’s current policy is frozen and stored in the growing policy pool. This maintains exposure to past strategies while prioritizing the more challenging opponents. Incidently, the opponent strategies being frozen during the training episode reduce the problem in each episode to a cooperative setting, and common MARL algorithms can apply. An overview of the training method is illustrated in Figure 2. Our agents play 50% of the episodes against the latest opponent policy to ensure a continuous improvement by facing the most advanced opponent, and 50% against the PFSP distribution.

The performance metric in our problem is not as simple as a binary catch/evade outcome. Drones can crash, which adds another possible outcome. And an evader that can avoid interception for an extended time is a stronger opponent than one that is quickly caught. During learning, pursuers should prioritize facing evader strategies that either make them often crash, or avoid capture for extended time and eventually reach timeout without being intercepted. However, weak evaders that crash a lot are less interesting and should be less sampled against. On the contrary, the evader should prioritize facing pursuers strategies that either make it often crash, or that catch it quickly. But avoid those that crash frequently without providing a challenge.

According to this definition, we derive the following performance metrics for a pursuers’ policy \(i\) and an evader’s policy \(e\): \[\begin{align} \rho_P(i, e) &= \begin{cases} 1 - \frac{t}{\tau}, & \text{if i catches e} \\ 0, & \text{if i crashes, or if e evades} \\ 1, & \text{if e crashes} \\ \end{cases}\tag{1} \\ \rho_E(e, i) &= \left(1 - \rho_P(i, e)\right)\tag{2} \end{align}\] in which \(t\) is the time step at which ends the episode, and \(\tau\) is the maximum episode length. For a current training pursuer \(i\), we sample the evader policy \(e\) from the pool of past policies \(\mathcal{E}\) with probability: \[\frac{f\left(\rho_P(i, e)\right)}{\sum_{e' \in \mathcal{E}} f\left(\rho_P(i, e')\right)}\] And similarly for a learning evader \(e\), replacing \(\rho_P(i, e)\) with \(\rho_E(e, i)\). \(f(x)=(1-x)^2\) is a weighting function that focuses PFSP on the hardest opponents.

4.5 Training Setup↩︎

Similarly to prior work [@gavinagile], we use a high-fidelity simulator of the quadrotor dynamics, which include a low-level control architecture with collective thrust and body rates as inputs, and collisions between quadrotors, the elements of the arena and the net carried by the pursuers. This simulator also include a high-level INDI controller [@smeur2016adaptive], which allow us to give alternate high-level commands to the quadrotors, such as position, velocity, or acceleration commands. This will be used to compare our method with classical baselines in Section 5.

Each policy network is a two‐layer multilayer perceptron with \(256\) ReLU units per hidden layer. The output layer produces the mean and standard‐deviation of a multivariate Gaussian, followed by a tanh squashing to obtain bounded continuous actions. The value networks mirror this architecture but ends with a linear output.

Rollouts are generated in parallel across \(1024\) environments. The entire pipeline, including the simulator, is written in Python using JAX [@Bradbury2018], enabling just‐in‐time compilation and parallelized execution on accelerated hardware. Running on a single machine equipped with an NVIDIA RTX 4090 (24 GB VRAM), an AMD Ryzen 9 7950X3D (16 cores, 4.2 GHz) and 128 GB RAM, the system collects and processes approximately \(3.5\times10^{5}\) environment steps per second. We train for a total of \(5\) billion environment steps for each side (\(10\) billion steps in total), which corresponds to roughly 5h30 of wall-clock training time.

5 Experiments↩︎

5.1 Experiment Setup↩︎

Similar to prior work [@gavinagile], the main comparison metrics are the catch rate of the pursuers, the evade rate of the evader, the time to catch and the crash rates. The catch rate is the percentage of episodes where pursuers catch the evader within 10 seconds without crashing. The evade rate is the percentage where the evader avoids capture for 10 seconds. Crash rates distinguish between pursuers crash rate and evader crash rate where either a pursuer or the evader crashes alone, and double-crash rate where both agents crash simultaneously. Finally, the time-to-catch measures how long pursuers take to capture the evader.

Time-to-catch is biased downward because it only includes successful catches: weaker pursuers may seem faster by catching only easy targets without crashing. We therefore use a right-censored metric, setting the time-to-catch to 10 seconds whenever an episode ends in a crash or timeout.

We train then evaluate the performances of our strategies in an arena of size \(32\times32\times16\) meters, with the evader constrained in a smaller volume of size \(12\times12\times6\) meters in the centre of the arena. For each combination of pursuers and evader strategies, we run 10,000 episodes and report the averaged metrics in Table ¿tbl:tab:performances?.

5.2 Comparison with baselines↩︎

We compare the performances of the trained policies in the 3v1 scenario against both heuristic baselines and learning-based methods. Many recent works have applied RL to multi-pursuer pursuit–evasion tasks [@Xiao2024; @Zhang2023; @Chen2024; @zhang2023dacoop], sometimes using advanced encoding mechanisms to predict the evader’s motion or account for obstacles [@Zhang2023; @Chen2024]. However, they train against deterministic evaders. Only a few addresses high-speed manoeuvring drones, and most use velocity control. To our knowledge, none combines agile drone dynamics with a learning evader. But, we believe that these learning methods are not incompatible with an SP framework, and could still perform well against learning evaders. However, our aim in this paper is to show that PFSP combined with thrust–body-rate control produces robust policies for agile pursuit–evasion. Since evader state estimation in high-speed flight is outside our scope, we restrict learning-based comparisons to ablations that isolate the effect of PFSP and thrust-body-rate control on performances. And we encourage future works to address the benchmarking of the existing RL-based pursuit-evasion methods in an SP framework.

In particular for the pursuers, we will compare our PFSP-CTBR policies with:

• Fast-Response Proportional Navigation (FRPN) [@pliska2024towards]: an evolution of Proportional Navigation for manoeuvring multi-rotors. This method is designed for high-speed interception of single agile target with a single pursuer using acceleration commands. There is no inter-pursuer collision avoidance in the original implementation.

• Artificial Potential Field (APF): a method where the pursuers are attracted by the evader and repelled by the arena boundaries and other pursuers. We use the implementation from [@zhang2023dacoop] which uses the normalized sum of the attractive and repulsive forces as the direction of a constant velocity command.

• FRPN + APF: we combine the FRPN method with an APF-based collision avoidance, where the acceleration command from FRPN is summed with the repulsive forces from APF to obtain the final acceleration command.

• noSP: an ablation of our method where the pursuers are trained with MAPPO against a deterministic evader that moves away from the pursuers using APF navigation.

• SP: an ablation of our method where the pursuers are trained with MAPPO and SP, but face only the latest version of their opponent at each episode.

• PFSP-vel: an ablation of our method where the pursuers are trained with PFSP but output a desired velocity instead of thrust and body rates. The desired velocity is converted to thrust and body rates using an INDI controller described in Section 4.5.

For the evader:

• APF: the evader is repelled by the pursuers and the arena boundaries. We use the implementation from [@zhang2023dacoop].

• noSP: an ablation of our method where the evader is trained with PPO against deterministic pursuers that use APF to intercept the evader.

• SP: an ablation of our method where the evader is trained with MAPPO and SP, but faces only the latest version of its opponent at each episode.

• PFSP-vel: an ablation of our method where the evader is trained with PFSP but outputs a desired velocity instead of thrust and body rates. The desired velocity is converted to thrust and body rates using an INDI controller described in Section 4.5.

For fairness, these methods only access the opponent’s position and velocity, as do our RL policies.

All the pursuers heuristic baselines performances drop when facing the agile learned evaders. FRPN crash rates is especially high, as it does not consider boundary constraints. This effect is exacerbated with multiple pursuers as they tend to collide into each other. APF repulsion from arena boundaries creates safer flights, but the simple attraction to the evader current position leads to a “Pure-Pursuit behaviour” with the pursuers chasing the target indefinitely never reaching it. Combining FRPN with an APF repulsion leads to a higher catch rate, but the pursuers still fails to catch our learned evader consistently.

In contrast, PFSP-CTBR, which was trained to avoid crashes, exhibits a much lower crash rate against the more agile learned evader. It achieves the highest catch rate and the lowest time-to-catch against all evaders, indicating that it learns effective interception strategies across diverse evasive manoeuvres while respecting arena boundaries and reducing inter-pursuer collisions. Its low time-to-catch further suggests that PFSP-CTBR succeeds even against the most challenging evaders, where other methods either fail or crash. PFSP-CTBR also produces the strongest evader policy, with higher evade rates and lower crash rates than the other evaders across all settings. This shows that it can consistently avoid capture for the full episode duration or until a pursuer crashes, the PFSP-CTBR pursuers strategy being the only one reliably able to catch it. It is particularly effective against PN pursuers, which tend to crash when facing agile maneuvers.

5.2.1 Impact of control level↩︎

The crash rate of the PFSP-vel evader increases when facing pursuers using CTBR or acceleration control. Although PFSP-vel pursuers maintain strong catch rates against most evaders, they are ineffective against the PFSP-CTBR evader. Across all settings, PFSP-CTBR achieves higher catch and evade rates while also reducing crash rates compared with PFSP-vel. These results indicate that velocity control constrains agility, whereas CTBR supports safer and more robust policies by enabling more agile maneuvers. This advantage is particularly pronounced for the PFSP-CTBR evader, which consistently combines high evade rates with low crash rates.

5.2.2 Effect of Self-Play↩︎

The no-SP strategies show strong performances against the specific opponents encountered during training: the pursuers reliably capture the APF evader, while the evader achieves a high evade rate against FRPN-APF, which as a result crashes a lot. But against other learned evaders, the no-SP pursuers degrades substantially, with crash rates rising sharply against the most agile policies. This indicates overfitting to deterministic training opponents and poor generalization to unseen strategies.

This issue is less pronounced for SP policies. Their catch rate stays high against all evaders except PFSP-CTBR, even against PFSP-vel. And their evade and crash rates remain consistent against all pursuers other than PFSP-CTBR. Training against agile opponents appears to produce strategies that are more robust to unpredictable trajectories and still effective against less agile adversaries. The 3v1 setting favors the pursuers by reducing the evader’s escape routes, and likely alleviates the issue of overfitting against the last opponent seen during training. In comparison, the PFSP-CTBR agents have uniformly strong performance. By prioritizing challenging opponents during training, it maintains a stronger learning signal and produces stronger strategies, that also generalize better to unseen opponents.

5.3 Qualitative Results↩︎

In this section we present qualitative results of our trained pursuers and evader in simulation. Similar to prior work on the 1v1 case [@gavinagile], the evader learned a diverse set of agile evasive manoeuvres that use the full 3D space to avoid capture, such evasive manoeuvres can be seen in Fig. 5 (a). In response, the pursuers learned to anticipate these manoeuvres and to maximize the surface of the catching net facing the evader to increase the chances of a successful capture.

They also learned to exploit their number to increase catching chances. The pursuers learned to avoid inter-agent crashes and split to attempts at capture in close succession to limit the risk of collisions. One pursuer speed up to make a first capture attempt, while the two others approach slower to wait for the opportunity of a second attempt if the first one fails, which is what happens in Fig. 4 (a).

The number of episodes where the pursuers adopt an encircling strategy to surround the evader and reduce its escape routes is low. The pursuers seem to have learned to rely more on their agility to attempt coordinated successive rapid interception attempts rather than to rely on a more strategic encircling strategy. During early training stages, the evader strategy being simpler and less agile could have led the pursuers to first adopt more reactive and direct interception strategies, that are slow to be replaced by more strategic encircling later in the training stages. This could also be explained by the evader being constrained in a smaller volume in the centre of the arena, which make it easier to catch without the need for encircling strategies as the evader has already limited escape routes. A bigger arena could alleviate this issue. On the contrary, the pursuers learned well to exploit the evader’s reluctance to approach its constrained boundaries. They will split up and push it towards the edges, blocking its escape routes. They will then anticipate the evader avoiding crossing them to capture it. This behaviour is shown in Fig. 3. Still, when in favourable positions, the pursuers did learn to profit of the encircling positions to approach from all directions and capture the target as shown in Fig. 4 (b).

Interestingly, the pursuers exhibit adaptive behaviour, quickly recovering from failed interception attempts and making multiple successive tries within the allowed capture time. As illustrated in Fig. 5 (b), after the evader narrowly escapes in Fig. 5 (a), one pursuer rapidly re-engages and successfully completes the interception.

In the rare case when the pursuers attempt a capture alone, it usually results in a failure as the evader can escape easily.

6 Conclusions↩︎

In this paper, we presented a Deep RL approach for the pursuit-evasion task with multiple pursuers and one evader, using MARL and PFSP to train high-speed manoeuvring policies for both teams. Our method achieves a high capture rate and low time to catch against a variety of agile evader strategies, while maintaining low crash rates for both pursuers and evaders. We show that our method outperforms several heuristic baselines in simulation achieving higher catch rates and lower crash rates when facing agile evaders. Ablation studies demonstrated the importance of self-play and the control level for learning robust policies in this task. Qualitative analysis of our trained policies reveals emergent behaviours, such as coordinated interception attempts and adaptive recovery from failed attempts. Furthermore, our simulation environment and training loop being fully within the JAX framework was essential, as it enabled massively parallelized execution and drastically reduced training times, making extensive RL training computationally feasible.

While our approach shows promising results, policies are evaluated only in simulation and may behave differently in real-world settings due to sensor noise, unmodeled dynamics, and disturbances. Future work will transfer them to real quadrotors and incorporate perception and state estimation for practical deployment.