Deep Reinforcement Learning for Spacecraft Attitude Control During Atmospheric Re-Entry
June 30, 2026
Abstract
Deep reinforcement learning has the potential to solve attitude control problems more adaptively, precisely, and robustly by handling nonlinear dynamics, uncertainties, and failure cases more effectively than traditional attitude control approaches. We explore reinforcement learning (RL) for attitude control in spacecraft re-entry. An industry-standard proportional-integral-derivative controller with gain scheduling serves as a strong baseline for model-free RL and hybrid controllers that combine these two approaches. We formalize the application in the RL framework to apply continuous, off-policy RL. State-of-the-art RL achieves comparable performance to traditional control approaches in this domain. However, its out-of-distribution generalization is not sufficient. Hence, we use dynamics randomization to introduce challenging task variations during training and enforce generalization in a predefined operational envelope. Finally, we assess the best obtained RL-based controllers with application-specific metrics to show superior performance in comparison to traditional controllers in the operational envelope, that is, hybrid controllers are able to track the angle of attack better and are more robust under variations of mass, inertia tensor, and flap actuator bandwidth.
1 Introduction↩︎
Deep reinforcement learning (RL) for continuous control [1] has the potential to handle nonlinear dynamics and failure cases more easily and to be more robust against and adaptable to unforeseen conditions than other attitude control approaches [2]–[4]. However, the introduction of RL into a safety-critical attitude control system requires a fundamental shift in verification and validation (V&V) for space-grade software. V&V for guidance, navigation and control traditionally relies on deterministic and formally verifiable control laws, e.g., proportional–integral–derivative (PID) controllers. For these, the system’s behavior is auditable through formal analysis and exhaustive simulation of well-defined corner cases. Core challenges with deep RL are the out-of-distribution generalization and opacity of the learned policy. It is a black box, making it infeasible to formally prove its stability and performance across the entire state space and parametric envelope. While explainable artificial intelligence advances transparency in the decision-making process of deep RL [5]–[7], its integration into safety-critical control applications is not fully established. We propose the use of hybrid control designs [8] to mitigate black-box concerns.
We specifically discuss attitude control for a hypersonic re-entry vehicle, as illustrated in Figure [fig:sketch]. Such spacecraft return personnel and cargo from space and, hence, are a key technology for future human spaceflight, planetary exploration, and for establishing an extraterrestrial resource extraction economy, e.g., lunar mining. We attempt on the one hand to replace classical control engineering by RL, reducing the expert’s effort, and on the other hand to combine PID control with RL-based control to form a hybrid controller that more effectively handles challenging changes to the environment dynamics and failure cases. We compare three different types of controllers. (1) Baseline controller: The previous solution is a PID controller with gain scheduling. (2) Only RL: RL without any additional controller, but with the model of the spacecraft to generate kinematically feasible reference values. (3) Hybrid controller: RL modifies the baseline controller. The hybrid controller is our primary candidate for a flight-ready system. This architecture is inherently safer because the PID controller acts as a verified baseline. The RL agent’s authority can be explicitly bounded by a supervisor logic that monitors the system’s state and the RL agent’s output, disengaging the RL agent when the vehicle approaches the edge of its validated safe domain. We aim to provide statistical evidence that the controller is reliable and safe within a strictly defined operational envelope.
Our contributions are the following. (1) We apply state-of-the-art deep reinforcement learning algorithms for continuous control to the problem of attitude control during hypersonic re-entry of a spacecraft. This is a challenging tracking problem that requires gain scheduling for changing atmospheric conditions. We design a simple and effective reward function for attitude control and find that MR.Q excels without task-specific tuning.
(2) Although the best learned controller performs better than the industry-standard baseline controller under nominal conditions, we explore hybrid control architectures that combine the baseline controller with reinforcement learning and compare these against pure reinforcement learning to enhance out-of-distribution generalization.
(3) To improve robustness and generalization explicitly, we compare dynamics randomization and task scheduling approaches for reinforcement learning in this attitude control problem and determine the best training strategy. We find that the resulting policies considerably improve robustness in comparison to the baseline controller.
2 Background: Spacecraft Re-Entry and Reinforcement Learning↩︎
2.1 Spacecraft Re-Entry Problem↩︎
The present paper considers flight path control for a lifting-body vehicle with low lift-to-drag ratio during hypersonic re-entry (see Figure [fig:sketch]). We focus on the endo-atmospheric part of the flight trajectory and disregard the preceding deorbit maneuver. Specifically, we focus on the descent from the upper mesosphere down to the troposphere (altitude 93 km to 10 km). The vehicle speed at the start is 7378 m s−1 and decreases to approximately 150 m s−1 at the end, when a parachute can be deployed for landing. Atmospheric flight creates aerodynamic forces and moments that act on the vehicle. The main objective is to control the aerodynamic forces such that they result in a desired translational motion of the re-entry vehicle. In addition, the magnitude of the forces must remain limited to not endanger structural integrity of the vehicle. These objectives are addressed by tracking a specific angle of attack and bank angle. The angle of attack \(\alpha\) describes the pitch attitude of the vehicle with respect to its direction of flight, while the bank angle \(\mu\) describes the roll attitude of the vehicle with respect to this direction. Furthermore, the nose of the vehicle needs to be aligned with its direction of flight, i.e., the sideslip angle \(\beta\) must remain small. Figure [fig:sketch] illustrates these angles. Attitude control must minimize the errors between the commanded and actual angles over the course of the trajectory. The vehicle in this study has two flaps at its tail that deflect to produce aerodynamic moments about the body-fixed axes \(x_b\) and \(y_b\), and thrusters arranged in a configuration to exert a moment about the axis \(z_b\). The actions of attitude controllers therefore manipulate deflection commands to the aerodynamic control surfaces (\(\delta_\text{e,cmd}\) for symmetric and \(\delta_\text{a,cmd}\) for antisymmetric deflection) and a thruster command \(\boldsymbol{\tau} = \left[\tau_x, \tau_y, \tau_z\right]^T\). The observations encompass command values for angle of attack (\(\alpha_\text{cmd}\)) and bank angle (\(\mu_\text{cmd}\)) as well as estimates of the actual angles \(\alpha,\beta,\mu\), and body-fixed angular rates from the navigation filter.
2.2 Simulation Model↩︎
In order to train and evaluate the controllers, we rely on a simulation that encompasses the vehicle dynamics, actuator and sensor lags, atmosphere, as well as guidance, navigation, and control algorithms that serve as a baseline for this study. The models are briefly described next with additional details given in Appendix 10. The step size of the simulator is \(1/140\,\mathrm{s}\) and it accounts for multi-rate subsystems and computational delay. For example, control algorithms are executed at 14 Hz.
A six-degree-of-freedom rigid-body flight dynamics model for a rotating, spherical earth is the core of the simulation [9], [10]. In addition to the notation from Figure [fig:sketch], the aerodynamic side force \(Y\) (perpendicular to aerodynamic lift \(L\) and drag \(D\)), and the vehicle mass \(m\) are needed in the evolution of the flight path, defined in terms of velocity \(V\), flight path angle \(\gamma\) (vertical direction relative to the local horizontal), and course angle \(\chi\) (horizontal direction relative to local north) as \[\begin{bmatrix} \dot{V} \\ \dot{\gamma}\\ \dot{\chi} \end{bmatrix} = \frac{1}{m} \begin{bmatrix} -D-mg\sin\gamma \\ \left(L\cos\mu - Y\sin\mu -mg\cos\gamma\right)/V \\ \left( L\sin\mu + Y\cos\mu\right)/V \end{bmatrix}. \label{eq:flightpath}\tag{1}\] The change of the aerodynamic angles \(\alpha\), \(\beta\), and \(\mu\) is \[\begin{bmatrix} \dot{\mu} \\ \dot{\alpha}\\ \dot{\beta} \end{bmatrix} = \begin{bmatrix} (\omega_x \cos\alpha+\omega_z\sin\alpha)/\cos\beta - \dot{\chi}\sin\gamma+ \tan\beta (\dot{\gamma}\cos\mu +\dot{\chi}\cos\gamma\sin\mu) \\ \omega_y - \tan\beta (\omega_x\cos\alpha+\omega_z\sin\alpha) - (\dot{\gamma}\cos\mu +\dot{\chi}\cos\gamma\sin\mu)/(\cos\beta) \\ \omega_x\sin\alpha-\omega_z\cos\alpha + \dot{\chi}\cos\gamma\cos\mu-\dot{\gamma}\sin\mu \end{bmatrix},\] where we denote angular rates about the body-fixed axes by \(\boldsymbol{\omega}=[\omega_x, \omega_y, \omega_z]^T\). The rotational dynamics are described by \(\boldsymbol{\dot{\omega}} = \boldsymbol{I}^{-1} \left( \boldsymbol{M} - \boldsymbol{\omega} \times \boldsymbol{I} \, \boldsymbol{\omega} \right)\), where \(\boldsymbol{M}\in\mathbb{R}^3\) denotes the moments resulting from aerodynamics and thruster usage and \(\boldsymbol{I}\) is the inertia tensor with respect to the center of mass. The aerodynamic forces and moments \(D,L,Y,\boldsymbol{M}\) are complicated to model accurately and are, in general, nonlinear functions of the vehicle state and environmental conditions. We employ an industry-grade high-fidelity aerodynamics model that represents these dependencies in terms of tabulated coefficients that depend on the flight conditions. This is a standard approach in flight dynamics modeling [10], [11]. For example, the lift is modeled as \(L = \bar{q} S C_L(M\!a,\alpha,\beta,\boldsymbol{\omega},\boldsymbol{\delta}).\) Here, \(\bar{q}=\frac{1}{2} \rho V_a^2\) is the dynamic pressure, resulting from the velocity of the vehicle relative to air \(V_a\) (i.e., \(V\) accounted for wind) with the local air density \(\rho\), and \(S\) is the effective surface area of the vehicle. The coefficient \(C_L\) depends on the Mach number \(M\!a\) (the ratio of flight speed and local speed of sound) and the flap deflections \(\boldsymbol{\delta}=[\delta_\text{e}, \delta_\text{a}]^T\) in addition to the state variables \(\alpha\), \(\beta\), and \(\boldsymbol{\omega}\). The simulation encompasses \(M\!a\in [0.5, 26.8]\) and \(\bar{q}\in [50, 5825]\,\mathrm{Pa}\), leading to considerable variations in the aerodynamic influence. The international standard atmosphere model is used to determine environmental conditions such as the density of air and speed of sound.
Trajectory generation is a complex, mission-specific endeavor that requires precalculation and predictor-corrector algorithms [12]–[14]. We consider a generic atmospheric re-entry mission with a simplified flight path guidance loop. One objective for the controller is to keep the sideslip angle small, \(\beta\approx0\), which implies \(Y\approx0\). For this case, Equation 1 simplifies and provides insight into the principal mechanics behind guidance for re-entry. The gravitational force (\(mg\)) accelerates the vehicle during descent (\(\gamma<0\)), while the aerodynamic drag \(D\) decelerates the vehicle. The change in \(\gamma\) depends on the term \(L\cos\mu\), which represents the upward component of the aerodynamic lift \(L\) that opposes the gravitational force (\(mg\cos\gamma\)). Hence, the vertical direction of flight can be controlled by changing the magnitude of the bank angle \(\mu\). However, the equation for \(\chi\) reveals that this will also change the horizontal direction due to the sideward lift component \(L\sin\mu\). We employ a simple guidance strategy: (1) The angle of attack determines the magnitude of lift and drag. The command value \(\alpha_\text{cmd}\) issued by the guidance is a setpoint that ensures structural integrity of the vehicle, while resulting in sufficient deceleration along the trajectory. (2) The magnitude of the bank angle command \(\mu_\text{cmd}\) is calculated by a feedback loop to track a predefined flight path angle \(\gamma\), i.e., to offset the gravity component \(mg\cos\gamma\) in Equation 1 . (3) Bank angle reversal is triggered, i.e., the sign of the command value \(\mu_\text{cmd}\) is switched, whenever a specified heading angle deviation from the reference is exceeded. This strategy leads to sufficiently rich variations in the reference commands for the present study.
2.3 Baseline Controller↩︎
We focus on the atmospheric control mode, which is more complex compared to exo-atmospheric attitude control. To provide a challenging baseline for benchmarking, a state-of-the-art control algorithm is included in the simulation. This baseline consist of separate control laws for the longitudinal motion (\(\alpha\to\alpha_\text{cmd}\) tracking) and for the lateral-directional motion (\(\mu\to\mu_\text{cmd}\) tracking and \(\beta\to0\) regulation). Longitudinal control uses a gain-scheduled proportional-integral-derivative (PID) law in combination with an inverse model \(f\) of the flight dynamics in feedforward: \[\begin{align} \delta_\text{e}(t) &=& f(M\!a(t),\bar{q}(t),\alpha_\text{cmd}(t)) + k_p(M\!a(t),\bar{q}(t))\cdot (\alpha_\text{cmd}(t)-\alpha(t)) \\ &&+\, k_i(M\!a(t),\bar{q}(t))\cdot \int^t_0 \alpha_\text{cmd}(s)-\alpha(s) ds - k_d(M\!a(t),\bar{q}(t))\cdot \dot{\alpha}(t) \end{align}\] This is a deterministic, time-varying control policy with continuous changes of the controller gains based on the current estimates of the Mach number \(M\!a(t)\) and dynamic pressure \(\bar{q}(t)\). Similarly, lateral-directional control laws are gain-scheduled PID for bank angle tracking and proportional-derivative (PD) control for sideslip angle regulation. The lateral directional control laws involve crossfeeds and hence they are multivariable. The controller aims at yielding as little as possible error between the commanded and the estimated values of the angle of attack, bank angle, and sideslip angle. The gains for the control laws were selected based on the well-established pole-placement technique that seeks to match a closed-loop response with desired characteristics specified by a low-order transfer function reference model [9], [15]. Designs were performed for 21 linear snapshot models obtained along the considered nominal flight trajectory. The control laws are implemented in terms of state feedback gain schedules with linear interpolation in between design grid points. The scheduling variables are calculated based on estimated Mach number and dynamic pressure [16].
2.4 Reinforcement Learning for Continuous Control and Generalization of Policies↩︎
Deep Deterministic Policy Gradients [1] enabled using neural networks for continuous model-free RL. Since then, many off-policy RL algorithms for continuous control were influenced by Soft Actor Critic [17] and DDPG’s extension TD3 [18]. TD3 adds clipped double Q-learning, target policy smoothing, and delayed policy updates to DDPG. SAC is similar to TD3, but uses a stochastic policy in the maximum entropy RL framework. TD3 was extended to TD7 [19] and MR.Q [20]. TD7 adds a state-action encoder, loss-adjusted prioritized experience replay [21], and checkpointing. Similarly, MR.Q uses model-based representation learning and LAP. SAC was extended to DroQ [22], CrossQ [23], BRO [24], and SimbaV2 [25]. Across various benchmarks for continuous control, model-free RL [20], [24], [25] currently seems to outperform model-based RL, i.e., DreamerV3 [26] and TD-MPC2 [27].
To obtain robust policies and enable sim-to-real transfer, dynamics randomization has been used in robotics [28]–[31]. In its simplest form, dynamics randomization varies parameters of the dynamics in the simulation randomly during training without informing the RL agent. The agent has to learn a policy that is robust under these unknown variations while only relying on observations.
We have a smoothly varying parametrization of the dynamic properties over which we want to generalize, e.g., actuation characteristics. Hence, we adopt the formalism of Contextual Markov Decision Processes [32], in which the variation of tasks is described as a tuple \(\left(\mathcal{C}, \mathcal{S}, \mathcal{A}, \mathcal{M}(c)\right),\) with context space \(\mathcal{C}\), state space \(\mathcal{S}\), action space \(\mathcal{A}\), and a function \(\mathcal{M}(c) = \left( \mathcal{S}, \mathcal{A}, P_c(s'|s, a), R_c(r|s, a, s'), \mu_c(s_0) \right)\) mapping contexts \(c \in \mathcal{C}\) to individual Markov Decision Processes (MDPs) with context-dependent state transition probabilities \(P_c\), reward function \(R_c\), and initial state distribution \(\mu_c\). In our case, the agent only has access to observations that depend on context and previous states \(o_t = \phi(c, s_t, s_{t-1}, \ldots)\), and it is supposed to select actions \(a \in \mathcal{A}\). Our goal is to find a policy \(\pi\) that maximizes the objective function \(\frac{1}{|\mathcal{C}|} \sum_{c \in \mathcal{C}} J(\mathcal{M}(c), \pi),\) where \(J(\mathcal{M}, \pi)\) is the expected return of policy \(\pi\) in task \(\mathcal{M}\). CMDPs allow us to define distinct sets of contexts for training and testing to evaluate zero-shot generalization [33].
Going one step further, we want to select task variations intelligently during training to improve sample efficiency and final performance. Task scheduling approaches have been proposed for multi-task RL [34], [35] and contextual RL [36]. [36] frame the task selection problem as a non-stationary multi-armed bandit problem and solve it via discounted upper-confidence bound [37], [38]. In this framework, they compare task selection strategies based on the episodic return: easiest tasks first (best reward), hardest task first, and focus on tasks with largest improvement in returns (e.g., monotonic progress). We call this active multi-task training (AMT). Scheduled multi-task training [35] focuses on solving hard tasks first.
3 Learning Attitude Control for Hypersonic Spacecraft Re-Entry↩︎
3.0.0.1 Action space (see also Appendix 7)
The vehicle uses its flaps to control pitch and roll motion, and thrusters for yaw motion. We compare several architectures for attitude control. The simplest learned controller (Only RL) will directly output yaw thrust and changes of the flap angles. The controller described in Section 2.3 serves as a baseline for evaluation, but also as a building block in two hybrid controllers. In additive hybrid control, the output of the RL policy network is added to the output of the baseline controller [8]. A more advanced hybrid control approach uses the policy for gain scheduling in the baseline controller, i.e., we compute twelve continuous gain factors \(f_\text{gain}\) to scale gains in the baseline control laws. We represent these multipliers as values of decibel in accordance with classical control notion, i.e., we transform each policy output by \(f_\text{gain}(x) = 10^{x/20}\) with \(x \in \left[-6, 6\right]\), such that a policy output of \(x=0\) results in a factor of 1. The maximum is \(f_\text{gain}(6)\approx2\), and the minimum is \(f_\text{gain}(-6)\approx 0.5\).
3.0.0.2 Observation space (see also Appendix 8)
The observable state of the vehicle includes the aerodynamic angles (\(\alpha, \beta, \mu\)), and angular velocities \(\boldsymbol{\omega}\) (see Section 2). In addition, we add the commanded aerodynamic angles (\(\alpha_{\text{cmd},t}, \beta_{\text{cmd},t}, \mu_{\text{cmd},t}\)), the previously commanded flap deflections (\(\delta_{e,\text{cmd},t-1},\delta_{a,\text{cmd},t-1}\)), and, for hybrid control, also the current baseline control command to the observation space. To mimic the structure of a conventional PID error-feedback controller, we add for each aerodynamic angle the current error, its derivative, and its integral to the observation space. This improves the ability of a feedforward policy network in this partially observable MDP. The dynamics of the environment are influenced by air density and speed of the vehicle, which determine the dynamic pressure. We measure the altitude, velocity, and dynamic pressure and provide it as observations to the controller. Observations are normalized to the range \(\left[-1, 1\right]\).
3.0.0.3 Reward function (see also Appendix 9)
Our main objective is to minimize the attitude cost \[c(t) = \left(\frac{\alpha_\mathrm{cmd}(t) - \alpha(t)}{\alpha_{\text{range}}}\right)^2 + \left(\frac{\beta_\mathrm{cmd}(t) - \beta(t)}{\beta_{\text{range}}}\right)^2 + \left(\frac{\mu_\mathrm{cmd}(t) - \mu(t)}{\mu_{\text{range}}}\right)^2,\] which we define based on commands and estimates of the aerodynamic angles (in radian). The individual ranges allow us to define a desired range for each error. This approach is known as Bryson’s rule in classical linear quadratic optimal control. We focus the attitude cost on the angle of attack \(\alpha\) with \(\alpha_{\text{range}} = 2^\circ, \beta_{\text{range}}=10^\circ,\mu_{\text{range}}=10^\circ\).
However, taking the negative attitude cost as a reward has the drawback that the reward is always negative, which makes early termination of an episode by running into failure states appealing. We solve this problem by defining a strictly positive reward \[\text{attitude reward}\, (t) = \max (0.001, \exp(-w_c \cdot c(t))) \in \left[0.001, 1\right]\] with weight \(w_c=2\). Since the attitude reward is at least 0.001, it discourages the agent from terminating early and shifts focus to the main objective.
We combine the attitude reward with a control cost to form the reward function in each step \(t'\) as \(r_{t'} = \text{attitude reward}\, (t) - \text{control cost}\, (t),\) where control cost is the sum of thruster and flap costs \[\text{control cost}\,(t) = w_{\tau_z} \left(\frac{\tau_z(t)}{\tau_{z,\max}}\right)^2 + w_{\delta_e} \left(\frac{\Delta\delta_{e,\text{cmd}}(t)}{\Delta \delta_{e,\max}}\right)^2 + w_{\delta_a} \left(\frac{\Delta\delta_{a,\text{cmd}}(t)}{\Delta \delta_{a,\max}}\right)^2,\] with weights \(w_{\tau_z} = 1\) and \(w_{\delta_e} = w_{\delta_a} = 0.05\). The difference between successive flap commands \(\Delta \delta_{\circ,\text{cmd}}(t) = \delta_{\circ,\text{cmd}}(t) - \delta_{\circ,\text{cmd}}(t-1)\) has a maximum of \(\Delta \delta_{\circ,\max} = \Delta t \cdot \dot{\delta}_{\circ,\max} = \frac{1}{14}\, \cdot\) 15 ° \(\approx\) 1.07 °. The yaw torque \(\tau_z(t)\) generated by the thrusters is limited by \(\tau_{z,\max} =\) 300 N.
3.0.0.4 Context space (see also Appendix 10)
For dynamics randomization, we parametrize the simulation with the mass of the vehicle \(m_0 \in \left[1312, 1968\right]\) kg, fractions of the principal moments of inertia \(\boldsymbol{f} \in \left[0.9, 1.1\right]^3\), misalignment of the principal axis by means of rotation \(\boldsymbol{\omega} \in \{\omega = \theta\hat{\omega} | \hat{\omega} \in S^2, \theta \in \left[-10^\circ, 10^\circ\right] \} \subset \mathbb{R}^3\), and flap actuator bandwidth \(\omega_0 \in \left[12, 30 \right]\) rad s−1. Hence, the parameter space is \(\mathcal{C} \subset \mathbb{R}^8\). Nominal conditions correspond to \(c_\text{nominal} = \left(1640, 1, 1, 1, 0, 0, 0, 30\right)^T\).
4 Experiments↩︎
In our experiments, we want to answer the following research questions:
Can we apply deep RL algorithms to spacecraft attitude control? (Section 4.1.1)
What is the best achievable performance with pure RL? (Section 4.1.1)
Which control architecture works best? (Section 4.1.2)
How well do controllers generalize to out-of-distribution conditions? (Section 4.1.3)
Can we enforce generalization with domain randomization or task scheduling? (Section 4.2.1)
Which is the most robust controller? (Section 4.2.2)
4.0.0.1 Evaluation of RL algorithms:
Following [39], we analyze the performance of RL algorithms with sample-efficiency curves using the interquartile mean (IQM) and bootstrapped 95% confidence intervals of the cumulative episode reward, i.e., undiscounted return. For direct comparison of policies, we compute the probability of improvement.
4.0.0.2 Application-specific performance metrics:
Key metrics for attitude control are the absolute errors of the angle of attack \(|e_\alpha(t)| = |\alpha_{\text{cmd}}(t) - \alpha(t)|\), sideslip angle \(|e_\beta(t)| = |\beta_{\text{cmd}}(t) - \beta(t)|\), and bank angle \(|e_\mu(t)| = |\mu_{\text{cmd}}(t) - \mu(t)|\). We prioritize the angle of attack to maintain structural and thermal integrity of the vehicle and create sufficient drag to decelerate. We want to avoid large errors, while small errors over a longer period of time are allowed. Hence, we analyze the error distribution and report percentiles. As secondary metrics, we report the control costs in all three action components. For multiple test contexts, we furthermore report the success rate, which is the percentage of contexts, in which the controller was able to follow the trajectory completely. Episodes terminate when any angle is outside of its safe domain, which is defined based on the aerodynamic angles: \(\alpha \in \left[0^{\circ}, 60^{\circ}\right]\), \(\beta \in \left[-20^{\circ}, 20^{\circ}\right]\), and \(\mu \in \left[-90^{\circ}, 90^{\circ}\right]\).
4.1 Training Under Nominal Conditions↩︎
We start episodes with constant initial attitude, initial mass, inertia, and flap actuator bandwidth, but randomize parameters of the trajectory to prevent overfitting the trajectory (see Appendix 10 for details). We analyze which control architecture and RL algorithm performs best and what is the maximum achievable performance. Since we do not vary dynamics during training, we can test out-of-distribution generalization of the learned controllers.
4.1.1 RL Algorithm Selection↩︎
Figure 2: MR.Q surpasses the baseline controller. Learning curves with interquartile means (IQM) and 95% bootstrap confidence intervals for 10 training seeds [39]..
We initially selected SAC and TD3 to explore the feasibility RL-based control. We tried CrossQ as an extension of SAC as well as TD7 and MR.Q as extensions of TD3. In this study, we focus on TD3-based algorithms as they showed consistent behavior in preliminary experiments without task-specific tuning. Hence, we evaluate only SAC, TD3, TD7, MR.Q. Hyperparameters are listed in Appendix 11.
Figure 3: Comparison under nominal conditions.. a — Probability of improvement [39]., b — Error distribution (kernel density estimation) for angle of attack. Density is plotted on a logarithmic scale to highlight the tails. The peak of the baseline controller around \(0\) is more pronounced on a linear scale.
With the learning curves in Figure 2, we evaluate the performance and sample efficiency of RL algorithms and control architectures during training. We evaluate combinations of learning algorithm and control architecture with 10 random seeds for training and 10 evaluation episodes without exploration noise. For comparison, we plot the performance of no control and the baseline controller averaged over 10 evaluation episodes. Individual learning curves per training run are in Appendix 19.
MR.Q excels without task-specific tuning. The main reasons for this are the multi-step return, which reduces bias in the target value for the critic loss, and the model-based representation (detailed analysis in Appendix 13). Most algorithms perform better in the hybrid control architecture than their non-hybrid counterpart, with the exception of MR.Q. For the non-obvious cases, we compare the best obtained policies per run with the probability of improvement and a 95% confidence interval in Figure 3 (a). We attribute that MR.Q outperforms the baseline controller to the fact that MR.Q learns to continuously adapt to environment conditions, which is more efficient than the gain scheduling of the baseline. In particular, the error distribution for our primary control objective, the angle of attack, is narrower for the learned controller than for the baseline (see Figure 3 (b), see also Appendix 16 for obtained trajectories and control commands). However, the baseline has a higher concentration of errors close to 0. Hence, MR.Q works without task-specific tuning and is able to achieve a higher return than the baseline controller by tracking the angle of attack more accurately.
4.1.2 Comparison of Control Architectures↩︎
Figure 4: Comparisons of control architectures..
With MR.Q, we compare control architectures (see Figure 4). Our simplest RL-based controller maps from observations to flap changes and thruster commands. We compare it to additive and a gain-scheduling hybrid control. In both hybrid architectures, the baseline controller’s command is in the observation space.
For maximum performance pure RL or additive hybrid control seem to be the best options under nominal evaluation conditions. Although the best possible performance of the gain-scheduling hybrid controller is lower than for the two other architectures, the bounded influence of the policy makes learning considerably more stable across runs and over time, and the performance is still better than the baseline.
4.1.3 Out-Of-Distribution Generalization of Policies Trained Under Nominal Conditions↩︎
Figure 5: Generalization of policies trained under nominal conditions. The solid lines show median performance over training runs and shaded areas the \(\left[5, 95\right]\)-percentile interval.. b — Performance evaluated under variations of \(\boldsymbol{I}\). On the left, we scale the principal moments of inertia along three axes (from top to bottom: x-, y-, z-axis). On the right, we rotate \(\boldsymbol{I}\) about the three basis vectors., c — Performance with respect to changes of the initial attitude without training for it. Performance is measured at the initial attitude during training \(+\, \epsilon \in \left[-10, -5, -2, -1, 0, 1, 2, 5, 10\right]\) degree per component., d — Performance evaluated under variations of flap actuator bandwidth.
To analyze generalization of the standard RL training paradigm with random exploration in action space, we test robustness with respect to variations that were not present during training. We compare the undiscounted return of an episode for the best obtained policies per run (see Figure 5).
Inertia tensor (Figure 5 (b)): The inertia tensor \(\boldsymbol{I}\) has six degrees of freedom. To systematically vary it, we multiply the principal moments of inertia \(I_{xx},I_{yy},I_{zz}\) by factors from \(\left[0.7,1.3\right]\) and apply rotations about the basis vectors with angles from \(\left[-30^\circ, 30^\circ\right]\). Rotation of \(\boldsymbol{I}\) about the z-axis has the most impact and the baseline controller is more robust than the learned controllers to it.
Initial attitude (Figure 5 (c)): We vary the initial values of the aerodynamic angles. Although policies trained on constant initial attitude outperform the baseline under nominal conditions, they do not generalize well. Even for a small envelope around nominal conditions, we cannot guarantee stability without explicit testing, which demonstrates that action noise is not enough to learn policies that are robust against deviations from nominal conditions. However, the additive hybrid controller’s performance deteriorates less drastically in most out-of-distribution cases than pure RL and the gain-scheduling hybrid controller is almost as reliable as the baseline.
Flap actuator bandwidth (Figure 5 (d)): Similarly, we test generalization to variations of the flap actuator bandwidth. Lower values make the control problem harder and simulate failures of the actuation. Nominal conditions correspond to 30 rad s−1. Hence, all of the tested values are out of distribution. The baseline controller’s performance drastically drops for actuator bandwidths below 14 rad s−1. Interestingly, completely learned controllers are consistently more robust to lower values. We suspect that random exploration in action space during training makes the resulting policy robust to this scenario although the exact stable range cannot be guaranteed. Both hybrid control architectures show more variation, but their median performance is similar to the baseline.
The bounded influence of the gain-scheduling hybrid controller makes its behavior more similar to the baseline with respect to variations of the initial conditions. However, the stability of learned controllers cannot be guaranteed in the same range as for the baseline controllers when we do not explicitly enforce the operational envelope during training. Although MR.Q is more robust to actuation failures than the baseline controller, it is less stable with respect to variations of the inertia tensor and considerably less stable under variations of the initial attitude than the baseline controller.
4.2 Training Under Dynamics Randomization↩︎
We randomize not just the trajectories, but also the initial attitude to ensure that the RL-based controllers generalize to these variations. Furthermore, we systematically vary the following dynamics parameters: initial mass, inertia tensor, and flap actuator bandwidth (see Appendix 10 for details). Each combination of dynamics parameters defines a context. We compare task scheduling approaches to select contexts during training and evaluate generalization of the obtained controllers.
To test generalization and robustness of controllers, we uniformly sample a test set \(\mathcal{C}_{\text{text}}\) of 100 contexts and measure the obtained return in each test context. We select the best controllers per training procedure for this analysis based on the minimum return in a smaller test set of 30 contexts from regularly saved checkpoints per run (checkpoint interval: 200k time steps).
4.2.1 Comparison of Task Scheduling Methods for Dynamics Randomization↩︎
We sample \(|\mathcal{C}_\text{train}|=50\) context vectors that define individual tasks. We compare SMT, which prefers hard tasks, and active task selection with the best-reward (AMT-B, prefers easy tasks) and monotonic progress (AMT-M, prefers tasks with high expected learning progress) strategies. Hyperparameters are listed in Appendix 12. We use continuous uniform random sampling of contexts (dynamics randomization, DR) and round robin (RR) as baselines for task scheduling.
Task scheduling methods do not perform better than dynamics randomization or round robin. Since the results are inconclusive, we report them in Appendix 18. Among task selection strategies, focusing on hard tasks in the beginning (SMT) is the worst strategy. Focusing on the easiest tasks first (AMT-B) and focusing on tasks with the maximum expected learning progress (AMT-M) work equally well and are comparable to the baselines DR and RR with no clear advantage.
Training with task scheduling is difficult for two reasons: (1) training on the full range of flap actuator bandwidths creates some problems that are considerably harder to solve, and (2) as we use a replay buffer for each task, infrequently selected tasks might contain transitions of low performing policies that are still selected relatively often, since we first sample the task uniformly and then the samples within the task for the batch that is required in the network updates. We mitigate the first issue by sampling flap actuator bandwidths from \(\left[25, 30\right]\) rad s−1, but still evaluate on the full range. The second problem causes instabilities after about 4M steps, which we avoid by stopping after 5M steps. A more detailed analysis is required to understand the problem and find solutions.
4.2.2 Generalization and Robustness of Best Controllers↩︎
Since task scheduling approaches do not provide any benefit in comparison to continuous uniform dynamics randomization (DR), we analyze more closely the performance of the best obtained controllers with DR. We compare DR for pure RL and additive hybrid control in Table ¿tbl:tab:robustness? with domain-specific evaluation metrics. We focus on the error distribution of the aerodynamic angles and success rate. See Appendix 18 for a detailed analysis including the control costs. Exemplary training curves are displayed in Figure 6).
In comparison to the baseline controller, which fails particularly often at low flap actuator bandwidths and large rotations of the inertia tensor, both additive hybrid control and pure RL improve the success rate considerably. The additive hybrid controller’s success rate seems to be limited by the baseline, although it obtains lower error 9x-percentiles in each aerodynamic angle. For the angle of attack, both architectures reduce the 90, 95, and 98 error percentiles in comparison to the baseline. The additive hybrid controller is even better at tracking the bank angle than the baseline in these percentiles. Improving these metrics comes at the cost of more expensive control, i.e., thruster usage and flap angle changes (see Appendix 18).
5 Related Work↩︎
RL has been applied to attitude control in various ways for vehicles inside [3], [4], [40]–[44] and outside of an atmosphere [2], [45]–[52]. Q-learning [50], PPO [2], [40], [46], [49], DDPG [42], [45], TD3 [4], [47], [48], and SAC [3], [43] were used in these applications. Most studies replace traditional control approaches by RL. However, [52] propose a residual RL architecture for online adaptation and [4] embed a policy for anti-disturbance control in a hybrid controller for hypersonic re-entry. Furthermore, [4] extend TD3 to EVE-TD3, which shares similarities with MR.Q: (1) a multi-step return (over a horizon of 30 steps) to improve the estimation of the target value for the critic, and (2) the policy network has a bottleneck (6–18 nodes) to force the policy to extract high-level features.
According to [47], robustness of attitude control can be defined with respect to (1) tumble: nonzero initial angular velocity, (2) single, impulsive disturbance torque, (3) constant disturbance torque, and (4) different inertia tensors \(\boldsymbol{I}\). RL can be robust against variations of \(\boldsymbol{I}\) by adapting to it online [50] or by handling variation of it as noise [46]. Robustness can also be defined in terms of handling disturbances caused by the applications of unknown external torques on the spacecraft [47], [51] or actuator faults [51]. Sources of disturbances might be (1) fault caused by bias (e.g., less available torque), (2) fault caused by failure to respond to control signals (e.g., motor fault), or (3) disturbance of environment: uncertainties in air resistance, gravity, solar pressure, etc. [45]. [3] measure robustness with respect to wind gusts in quadcopter control, which is similar to a single, impulsive disturbance force. Similarly, [42] verified a deep RL algorithm for guidance and control of a reusable launch vehicle in the landing phase and tested under various wind conditions.
6 Conclusion↩︎
The MR.Q algorithm excels without task-specific tuning in the challenging attitude control problem of a hypersonic re-entry vehicle (see Section 4.1.1). Specifically, its model-based representation and multi-step returns are pivotal. Pure MR.Q can surpass the performance of the baseline controller and obtain a similar performance as hybrid controllers under nominal conditions. However, training under nominal conditions does not yield sufficient generalization, although hybrid control architectures often work better under out-of-distribution settings than pure RL (see Section 4.1.3). Hence, we enforce generalization with dynamics randomization and find that continuous uniform sampling of dynamic parameters is sufficient, i.e., task scheduling approaches do not perform better (see Section 4.2.1). Surprisingly, MR.Q is able to generalize over each condition that we varied during training. With dynamics randomization, obtained controllers are considerably more robust than the baseline with respect to uncertainties in dynamics parameters (see Section 4.2.2).
The additive hybrid controller is the most promising controller. It generalizes better than pure MR.Q in the out-of-distribution experiments with training under nominal conditions (see Section 4.1.3). It improves the robustness in comparison to the baseline by reducing the probability of larger errors of the angle of attack at the cost of using the thrusters and changing the flap angles more (see Section 4.2.2). The lower bound of the additive hybrid controller’s performance is the baseline controller’s performance if we deactivate the learned component in out-of-distribution settings.
To ensure stability of hybrid controllers, out-of-distribution conditions must be detected to disable the learned component and fall back to the formally verified baseline. This is straightforward to implement through outlier detection with data from the replay buffer. To enable real-time execution of policies in deployment, we plan to run them on Field Programmable Gate Arrays (FPGAs).
Being able to adapt to unforeseen conditions online would be a major advantage of reinforcement learning for attitude control [52]. The gain-scheduling controller, which exhibits less variance across training steps and runs (see Section 4.1.2), is our preferred candidate for online adaptation in a real scenario. However, the analysis is out of scope for this study. More experiments are needed to evaluate stability and performance of the online adaptation process.
Acknowledgments↩︎
This work was funded by the European Aerospace Agency under the GSTP programme, activity GT1I-602SA “Artificial Intelligence techniques for spacecraft attitude control and estimation” (project acronym: AI4AOCS), contract number 4000145154/24/NL/MGu, lead by Airbus Defence and Space GmbH. We thank Octavio Arriaga for feedback on our implementation of MR.Q, Shubham Vyas for feedback on the manuscript, and Arthur de Freitas Precht for technical support for the interface to the simulation software. This work was partially supported by the German Federal Ministry of Research, Technology and Space (BMFTR) under the Robotics Institute Germany (RIG).
7 Action Space↩︎
| Action | Description | Unit | Limits |
|---|---|---|---|
| Control Commands (Only in Pure RL or Additive Hybrid Control Mode) | |||
| \(\Delta\delta_{e,\text{cmd}}(t)\) | Change (\(\delta_{e,\text{cmd}}(t) - \delta_{e,\text{cmd}}(t-1)\)) of symmetric deflection (both flaps trailing edge down). | rad | \(\left[-\frac{15}{14}^\circ, \frac{15}{14}^\circ\right]\) |
| \(\Delta\delta_{a,\text{cmd}}(t)\) | Change of antisymmetric deflection of both flaps. Right flap is deflected trailing edge down, left flap is deflected trailing edge up. | rad | \(\left[-\frac{15}{14}^\circ, \frac{15}{14}^\circ\right]\) |
| \(\tau_{z}(t)\) | The available four thrusters are fired in a group that produces a (yawing) torque about the body-fixed vertical (z) axis. The command is a torque demand that is converted to thruster opening times by low-level functions. | N | \(\left[-300, 300\right]\) |
| PID Gains (Only in Gain-Scheduling Hybrid Control Mode) | |||
| \(f_{k_{p,\alpha}}\) | Factor for proportional gain of angle of attack error. | decibel | \(\left[-6, 6\right]\) |
| \(f_{k_{i,\alpha}}\) | Factor for integral gain of angle of attack error. | decibel | \(\left[-6, 6\right]\) |
| \(f_{k_{d,\alpha}}\) | Factor for derivative gain of angle of attack error. | decibel | \(\left[-6, 6\right]\) |
| \(f_{k_{p,\beta,\beta}}\) | Factor for proportional gain of yaw angle error to thruster command. | decibel | \(\left[-6, 6\right]\) |
| \(f_{k_{d,\beta,\beta}}\) | Factor for derivative gain of yaw angle error to thruster command. | decibel | \(\left[-6, 6\right]\) |
| \(f_{k_{p,\beta,\mu}}\) | Factor for proportional gain of bank angle error to thruster command. | decibel | \(\left[-6, 6\right]\) |
| \(f_{k_{d,\beta,\mu}}\) | Factor for derivative gain of bank angle error to thruster command. | decibel | \(\left[-6, 6\right]\) |
| \(f_{k_{p,\mu,\mu}}\) | Factor for proportional gain of bank angle error to flap command. | decibel | \(\left[-6, 6\right]\) |
| \(f_{k_{i,\mu,\mu}}\) | Factor for integral gain of bank angle error to flap command. | decibel | \(\left[-6, 6\right]\) |
| \(f_{k_{d,\mu,\mu}}\) | Factor for derivative gain of bank angle error to flap command. | decibel | \(\left[-6, 6\right]\) |
| \(f_{k_{p,\mu,\beta}}\) | Factor for proportional gain of yaw angle error to flap command. | decibel | \(\left[-6, 6\right]\) |
| \(f_{k_{d,\mu,\beta}}\) | Factor for derivative gain of yaw angle error to flap command. | decibel | \(\left[-6, 6\right]\) |
8 Observation Space↩︎
| Observation | Description | Unit | Limits |
|---|---|---|---|
| Environment Conditions | |||
| Altitude | Altitude above the WGS84 geoid. | m | \(\left[0, 150k\right]\) |
| Mach number \(M\!a\) | Velocity as ratio of absolute speed and local speed of sound | unitless | \(\left[0, 35\right]\) |
| Dynamic pressure \(\bar{q}\) | Dynamic pressure depends on air speed \(V_a\) and air density \(\rho\) through \(0.5 V_a^2 \rho\) | Pa | \(\left[0, 10k\right]\) |
| Commanded Aerodynamic Angles | |||
|
|||
| \(\alpha_{\text{cmd}}(t)\) (angle of attack) | Requested angle between the direction of flight relative to air and the body-fixed frame in the vertical plane. | rad | \(\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]\) |
| \(\beta_{\text{cmd}}(t)\) (sideslip angle) | Requested angle between the direction of flight relative to air and the body-fixed frame in the horizontal plane. The command can be assumed to be zero all of the time. | rad | \(\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]\) |
| \(\mu_{\text{cmd}}(t)\) (bank angle) | Requested angle between the direction of lift and the vertical plane around the direction of flight relative to air. | rad | \(\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]\) |
| Measured Angles and Angular Rates | |||
| \(\alpha(t)\) (angle of attack) | Measured angle of attack. | rad | \(\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]\) |
| \(\beta(t)\) (sideslip angle) | Measured angle of sideslip. | rad | \(\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]\) |
| \(\mu(t)\) (bank angle) | Measured bank angle. | rad | \(\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]\) |
| \(\alpha(t-1)\) | Last measured angle of attack. | rad | \(\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]\) |
| \(\beta(t-1)\) | Last measured angle of sideslip. | rad | \(\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]\) |
| \(\mu(t-1)\) | Last measured bank angle. | rad | \(\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]\) |
| \(\dot{\phi}(t)\) (roll rate) | Angular velocity about the forward (x) axis of the vehicle. | rad/s | \(\left[-10, 10\right]\) |
| \(\dot{\theta}(t)\) (pitch rate) | Rate of change of the pitch angle. Corresponds to angular velocity about the intermediate y’ axis in a standard Euler angle zyx-rotation sequence. | rad/s | \(\left[-10, 10\right]\) |
| \(\dot{\psi}(t)\) (yaw rate) | Rate of change of the yaw angle. Corresponds to angular velocity about the geodetic z axis in a standard Euler angle zyx-rotation sequence. | rad/s | \(\left[-10, 10\right]\) |
| Last Commands | |||
| \(\delta_{\text{e},\text{cmd}}(t-1)\) | Flap command from last step. See actions for details. | rad | \(\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]\) |
| \(\delta_{\text{a},\text{cmd}}(t-1)\) | Flap command from last step. See actions for details. | rad | \(\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]\) |
| Baseline Control Commands (Only in Hybrid Control Mode) | |||
| \(\delta_{\text{e},\text{basecmd}}(t)\) | Flap command of baseline controller. See actions for details. | rad | \(\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]\) |
| \(\delta_{\text{a},\text{basecmd}}(t)\) | Flap command of baseline controller. See actions for details. | rad | \(\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]\) |
| \(\tau_{\text{z},\text{basecmd}}(t)\) | Thruster command of baseline controller. See actions for details. | N | \(\left[-300, 300\right]\) |
| PID Error Components | |||
| \(e_\alpha(t)\) | Proportional error for angle of attack. | rad | \(\left[-\pi,\pi\right]\) |
| \(e_\beta(t)\) | Proportional error for sideslip angle. | rad | \(\left[-\pi,\pi\right]\) |
| \(e_\mu(t)\) | Proportional error for bank angle. | rad | \(\left[-\pi,\pi\right]\) |
| \(\int_0^t e_\alpha(\tau) d\tau\) | Integral error for angle of attack. | rad s | |
| \(\int_0^t e_\beta(\tau) d\tau\) | Integral error for sideslip angle. | rad s | |
| \(\int_0^t e_\mu(\tau) d\tau\) | Integral error for bank angle. | rad s | |
| \(\frac{d e_\alpha(t)}{dt}\) | Derivative error for angle of attack. | rad s−1 | |
| \(\frac{d e_\beta(t)}{dt}\) | Derivative error for sideslip angle. | rad s−1 | |
| \(\frac{d e_\mu(t)}{dt}\) | Derivative error for bank angle. | rad s−1 | |
9 Design of Reward Function↩︎
9.1 Sensitivity Analysis of Reward Function↩︎
Figure 7: Sensitivity analysis of for weights in the reward function.. a — Influence of flap cost weight on control cost and absolute error of aerodynamic angles., b — Influence of thruster cost weight on control cost and absolute error of aerodynamic angles. In the gray region, the policy was not able to follow the complete trajectory.
For various configurations of the control cost weights (flap cost weights \(w_{\delta_e} = w_{\delta_a}\) and thruster cost weight \(w_{\tau_z}\)), we report the interquartile mean of absolute errors and control costs of the obtained controllers over ten randomly sampled evaluation trajectories after 2.4M training steps with one training run of MR.Q in Figure 7.
We see that increasing the flap cost weights \(w_{\delta_e} = w_{\delta_a}\) has the desired effect of reducing the flap control cost without sacrificing tracking performance around a sweet spot. Good policies with low control costs are obtained in the range of \(\left[0.03, 0.2\right]\). However, a weight of \(0.2\) considerably reduces sample efficiency in comparison to lower values, indicating that it makes the learning task harder. Usually about \(250k\) to \(600k\) training steps are required to follow the trajectory completely during training. With \(w_{\delta_e} = w_{\delta_a} = 0.2\), about \(1M\) training steps were required. For a weight of \(0.5\), \(2.2M\) training steps are required. Although this analysis was done after the experiments under nominal conditions, \(w_{\delta_e} = w_{\delta_a} = 0.05\) seems to be a good choice in hindsight as it balances sample efficiency, control cost, and control performance.
Similarly, increasing the thruster cost weight has the desired effect of reducing control cost. However, the errors of the aerodynamic angles decrease at the same time. The best possible weight is about \(w_{\tau_z} = 1\) as increasing the weight further makes the learning problem so hard that it is not possible to follow the whole trajectory after \(2.5M\) training steps.
9.2 Monitoring Training↩︎
Since the maximum value of \(r_t\) is 1, the maximum value of the expected return approximated by the value function is \(\frac{1}{1-\gamma}\) under the discounted infinite horizon model, e.g., for \(\gamma=0.99\) the maximum return is 100 if we assume no control costs and that the attitude is tracked perfectly. Hence, we are able to directly track the performance of the policy in comparison to the optimum during training in algorithms based on DDPG (i.e., TD3, TD7, MR.Q), since the loss of the policy is the negative expected return approximated by the value function. The best possible value would be -100 under the condition of perfect approximation.
9.3 Considered Reward Functions↩︎
In non-systematic, preliminary experiments we explored a wide range of reward functions and components of reward functions. We tried the following:
Negative attitude cost: sum of squared differences between commanded and measured aerodynamic angles.
Adding a penalty for thruster forces.
Adding a penalty for flap angle changes.
Adding a penalty for derivative of the aerodynamic angles. We do not use this component anymore because it is often required to change the aerodynamic angles considerably to track the commanded trajectory.
Adding an extra penalty for large deviations from commanded angles (unsafe state penalty).
Terminating an episode when the orientation is out of the safe range and adding a large penalty of -1000 or -8000 (stop penalty).
Adding a reward for each step that the episode is not aborted (healthy reward), which results in significantly longer simulations, however, sometimes at the cost of less accurate tracking of the orientation.
Replacing angle cost by forward reward, which rewards moving closer to the commanded attitude and is inspired by the reward function of the MuJoCo environments from Gymnasium.
Initially, we set the weight of the control cost components to 0.001 and then increased it to 0.01, 0.1, 0.5, and 1. Higher weights lead to less switching between flap angles, less extensive use of thrusters, and generally less oscillation of commands. The current individual weights are a good compromise between tracking performance and avoiding excessive overcompensation of errors.
We tried to remove the healthy reward and use a terminal cost instead, that rewards low altitude of the vehicle when the episode terminates. This did not lead to better results, even when evaluating policies obtained with the previous reward under the new reward.
We tried to remove the healthy reward and use an attitude that is always positive and more positive when the attitude cost is lower. We call this the attitude reward and it is included in the final version of the reward function. The idea behind this is to substitute the healthy reward and the attitude cost component, since it is always better to survive longer and it is better to have a low attitude cost. The approach was successful, so we do not use any attitude cost, terminal cost, or stop penalty anymore.
10 Simulation Details↩︎
10.1 Common Configuration↩︎
10.1.0.1 Initial conditions:
The vehicle starts at the outer atmosphere of Earth with a fixed altitude of about 93 km and fixed velocity of approximately Mach 26.75. The dynamic pressure is low with approximately 54 Pa in this state.
10.1.0.2 Flight path:
To avoid overfitting to a specific flight path, we randomize the trajectory given by the guidance system in each episode by setting two parameters \(\gamma\) and \(\Delta \chi_{\max}\) (see Table 1). The flight path angle \(\gamma\) determines the desired speed of descent. The maximum deviation from the flight course \(\Delta \chi_{\max}\) controls bank angle reversals. These reversals are necessary because the bank angle of the vehicle influences not just the speed of descent, but also the direction of the flight. In order to follow a straight path, the bank angle has be reversed when \(\Delta \chi\) exceeds the given threshold. This has the effect of modifying the environment dynamics and reward function from the perspective of the agent because it modifies the commanded aerodynamic angles that are part of the state space as well as the PID error components that are part of the observation space.
| \(\gamma\) | -1.0° | \(\mathcal{U}(-1.1^\circ, -0.9^\circ)\) | \(\mathcal{U}(-1.1^\circ, -0.9^\circ)\) |
| \(\Delta \chi_{\max}\) | 3.25° | \(\mathcal{U}(1.5^\circ, 5^\circ)\) | \(\mathcal{U}(1.5^\circ, 5^\circ)\) |
| \(\alpha(0)\) | 45.024° | constant | \(\mathcal{U}(\alpha_{0,\text{nominal}} - 5^\circ, \alpha_{0,\text{nominal}} + 5^\circ)\) |
| \(\beta(0)\) | 0.046° | constant | \(\mathcal{U}(\beta_{0,\text{nominal}} - 5^\circ, \beta_{0,\text{nominal}} + 5^\circ)\) |
| \(\mu(0)\) | 61.141° | constant | \(\mathcal{U}(\mu_{0,\text{nominal}} - 5^\circ, \mu_{0,\text{nominal}} + 5^\circ)\) |
| \(m_0\) | 1640 kg | constant | context, \(m_0 \in \left[ 1312, 1968 \right]\) kg |
| \(\boldsymbol{I}\) | \(\text{diag}\left(\begin{array}{c} 492 \\ 2247 \\ 2358 \\ \end{array}\right) \text{kg} \cdot \text{m}^2\) | constant | context, see main text |
| \(\omega_0\) | 30 rad s−1 | constant | context, \(\omega_0 \in \left[12, 30\right]\) rad s−1 |
10.2 Nominal Conditions↩︎
Under nominal conditions, the initial attitude, vehicle mass, the vehicle’s inertia tensor, and the actuation are constant as summarized in Table 1.
10.3 Dynamics Randomization↩︎
In comparison to nominal conditions, we additionally randomize the initial attitude for each episode according to Table 1 and the context vector contains the vehicle’s mass, inertia tensor, and flap actuator bandwidth.
To model uncertainty in the actuation of the flaps, we modify the flap actuator bandwidth \(\omega_0\). Lower values make controlling the vehicle harder. The baseline controller becomes unstable for values under \(14\) rad s−1.
The initial mass is varied in a predefined range according to Table 1. The vehicle’s mass defines its gravitational force and, hence, influences the evolution of the flight path. The flight path then defines the commanded aerodynamic angles, as it is required to increase or decrease the lift to compensate for the change in mass.
To generate physically plausible inertia tensors, we define perturbations of the nominal inertia tensor with the following procedure. Since the nominal inertia tensor is diagonal, we apply a perturbation directly to the principal moments and then rotate the inertia tensor. To modify the principal moment, we use a vector \(\boldsymbol{a} \in \left[-0.1, 0.1\right]^3,\) and to modify the rotation, we use a rotation vector \(\boldsymbol{\omega} = \theta \hat{\boldsymbol{\omega}} \in \mathbb{R}^3\) with an angle \(\theta \in \left[-10^\circ, 10^\circ\right]\) and axis \(\hat{\boldsymbol{\omega}} \in S^2 \subset \mathbb{R}^3\). With the exponential map \(\text{Exp}: \mathbb{R}^3 \rightarrow SO(3)\) [53], we compute the rotation matrix from the rotation vector so that the nominal inertia is perturbed with \[\boldsymbol{I} = \text{Exp}(\boldsymbol{\omega}) \cdot \text{diag}(\boldsymbol{1}_3 + \boldsymbol{a}) \cdot \boldsymbol{I}_{\text{nominal}} \cdot \text{Exp}(\boldsymbol{\omega})^T,\] where \(\text{diag}(\boldsymbol{x})\) transforms a vector \(\boldsymbol{x} \in \mathbb{R}^3\) to a diagonal matrix. We generate a random set of rotation axes by sampling from a 3D standard normal distribution \(\mathcal{N}(\boldsymbol{0}_3, \boldsymbol{I}_{3\times 3})\) and normalizing the vector.
11 Hyperparameters of RL Algorithms↩︎
| Hyperparameter | MR.Q | TD7 | TD3 | SAC | |
|---|---|---|---|---|---|
| Discount factor \(\gamma\) | \(0.99\) | \(0.99\) | \(0.99\) | \(0.99\) | |
| Replay buffer capacity | \(2\)M | \(2\)M | \(2\)M | \(2\)M | |
| Mini-batch size | \(256\) | \(256\) | \(256\) | \(256\) | |
| Target update frequency \(T_\text{target}\) | \(250\) | \(250\) | - | - | |
| Target update rate \(\tau\) | - | - | \(5\cdot 10^{-3}\) | \(5\cdot 10^{-3}\) | |
| Dynamics loss weight \(\lambda_\text{Dynamics}\) | \(1\) | - | - | - | |
| Reward loss weight \(\lambda_\text{Reward}\) | \(0.1\) | - | - | - | |
| Terminal loss weight \(\lambda_\text{Terminal}\) | \(0.1\) | - | - | - | |
| Activation loss weight \(\lambda_\text{pre-activ}\) | \(1\text{e}-5\) | - | - | - | |
| Encoder horizon \(H_\text{Enc}\) | \(5\) | - | - | - | |
| Multi-step returns horizon \(H_Q\) | \(3\) | - | - | - | |
| Target policy noise \(\sigma\) | \(\mathcal{N}(0,0.2^2)\) | \(\mathcal{N}(0,0.2^2)\) | \(\mathcal{N}(0,0.2^2)\) | - | |
| Target policy noise clipping \(c\) | \((-0.3, 0.3)\) | \((-0.5, 0.5)\) | \((-0.5, 0.5)\) | - | |
| Initial random exploration steps | \(10\)k | \(25\)k | \(25\)k | \(5\)k | |
| Exploration noise | \(\mathcal{N}(0,0.2^2)\) | \(\mathcal{N}(0,0.1^2)\) | \(\mathcal{N}(0,0.2^2)\) | - | |
| Policy delay | \(1\) | \(2\) | \(2\) | \(1\) | |
| Probability smoothing \(\alpha\) | \(0.4\) | \(0.4\) | - | - | |
| Minimum priority | \(1\) | \(1\) | - | - | |
| Optimizer | AdamW1 | Adam2 | Adam | Adam | |
| Learning rate | \(3\text{e}-4\) | \(3\text{e}-4\) | \(3\text{e}-4\) | \(1\text{e}-3\) | |
| Hidden dim | \(512\) | \(256\) | \(256\) | \(256\) | |
| Activation function | ELU3 | ELU | ReLU | ReLU | |
| Weight initialization | Xavier uniform4 | LeCun normal5 | LeCun normal | LeCun normal | |
| Bias initialization | \(0\) | \(0\) | \(0\) | \(0\) | |
| Gradient clip norm | \(20\) | - | - | - | |
| Optimizer | AdamW | Adam | Adam | Adam | |
| Learning rate | \(3\text{e}-4\) | \(3\text{e}-4\) | \(3\text{e}-4\) | \(3\text{e}-4\) | |
| Hidden dim | \(512\) | \(256\) | \(256\) | \(256\) | |
| Activation function | ReLU6 | ReLU | ReLU | Swish7 | |
| Weight initialization | Xavier uniform | LeCun normal | LeCun normal | LeCun normal | |
| Bias initialization | \(0\) | \(0\) | \(0\) | \(0\) | |
| Optimizer | AdamW | Adam | - | - | |
| Learning rate | \(1\text{e}-4\) | \(3\text{e}-4\) | - | - | |
| Weight decay | \(1\text{e}-4\) | - | - | - | |
| \(\mathbf{z}_s\) dim | \(512\) | \(256\) | - | - | |
| \(\mathbf{z}_{sa}\) dim | \(512\) | \(256\) | - | - | |
| \(\mathbf{z}_a\) dim | \(256\) | \(256\) | - | - | |
| Hidden dim | \(512\) | \(256\) | - | - | |
| Activation function | ELU | ELU | - | - | |
| Weight initialization | Xavier uniform | LeCun normal | - | - | |
| Bias initialization | \(0\) | \(0\) | - | - | |
| Reward bins | \(65\) | - | - | - | |
| Reward range | \([-10,10]\) (effective: \([-22\text{k}, 22\text{k}]\)) | - | - | - | |
| Entropy optimizer | - | - | - | Adam | |
| Entropy learning rate | - | - | - | \(1\text{e}-3\) |
While the MR.Q algorithm is a theoretically sound approach [20], we had to modify it slightly in comparison to its original implementation. In the original implementation, the dynamics loss compares the latent state computed from unrolled dynamics \(z_{sa}^T W_p \in \mathbb{R}\) to the latent target state obtained by applying the state encoder \(f_\omega\) to a state \(s'\) from the replay buffer \(f_\omega(s') = z_{s'}\), in which the state encoder applies layer normalization and ELU, hence, limits the output to \(z_{s'} \in \mathbb{R}_{\geq -1}\). Initially, we had this as a bug in our implementation of MR.Q, but we found that removing ELU from the last layer of the state encoder enhances performance. Hence, our implementation does apply layer normalization without ELU after the last layer of the state encoder, which resembles the behavior of the state encoder in TD7 [19], in which AvgL1Norm is supposed to protect from monotonic growth of the features.
12 Hyperparameters of Task Scheduling Algorithms↩︎
For task scheduling, we sample a discrete set of contexts and create one replay buffer per context. For each update, we first randomly sample the replay buffer and then a batch of samples from this replay buffer. This ensures that we continue training each task even though a task is not selected anymore.
In comparison to the original SMT [35], we do not perform any network resets, which does not seem to be necessary in MR.Q. Furthermore, we do not learn a task encoding.
| Hyperparameter | Active MT | SMT | |
|---|---|---|---|
| RL algorithm | MR.Q | MR.Q | |
| Number of tasks \(|\mathcal{T}|\) | 50 | 50 | |
| Training steps | \(15M\) | \(15M\) | |
| Scheduling interval | 1 episode | 1 episode | |
| Replay buffer size per task | \(\max\left(200k, \frac{2M}{|\mathcal{T}|}\right)\) | \(\max\left(200k, \frac{2M}{|\mathcal{T}|}\right)\) | |
| Upper bound for task selection reward \(r_{\max}\) | \(10.8k\) | - | |
| Discount factor \(\gamma_{\mathcal{T}}\) | \(0.95\) | - | |
| Padding function strength \(\xi\) | \(1\text{e}-4\) | - | |
| Stage 1 budget \(B_1\) | - | \(12.75M\) | |
| Stage 2 budget \(B_2\) | - | \(2.25M\) | |
| \(\kappa\) | - | \(0.7\) | |
| \(K\) | - | \(8\) | |
| Threshold for unsolved tasks \(m\) | - | \(-100\) | |
| Threshold for solved tasks \(M\) | - | \(10k\) | |
| Reset interval | - | No resets |
13 Ablation Studies↩︎
We perform ablation studies to find out why MR.Q works in this application. To evaluate the effect of algorithm components and hyperparameters in the MR.Q algorithm, we train for \(2.4M\) steps in the single task setting with one seed, and measure the accumulated reward over ten evaluation episodes. Although the results of these studies are not statistically robust with only one training seed, we expect the conclusions to be correct, since MR.Q is a robust algorithm with mostly consistent behavior across random seeds and the results are unambiguous.
13.1 Effect of Policy Activation Penalty↩︎
Similarly to the control cost of the reward function, we expected that the activation loss of the policy forces the policy to reduce the magnitude of control commands. Other algorithms do not have this component and learn policies that often result in bang-bang control. However, setting \(\lambda_\text{pre-activ}=0\) does not decrease the performance. The activation penalty is not contributing to the considerable performance improvement of MR.Q in comparison to TD3 and TD7.
13.2 Effect of Model-Based Representation Learning↩︎
| Encoder loss | \(\lambda_\text{Dynamics}\) | \(\lambda_\text{Reward}\) | \(\lambda_\text{Terminal}\) | Return (IQM \(\pm\) interquartile std. dev.) |
|---|---|---|---|---|
| No loss | 0 | 0 | 0 | 2987\(\pm\)277 |
| No reward loss | 1 | 0 | 0.1 | 10455\(\pm\)212 |
| No dynamics loss | 0 | 0.1 | 0.1 | 10630\(\pm\)193 |
| Default | 1 | 0.1 | 0.1 | 10492\(\pm\)181 |
The model-based representation learned by the encoder is the main difference of MR.Q in comparison to TD3 and TD7. Deactivating the loss for the encoder completely, i.e., using random features, results in a performance similar to TD3 and TD7, which confirms that the model-based representation of MR.Q is important for the application. Interestingly, deactivating either the dynamics loss or the reward loss has a negligible effect on the final performance. Note that these results are specific for this application, in which a useful and meaningful reward can be computed in every step. The behavior might be different for environments with sparse reward.
13.3 Effect of Horizon for Multi-Step Returns in Critic Loss↩︎
| Multi-step returns horizon \(H_Q\) | 1 | 3 (default) | 5 | 10 |
|---|---|---|---|---|
| Return (IQM) | 3184 | 10492 | 9250 | 4483 |
As EVE-TD3 [4] uses \(H_Q=30\), we are interested in the effect of the hyperparameter in this application with MR.Q. The horizon controls the bias-variance trade-off in the target value for the action-value function (see also [61] for a discussion in the context of advantage function estimation). The results demonstrate that the multi-step return target for the critic is a main reason for the good performance of MR.Q in this application. Furthermore, MR.Q’s default value of \(H_Q=3\) is also an optimum in this application. Considering the control frequencies of 14 Hz in our controller and 50 Hz in the controller of [4] (control step size of 20 ms), we use a horizon of ca. 0.214 s and [4] use a horizon of 0.6 s in simulation time, which is in the same order of magnitude.
14 Code↩︎
Although we cannot make the code for the simulation and experiments publicly available, the source code for all reinforcement learning algorithms used in our experiments is available at https://github.com/mlaux1/rl-blox (version 0.5.7, JAX version 0.7.0, flax version 0.11.1).
For statistical comparison of RL algorithms, we rely on the open-source library rliable of [39] available at https://github.com/google-research/rliable.
15 Computation Time↩︎
One run with 10 million time steps in the single task setting takes about 48 h with MR.Q and TD7, 24 h with TD3, and 30 h with SAC. In total, we analyzed 10 training seeds and two control architectures (pure RL and additive hybrid) for each algorithm, resulting in \((48 + 48 + 24 + 30) \times 10 \times 2 = 3000\) hours of wall-clock training time to generate the learning curves. However, the runs could be parallelized. In addition, we used another script to run the performance evaluation of the stored checkpoints.
16 Application-Specific Performance Analysis Under Nominal Conditions↩︎
16.1 Tracking Performance Over Full Trajectory↩︎
Figure 8: Comparison of commanded and measured aerodynamic angles for pure RL controller based on MR.Q trained under nominal conditions and baseline controller..
16.2 Tracking Performance at Critical Points↩︎
Figure 9: Comparison of commanded and measured aerodynamic angles for pure RL controller based on MR.Q trained under nominal conditions and baseline controller. Zoomed in at critical points of the trajectory..
16.3 Commands↩︎
Figure 10: Control commands for pure RL controller based on MR.Q trained under nominal conditions and baseline controller..
16.4 Error Distribution↩︎
Figure 11: Distributions of aerodynamic angle errors. The results are obtained for the best policy and 10 evaluation seeds. A kernel density estimation was applied to the sampled errors. Note that density is plotted on a logarithmic scale to highlight the tails. The peak of the baseline controller around \(0\) is more pronounced on a linear scale..
16.5 Distribution of Control Cost↩︎
Figure 12: Distributions of control commands. The results are obtained for the best policy and 10 evaluation seeds. A kernel density estimation was applied to the sampled control commands. Note that density is plotted on a logarithmic scale to highlight the tails. The peak of the baseline controller around \(0\) is more pronounced on a linear scale..
17 Error Percentiles After Training With Dynamics Randomization↩︎
Figure 13: Distribution of aerodynamic angle errors evaluated over 100 test contexts. The overlapping bars correspond to the following percentiles of the distribution: 1, 2, 5, 10, 20, 80, 90, 95, 98, 99..
18 Domain-Specific Performance Analysis↩︎
Evaluation of robustness of controllers with domain-specific metrics. Cell background color indicates best values or values better than the baseline (only if success rate is \(>\)0%). Appendix [sec:app:mt95percentiles] contains more details on the error distributions.
19 Individual Learning Curves Under Nominal Conditions↩︎
Plots throughout the paper show the IQM for the returns, which remove outliers, e.g., failed runs. Here we plot individual learning curves for training under nominal conditions without outlier filtering (experiments from Section 4.1.1). For each training run, we average the performance over ten evaluation episodes. For comparison, we plot the performance of the baseline controller, no control, and the IQM of all training runs. Individual learning curves for MR.Q indicate that the training process is slightly unstable, but all runs surpass the baseline controller at some point.
