1 Introduction↩︎

The rapid modernization of industrial automation has brought forward a critical design paradigm: systems must no longer be optimized solely for operational performance, but must concurrently adhere to strict energy efficiency and sustainability constraints [1]. Traditional control methodologies, while robust, often struggle to adapt to the highly nonlinear dynamics, complex environmental interactions, and shifting objectives present in modern mechatronic systems. Consequently, rl has emerged as a powerful, data-driven framework capable of learning optimal control policies through continuous interaction with an environment [2]. Indeed, recent empirical evaluations have demonstrated that policy gradient methods can offer highly competitive adaptability and rapid response characteristics when benchmarked against traditional Model Predictive Control (MPC) on non-linear mechatronic systems [3].

However, integrating energy efficiency into the rl framework fundamentally transforms the control task into a multi-objective optimization problem. The agent must balance at least two inherently conflicting goals: maximizing primary operational performance, such as precise setpoint regulation, dynamic trajectory tracking, or process throughput, and minimizing the physical energy expended during actuation. Traditional approaches frequently treat energy consumption as a secondary heuristic via fixed, manually tuned reward penalties. While our approach utilizes a similar linear scalarization to combine competing objectives, we address the critical limitation of manual tuning by formulating a framework that automates the discovery of the optimal trade-off surface [4].

In industrial practice, this selection is predominantly achieved through manual tuning. A control engineer intuitively selects a weight, trains a complete rl agent, evaluates the physical performance, and iteratively adjusts the weight based on the observed outcome. This methodology is incredibly time-consuming, highly susceptible to human bias, and frequently fails to uncover the true Pareto front, i.e., the set of optimal solutions where no single objective can be improved without degrading another [5]. Because training an rl agent on physical hardware or high-fidelity simulators is computationally expensive and subjects machinery to physical wear, there is a pressing need for sample-efficient automation of this process.

To address these challenges, this paper proposes an automated framework for modeling the optimal trade-off surface in energy-aware continuous control. Our main contributions are as follows:

• We formulate the multi-objective reward weighting process as a black-box optimization problem and apply mobo to automate the discovery of the Pareto front.

• We conduct a rigorous empirical comparison between the proposed mobo approach and a standard uniform grid search baseline, utilizing a high-fidelity environment of a Quanser Aero 2 mechatronic testbed.

• We demonstrate that mobo achieves superior Pareto front approximations across multiple quantitative metrics (hypervolume, spacing, maximum spread) while drastically reducing the required computational budget and environment interactions.

The remainder of this paper is structured as follows. 2 provides the necessary theoretical background and discusses related work in the fields of multi-objective RL and Bayesian optimization. 3 details the system architecture of the mechatronic testbed and formally defines the reinforcement learning problem. 4 outlines our implementation and evaluation methodology. Finally, 5 discusses the empirical results, followed by concluding remarks and directions for future research in 6.

2 Background and Related Work↩︎

2.1 Multi-Objective Reinforcement Learning↩︎

Standard rl operates on the premise of maximizing a single scalar reward signal, traditionally modeled as a mdp [6]. However, industrial applications inherently involve multiple conflicting feedback signals. As Roijers et al. [4] establish, converting a multi-objective problem into a single-objective one via fixed scalarization is often undesirable when the end-user’s specific performance-versus-energy preferences are unknown a priori. In such scenarios, the objective shifts from finding a single optimal policy to identifying the set of Pareto-optimal policies. Recent literature emphasizes that explicitly multi-objective methods are required to solve complex planning problems efficiently [7]. Let the performance of a policy \(\pi\) be evaluated by a vector of objective functions \(\mathbf{J}(\pi) = [J_1(\pi), J_2(\pi), \dots, J_k(\pi)]^T\). A policy \(\pi^*\) is considered Pareto optimal if there does not exist another policy \(\pi\) such that \(J_i(\pi) \leq J_i(\pi^*)\) for all objectives \(i\), and \(J_j(\pi) < J_j(\pi^*)\) for at least one objective \(j\) [8]. The set of all such non-dominated policy evaluations forms the Pareto front.

2.2 Energy-Aware Control in Industrial Automation↩︎

In modern industrial robotics, balancing system performance, such as precise trajectory tracking or manipulation, with energy efficiency is critical. Traditional approaches frequently treat energy consumption as a secondary heuristic or a simple reward penalty. However, sustainable operation demands that actuation energy is explicitly regulated. Recent frameworks address this by treating the task as a Constrained mdp or a multi-objective problem, seeking to minimize long-term energy budgets while preserving strict operational reliability [9]. Our approach aligns with this paradigm by directly modeling the trade-off between tracking error and power consumption, focusing specifically on automating the discovery of the optimal trade-off surface.

2.3 Bayesian Optimization for Pareto Front Modeling↩︎

To map the Pareto front without exhaustively training rl policies for every possible weight combination, sample-efficient black-box optimization is necessary. bo [10] has become a standard approach for optimizing expensive functions, utilizing a gp [11] surrogate model to approximate the objective landscape and quantify uncertainty. In the multi-objective context (mobo), Gaussian Processes are combined with specialized acquisition functions, such as Expected Hypervolume Improvement (qEHVI) [12], to intelligently select query points that maximally expand the non-dominated solution set. Recent advancements focus on efficiently evaluating Pareto-frontier entropy [13] and selecting diverse batches of parameters to ensure broad coverage [14]. Our work directly applies these mobo principles to automate the hyperparameter weighting of an rl reward function.

3 System Architecture and Problem Formulation↩︎

3.1 The Quanser Aero 2 Testbed↩︎

To rigorously validate our approach, we require a mechatronic system featuring non-linear dynamics and measurable physical energy states. We utilize the Quanser Aero 2 system (1). For the scope of this evaluation, the testbed is configured in a 1-dof setup: the yaw axis is mechanically locked, while the pitch axis remains unlocked and free to rotate.

The system is actuated by two DC motors equipped with propellers. Let \(u_0\) and \(u_1\) represent the voltage signals applied to the respective motors. In the 1-dof pitch configuration, the control input \(u\) is symmetric and defined as \(u = u_0 = -u_1\). The primary output \(y = \Theta\) represents the pitch angle of the main beam. The system dynamics are inherently non-linear due to the aerodynamic drag, friction, and thrust characteristics of the propellers, providing an established, highly representative benchmark for evaluating advanced learning-based control algorithms [3].

3.2 State and Action Space Formulation↩︎

The task assigned to the ppo agent [15] is to orient the beam to a desired, time-varying target angle while minimizing the electrical power consumed by the DC motors. The state space \(\mathcal{S}\) is fully observable and represented as a continuous vector containing the current pitch angle, the angular velocity, and the current target reference angle: \[s_t = \left[ \Theta_t, \dot{\Theta}_t, \Theta^{ref}_t \right]^T\] The action space \(\mathcal{A}\) comprises the continuous voltage signal applied to the motors, bounded by the physical limits of the hardware: \[a_t = [ u_t ] \in [-24\text{V}, 24\text{V}]\]

3.3 Reward Function Design↩︎

The behavior of the agent is entirely dictated by the parameter \(\alpha\) in the scalarized reward function. At each time step \(t\), the reward \(R_t\) is computed as: \[R_t = -(1-\alpha) \cdot |\Delta_t| - \alpha \cdot P_t \label{eq:scalar95reward}\tag{1}\] To ensure stable neural network updates and meaningful scalarization, the terms \(\Delta_t\) (absolute tracking error) and \(P_t\) (power consumption) are normalized to a \([0, 1]\) scale. As demonstrated in our prior investigations regarding the formulation of RL environments, strict normalization of observation spaces and actions is a crucial prerequisite for stable policy convergence and the successful transfer of simulated policies to the physical system [16]. The power consumption \(P_t\) is calculated dynamically based on the applied voltage and the internal resistance characteristics of the Aero 2 motors. By manipulating \(\alpha \in [0, 1]\) in our previous work [17], we observed that the problem spans the entire spectrum from maximum tracking performance to maximum energy conservation, validating its use as the core tunable hyperparameter.

4 Methodology↩︎

4.1 Software and Implementation Details↩︎

The rl environment was built as a high-fidelity digital twin of the physical Quanser Aero 2 system to ensure that all trained policies are highly deployable. This environment utilizes our previously established Python-Simulink co-simulation framework [18], which enables the seamless integration of numerically robust Simulink models with flexible Python-based machine learning environments. The ppo [15] agent was implemented utilizing the Stable Baselines3 framework [19], backed by PyTorch [20] for automatic differentiation and neural network optimization. Both the actor and critic networks feature continuous multi-layer perceptron (MLP) architectures.

4.2 MOBO-RL Integration Workflow↩︎

To automate the discovery of the Pareto front, we integrate the rl training process within an outer-loop optimization framework managed by the BoTorch/Ax library [21]. The optimization is initialized by defining the continuous parameter bounds \(\alpha \in [0.0, 1.0]\) and the two minimization objectives: tracking error (\(f_1\)) and power consumption (\(f_2\)).

The structure of this framework is illustrated in 2. The surrogate model approximates the objective functions using a gp. For each sequential trial, the optimizer queries this surrogate model to suggest the next most promising \(\alpha\) weight utilizing the qnehvi acquisition function. A fresh ppo agent is instantiated and trained to convergence using this specific scalarized reward formulation. Following training, the resultant policy \(\pi\) is deployed in a deterministic evaluation environment to record the true tracking error and electrical power draw. Finally, these empirical results are passed back to the optimizer via the API, dynamically updating the posterior distribution of the surrogate model and informing the subsequent iteration.

4.3 Exploration Strategies↩︎

To approximate the Pareto front, we enforce a strict, heavily constrained maximum budget of 11 full rl training runs (evaluations). This budget simulates the real-world constraints of hardware training times. We compare two distinct strategies for exploring the parameter space of \(\alpha\):

• Uniform Grid Search (Baseline): We manually select 11 equally spaced values for \(\alpha \in [0, 1]\). While providing simple, uniform coverage of the parameter space, this method is fundamentally inefficient as it cannot adapt to the underlying non-linear mapping between \(\alpha\) and the bi-objective performance.

• Multi-Objective BO (MOBO): To prevent premature convergence and provide a baseline statistical variance, the mobo optimization is initialized with 5 quasi-random Sobol points. This is followed by 6 sequentially guided bo selections. The total remains exactly 11 trials to ensure a mathematically fair comparison against the baseline.

4.4 Quantitative Evaluation Metrics↩︎

We quantitatively assess the quality of the generated Pareto fronts using three standard multi-objective metrics. Let \(\mathcal{P}\) be the approximated Pareto front.

1. Hypervolume (HV): Evaluates the total dominated n-dimensional volume in the objective space between the Pareto solutions and a predefined, worst-case reference point \(\mathbf{r}\). \[\text{HV}(\mathcal{P}) = \Lambda \left( \bigcup_{\mathbf{x} \in \mathcal{P}} [f_1(\mathbf{x}), r_1] \times [f_2(\mathbf{x}), r_2] \right)\] where \(\Lambda\) denotes the Lebesgue measure. A larger volume indicates a front that is closer to the true Pareto optimal set and possesses a better spread.

2. Spacing (\(S\)): Measures the uniformity of the solution distribution along the front. It is defined as the variance in the Euclidean distance between adjacent solutions: \[S = \sqrt{\frac{1}{n-1} \sum_{i=1}^{n} (\bar{d} - d_i)^2}\] where \(d_i\) is the distance to the nearest neighbor for the \(i\)-th solution, and \(\bar{d}\) is the mean of all \(d_i\). Lower values indicate a more evenly distributed set of choices.

3. Maximum Spread (\(MS\)): Assesses the maximal range coverage by calculating the Euclidean distance between the extreme boundary points of the discovered front in the objective space. Let \(K\) be the number of objectives. The maximum spread is defined as the Euclidean norm of the objective ranges: \[MS = \sqrt{\sum_{k=1}^{K} \left( \max_{\mathbf{x} \in \mathcal{P}} f_k(\mathbf{x}) - \min_{\mathbf{x} \in \mathcal{P}} f_k(\mathbf{x}) \right)^2}\] A higher \(MS\) value indicates that the algorithm successfully explored the absolute outer limits of the performance trade-off, providing a broader spectrum of operational modes.

5 Results and Discussion↩︎

5.1 Quantitative Results↩︎

1 summarizes the final performance of both exploration strategies after the strict budget of 11 evaluations was exhausted. The automated mobo approach consistently outperforms the uniform grid sampling across all three defined multi-objective metrics.

The higher Hypervolume demonstrates that mobo discovered policies that objectively dominate those found by the uniform grid, as visualized in 3. Crucially, the non-linear relationship between the weight \(\alpha\) and the actual physical system performance causes a uniform grid search in the parameter space to map to a highly non-uniform cluster of points in the objective space. As observed in our experiments, the Grid Search wasted a significant portion of its 11 evaluations on adjacent \(\alpha\) values that resulted in similar physical system behavior, largely failing to map the critical “knee” of the Pareto front where the most viable trade-offs reside.

5.2 Sample Efficiency and Progression↩︎

The most significant and practical advantage of the mobo approach is its remarkable sample efficiency, which directly translates to saved computational time and reduced hardware degradation during physical training. Analyzing the progression of the hypervolume (4), mobo rapidly identifies high-quality solutions. Because the qEHVI acquisition function explicitly targets areas of the objective space that maximize expected improvement, it intelligently avoids redundant regions of the parameter space.

Remarkably, the mobo approach achieves the final, maximum hypervolume of the 11-trial Grid Search baseline in just 4 evaluation trials. In a real-world industrial context, where training a single policy to convergence might take hours or days of continuous operation, matching baseline performance in 4 trials represents a reduction in computational and temporal cost of approximately 64%.

Similarly, the maximum spread progression (5) highlights the capacity of bo for active boundary exploration. The algorithm matched the grid search’s maximum spread by trial 8. This proves that mobo does not merely exploit a single high-performing region, but actively pushes the boundaries of the known objective space. It ensures that fundamentally diverse trade-off policies, spanning from ultra-precise, high-energy trajectory tracking to loose, highly conservative energy-saving modes, are systematically discovered and made available to the control engineer without requiring tedious manual intervention.

6 Conclusion and Future Work↩︎

This work rigorously validates Multi-Objective Bayesian Optimization as a highly effective, sample-efficient, and automated method for multi-objective weight selection in industrial rl continuous control tasks. By successfully balancing electrical energy consumption and tracking performance on the Quanser Aero 2 testbed, we demonstrated that mobo fundamentally eliminates the biases and inefficiencies of manual heuristic weighting. The approach consistently creates a superior approximation of the Pareto front while significantly reducing the required evaluation budget, marking a critical step toward accessible, sustainable intelligent automation.

Future work will focus on extending this methodology to the full, highly coupled 2-dof (pitch and yaw) configuration of the Quanser Aero 2 to handle more complex trajectory tracking tasks. Furthermore, because exploring the boundaries of the weight parameter space, specifically where tracking performance is heavily prioritized over energy conservation (\(\alpha \to 0\)), can lead to aggressive, potentially hardware-damaging “bang-bang” policies, we intend to integrate strict Safe RL constraints into the optimization framework. This ensures that boundary exploration remains physically safe during early training phases. Finally, we will investigate the transferability of these findings, utilizing the identified Pareto optimal weights from the Aero 2 to jump-start and accelerate rl training processes on similar, yet mathematically distinct, industrial mechatronic systems.

Acknowledgments↩︎

Financial support for this study was provided by the Christian Doppler Research Association (CDG) through the Josef Ressel Centre for Intelligent and Secure Industrial Automation, the corresponding WISS Co-project of Land Salzburg, and by the European Interreg project BA0100172 AI4GREEN.

References↩︎