1 Introduction↩︎

Reinforcement learning (RL) [1] builds decision-making policies by trial and error to optimize rewards associated with a control task. It substantially advanced in the last decade by the integration of deep neural networks into training; such deep RL approaches have managed to learn to play computer games at a human level [2] and reached superhuman performance in complex board games, like Go [3]. Deep RL has also ventured into areas where reward-based task specifications are simpler than alternative formulations. Recent applications include fine-tuning LLMs [4], learning policies for control problems in the natural sciences [5], and use in several autonomous systems [6].

Despite this success, the application of deep RL is hampered by the opacity of learned control policies, which are difficult to verify and understand. This has led regulators to require greater transparency in AI systems used in critical domains, e.g.the [7] and the [8]. For example, the EU AI Act specifies in Article 13.3 “The instructions for use shall contain [...] technical capabilities and characteristics of the high-risk AI system to provide information relevant to explain its output.” Explainable RL (XRL) methods [9] address this by providing human-interpretable explanations of RL policies. Explanations are often symbolic models, such as decision trees [10], which can be verified when sufficiently small, paving the way for deploying RL in safety-critical domains. However, much XRL work targets tasks with discrete control actions, as in early deep RL applications with discrete (yet very large) state spaces [2], [3].

In this paper, we address the challenge of learning small, interpretable surrogate models of RL policies with continuous action spaces and continuous-discrete state spaces. We propose Orcaid(Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies) to construct rule-based models from RL policies. Orcaid first partitions the state space by learning an oblique decision tree [11] (extended for discrete/categorical features). Unlike axis-aligned trees, oblique trees use hyperplanes that are linear combinations of features, allowing them to capture non-orthogonal boundaries compactly. As illustrated in Figure 1, an oblique decision tree can approximate a curvilinear boundary with much fewer splits than an axis-aligned decision tree.

To model continuous actions, we fit linear regressions for each action dimension inside every state-space region (leaf). We then simplify the learned tree by merging similar regions of the state space and using backward elimination to drop unnecessary features. This process generates oblique decision rules [12], known to be explainable [13], extended to produce continuous outputs via a linear model over features in each rule’s head. Experiments on nine control tasks show that Orcaid models outperform SOTA baselines by more accurately representing the original policy (fidelity), maintaining task-solving capability (performance), and achieving a smaller model size (complexity).

Our contributions can be summarized as follows:

2 Related Work↩︎

Our work focuses on explainable reinforcement learning (XRL) through decision trees and rules. Milani et al. [9] categorize XRL methods into various ways. We specifically target post-hoc feature importance explanations, aiming to interpret actions based on the current state.

VIPER [10] is an example in this category, generating Decision Trees (DTs) through imitation learning. However, VIPER and its multi-agent extension MAVIPER [14] use the CART algorithm [15] that generates DTs with axis-aligned predicates, yielding large models, and are limited to Q-value-based policies. Recently, Dhebar et al. [16] have investigated the use of DTs with nonlinear predicates using evolutionary search, but they consider only discrete actions. The same limitation also appears in [17] on policy extraction using oblique DTs, [18] on differentiable DTs for XRL, [19] on soft DTs, and [20] on IF-THEN fuzzy rules for XRL.

Similarly to us, [21] considered continuous states and actions, but using genetic programming to derive interpretable policies in the form of loop-free programs, and limited to simpler RL problems. [22] also proposed learning rules in continuous states. However, they only considered discrete actions and axis-aligned predicates. In contrast, our Orcaid approach supports mixed continuous-discrete states and continuous actions.

Additionally, some XRL methods provide insight into the temporal behavior of RL policies in relation to state transitions, either in the environment [23]–[25] or within policies using memory [26], [27]. We only focus on non-temporal explanations in this paper. Another area of research considers learning inherently interpretable policies [28], whereas we aim to explain opaque policies.

3 Preliminaries↩︎

3.0.0.1 Reinforcement learning

is a machine learning approach where an agent learns to maximize cumulative reward through interaction [1]. Deep RL with neural networks yields powerful but opaque policies, challenging interpretability (e.g., Go [3]). We formalize continuous control as an MDP \(\langle \mathcal{S}, \mathcal{A}, p, \rho_0, r, \gamma\rangle\) [29] with state space \(\mathcal{S}\subseteq \mathbb{R}^n\), action space \(\mathcal{A}\subseteq\mathbb{R}^m\), transition density \(p(\mathbf{s}'|\mathbf{s},\mathbf{a})\), initial state density \(\rho_0(\mathbf{s})\), reward \(r\), and discount factor \(\gamma\). The goal is to learn a policy \(\boldsymbol{\pi}: \mathcal{S}\to\mathcal{A}\) maximizing the expected discounted return \(\mathbb{E}\bigl[\sum_{t=0}^\infty \gamma^t\,r(\mathbf{s}_t,\mathbf{a}_t)\bigr]\). We use deep deterministic policy gradient (DDPG) [30] to learn policies in our experiments.

3.0.0.2 Explainability.

We focus on explainable RL (XRL), which aims to create explanations for the decisions made by RL policies. Specifically, we focus on post-hoc explanations, especially through feature importance [9] methods. Post-hoc methods generate interpretable surrogate models that mimic non-interpretable policies, emphasizing the link between individual states and actions as opposed to long-term behavior and training influences.

We assume a given policy \(\boldsymbol{\pi}\,{:}\, \mathcal{S} \,{\rightarrow }\, \mathcal{A}\) encoded by a deep neural network. The goal is to learn a small surrogate \(\boldsymbol{\hat{\pi}}\,{:}\,{\mathcal{S}}\,{\rightarrow}\, \mathcal{A}\) producing actions close to those of \(\boldsymbol{\pi}\). As common [16], [17], [28], we measure model complexity by parameter count to assess interpretability. Closeness between actions (fidelity [9]) is measured by an error function \(\mathcal{L}(\boldsymbol{\pi}(\mathbf{s}), \boldsymbol{\hat{\pi}}(\mathbf{s}))\), like mean-squared error (MSE). We also assess performance [9], which refers to how well the generated surrogate model solves the task.

As we consider finite-horizon tasks (\(H\,{\in}\,\mathbb{N}\) steps), we measure performance by the undiscounted return \(G_\pi = \sum_{t=0}^H r_t\); hence the ratio \(G_{\boldsymbol{\hat{\pi}}}/G_{\boldsymbol{\pi}}\) should be close to \(1\). To demonstrate actionability, we adapt the Legible approach [33] from discrete to continuous environments, effectively showing that Orcaid rules provide actionable insights to improve RL policies. Furthermore, we use LLM-as-a-Judge [34] as an extra way to assess the explainability of \(\boldsymbol{\hat{\pi}}\). Finally, complementing this work, an interactive tool is developed to help explain Orcaid models (App.  9 ).

3.0.0.3 Rule mining

discovers IF-THEN rules that identify patterns (descriptive) or make predictions (predictive) [35]. Various approaches to rule mining exist, like covering algorithms [35] and tree-based methods. RuleFit [36] is an example of the latter, which uses decision trees to generate rule predicates and then incorporates them as binary features into a single, global linear model with the original features. We use trees (albeit oblique) to discover predicates that define regions within the state space and fit separate linear models per region using only the original features.

4 Orcaid Rule Mining↩︎

Orcaid learns a rule-based surrogate model that approximates the decision-making behavior of an RL policy over continuous state and action spaces. It operates in three main phases, shown in Fig. 2:

We further embed these steps into a DAgger (Dataset Aggregation) [37] loop to improve accuracy. Ablation results (App. 13) also justify our approach by analyzing the effect of isolating each component’s effect and varying hyperparameters.

4.0.0.1 Method Outline.

Given an RL policy \(\boldsymbol{\pi} : \mathcal{S} \rightarrow \mathcal{A}\) over continuous states \(\mathcal{S} \subseteq \mathbb{R}^n\) and actions \(\mathcal{A} \subseteq \mathbb{R}^m\), Orcaid computes a policy \(\boldsymbol{\hat{\pi}}: \mathcal{S} \rightarrow \mathcal{A}\) that approximates \(\boldsymbol{\pi}\) consisting of rules \(r_i\) of the form \(c_i \Rightarrow \boldsymbol{\hat{\pi}}_i\), where \(c_i\) is a triggering condition from the set \(C(\mathcal{S})\) of Boolean formulas over linear inequalities over states, and \(\boldsymbol{\hat{\pi}}_i\) is an action proxy from the set \(F(\mathcal{S},\mathcal{A})\) of affine functions from \(\mathcal{S}\) to \(\mathcal{A}\). The conditions \(c_i\) collectively cover \(\mathcal{S}\), and Orcaid aims to minimize both the approximation error \(\mathcal{L}(\boldsymbol{\pi},\boldsymbol{\hat{\pi}})\) and the model size of \(\boldsymbol{\hat{\pi}}\), measured by the number of parameters in the rules \(r_i\).

4.1 Data collection↩︎

The initial data collection involves the following three steps.

Clustering accelerates computation, and sampling helps prevent overfitting by covering more of the state space. This makes the surrogate model more robust to errors that accumulate during execution. In the experiments, we observed that small noise (viz. \(\sigma=0.15\), see App. 11) was enough to obtain good results without deviating too much from the original trajectories.

Random state space sampling to collect data could be an alternative, but it scales poorly with dimension. Instead, we focus on regions frequently visited by the RL agent and their neighborhoods, i.e., the relevant states for task performance.

4.2 Region Definition↩︎

Orcaid partitions the state space via oblique decision trees (i.e., using hyperplanes as branching conditions), which are then merged to form disjoint regions. To formalize this, we first introduce a condition, which is a hyperplane inequality that serves as a branching condition.

By imposing constraints on the weights, the same linear form can represent three kinds of splits:

If the binary features originate from one‑hot encoding a categorical variable, case 2 checks that the category is not the one represented by feature \(i\), and case 3 checks that it is that category. For a native binary variable (already 0/1), the same logic applies directly.

We now introduce regions. Each leaf of an oblique decision tree defines a simple region induced by the conditions at its ancestor nodes. Formally, simple regions are defined as follows.

That is, \(\mathbf{x}\,{\in}\,R_C\) iff \(\mathbf{w}\,{\cdot}\,\mathbf{x}\,{\leq}\,b\) holds for each condition \(c_{\mathbf{w},b}\) in \(C\). For merging leaves, we need more expressive regions.

Thus \(\mathbf{x} \in R\) iff for some \(R_{C_i}\) forming \(R\), \(\mathbf{w}\cdot \mathbf{x}\leq b\) holds for each condition \(c_{\mathbf{w},b}\in C_i\). Orcaid efficiently assigns points to regions via matrix multiplication, thanks to the structure of the conditions.

4.3 Tree building with hyperplane optimization↩︎

Given a set of states \(X \in \mathbb{R}^{N \times n}\) and corresponding multi‑dimensional actions \(Y \in \mathbb{R}^{N \times m}\), the oblique tree recursively partitions the input space with hyperplanes, which is built using Algorithm 3. In each leaf, a pruned linear model is stored for every action dimension. The quality of a split is measured by the weighted mean squared error after fitting separate ordinary least‑squares (OLS) regressions in each child region. Each recursive step includes: (1) dimensionality reduction and scaling to handle high-dimensional state space, (2) random hyperplane initialization to find a good partitioning fast (warm start), and (3) local split optimization to partition states optimally.

Algorithm 3 recursively partitions the data until a stopping criterion is met – reaching max_depth, having fewer than min_spl samples (line 2), or achieving a low mean MSE by an OLS model (line 5–6) – and a leaf node is created. The algorithm creates branch conditions using FindSplit (line 9), which searches for a hyperplane \((\mathbf{w},b)\) minimizing the weighted split MSE. The data is partitioned via \(\mathbf{w}\cdot\mathbf{x}_i \le b\), and BuildNode recurses on each side (lines 10–13). The search proceeds in three stages in a reduced PCA space to improve numerical stability and speed.

Leaf construction (LeafNode). For every action dimension \(j\), an OLS model is fitted (line 37). Backward elimination then removes features whose absolute coefficient is smaller than \(\epsilon\) or whose removal increases the MSE by less than \(\delta\) (lines 39–40). The pruned model is stored, and the leaf node returns the set of all \(m\) linear models (line 43).

4.4 Simplification↩︎

After constructing the oblique decision tree, we merge adjacent regions to create a simplified rule-based model. Each rule body defines a state-space region in which it triggers, and the corresponding head defines the actions via linear models. This gives us three major benefits: (1)reduced complexity: by decreasing the number of regions, we reduce the number of linear regressions, leading to fewer parameters; (2) improved interpretability: a smaller set of regions makes the surrogate model easier to analyze; and (3) diminished overfitting: merging similar regions discourages overfitting and improves generalization. For region merging, we use the notion of active condition.

This definition also applies to binary features. Any leaf \(R_C\) has at least one active condition in its \(C\) (the last split), and can have all of them active. We next define when two simple regions are adjacent.

Sibling leaves \(R_{C_1}\) and \(R_{C_2}\) (i.e., having the same parent \(R_C\)) are always adjacent, as they differ only by the split condition at \(R_C\). Non-sibling leaves can also be adjacent; this happens when the split condition is active in both regions.

After identifying adjacent regions, we evaluate potential merges in an iterative procedure. For each pair \(R_i\) and \(R_j\) of adjacent regions, we fit single linear regressions for each target variable on \(R=R_i \cup R_j\) and compute its mean MSE, as well as the MSE of the new and the original regressions on the subparts. Adjacent regions \(R_i\) and \(R_j\) can be merged if (1) \(\text{MSE} < \tau\) (global error bound), (2) \(\text{MSE}_{i} < \tau\) and \(\text{MSE}_{j} < \tau\) (local error bound), or (3) \(\text{MSE}_{k}\,{<}\,(1\,{+}\,\epsilon) \,{\cdot}\,\text{MSE}_k^{\text{orig}}\), \(k\,{=}\,i,j\) (rel.improvement), where \(\epsilon=0.05\) and \(\tau=2\cdot\)max_mse. The resulting merged region is the union of \(R_i\) and \(R_j\). This process repeats until no further merging is possible.

After region merging, each resulting region \(R\) is defined by one or more sets \(C_i\) of conditions that define a simple region for a leaf node. In logical terms, the definition of \(R\) amounts to a disjunction of conjunctions of conditions, i.e., a disjunctive normal form (DNF) over the set of conditions. Hence, we can simplify the merged region definitions using the QM method [39], [40]. Furthermore, for the final linear regressions, we perform backward elimination to remove irrelevant features.

The following definition is used for a fair complexity comparison of different surrogates. We measure complexity with two components: what it takes to (i) get to a decision (path conditions) and to (ii) make the decision (action logic cost).

4.5 Iterative Improvement via DAgger↩︎

To remedy the accumulation of errors between the original deep RL policy and rules, we iteratively improve rules through DAgger (Dataset Aggregation) [37] by the following steps (cf.Appendix  7 ):

This repeats until performance stops improving or Orcaid’s actions and those of the RL agree.

5 Experiments↩︎

We empirically evaluate Orcaid on nine control problems, comparing it to (1) standard decision trees (DT) learned using CART [15] implemented in scikit-learn [41], (2) rules learned with Cubist [42], [43], and (3) RuleFit [36]. We also combine each baseline with DAgger. For DTs, this setup is effectively a Q-value-free variant of VIPER [10], including iterative aggregation and learning, but replacing Q-weighted supervision with policy labels to support general deterministic RL policies.

Our research questions are the following:

For RQ1 and RQ2, we compare the average cumulative reward and MSE of each surrogate and contrast it with its model size (Def. 7). For RQ3, we adapt Legible [33]. RQ4 is related to RQ1, as model size is often used for interpretability [44]. Human evaluation can provide further insights into comprehensibility. However, it is very costly, audience-specific, and potentially biased. We thus follow the LLM-as-a-Judge framework [34] to complement model size with a comprehensive LLM-based evaluation of interpretability.

5.0.0.1 Experimental Setup & Hyperparameter Selection.

We conducted all experiments on a computing cluster with four NVIDIA A100-SXM4-40GB GPUs and two AMD EPYC CPUs. Every RL policy was trained with 3 different seeds, using the DDPG implementation of SB3 [45] (cf.App.  11 Table  4 ). Every surrogate was also trained with 3 different seeds. Thus, every result for RQ1 – RQ3 is the average of 9 runs. For RQ1, performance was evaluated by running 1,000 episodes per model.

For a fair comparison, we use identical sampling hyperparameters across all surrogates and tuned each surrogate’s main hyperparameter: tree depths for Orcaid and DT, number of rules for Cubist and RuleFit, and additionally the max_mse for Orcaid. App. 12 lists all results for all surrogate model sizes, and App.  11 , Table 5, all the sampling hyperparameters, and Orcaidmax_mse.

We evaluate Orcaid and the baselines in nine control problems [32]: from simple like Mountain Car (****MC****), to complex like Hopper (****H****) and Half Cheetah (****HC****). Lunar Lander (****LL****) was a case of mixed continuous-discrete state space. The other problems are Pendulum (****P****), Inverted Pendulum (****IP****), Inverted Double Pendulum (****IDP****), Reacher (****R****), and Swimmer (****S****).

r0.59

5.1 Empirical Results↩︎

For each environment, we learn Orcaid and DT models with increasing maximum depth and increasing maximum rules for RuleFit and Cubist, selecting the best-performing model for each surrogate until test MSE stops improving (cf. App.  12 for all results). As examples of approximated training times, for MC, DT takes \(5\)s, Cubist \(40\)s, RuleFit \(10\)s, and Orcaid\(20\)s. For HC, DT takes \(15\)s, Cubist \(22\)min, RuleFit \(2\)min, and Orcaid\(30\)min. These values also consider the time spent on model evaluation for DAgger.

Model size for baselines aligns with Def. 7 by summing (1) path conditions and (2) action logic cost. For DTs, (1) #features\(\times\)#leaves, (2) depth per leaf. For Cubist: analogous to Orcaid, but with single-feature conditions and one model per action dimension. For RuleFit, with one model per action dimension, (1) combined length of all rules, (2) #coefficients.

5.1.0.1 Performance.

Fig. [fig:plot95ratio95model95size] addresses RQ1 by contrasting performance with model size. Performance is measured by the average ratio between the cumulative reward of each surrogate and the RL policy. Orcaid performs better than or equal to the other surrogates, with RuleFit and DT being far less performant, sometimes failing to reach \(75\%\). For IDP, this can be due to the high impact of small deviations in actions on the stick’s stability, which also affects Cubist, leading to a high variance in its reward ratio. Orcaid’s model size is much smaller than DT’s (except in HC, where performance differs substantially) and smaller than RuleFit’s when performance is similar. While Cubist shows competitive ratios across most environments, even better for HC(non-significant, by the confidence intervals), its size is consistently larger than Orcaid’s (note the log scale). Overall, for RQ1, Orcaid learns smaller surrogate models with strong performance.

5.1.0.2 Fidelity.

To assess fidelity, we computed the MSE on test data (Fig. 4). Similar to performance, Cubist learns a surrogate that is closest to Orcaid. The other surrogates usually have higher MSEs, deviating more from the RL policy. We thus can answer RQ2 affirmatively, i.e., explanations derived by Orcaid closely represent the RL policy with smaller model sizes. In contrast with Fig. [fig:plot95ratio95model95size], lower MSE does not necessarily mean higher performance due to potential model drift, where small deviations compound over time and lead to different trajectories.

5.1.0.3 Policy Improvement.

We adapt Legible[33] to continuous state and action spaces. Legibletransforms rules to related situations, e.g., symmetric situations, then locally overrides policy decisions with selected transformed rules, yielding a new policy \(\pi'\). If \(\pi'\) achieves a higher return than \(\pi\), it exposes a weakness in \(\pi\) and improves it. This testing approach may be ineffective for perfect policies, but as in model-based software testing, the quality of surrogate models matters; we therefore study which surrogates are most effective.

To test for policy improvements, we (1) transform rules produced by surrogates by mirroring them along selected axes, and (2) enforce them by overriding RL policy decisions to identify weaknesses. We confine to symmetry, as this is the most relevant property in continuous environments. App. 8 includes the formalization of transformations, the complete results and an extra analysis.

Every surrogate was evaluated using four mirroring strategies (1 or 2 mirrored regions for 2 axes, and 1 mirrored region for 1 or 4 axes) with a random selection of \(50\) transformations for each strategy. Table 1 shows how often the transformations led to better performance (Rate) and the maximum relative improvement w.r.t. the RL policy (Best). For Cubist and RuleFit, only single-action environments can be evaluated, since they build one model per action, which prevents mirroring across regions that contain different models. RL policies for Pand MCare optimal and do not improve; thus, they are excluded. For the other environments, Orcaid-based testing finds weaknesses more consistently than other methods, and the maximum improvement is higher or comparable.

We can thus answer RQ3 positively as we could use Orcaid models to improve the RL policies – in general, more effectively than with other models. Furthermore, as Orcaid models are smaller, their mirrored rules contain fewer parameters. This makes it easier to understand both the mirrored region and how to improve the RL policy.

5.1.0.4 Interpretability.

As shown in Fig. [fig:plot95ratio95model95size], Orcaid models are smaller, strongly suggesting better interpretability. As a further interpretability analysis, and leaving a human survey for future work, we adopted the LLM-as-a-Judge framework. Based on the principles and checklist of [46], we selected 7 explainability questions relevant to feature-importance methods beyond performance, added 8 domain-specific questions for rule- and DT-based control models, and a final verdict question. We omitted performance values so LLMs focused solely on explainability. Six LLM judges rated each surrogate on a 1–5 Likert scale using a rubric, with thinking mode enabled when available. We compared Orcaid with DTs, which are typically considered interpretable, and Cubist, its closest competitor from earlier results. Table ¿tbl:tab:results95llms? reports average scores and verdict rates across the 6 judges and 15 questions (details in App.  10 ). Orcaid outperformed DTs given comparable performance, and outperformed Cubist in all environments except H, where they tied due to its small model (20 rules per action dimension), but Cubist showed worse performance. We thus have evidence for RQ4 that Orcaid rules are usually more interpretable than other methods with comparable performance.

6 Conclusion↩︎

We presented Orcaid, the first XRL method for constructing interpretable rule-based surrogate models of deep RL policies with continuous actions over mixed continuous-discrete state spaces using oblique decisions. Empirically, Orcaid achieves high fidelity and strong task performance while using few parameters, and it is typically more interpretable than standard DT (effectively a Q-value-free variant of VIPER), Cubist, and RuleFit at comparable performance.

Discussion. Our current evaluation spans tasks up to \(17\)-dimensional state and \(6\)-dimensional action spaces (HC); further scaling may require dimensionality reduction or feature selection. We model deterministic policies and collect data via environment queries (DAgger); this can limit fidelity for stochastic policies and risks model drift without query access, although the core oblique-tree learning and simplification steps remain applicable. Extending leaves with independent linear-Gaussian models per action dimension (diagonal covariance) is a natural next step toward supporting stochastic policies. Beyond quantitative metrics, we used an LLM-as-a-Judge protocol to assess interpretability; establishing agreement with human judgments remains important future work.

Our experiments suggest that rule models can match or improve the original policy, so they may be further improved by training them in the environment, i.e., re-optimization [16]. Such models could be deployed in place of deep RL policies while offering transparency. Finally, we plan to compose learned rules with environment models to verify compliance with requirements based on [47].

Appendix	Title	Page
7
8
9
10
11
12
13

7 Dagger-like Iterative Rule Learning↩︎

Based on the Dataset Aggregation (DAgger) approach, we iteratively refine the model by collecting new data points that are annotated with the RL policy. We filter in the points at which the distilled model deviates from the original RL policy. This way, we iteratively build a policy that tries to learn from previous errors.

Algorithm 5 consists of two main stages for each iteration:

7.1 Key Components↩︎

7.1.1 Trajectory Collection↩︎

This generates a trajectory of observation-actions triples: \(\{\dots, (o_t, a^{M_j}_t, a^{\pi}_t), \dots\}\).

7.1.2 Data Filtering and Augmentation↩︎

To ensure we do not add unnecessary data points to the training dataset, we include these steps:

7.1.3 Model fitting↩︎

After adding the new data points to the training dataset, the model is fitted to this new dataset. The retraining process is designed to be efficient.

Instead of training from scratch until depth \(j\), the algorithm starts from the previous model of depth \(j-1\) by extending it by one depth. It extends that model, optimizing oblique splits as described in Section 4.3. This increases the tree’s depth to \(j\). This only applies to Orcaid.

7.1.4 Termination↩︎

The refinement loop contains an early stopping mechanism. After each refinement attempt, the average reward of the new model \(M'_j\) is estimated.

8 Transformations for Policy Improvement↩︎

8.1 Method for Transformation-based Improvement & Evaluation↩︎

In our evaluation for RQ3, we present an application of Orcaid-generated rules to evaluate and improve reinforcement learning (RL) policies. Following the approach proposed for rules with discrete conditions in [33], we generate alternative rules \(\mathcal{R}'\) by adapting rules \(\mathcal{R}\) learned from a given policy. By enforcing \(\mathcal{R}'\), that is, by overriding the decisions of a policy \(\pi\) according to \(\mathcal{R}'\), we can identify weaknesses in \(\pi\): if enforcing \(\mathcal{R}'\) leads to a higher return, this indicates that \(\pi\) makes suboptimal decisions in the corresponding situations. The alternative rules \(\mathcal{R}'\) capture counterfactual decisions, which we generate by exploiting domain knowledge, such as known symmetries in the environment.

This application further enhances the explainability provided by Orcaid. Instead of solely constructing a rule-based surrogate model, Orcaid-guided policy evaluation identifies a concise set of rules that pinpoint situations in which the evaluated policy makes suboptimal decisions. While a complete formalization of this evaluation approach is beyond the scope of this paper, we focus on exploiting domain symmetries by mirroring rules across selected axes.

8.1.0.1 Rule Adaptation for Orcaid.

Let \(c_i \Rightarrow \hat{\pi}_i\) be a rule \(i\), where \(c_i\) is a region and let \(MI \subseteq \{1,\ldots,n_c\}\) be a set of indices. We define \(\mathrm{\small Mirror}(c_i \Rightarrow \hat{\pi}_i,MI) = m_i \Rightarrow \hat{\pi}_i\) as function mirroring the conditions \(c_i\) across the axes given by \(MI\). To this end, Mirror inverts the coefficients for these axes, i.e., for every \(c_{\mathbf{w},b}\) in \(c_i\), we create \(c_{\mathbf{w'},b}\) such that for every \(m\in MI\): \(\mathbf{w'}[m]\) = \(-\mathbf{w}[m]\). We do not invert \(\hat{\pi}_i\) because we assume that if symmetries exist in an environment, the same linear model captures the required dependencies between states and actions. It suffices to mirror the conditions and apply the same local policy. Note that we assume that \(MI\) does not contain discrete features, because mirroring one-hot-encoded features generally does not make sense.

8.1.0.2 Rule Adaptation for DTs.

For DTs, we consider individual paths from the root to a leaf. The function \(\mathrm{\small MirrorDT}((C,l),MI)\) takes a list of axis-aligned conditions \(C\) as input and a leaf-value \(l\). It mirrors every \(c \in C\) that applies to a feature with an index in \(MI\), by inverting the relation and comparison value of \(c\). Additionally, it multiplies \(l\) by \(-1\), as DTs predict a constant value rather than using a linear function.

8.1.0.3 Rule Adaptation for CUBIST.

For CUBIST, we combine the above transformation approaches. Like DT, CUBIST uses axis-aligned conditions; therefore, we mirror them analogously. As for Orcaid, we leave the linear model for action prediction unchanged.

8.1.0.4 Rule Adaptation for RuleFit.

RuleFit uses axis-aligned rules, which we mirror in the same way as paths in DTs. However, RuleFit rules cannot be used on their own. For this reason, we create a tuple \((C,R)\) from mirroring RuleFit rules, \(C\) is a list of mirrored conditions, and \(R\) is a copy of the original RuleFit models transformed by replacing one list of rules with the mirrored list \(C\).

8.1.0.5 Mirrored Rule Selection.

Given a surrogate model, we generally cannot exhaustively mirror and test all possible rule-axis combinations, especially considering that multiple rules should be mirrored at once along multiple axes. For this reason, we apply a random selection approach parameterized by \(rs\), \(as\), and \(nm\), where \(rs\) specifies how many rules we select and mirror at once, \(as\) specifies along how many axes we mirror at most, and \(nm\) specifies the overall number of selections. For instance, if \(rs = 2\) and \(as = 2\), and \(nm = 50,\) we randomly select \(50\) pairs of rules and mirror them along \(2\) or \(1\) randomly selected axes.

Note that for DTs, we consider paths in the tree, and we ensure that all selected axes-rule combinations make sense, i.e., that a rule includes conditions on the selected axes. While we use both DTs and rules, from now on, we simply refer to sets of rules. Let \(\mathcal{MR}\) be a set of \(nm\) mirrored rule sets.

8.1.0.6 Enforcing Rules.

Given an RL policy \(\pi\), and a rule set \(\mathcal{R} \in \mathcal{MR}\), and state \(s\), we define \(\mathrm{\small Enforce}(\pi,\mathcal{R},s)\) as follows: if there is an \(r\in \mathcal{R}\) that triggers in \(s\), i.e., its conditions evaluate to true, we \(r\) to predict the action value and return it from Enforce. Otherwise, we return \(\pi(s)\). If several rules trigger, we select one of them at random. In the case of RuleFit, we check for each rule if the corresponding conditions \(C\) trigger and apply the transformed model \(R\) if they do.

Note that by the rule construction for Orcaid above, mirrored rule conditions do not overlap, which is harder to guarantee for the other models.

8.1.0.7 Rule-based Evaluation & Improvement.

We propose to combine, mirror, and enforce sets of rules to determine regions where the policy under consideration could be improved. To implement this form rule-based evaluation, we first select \(\mathcal{MR}\) using the random selection described above. Then, we execute \(n_{eval}\) episodes with \(\pi\) and without any interference to collect samples of the average undiscounted return \(G_\pi\). That is, we determine a baseline for the return.

After that, for every \(\mathcal{R} \in \mathcal{MR}\), we perform another \(n_{eval}\) episodes with \(\pi\) while enforcing the rules in \(\mathcal{R}\) via Enforce to collect samples of \(G_{\pi,\mathcal{R}}\). Overall, we get \(|\mathcal{MR}| = nm\) sets of samples. Based on these, we statistically test if and how many \(\mathcal{R}\) improve the RL policy \(\pi\). For this purpose, we use a variation of Dunnett’s test [48]. Since the basic version of Dunnett’s test for multiple comparisons assumes equal variances, we use a non-parametric bootstrapped version, which is a non-parametric adaptation of the test proposed by Alver and Zhang [49]. With this test, we determine for every \(\mathcal{R}\) if it yields a significantly larger return than the baseline policy \(\pi\), where we generally apply a significance threshold of \(p=0.05\).

8.2 Experiments↩︎

The results reported in Table 1 are different than the ones from this section because the first ones were only conducted for a single model configuration (fixed maximum depth or number of rules). The ones in this section present are cumulative, increasing the depth or the number of rules. For example, for IDP, Table 1 shows no improvement for the baselines, but in this section, we find that improvements are possible with smaller models. This occurs when the mirrored regions are too small, which is equivalent to having large models.

Tables ¿tbl:tab:mirrored95orcaid95ll?- ¿tbl:tab:mirrored95dt95hc? present the results for this analysis. For each environment, we tested four mirroring configurations, with varying numbers of selected mirrored regions \(rs\) and number of selected axes, i.e., features, \(as\). The tables are labeled at the top with “comb-\(as\) & random-\(rs\)-\(nm\)” to denote the mirroring configuration. In each case, we set the number of mirroring evaluations to \(nm=25\). For each evaluation with mirrored rules, we ran the environment for \(n_{eval} = 500\) episodes in parallel, whereas for the base agent, we performed \(n_{eval} =1,500\) episodes. A significant improvement is detected using a bootstrapped version of Dunnett’s test as previously mentioned. That is why the base agent is run for more episodes.

To limit the number of tests, we followed an incremental approach. For a given \(rs\) and a given \(as\), we tested whether there was an improvement for the smallest model. If there was, we stopped; if there was not, we continued to the next bigger model by increasing the maximum depth (Orcaid and DT) or the maximum number of rules (Cubist and RuleFit). The results are therefore cumulative. Rates indicate that an improvement was found at that maximum depth or maximum number of rules, or at a lower one. The best improvement follows the same idea.

We can see that for many environments (IP, S, H), an Orcaid model of depth 1 often finds a weakness. Recall that an Orcaid model with depth 1 divides the state space into just two parts. That is, enforcing a mirrored rule in one part of the space and letting the RL policy decide actions in the other part improves performance compared to having the RL policy for the entire state space. This also means that we do not need to learn a perfect Orcaid model to improve the original RL policy.

Another example of this situation appears with Cubist for IDP(Table ¿tbl:tab:mirrored95cubist95idp?), where improvements are found with smaller depths than the one that maximizes the reward ratio.

The results for a similar analysis leading to the identification of a weakness could help a user in, for example, the following two ways:

The following section shows how to understand the results with an example.

8.3 Explanation example for Swimmer↩︎

Table ¿tbl:tab:mirrored95orcaid95s? shows that weaknesses were often found with an Orcaid model of depth 1. One of such models has the following condition \(C\) to split the state space into two regions:

\[\begin{align} C \equiv &\;0.2\cdot \mathit{motor1\_rot} + 0.7\cdot\mathit{motor2\_rot} \\ & + 0.2\cdot \mathit{slider1dot} \\ & - 0.6\cdot\mathit{slider2dot} + 0.2\cdot\mathit{motor1\_rotdot} \geq 0.4 \end{align}\]

Region 0 satisfies \(C\) and has the following torques:

Region 1 does not satisfy \(C\) and has the following torques:

The feature in \(C\) with the highest positive weight is \(\mathit{motor2\_rot}\) (angle of the second rotor), and the feature with the highest negative weight is \(\mathit{slider2dot}\) (velocity of the tip along the y-axis). Thus, Region 0 corresponds to states in which the Swimmer is folded and has low upward velocity, possibly during the power stroke (thrust-generating phase). Fig. 6 shows this type of position when setting the state with our interactive tool (cf. App 9). The torque for this region is positive for both rotors, confirming that the Swimmer is generating thrust.

On the other hand, Region 1 covers states in which the Swimmer is extended with high upward motion, during a recovery stroke (reset phase). Here, therefore, the torques are negative, as one can see in their formulas.

Clearly, the Swimmer is a symmetric environment. Mirroring \(C\) by the axis given by a feature (e.g., \(\mathit{motor2\_rot}\)), we get \[\begin{align} C' \equiv &\;0.2\cdot \mathit{motor1\_rot} \mathbf{-} 0.7\cdot\mathit{motor2\_rot} \\ & + 0.2\cdot \mathit{slider1dot} \\ & - 0.6\cdot\mathit{slider2dot} + 0.2\cdot\mathit{motor1\_rotdot} \geq 0.4. \end{align}\]

As explained previously, the actions (torques) remain the same, as the linear regression should already capture variations in their inputs. In Fig. [fig:swimmer95mirrored], we can now see what the Swimmer would look like when \(C'\) is satisfied. It makes sense that this mirrored region corresponding to the satisfaction of \(C'\) behaves the same way as the original Region 0. That is, this mirrored region should also correspond to a thrust-generating phase, since the states within it are the same; only the angle has been mirrored, without affecting the region’s general purpose.

When we enforce this mirrored condition on the RL agent and observe a reward improvement, we can suspect that the RL agent has a weakness, since it has not fully learned this symmetry. Thus, we could try retraining with demonstrations that satisfy this condition. Also, we could incorporate this rule on top of the RL agent to enforce this behavior in production, and the user would know exactly how it behaves in this region. This would enhance the trust in the agent.

9 Interactive State Visualization Tool↩︎

The code in interactive_representation.py implements an interactive tool for exploring the regions of a learned Orcaid model. The tool enables users to manually manipulate the state of some Gymnasium environments and observe which region the model assigns the state to, together with its corresponding action formula. Fig. 11 displays the Hopper environment in a position set manually by the user via the interface in Fig. [fig:interface95hopper]. It shows that it is in Region 1 from an Orcaid model whose linear models are displayed at the bottom of the interface.

9.1 Core Functionality↩︎

The tool’s primary purpose is to facilitate the understanding of an Orcaid learned model. It achieves this by:

9.2 Workflow and User Interaction↩︎

The typical workflow for a user of this tool is as follows:

10 LLM-as-a-Judge↩︎

Quantitative metrics for model explainability, such as tree depth or number of rules, and model size, often fail to capture the semantic quality of an explanation. A model with a thousand simple but arbitrary rules is not necessarily more explainable than a model with a few complex but physically meaningful partitions. Human evaluation remains the gold standard but is costly and potentially biased.

To address this problem, we have taken the LLM-as-a-Judge framework. We have created a domain-specific evaluation questionnaire with associated rubrics to enable an LLM to act as a control expert and determine whether a model is globally explainable. We demonstrate this by comparing Orcaid with decision trees, which are typically assumed to be explainable, and with Cubist, which has the most similar reward ratio w.r.t. Orcaid.

10.1 Methodology↩︎

Our methodology consists of three main steps: (1) Context generation, (2) prompt and rubric design, and (3) execution and analysis.

10.1.1 Evidence and Context Generation↩︎

For a fair comparison, we standardize the “case file" provided to the LLM judge for each model. The case file contains:

10.1.2 Prompt and Rubric Design↩︎

We first assigned a persona (a control expert), then provided the model’s case file, and finally asked the LLM to evaluate the models based on the following questionnaire. Based on the list suggested by [46], we included 7 explainability-related questions. Additionally, we designed 8 domain-specific questions, as shown below. We assigned a meaning to each question’s score so that each LLM’s score would have the same meaning, allowing us to later summarize them all together. Each criterion is scored on a 1-5 Likert scale (5: best, 1: worst). LLMs must also provide a justification for each score, forcing them to “think” more and produce more accurate results.

10.1.2.1 Principles:

10.1.2.2 Good explanations:

10.1.2.3 Domain-specific:

10.1.2.4 Final:

10.1.3 Execution and Analysis↩︎

We used six SOTA LLMs available for free for a limited number of requests at https://openrouter.ai/:

Every surrogate for every LLM was evaluated at a low temperature of 0.2 to get more coherent and reproducible answers. The average scores and standard errors are shown in Table 2 and Table 3.

10.1.4 Model A: Orcaid Case File↩︎

The following lines show the case file provided to the LLMs to represent an Orcaid model for Pendulum.

Model Representation:
C0: -0.24y - 0.97ang_vel <= -0.60
C1: -0.19x - 0.41y - 0.89ang_vel <= -0.87
C2: 0.30x - 0.39y - 0.87ang_vel <= -0.33
Region 0:
C0 & C1
Model: z = -4.94x + -4.66y + 6.07
Region 1:
C0 & not C1
Model: z=-1.11x-1.04y-1.41ang_vel+3.47
Region 2:
C2 & not C0
Model: z = 2.54x - 1.05y - 1.55
Region 3:
not C0 & not C2
Model: z = 2.74x + 3.20*y - 2.80

10.1.5 Model B: Decision Tree Case File↩︎

The following text shows the case file provided to the LLMs to represent a DT model for Pendulum, simplified for readability purposes.

Model Representation:
|--- ang_vel <= 0.48
| |--- x <= 0.40
| | |--- y <= 0.66
| | | |--- x <= 0.33
| | | | |--- ang_vel <= 0.06
| | | | | |--- x <= 0.16
| | | | | | |--- x <= 0.13
| | | | | | | |--- value: [-1.96]

[... removed 204 lines ...]

| | | |--- x > 0.78
| | | | |--- ang_vel <= 0.48
| | | | | |--- value: [0.49]
| | | | |--- ang_vel > 0.48
| | | | | |--- ang_vel <= 0.51
| | | | | | |--- value: [-1.41]
| | | | | |--- ang_vel > 0.51

10.1.6 Full LLM Evaluation Prompt↩︎

The following prompt was given to the LLMs for each environment. The prompt also included instructions to format their answers according to a given JSON structure, which is omitted for better readability.

You are an expert in robotics and control systems, tasked with auditing the explainability and trustworthiness of two different surrogate models that were learned from a reinforcement learning model using imitation learning.

Model A is defined as follows:

Model B is defined as follows:

Your goal is to determine which model is more understandable, verifiable, and ultimately more trustworthy for deployment. Please evaluate both using the following rubric. Each criterion is scored on a 1-5 Likert scale, with 5 indicating a strong positive answer (’Excellent’) and 1 indicating a strong negative answer (’Very Poor’). You are also required to provide a brief justification for each score.

10.2 Results↩︎

Table 2 shows the average score across the 6 LLMs per question and per environment when comparing Orcaid with DT. One can see that Orcaid scores better on most questions across all environments, except Hopper and Half Cheetah. For Hopper, the scores are mixed, and for Half Cheetah, DT shows slightly better results. However, Orcaid and DT are not comparable for Half Cheetah since Orcaid performs much better (reward ratio \(\sim\!75\%\) vs \(\sim\! 47\%\)). Analogously, Table 3 shows the score average for the comparison between Orcaid and Cubist, where Orcaid outperforms Cubist across all environments, with the exception of Hopper, where results are mixed.

These results clearly indicate that Orcaid might be more interpretable than DTs and Cubist, especially when the latter have similar performance to Orcaid. This is even more noticeable in the examples when Orcaid models are built from oblique trees with a small depth.

1.75pt

1.75pt

10.2.1 Qualitative Analysis↩︎

LLM’s justifications explained the scores. As an example, for Pendulum with Orcaid depth 2 (Model A) and DT depth 7 (Model B), a judge explained its verdict towards Orcaid as follows:

“Model A [Orcaid]’s strategy is clear, sound, and physically motivated. It decomposes the complex pendulum problem into a small number of physically meaningful regimes, each governed by a simple, analyzable law. This structure provides a direct path to understanding, verification, and debugging. Its global policy is coherent and robust".

On the other hand, the DT model was criticized:

“Model B [DT]’s strategy is non-existent. It is a high-resolution, data-driven lookup table that has memorized a solution. Its individual rules are simple, but the resulting global policy is a chaotic patchwork with no discernible physical basis. This lack of an overarching strategy makes it a black box, rendering it unverifiable, brittle, and ultimately untrustworthy for a real-world deployment".

Judges usually highlighted the small number of regions and parameters of Orcaid. In contrast, they criticized the excessive number of conditions/rules in the DT and Cubist models. They suggested that Orcaid regions could more likely be associated with different modes of operations, whereas DT and Cubist partitions seemed to be arbitrary and too granular.

Therefore, the results support our hypothesis that Orcaid models can be interpretable. The LLM-as-a-Judge successfully identified and valued the semantic quality of Orcaid’s global strategy.

1.5pt

10.3 Limitations↩︎

Our methodology depends on prompt design and may be sensitive to the specific LLM version. We mitigated these dependencies by using a subset of the given list in [46], homogenizing the questions across all LLMs by providing explanations for the scores, and reducing randomness by setting a low temperature and using multiple LLMs. A more extensive analysis of different LLMs and learned models, together with a human evaluation, is an important direction for future work.

11 Hyperparameters↩︎

Table 4 shows the hyperparameters used for training an agent using RL for each environment. They are the ones that can be adjusted with Stable Baselines3 (SB3). Hyperparameters not listed in the table use their default values.

These hyperparameters were selected using RL Baselines3 Zoo and Hugging Face (https://huggingface.co/) models as references. However, they were slightly fine-tuned to reduce the number of training steps and speed up convergence. One can see that most of the values are similar across environments. Convergence was reached before the end of the training steps, ensuring the RL agent had sufficient performance.

Table 5 presents the hyperparameters for Orcaid used for the experiments in the paper. Each of the hyperparameters is described as follows:

Recall that \(\epsilon\), the relative error threshold to merge adjacent regions (stop condition no. 3, “rel. improvement” in the paper), is set to \(0.05\) for all the experiments. Also, the MSE threshold to merge regions is set to \(\tau=2\cdot\)max_mse for all the experiments. There are two main reasons to set a higher MSE threshold like this. First, this creates a more granular partition of the state space by allowing finer splits. Second, it provides flexibility to later combine smaller regions into larger ones. If both thresholds were too similar, we would have trouble finding suitable regions to merge.

1.2pt

12 Performance, MSE, model size, and time↩︎

Table 6 represents the underlying data for Fig.  [fig:plot95ratio95model95size] and Fig.  4 of the paper. They include the mean and standard error for each metric. To compute the confidence intervals with a probability of \(\alpha=0.05\) shown on the plots of the paper, we used the following formula since the number of samples was relatively small (\(N=9\)): \[\bar{x}\pm t_{0.05/2,8}\cdot SE,\] where \(SE\) is the standard error, \(SE=\sigma/\sqrt{N}\), \(t_{0.05/2,8}\) is the value of the Student’s t-distribution with \(9-1=8\) degrees of freedom whose cumulative distribution function equals \(1-\alpha/2=0.975\), i.e., \(t_{0.05/2,8}\approx2.3\).

Table 6 also includes the maximum depth (MD for Orcaid and Decision Trees) or maximum number of rules that were used (MNR for Cubist and RuleFit). The depth that is needed to reach an equal or better performance for Orcaid in comparison to Decision Trees is much smaller. Since MD with MNR are not comparable, we compared model sizes in the paper, showing that Orcaid model sizes are smaller than those of the other surrogates when the reward ratios were similar.

Tables ¿tbl:tab:orcaid95mc?- ¿tbl:tab:rulefit95hc? display RL reward/surrogate reward ratio in %, MSE for the training and test sets, model size calculated as explained in the paper (Def.  7 and Section  5 ), and the total training time, which includes iterative trainings and evaluations as for the DAgger methodology.

For Orcaid, neither the training time nor the model size increases exponentially. This is thanks to the stopping conditions and the simplification of regions, which reduces the number of linear regressions and conditions, thus reducing the number of parameters.

Clearly, Decision Trees are the fastest to learn due to the efficient implementation from scikit-learn. Cubist and Orcaid have similar training times, and RuleFit lies in between. However, for Cubist and RuleFit, selecting the maximum number of rules requires trying several possibilities. For example, for the extreme case of swimmer (S), for Orcaid, the best maximum depth was just \(2\), and we tried up to \(5\), whereas for Cubist we tried up to \(150\), selecting \(110\) as the best, and for RuleFit we tried up to \(14\), selecting \(14\) as the best. This adds an extra computation time that is not accounted for per single training maximum depth (resp. maximum number of rules).

For Decision Trees, one can see that they tend to overfit: the MSE train keeps decreasing, while the MSE test remains the same or slightly decreases. This seems to be avoided by Orcaid, where MSE train and test remain closer. This is probably thanks to the measures that Orcaid includes to avoid overfitting, such as early stopping, merging, and feature removal.

13 Ablation study↩︎

2.2pt

2.2pt

2.2pt

2.2pt

In this section, we learn different Orcaid models by modifying the hyperparameters shown in Appendix 11, Table 5:

We also learn models without certain components of our approach:

We keep all other components fixed using the values from Appendix 11, Table 5, as default and change only one of the mentioned hyperparameters or components per learned model. For this study, we selected the environments Pand Sto include both a simpler and a more complex environment. The default parameters for

The maximum depth selected for Pin the paper was \(3\), and \(2\) for S.

Table 15 shows Orcaid’s reward ratio for P when varying the different components as described, keeping the column default as a reference for the Orcaid model with the hyperparameters and approach as presented in the paper. Table 16 shows Orcaid’s model sizes for every ablation. Similarly, Table 17 and Table 18 show Orcaid’s reward ratios and model sizes for Sfor every ablation.

Reducing the max_mse makes Orcaid adjust better or equally to the resampled data. However, this comes at the cost of needing more regions as the number of merged regions decreases, and it also carries a potential for overfitting. This is also what happens when no simplification is included, i.e., the model still learns, but its size grows.

Using fewer clusters causes Orcaid to miss trajectory segments (points become over-grouped), reducing performance. The models become smaller since the data points are more concentrated around the clusters. A similar situation occurs when the number of samples is insufficient, i.e., some important points may be excluded from the training process. The model is smaller but has worse performance.

Removing leaf regressions reduces adaptability, requiring deeper trees to match the original performance. Linear regressions help train models with strong performance more quickly by avoiding the need to create more regions.

The absence of the DAgger loop makes the model learn more slowly, needing more depth to reach the original performance. This is emphasized in S, where it is not able to learn a good enough model.

Finally, having no linear regression and no oblique decisions creates models similar to DTs. Hence, the comparisons to DTs in the main part of the paper constitute another ablation study.

References↩︎