1 Introduction↩︎

Reinforcement learning methods require many interactions with their environment to successfully learn and evaluate policies. Therefore, they are rarely applied in challenging real-world applications such as medicine [1], education [2] or autonomous driving [3], where a policy can only be deployed in the environment if it exceeds a pre-specified performance threshold or fulfills certain safety criteria. This leaves us with a challenging problem: How do we know whether a policy fulfills the necessary criteria so that it can safely interact with the environment, without testing it on the environment, and in the process, compromising safety?

Off-policy evaluation (OPE) aims to solve this problem by estimating the performance of an evaluation policy, using only offline data that was previously collected by other agents [4], [5]. In practice, offline datasets are often recorded interactions of a human expert with the environment. Since the evaluation policy typically induces a different action-state distribution than offline data, OPE methods often have to make predictions under strong distribution shifts. As a result, most existing OPE estimators suffer from high variance and are prone to overestimating the performance of the policy [6]. In safety-critical applications, we can not risk and deploy a policy that is potentially much worse than what the OPE estimate suggests. Therefore, we aim for conservative off-policy evaluation (COPE) which seeks a (tight) lower bound on the evaluation policy’s expected return that holds with high probability. Once deployed, the policy may end up exploring areas that were not included in the offline data. Thus, reliably bounding the worst-case performance can be quite challenging.

We develop a novel model-based COPE approach that hinges upon two key ideas: epistemic uncertainty and pessimism. In particular, our approach, Hallucinated Adversarial Model-Based Off-policy evaluation (HAMBO) (HAMBO), builds on a learned statistical model of the transition dynamics that is able to quantify epistemic uncertainty. To obtain a valid lower bound on the policy performance, HAMBO hallucinates adversarial/worst-case trajectories the agent may take within the epistemic confidence sets of the model.

We prove that HAMBO reliably yields a high-probability bound on the true expected return of the policy, even when the offline data does not cover the areas explored by the evaluation policy (Proposition 1). Under regularity conditions, we further show that our conservative estimate converges from below to the true expected return (Theorem 3). To the best of our knowledge, HAMBO is the first provably consistent and conservative approach for OPE in continuous action-state spaces. We then propose scalable Bayesian neural network (BNN) variants of HAMBO and empirically evaluate them on various continuous control tasks. Importantly, we demonstrate that, even when the regularity conditions are not met, HAMBO reliably provides tight lower bounds on the true expected return.

2 Problem Setting↩︎

We consider a finite horizon Markov decision process (MDP) \(\mathcal{M} = (\mathcal{S}, \mathcal{A}, p_0, p, r, T)\) with continuous state and action spaces \(\mathcal{S} \subseteq \mathbb{R}^{d_s}\) and \(\mathcal{A} \subseteq \mathbb{R}^{d_a}\), initial state distribution \(p_0(\boldsymbol{s}_0)\), reward function \(r(\boldsymbol{a}_t, \boldsymbol{s}_t)\) and horizon \(T \in \mathbb{N}\). In particular, we consider stochastic transition dynamics that are governed by \(\boldsymbol{s}_{t+1} = {\boldsymbol{f}}(\boldsymbol{s}_t, \boldsymbol{a}_t) + \boldsymbol{\epsilon}_t\) where \({\boldsymbol{f}}: \mathcal{S}\times \mathcal{A}\rightarrow \mathcal{S}\) is unknown and \(\boldsymbol{\epsilon}_t \in \mathbb{R}^{d_s}\) is independent, additive transition noise with distribution \(p_{\boldsymbol{\epsilon}}(\boldsymbol{\epsilon}_t|\boldsymbol{s}_t, \boldsymbol{a}_t)\). Hence, the transition distribution \(p\) follows as \(p(\boldsymbol{s}_{t+1}| \boldsymbol{s}_t, \boldsymbol{a}_t) = p_{\boldsymbol{\epsilon}}(\boldsymbol{s}_{t+1} - f(\boldsymbol{s}_t, \boldsymbol{a}_t) | \boldsymbol{s}_t, \boldsymbol{a}_t)\). For simplicity, we assume that the reward function is known. However, all results can straightforwardly be extended to unknown rewards.

The agent interacts with the environment according to a policy \(\pi(\boldsymbol{a}_t | \boldsymbol{s}_t)\), which is a distribution over actions, conditioned on the current state \(\boldsymbol{s}_t\). The performance of a policy is typically measured by its expected return \(J(\pi) := J_p(\pi) := \mathbb{E}_{\boldsymbol{s}_0 \sim p_0}[V_{p, 0}^\pi(\boldsymbol{s}_0)]\) where \(V_t^\pi(\boldsymbol{s}) := V_{p, t}^\pi(\boldsymbol{s}) := \mathbb{E}_{p,\pi}[G_t|S_t = \boldsymbol{s}]\) is the value function and \(G_t := \sum_{t'=t+1}^T r(\boldsymbol{s}_{t'}, \boldsymbol{a}_{t'})\) is the return. For simplicity, we omit a discount factor in the return computation. However, all results presented can be straightforwardly adapted to discounted rewards. Furthermore, we denote the occupancy measure of policy \(\pi\) as \[ \rho^{\pi}(\boldsymbol{s}, \boldsymbol{a}) := \frac{1}{T} \sum_{t=0}^{T-1} p(\boldsymbol{s}_t=s, \boldsymbol{a}_t=a | \pi, \mathcal{M}) \;, \label{eq:state95occ95measure}\qquad{(1)}\] that is, the probability density function of being in state \(\boldsymbol{s}\) and performing action \(\boldsymbol{a}\) at any point of time \(t = 0, ..., T-1\).

We study the problem of offline policy evaluation where the task is to evaluate the performance, i.e. estimate the expected return \(J(\pi_e)\), of a given evaluation policy \(\pi_e\) while only using an offline dataset \(\mathcal{D}_b = \{(\boldsymbol{s}_i, \boldsymbol{a}_i, r_i, \boldsymbol{s}'_i)\}_{i=1}^n\) of observed transitions. The key challenge in OPE is the distribution shift between the (unknown) behavior policy \(\pi_b\) which generated the dataset \(\mathcal{D}_b\) and the policy \(\pi_e\) which we would like to evaluate. If \(\pi_b\) differs from \(\pi_e\), their state occupancy measures \(\rho^{\pi_b}\) and \(\rho^{\pi_e}\) can look significantly different. As a result, the dataset \(\mathcal{D}_b\) which is generated based on \(\rho^{\pi_b}\) may contain many samples in regions of the state-action space which \(\pi_e\) is unlikely to visit and limited data in regions that are relevant for accurately evaluating \(\pi_e\). In some cases, the support of \(\rho^{\pi_b}\) might not even contain the support of \(\rho^{\pi_e}\), i.e., \(\exists (\boldsymbol{s}, \boldsymbol{a}) \in \mathcal{S} \times \mathcal{A}: \rho^{\pi_e}(\boldsymbol{s}, \boldsymbol{a}) > 0 \wedge \rho^{\pi_b}(\boldsymbol{s}, \boldsymbol{a}) = 0\). Since OPE methods have to make predictions under such strong distribution shifts their estimates suffer from high variance and are prone to overestimate the performance of the policy.

OPE is particularly relevant in applications where we need to ensure a certain level of performance before a policy can be deployed online. Hence, it is often important to reliably determine whether or not the policy \(\pi_e\) meets its minimum performance requirements. We formalize this problem as conservative offline policy evaluation (see Definition 1) where we want to ideally find a tight lower bound on the expected return that holds with high-probability:

In some applications [7], safety criteria are not directly encoded in the reward and instead, are expressed as additional constraints in the form of \(\mathbb{E}_{(\boldsymbol{s}, \boldsymbol{a})\sim \rho^{\pi_e}} \left[ c_i(\boldsymbol{s}, \boldsymbol{a})\right] \geq 0\). To determine with high confidence whether \(\pi_e\) meets these constraints, we can apply COPE to each \(c_i\) individually.

3 COPE via Adversarial Transition Models↩︎

We take a model-based approach to COPE, and use a statistical model to estimate which transition functions \(\boldsymbol{h}: \mathcal{S}\times \mathcal{A}\rightarrow \mathcal{S}\) from a hypothesis space \(\mathcal{H}\) are plausible given the offline data \(\mathcal{D}_b\) of size \(n\). Then, we employ this statistical model of the transition dynamics to estimate the policy value \(J(\pi_e)\). For this estimate, we want to guarantee with high probability that it does not exceed the true policy value. To ensure this, we need to be able to reliably quantify the epistemic uncertainty of our model estimates.

Uncertainty quantification can be done with either a frequentist approach that produces mean and confidence estimate \(\boldsymbol{\mu}_n(\boldsymbol{s}, \boldsymbol{a})\) and \(\boldsymbol{\sigma}_n(\boldsymbol{s}, \boldsymbol{a})\) or with a Bayesian model that maintains a posterior distribution \(p({\boldsymbol{h}}|\mathcal{D}_b)\) over dynamics models in \(\mathcal{H}\). In the Bayesian case, we denote \(\boldsymbol{\mu}_n(\boldsymbol{s}, \boldsymbol{a}):= \mathbb{E}_{{\boldsymbol{h}}\sim p(\boldsymbol{h}|\mathcal{D}_b)} [\boldsymbol{h}(\boldsymbol{s}, \boldsymbol{a})]\) as the posterior mean and \(\boldsymbol{\sigma}^2_n(\boldsymbol{s}, \boldsymbol{a}) :=\mathop{\mathrm{diag}}(\mathbb{E}_{{\boldsymbol{h}}, {\boldsymbol{h}}' \sim p(h|\mathcal{D}_b)}[{\boldsymbol{h}}(\boldsymbol{s}, \boldsymbol{a}) {\boldsymbol{h}}'(\boldsymbol{s}, \boldsymbol{a})^\top])\) as the posterior variance. In either case, we require that our statistical model of \(\boldsymbol{h}\) is calibrated:

Popular statistical models for transition dynamics that capture epistemic uncertainty are Gaussian Processes (GPs) [8], Probabilistic Neural Network Ensembles [9] and Bayesian Neural Networks [10]. In later sections, we will attend to these specific choices of model in more detail and discuss when they are calibrated.

3.1 The HAMBO Framework↩︎

If our model is calibrated, we can, with high probability, use the confidence region \[[\boldsymbol{\mu}_n(\boldsymbol{s}, \boldsymbol{a}) - \beta_n(\delta) \boldsymbol{\sigma}_n(\boldsymbol{s}, \boldsymbol{a}), \boldsymbol{\mu}_n(\boldsymbol{s}, \boldsymbol{a}) + \beta_n(\delta) \boldsymbol{\sigma}_n(\boldsymbol{s}, \boldsymbol{a})]\] which is a \(d_s\)-dimensional hypercube, as a proxy for the true dynamics \({\boldsymbol{f}}(\boldsymbol{s}, \boldsymbol{a})\). We then pessimistically select transitions within this region, to guarantee a high probability lower bound on the policy value \(J(\pi_e)\). We do so, by introducing an adversary \(\boldsymbol{\eta}: \mathcal{S}\times \mathcal{A}\rightarrow [-1,1]^{d_s}\) that, for every \((\boldsymbol{s}, \boldsymbol{a}) \in \mathcal{S}\times \mathcal{A}\) picks a transition from the confidence region, thereby inducing the following hallucinated transition distribution: \[\begin{align} \label{eq:hallucinated95model} \begin{aligned} \tilde{p}_{\boldsymbol{\eta}}(\boldsymbol{s}_{t+1}| \boldsymbol{s}_t, \boldsymbol{a}_t) := p_{\boldsymbol{\epsilon}}\big(&\boldsymbol{s}_{t+1} - \boldsymbol{\mu}_n(\boldsymbol{s}_t, \boldsymbol{a}_t)\\ &- \beta_n \boldsymbol{\eta}(\boldsymbol{s}_t, \boldsymbol{a}_t) \boldsymbol{\sigma}_n(\boldsymbol{s}_t, \boldsymbol{a}_t) \big). \end{aligned} \end{align}\tag{1}\] This allows us to obtain a conservative value estimate for \(\pi_e\) \[\label{eq:pessimistic95value}\tilde{J}(\pi_e) := \min_{\boldsymbol{\eta}} J_{\tilde{p}_{\boldsymbol{\eta}}}(\pi_e) ~. \tag{2}\] This equation summarizes our approach hallucinated adversarial model-based off-policy evaluation (HAMBO) and Algorithm 5 presents the pseudo-code. Here, the expected reward \(J_{\tilde{p}_\eta}(\pi_e)\) of \(\pi_e\) under the hallucinated transition model \(\tilde{p}_\eta\) can, e.g., be estimated via Monte Carlo estimation (i.e., generating trajectory rollouts and averaging the respecting returns). To find the adversary \(\boldsymbol{\eta}(\boldsymbol{s}, \boldsymbol{a})\) which minimizes (2 ), we can view \(\boldsymbol{\eta}(\boldsymbol{s}, \boldsymbol{a})\) as policy that aims to maximize \(- J_{\tilde{p}_\eta}(\pi_e)\) and solve the corresponding optimal control problem. Importantly, with high probability, \(\tilde{J}(\pi_e)\) is a lower bound on the true policy value \(J(\pi_e)\):

While Proposition 1 shows that our estimate \(\tilde{J}(\pi_e)\) fulfills the requirements of COPE, \(\tilde{J}(\pi_e)\) could potentially be very loose. However, we can further establish a worst-case lower bound on \(\tilde{J}(\pi_e)\), if \({\boldsymbol{f}}\), \(r\), \(\boldsymbol{\sigma}_n\) and \(\pi_e\) are continuous. Formally, we make the following Lipschitz continuity assumption:

Here, the continuity assumption on \(\pi\) is expressed in terms the Wasserstein-1 distance and implies that a small change in the state space only induces a proportionally small change in the conditional action distribution of the policy. For instance, this is the case for policies that can be reparametrized with a Lipschitz function which is very common in practice:

Such Lipschitz assumptions are common in model-based OPE [11], [12] and RL more broadly [13], [14], and, e.g, hold in many real-world control problems. With these regularity assumptions, we bound how far away the HAMBO estimate \(\tilde{J}(\pi_e)\) is from the true policy value:

This theorem shows how by tuning the confidence level \(\delta\), we can trade-off accuracy with reliability. In particular, choosing a small \(\delta\) will ensure that the upper-bound on \(J(\pi_e)\) holds. However, it also increases \(\beta_n(\delta)\) and loosens the bound, indicating the \(\tilde{J}(\pi_e)\) estimate will be less accurate. Tightness of the HAMBO lower bound \(\tilde{J}(\pi_e)\) depends on the following key factors: Lipschitz-regularity, episode horizon \(T\), and epistemic uncertainty. Mainly, smaller Lipschitz constants and shorter episode lengths improve the bound. Moreover, the smaller the expected epistemic standard deviation \(\boldsymbol{\sigma}_n(\boldsymbol{s}_t, \boldsymbol{a}_t)\) under the state occupancy measure of \(\pi_e\), the tighter the bound. While the first two factors are generally dictated by the problem instance, the epistemic uncertainty can be reduced by using more offline data (in the relevant areas of the state-action space). If we can show that the epistemic uncertainty shrinks sufficiently fast with the number of offline data points \(n\) (i.e., faster than \(\mathcal{O}(\beta_n^{T})\)), then we can prove that \(\tilde{J}(\pi_e)\) converges to the true policy value as \(n \rightarrow \infty\). In the following, we discuss corresponding sufficient convergence conditions for GP models.

3.2 HAMBO with Smooth GP functions↩︎

In this section, we discuss the application of GPs for constructing calibrated confidence regions to be used for HAMBO. For the transition dynamics, we consider vector-valued functions \({\boldsymbol{f}}(\boldsymbol{s}, \boldsymbol{a}) \mapsto (f_1(\boldsymbol{s}, \boldsymbol{a}), ..., f_{d_s}(\boldsymbol{s}, \boldsymbol{a}))\) such that the scalar-valued functions \(f_j \in \mathcal{H}_k\) reside in a Reproducing Kernel Hilbert Space (RKHS) \(\mathcal{H}_k\) with kernel function \(k(\cdot, \cdot)\) and have bounded RKHS norm, i.e. \({\lVert f_j\rVert}_k \leq B\). We denote this space by . We assume that transition noise \(\boldsymbol{\epsilon}\sim \mathcal{N}(0, \sigma^2_\epsilon \boldsymbol{I})\) is normally distributed with variance \(\sigma^2_\epsilon\).

By fitting a zero-mean Gaussian Process \(\mathcal{GP}(0, k)\) on each dimension \(j=1, ..., d_s\) of the next state \(\boldsymbol{s}_{t+1}\), we can use the posterior means and variances to construct calibrated confidence sets. For brevity, we denote \(\boldsymbol{x}:= (\boldsymbol{s}, \boldsymbol{a})\), so that \[\begin{align} \begin{aligned} \mu_{n,j} (\boldsymbol{x})& = {\boldsymbol{k}}_{n}^\top(\boldsymbol{x})({\boldsymbol{K}}_{n} + \sigma^2_\epsilon \boldsymbol{I})^{-1}\boldsymbol{y}_{n, j} \label{eq:GPposteriors}\\ \sigma^2_{n, j}(\boldsymbol{x}) & = k(\boldsymbol{x}, \boldsymbol{x}) - {\boldsymbol{k}}^T_{n}(\boldsymbol{x})({\boldsymbol{K}}_{n}+\sigma^2_\epsilon \boldsymbol{I})^{-1}{\boldsymbol{k}}_{n}(\boldsymbol{x}) \end{aligned} \end{align}\tag{3}\] where \(\boldsymbol{y}_{n, j} = [s'_{i, j}]^\top_{i \leq n}\) is the vector the \(j\)-th element of the observed next states \(\boldsymbol{s}'_i\), \(\boldsymbol{k}_{n}(\boldsymbol{x}) = [ k(\boldsymbol{x}, \boldsymbol{x}_i)]^\top_{i \leq n}\), and \({\boldsymbol{K}}_{n} = [ k(\boldsymbol{x}_i, \boldsymbol{x}_l)]_{i,l \leq n}\) is the kernel matrix. By concatenating the element-wise posterior mean and standard deviation, we obtain \(\boldsymbol{\mu}_n(\boldsymbol{x}) = [\mu_{n,j}(\boldsymbol{x})]^\top_{j \leq d_s}\) and \(\boldsymbol{\sigma}_n(\boldsymbol{x}) = [\sigma_{n,j}(\boldsymbol{x})]^\top_{j \leq d_s}\). Using this, we can construct calibrated confidence intervals that fulfill Assumption 1:

In Appendix 9.2 we prove this lemma using results of [15] and give the exact expression for a \(\beta_n(\delta)\) that satisfies it. Generally, \(\beta_n(\delta)\) depends on the maximum information capacity \(\gamma_n\) of the kernel (see Appendix 9.2 for definition and details). In the GP setting, we can also show Lipschitz continuity of \({\boldsymbol{f}}\) and \(\boldsymbol{\sigma}\), if the kernel function \(k\) is sufficiently regular:

For common kernels, the kernel metric is Lipschitz continuous, and thus Lemma 2 applies. For instance, for the linear kernel we have \(L_k = 1\), for the RBF kernel we have \(L_k = 1 / \ell\) and for the Matern-\(\nu\) kernel we have \(L_k = \sqrt{\nu / (\nu -1)} / \ell\), where \(\ell\) is the lengthscale and \(\nu\) the smoothness parameter of the Matern kernel.

We can conclude that conditions of Proposition 1 and Theorem 2 are met when a GP is used for learning the transition dynamics from offline data. Hence, when the reward and the policy are Lipschitz, the HAMBO estimate satisfies \[J(\pi_e) - C_n \underset{\rho^{\pi_e}}{\mathbb{E}}\left[ {\lVert\boldsymbol{\sigma}_n(\boldsymbol{s}, \boldsymbol{a})\rVert}_2\right] \leq \tilde{J}(\pi_e) \leq J(\pi_e)\] with high probability. We can show that given a dataset of i.i.d. trajectories, the difference term shrinks with \(n\) sufficiently fast:

The theorem implies that \(\tilde{J}(\pi_e)\) is not only a conservative estimator for \(J(\pi_e)\), but under certain regularity conditions, it is also a consistent estimator of the policy’s true value. Meaning that for large \(n\), the trade-off between reliability and accuracy vanishes. In Appendix 9.3 we prove this theorem and give the exact rate at which the HAMBO estimate converges to the true value of \(\pi_e\). This rate depends on the choice of kernel, time horizon \(T\), and dimensions of the environment \((d_a, d_s)\). As an example, if \(k\) is a Linear or RBF kernel, with high probability \(\vert \tilde{J}_n(\pi_e) - J(\pi_e)\vert = \tilde{\mathcal{O}}\left(n^{-1/2}\right)\) where \(\tilde{O}\) omits polylogarithmic factors.

The kernel assumptions in Theorem 3 hold for many popular kernels such as any inner product kernel \(k(\boldsymbol{x}, \boldsymbol{x}') = \phi(\boldsymbol{x})^\top\phi(\boldsymbol{x}')\) with Lipschitz continuous finite-dimensional feature maps \(\phi(\cdot)\) or smooth kernels such as the RBF. On the other hand, Theorem 3 does hold for non-smooth functions, e.g., those corresponding to a Matern since their maximum information capacity \(\gamma_n\) grows polynomially with \(n\).

To the best of our knowledge, Theorem 3 is the first result that shows the consistency of a model-based finite-horizon OPE method for a continuous environments.

4 HAMBO with Neural Networks↩︎

In practice, we often want to evaluate policies in settings where the state and action spaces are higher-dimensional, and have access to larger amounts of offline data. In such environments, GPs become unpractical as they tend to generalize poorly in high-dimensional domains and their inference becomes prohibitively expensive for larger datasets.

The NN-based Statistical Model. In this section, we discuss practical variants of HAMBO which employ neural networks that scale more favorably to large datasets and high-dimensional domains. Crucially, we need to be able to quantify epistemic uncertainty. For this purpose, we employ Bayesian Neural Networks (BNNs) which model \(\boldsymbol{h}_{\boldsymbol{\theta}}(\boldsymbol{s}, \boldsymbol{a})\) as a neural network function where \(\boldsymbol{\theta}\) are the parameters of the neural network. BNNs presume a prior distribution \(p(\boldsymbol{\theta}) = \mathcal{N}(\boldsymbol{\theta}; 0, \lambda \boldsymbol{I})\) and maintain an approximation of the posterior \(p(\boldsymbol{\theta}| \mathcal{D}_b) \propto p(\mathcal{D}_b | \boldsymbol{\theta}) p(\boldsymbol{\theta})\) over neural network parameters. We use an independent Gaussian likelihood \(p(\mathcal{D}_b | \boldsymbol{\theta}) = \prod_{i=1}^n \mathcal{N}(\boldsymbol{s}_i'; \boldsymbol{h}_{\boldsymbol{\theta}}(\boldsymbol{s}_i, \boldsymbol{a}_i), \boldsymbol{\nu}^2_{\boldsymbol{\theta}}(\boldsymbol{s}_i, \boldsymbol{a}_i))\) where \(\boldsymbol{\nu}^2_{\boldsymbol{\theta}}(\boldsymbol{s}, \boldsymbol{a})\) is the vector of transition noise variances which is also predicted by the BNN. We use Stein Variational Gradient Descent (SVGD) [16] to approximate the posterior as a set of \(K\) particles \(\Theta = \{ \boldsymbol{\theta}_1, ..., \boldsymbol{\theta}_K \}\). We form the mean prediction of our model as the average prediction of the \(K\) NNs: \[\begin{align}\boldsymbol{\mu}_\Theta(\boldsymbol{s}, \boldsymbol{a}) &= \frac{1}{K} \sum_{k=1}^K \boldsymbol{h}_{\boldsymbol{\theta}_k}(\boldsymbol{s}, \boldsymbol{a}).\end{align}\] Similarly, we estimate the epistemic variance as \[\begin{align}\boldsymbol{\sigma}^2_{\Theta, e}(\boldsymbol{s}, \boldsymbol{a}) &= \frac{1}{K} \sum_{k=1}^K (\boldsymbol{h}_{\boldsymbol{\theta}_k}(\boldsymbol{s}, \boldsymbol{a}) - \boldsymbol{\mu}_\Theta(\boldsymbol{s}, \boldsymbol{a}))^2.\end{align}\] The overall predictive distribution is the equally weighted mixture of all \(K\) NN-based conditional Gaussians, i.e., \[\label{eq:predictive95gmm}p(\boldsymbol{s}'|\boldsymbol{s}, \boldsymbol{a}, \mathcal{D}_b) = \frac{1}{K} \sum_{k=1}^K \mathcal{N}(\boldsymbol{s}'; \boldsymbol{h}_{\boldsymbol{\theta}_k}(\boldsymbol{s}, \boldsymbol{a}), \boldsymbol{\nu}^2_{\boldsymbol{\theta}_k}(\boldsymbol{s}, \boldsymbol{a})) \tag{4}\] whose variance is \(\boldsymbol{\sigma}_\Theta^2(\boldsymbol{s}, \boldsymbol{a}) = \boldsymbol{\sigma}^2_{\Theta, e}(\boldsymbol{s}, \boldsymbol{a}) + \boldsymbol{\sigma}^2_{\Theta, a}(\boldsymbol{s}, \boldsymbol{a})\), where represents aleatoric and \(\boldsymbol{\sigma}^2_{\Theta, e}(\boldsymbol{s}, \boldsymbol{a})\) the epistemic uncertainty.

Calibrating the Model. Since our BNN model uses approximate inference and a potentially misspecified prior, it may not satisfy the calibration condition of Assumption 1. Thus, we re-calibrate the model’s uncertainty estimates with a calibration set \(\mathcal{D}_c \subset \mathcal{D}_b\) that is withheld from the training. In particular, we use temperature scaling which chooses \(\tau > 0\) such that the scaled predictive distribution (4 ) with variance \(\tau^2 \boldsymbol{\sigma}_\Theta^2(\boldsymbol{s}, \boldsymbol{a})\) has a minimal empirical calibration error on \(\mathcal{D}_c\) [17]. Algorithm 6 formalizes this technique. Note that re-calibrating the BNN model does not guarantee formal calibration in the sense of Assumption 1. However, in our experiments, we found it to reliably yield a conservative value estimate \(\tilde{J}(\pi_e)\).

4.1 Practical NN-Based HAMBO Variants↩︎

In the following, we discuss three ways of constructing adversarially hallucinated transition models based on our BNN model described in Equation (4 ). The formal pseudo-code of all algorithms is presented in Appendix 8.

Continuous Adversary (HAMBO-CA). This approach directly reflects the hallucinated adversarial transition model, introduced in (1 ) and (2 ). The adversary \(\boldsymbol{\eta}(\boldsymbol{s}, \boldsymbol{a}) \in [-1, 1]^{d_s}\) chooses the mean of the Gaussian transition probability from the epistemic confidence set, i.e., \[\begin{align} {!}{\tilde{p}_{\boldsymbol{\eta}}(\boldsymbol{s}'| \boldsymbol{s}, \boldsymbol{a}) \mathrel{\vcenter{:}}= \mathcal{N}\big(\boldsymbol{s}'; \boldsymbol{\mu}_\Theta(\boldsymbol{s}, \boldsymbol{a}) + \tau^2 \boldsymbol{\eta}(\boldsymbol{s}, \boldsymbol{a}) \boldsymbol{\sigma}^2_{\Theta, e} , \boldsymbol{\sigma}^2_{\Theta, a}(\boldsymbol{s}, \boldsymbol{a}) \big) . } \end{align}\] To get the corresponding conservative value estimate \(\tilde{J}(\pi_e)\), we need to solve the minimization problem \(\min_{\boldsymbol{\eta}} J_{\tilde{p}_{\boldsymbol{\eta}}}(\pi_e)\). For this, we parameterize the adversary \(\boldsymbol{\eta}(\boldsymbol{s}, \boldsymbol{a})\) as a neural network policy and use Soft Actor-Critic (SAC) [18] to maximize the negative return.

Discrete Adversary (HAMBO-DA). Our BNN posterior is approximated by a set of \(K\) NNs whose mean squared error difference corresponds to epistemic uncertainty. Thus, we can also construct a pessimistic transition model by letting the adversary choose which of the \(K\) NNs to pick. In this case, the adversary \(\vartheta(k | \boldsymbol{s}, \boldsymbol{a})\) is a categorical distribution over the NN indices \(\{1, ..., K\}\). The hallucinated transition model follows as: \[\begin{align}\tilde{p}_{\vartheta}(\boldsymbol{s}'| \boldsymbol{s}, \boldsymbol{a}) := \sum_{k=1}^K \vartheta(k | \boldsymbol{s}, \boldsymbol{a}) \mathcal{N}\big(\boldsymbol{s}' ; \boldsymbol{h}_{\boldsymbol{\theta}_k}(\boldsymbol{s}, \boldsymbol{a}) , \boldsymbol{\nu}^2_{\boldsymbol{\theta}_k}(\boldsymbol{s}, \boldsymbol{a}) \big).\end{align}\] Here, the adversary stochastically picks one of NN models at every step \(t=0, ..., T-1\). For this reason, we refer to this variant as \(\mathrm{\small DA1}\) (Algorithm 8). The corresponding value estimate follows as \(\tilde{J}_{\mathrm{\small DA1}}(\pi_e) = \min_{\vartheta} J_{\tilde{p}_{\vartheta}}(\pi_e)\). We solve the optimization problem by parameterizing the adversary \(\vartheta\) as a NN policy and use the clipped double DQN algorithm [19] to maximize the negative return.

Alternatively, we can constrain the adversary so that it has to commit to one of the \(K\) NN models for the entire trajectory. We refer to this variant as \(\mathrm{\small DAinf}\) (Algorithm 9). In this case, the transition model corresponds to the predictive distribution of one of the NNs \(p_{\boldsymbol{\theta}_k}(\boldsymbol{s}'| \boldsymbol{s}, \boldsymbol{a}) = \mathcal{N}\big(\boldsymbol{s}' ; \boldsymbol{h}_{\boldsymbol{\theta}_k}(\boldsymbol{s}, \boldsymbol{a}), \boldsymbol{\nu}^2_{\boldsymbol{\theta}_k}(\boldsymbol{s}, \boldsymbol{a}) \big)\), and the value estimate follows as the minimum the policy values under each of the models, i.e., \(\tilde{J}_{\mathrm{DAinf}}(\pi_e) = \min_{k \in \{1, \dots, K\}} J_{p_{\boldsymbol{\theta}_k}}(\pi_e)\). If \(K\) is larger (e.g., \(K > 20\)), we recommend taking the empirical \(\delta\) quantile of the policy values \(\{J_{\bar{p}_{k}}(\pi_e)\}_{k=1}^K\) instead of the minimum. In this case, \(\mathrm{\small DAinf}\) has similarities to the model-based bootstrap approach of [20].

Naturally, the value estimates of \(\mathrm{\small DAinf}\) are less pessimistic than those of \(\mathrm{\small DA1}\), i.e. \(\tilde{J}_{\mathrm{DAinf}}(\pi_e) \geq \tilde{J}_{\mathrm{DA1}}(\pi_e)\), because the adversary cannot change which model it picks throughout the trajectory. In the experiment section, we investigate whether \(\mathrm{\small DAinf}\) is still conservative enough to reliably yield lower bounds on the true policy values \(J(\pi_e)\).

5 Experiments↩︎

We start this section by illustrating the inner workings of HAMBO with a toy example to show why pessimism is crucial for COPE. We demonstrate that the convergence guarantees from Section [section:hambo95gp95models] materialize in practice for GP models. Finally, we empirically evaluate and compare the practical variants of HAMBO with BNNs on various continuous control tasks. For comparability between our environments, we shift and scale all our results so that the true policy return value \(J(\pi_e)\) is 1.

5.1 Illustrative Example↩︎

To illuminate the core idea of HAMBO and why pessimism is crucial for COPE, we conduct experiments on a toy environment which we call PointSafety (see Figure 1). In this environment, the agent navigates in the two-dimensional plane by applying actions \(\boldsymbol{a}\in [-0.5, 0.5]^2\) such that its position (i.e, state \(\boldsymbol{s}\in \mathcal{S}= \mathbb{R}^2\)) changes to \(\boldsymbol{s}_{t+1} = \boldsymbol{s}_t + \boldsymbol{a}_t\). The agent always starts on the left \(\boldsymbol{s}_0 = (-2, 0)\) and aims to go to its goal on the right \(\boldsymbol{s}_{\mathrm{fin}} = (2,0)\). However, the unit circle is a danger zone, in which the agent is subject to highly negative rewards (red shaded area).

We consider evaluation policies \(\pi_y\) with an intermediate goal \(\boldsymbol{s}_{\mathrm{im}} = (0, y)\) on the y-axis that goes in a straight line from \(\boldsymbol{s}_0\) to \(\boldsymbol{s}_{\mathrm{im}}\) and then in a straight line from \(\boldsymbol{s}_{\mathrm{im}}\) to the goal \(\boldsymbol{s}_{\mathrm{fin}}\). Note that policies \(\pi_y\) with \(|y| \leq 1.155\) are unsafe.

We generate an offline dataset by rolling out the behavior policy \(\pi_{1.6}\) with Gaussian action noise with a standard deviation of \(0.1\). Then, we evaluate \(\pi_{1.1}\), which is unsafe (see black trajectory), by rolling it out using HAMBO-CA.

We compare this to a neutral variant that predicts the next state with the predictive mean \(\boldsymbol{\mu}_\Theta(\boldsymbol{s}, \boldsymbol{a})\), i.e., without pessimism. As we can observe from the yellow trajectory, it falsely estimates \(\pi_{1.1}\) as safe, that is, it predicts that the trajectory lies outside of the danger zone. The trajectories with the adversarial transition model and the corresponding epistemic confidence sets for every step are depicted in Fig 1. The adversary successfully moves the prediction towards the danger zone within the confidence set, and, thus, correctly estimates the policy to be unsafe. Overall, this demonstrates a failure case of (neutral) off-policy evaluation and shows how HAMBO reliably gives a conservative estimate of the policy value through its pessimistic transition model.

5.2 Empirical convergence of HAMBO↩︎

For GP models, we show that HAMBO estimates converge to the true policy values (Theorem 3). Now, we empirically evaluate the behavior of GP-based HAMBO with an RBF kernel, as the number of offline data points grows. To this end, we consider two environments; a simple 2D PointEnv (\(\mathcal{S}=\mathbb{R}^2\), \(\mathcal{A}= [-1, 1]^2\)), similar to the PointSaftey environment, and the Pendulum-v1 environment from the OpenAI Gym [21]. In the PointEnv, the agent has to navigate the origin and accordingly receives the negative distance to the origin as a reward.

To generate the offline dataset, we collect transition data by uniformly sampling states and actions from the state and action space respectively. For the PointEnv, we restrict the sampled states to \([-40, 40]^2\) which covers the relevant part of the state space. As the evaluation policy, we use a proportional controller for the PointEnv, and a controller learned with SAC for the Pendulum.

Figure 2 plots the HAMBO estimates \(\tilde{J}(\pi_e)\) for a varying number of offline datapoints \(n = |\mathcal{D}_b|\). We notice that in the PointEnv, when we have insufficient data (here, ca. \(n \leq 150\)), the epistemic confidence regions of our GP model are large enough so that the transition model adversary sometimes manages to steer the policy outside the data support where the epistemic uncertainty is even higher. As a result, we see that \(\tilde{J}(\pi_e)\) are initially far below the true expected return \(J(\pi_e)\). However, as \(n\) increases, the GP uncertainty regions become smaller, and, as we can observe in Figure 2, \(\tilde{J}(\pi_e)\) becomes an increasingly tighter lower bound, approaching \(J(\pi_e)\) for both the environments.

5.3 HAMBO for Continuous Control↩︎

We evaluate the NN-based HAMBO methods from Section [sec:hambo95nn] on the continuous control tasks Pendulum-v1, Hopper-v3, and HalfCheetah-v3 from the OpenAI Gym and compare them to respective neutral (non-pessimistic) OPE methods.

Our general methodology is as follows: For a given environment, we first train a policy using the SAC algorithm [18], [22] and save several checkpoints of the agent. Then, some of the mediocre-performing checkpoints are rolled out to generate an offline dataset. After that, a given policy (usually one of the best checkpoints) is evaluated with the NN-based HAMBO variants.

We compare our approach to neutral OPE variants that do not use a pessimistic transition model [23]. In particular, we consider various trajectory uncertainty propagation methods from [24], employed in the context of OPE: First, we consider OPE-DS, where the transition model is approximated by a Gaussian \(p(\boldsymbol{s}'|\boldsymbol{s}, \boldsymbol{a}) = \mathcal{N}(\boldsymbol{s}' ; \boldsymbol{\mu}_\Theta(\boldsymbol{s}, \boldsymbol{a}); \boldsymbol{\sigma}_\Theta^2(\boldsymbol{s}, \boldsymbol{a}))\), here the variance is the sum of the epistemic and aleatoric variance. Second, we consider OPE-TS1 where the transition model is the mixture of predictive Gaussians in (4 ). This means that, in every step, one of the NN models is chosen uniformly at random to compute the next state distribution. Third, we consider OPE-TSinf, where, for every episode, we randomly commit to one of the \(K\) NNs.

We investigate the following three aspects: 1) whether a method yields reliable lower bounds, 2) the effect of the offline dataset size, and 3) the curse of long horizons. Figure 3 and 4 report the estimated expected policy returns, averaged over 5 seeds, alongside the corresponding confidence intervals. Reliable Lower Bounds. The HAMBO variants are designed to give reliable lower bounds on the true expected return. The results in Figure 3 and 4 empirically confirm that, across all seeds, all NN-based HAMBO variants reliably provide lower bounds on \(J(\pi_e)\), and, thus, fulfill the COPE requirements from Definition 1. In contrast, the neutral OPE variants which do not introduce pessimism w.r.t. the epistemic uncertainty of the transition model fail to do so. In many cases, they overestimate the true policy value, particularly in the Hopper environment. This demonstrates the importance of pessimism in model-based COPE and affirms the validity of HAMBO, even with BNN models, where calibration (Definition 1) cannot be formally proven.

Offline Dataset Size and Tightness. The difference between HAMBO estimates \(\tilde{J}(\pi_e)\) and the true expected reward \(J(\pi_e)\) depends on the strength of the transition adversary, which is limited by the size of the epistemic confidence sets. As the size of the offline datasets \(\mathcal{D}_b\) increases, we can generally expect the epistemic uncertainty to shrink. Thus, the adversary \(\boldsymbol{\eta}\) becomes less powerful and the HAMBO estimates become an increasingly tight lower bound.

In Figure 3, we empirically investigate this effect by varying the offline dataset size \(n\). As we hypothesized, we can observe the general trend that the HAMBO estimates come close to the true policy value, as \(n\) increases. Moveover, we observe that the HAMBO-DA1 estimates are always strictly smaller than those of the HAMBO-DAinf variant. This is expected, since in the DA1 variant, the adversary can pick the worst-case NN transition model at every step while in the case of DAinf the adversary can only do so per trajectory, and, thus has less power. Since our experiment results indicate that the pessimism in HAMBO-DAinf is sufficient to obtain reliable lower bounds in practice, we conclude that HAMBO-DAinf is the preferred choice among the two. While HAMBO-DAinf performs better in Hopper and HalfCheetah, HAMBO-CA yields the tightest lower bounds in the Pendulum environment.

The Curse of the Long Horizons. Finally, we investigate the effect of the horizon length \(T\) on our COPE estimates. Over the course of a trajectory, the transition model estimation errors can compound and lead to large discrepancies. This is a well-studied phenomenon in model-based RL [25]. In our case, this is reflected by the worst-case lower bound in Theorem 2 which depends exponentially on \(T\).

To evaluate the empirical effect of horizon length, we report the (C)OPE estimates for an offline dataset of size \(n= 10^5\) across varying horizon lengths: \(T\) = 100, 200 and 400 for the Pendulum and Hopper. For HalfCheetah, we only report horizon lengths of \(T\) = 100 and 150. Figure 4 displays the corresponding results. For an increasing horizon length, the variance of the neutral variants increases and the lower bounds of the conservative HAMBO estimates become looser. However, the observed decline in tightness in Figure 4 is much less pronounced than the exponential decline of the worst-case bound in Theorem 2.

For large horizon lengths, it can happen that the hallucinated trajectory under the pessimistic transition model strives far outside the support of the offline data. In such cases, unlike neutral OPE methods, HAMBO will still provide lower bounds on the true expected return. However, these bounds can be very pessimistic. For instance, this can be observed in the case of Pendulum, where for \(T=400\) the estimates of HAMBO-DA1 and HAMBO-DAinf go out of the chart. Making accurate long-horizon predictions is generally very hard. For instance, this is discussed extensively in the context of model-based RL in [25]. Often, a discount factor is used when computing returns to alleviate these issues. We highlight that we work with undiscounted returns and continuous state-action spaces, and, thus, operate in the most challenging setting for OPE.

6 Related Work↩︎

This work mainly contributes to the literature on off-policy evaluation for MDPs, which we divide to three categories.

Model-Free OPE. The key challenge in OPE is to the distribution shift between behavior and evaluated policy. A popular natural approach to correct the distribution mismatch is to use importance sampling (IS) ratios to re-weight the rewards collected by the behavior policy [5], [26] or to adjust the recursive updates when estimating the values directly via the Bellman equation [4], [27], [28]. Some work also combine both approaches to obtain a more favorable bias-variance trade-off [29], [30]. Unlike HAMBO, these approaches are model-free, i.e., they do not learn a model of the state transitions. However, they suffer from three key disadvantages: First, they have notoriously high variance, especially if the evaluated policy differs a lot from the behavior policy [31]. Second, they require the support of the behavior occupancy measure \(\rho^{\pi_b}\) to contain the support of \(\rho^{\pi_e}\) which is often not the case. In contrast, HAMBO still provides valid estimates in this scenario. Third, to compute the importance ratios, they assume access to the distribution of behavior policy which is almost never the case in practical applications where data is often collected by human experts. HAMBO does not require access to the behavior policy and, thus, is much more broadly applicable.

A recent line of work [32]–[35] estimates the state occupancy correction ratios via a form of fixed point iteration, and does not require access to the behavior policy. However, the Bellmann-like fixed point iteration is not applicable to the finite horizon case that we study in this paper. In addition, due to the fixed point iteration, it is very hard to quantify the uncertainty or bound error that is associated with such OPE estimates, making them poorly suited to COPE.

Model-Based OPE. This approach first learns the transition dynamics, to then simulate rollouts with the evaluation policy \(\pi_e\) and thereby estimate the expected reward of \(\pi_e\) [20], [23], [36]. Due to error in predicting the transitions, the resulting OPE estimate may overestimate the policy’s performance which is prohibited in safety-critical applications. Our approach additionally simulates pessimistic trajectories using the model’s epistemic uncertainty, to avoid overestimation. Further, to the best of our knowledge, Theorem 3 is the first consistency result for model-based OPE.

COPE and High-Confidence OPE. We study the problem of COPE which seeks a high-probability lower bound on the expected return. This is is closely related to estimating confidence bounds for OPE. [6] provide such confidence bounds for IS-based OPE estimates. However, due to the high variance of IS estimates, such bounds are often very loose [31]. [37] and [38] propose a model-free approach to give asymptotically normal confidence intervals for directly \(J(\pi)\). Assuming that the \(Q\)-function resides in an RKHS, [39] and [40] present rates of convergence, under theoretically unverified assumptions about the MDP. These model-free approaches only work for discounted, infinite-horizon MDPs, thus, are not generally applicable to our finite-horizon setting.

[20], [36] use model-based bootstrapping to construct confidence intervals for the OPE estimates. [20] prove the asymptotic correctness of the bootstrap confidence intervals only for finite state-action spaces. In contrast, we show the validity of our COPE estimates non-asymptotically for any \(|\mathcal{D}_b|\), and in continuous state-action spaces. Alternatively, [11] and [12] employ a Lipschitz argument to obtain valid COPE estimates. Our derivation of the worst-case lower bound in Theorem 2 also uses Lipschitz continuity. However, the HAMBO estimate provide a tighter lower bound on the true policy value, as we use the local confidence intervals rather than the global Lipschitz constants to introduce pessimism. Furthermore, unlike the mentioned work, HAMBO does not require knowledge of the Lipschitz constant and works with sub-Gaussian noise.

7 Conclusion↩︎

HAMBO, a novel approach for COPE that forms a pessimistic estimate of the expected return by hallucinating adversarial trajectories within the epistemic confidence regions of the estimated transition model. We formally prove the validity and consistency of the resulting COPE estimates. We propose various scalable NN-based variants of HAMBO and empirically demonstrate that they give reliable and tight lower bounds on the true expected return.

Importantly, our approach does not require access to the probability distribution of the behavior policy and gives reliable estimates, even when the support evaluation policy’s occupancy measure is not contained in the offline data distribution. This makes HAMBO particularly relevant for safety-critical real-world applications, where the offline data is mostly collected by human experts and we need to make reliable decisions about whether a given policy is good enough to be deployed.

HAMBO can be naturally combined with other offline reinforcement learning (ORL) algorithms to solve safety-critical ORL tasks. We leave this for future work to investigate.

8 Algorithm and Experiment Details↩︎

In the following, we provide algorithmic formalizations and implementation details of the HAMBO framework and its practical variants which were discussed in the main paper.

8.1 Generic HAMBO algorithm from Section [subsec:general95hambo95approach]↩︎

First, we formalize the general HAMBO framework from Section [subsec:general95hambo95approach]:

We estimate \(J_{\tilde{p}_{\boldsymbol{\eta}}}(\pi_e) = \mathbb{E}_{\boldsymbol{s}_0 \sim p_0}[\mathbb{E}_{p_{\boldsymbol{\eta}},\pi}[\sum_{t=0}^T r(\boldsymbol{s}_{t}, \boldsymbol{a}_{t})]]\) via Monte Carlo estimation, i.e., we roll out \(L\) trajectories and estimate the expectation as the average of the trajectory return: \[\label{eq:mc95estimation95return} \hat{J}_{\tilde{p}_{\boldsymbol{\eta}}}(\pi_e) = \frac{1}{L} \sum_{l=1}^{L} \sum_{t=0}^T r(\boldsymbol{s}_{l,t}, \boldsymbol{a}_{l, t}) ~~ \text{where} ~~ \boldsymbol{s}_{l,0} \sim p_0, ~ \boldsymbol{a}_{l,t} \sim \pi(\boldsymbol{a}| \boldsymbol{s}_{l,t}), ~ \boldsymbol{s}_{l,t+1} \sim \tilde{p}_{\boldsymbol{\eta}}(\boldsymbol{s}' \vert \boldsymbol{s}_{l,t}, \boldsymbol{a}_{l,t})\tag{5}\]

The optimization of the advesary corresponds to a standard optimal control problem for which we use traditional methods such as trajectory optimization or model-free RL algorithms such as SAC.

8.2 BNN Based HAMBO Variants↩︎

8.2.1 The BNN model↩︎

We use fully connected neural networks with 4 hidden layers each of size 256 with ReLU activation functions. Before training, the offline data inputs and targets are standardized. The NN takes the concatenated state and action as input (i.e., \(d_s + d_a\) dimensional) and outputs a vector of size \(2 d_s\) which is split into two vectors of size \(d_s\). The first one corresponds to the mean prediction \(\boldsymbol{h}_{\boldsymbol{\theta}}(\boldsymbol{s}, \boldsymbol{a})\) and the second one is the raw the aleatoric standard deviation which is fed through a softplus function to ensure positivity of \(\boldsymbol{\nu}^2_{\boldsymbol{\theta}}(\boldsymbol{s}, \boldsymbol{a})\)

As BNN prior we use a standard Normal distribution over the NN parameters \(\boldsymbol{\theta}\), i.e., \(p(\boldsymbol{\theta}) = \mathcal{N}(\boldsymbol{\theta}; \boldsymbol{0}, \boldsymbol{I})\). However, as commonly done for BNNs to alleviate the problems of prior misspecification, we add a temperature parameter \(\tau\) to the prior, so that we have \(p(\boldsymbol{\theta}| \mathcal{D}_b) \propto p(\mathcal{D}_b | \boldsymbol{\theta}) p(\boldsymbol{\theta})^\tau\). This hyper-parameter is chosen to as \(\tau=0.0001\) for Pendulum and Hopper and \(\tau=0.01\) for the HalfCheetah control environment.

We use Stein Variational Gradient Descent (SVGD) [16] for approximate posterior inference. In particular, we approximate the posterior \(p(\boldsymbol{\theta}| \mathcal{D}_b)\) with \(K\) NN particles \(\{ \boldsymbol{\theta}_1, ..., \boldsymbol{\theta}_K\}\). After randomly initializing the parameters of the \(K\) NNs, the parameters are iteratively updated with the SVGD update rule: \[\boldsymbol{\theta}_k \leftarrow \boldsymbol{\theta}_k+ \frac{1}{K} \sum_{k'=1}^K \left [k(\boldsymbol{\theta}_{k'}, \boldsymbol{\theta}_k) \nabla_{\boldsymbol{\theta}_{k'}} \log p(\boldsymbol{\theta}_{k'}| \mathcal{D}_b) + \nabla_{\boldsymbol{\theta}_{k'}} k(\boldsymbol{\theta}_k', \boldsymbol{\theta}_k) \right] ~~ \forall k = 1, ..., K ~.\] Here \(k(\cdot, \cdot)\) is a kernel function on the space of NN parameters vectors. In our experiments, we use an RBF kernel \(k(\boldsymbol{\theta}, \boldsymbol{\theta}') = \exp \left( - {\lVert\boldsymbol{\theta}- \boldsymbol{\theta}'\rVert}_2^2 / (2 \ell) \right)\) with a length scale of \(\ell = 10\) and \(K=5\) NN particles. Note that the kernel here is different from the one in Section [section:hambo95gp95models]. Algorithm [alg:SVGD] summarizes how to obtain the SVGD BNN posterior approximation.

8.3 Recalibration of the BNN uncertainty estimates↩︎

To obtain well-calibrated confidence sets for HAMBO, we recalibrate the BNNs predictive distribution. In particular, we use temperature scaling based on the regression calibration error [17]. We perform re-calibration based on the predictive distribution \(\mathcal{N}(\boldsymbol{\mu}_\Theta(\boldsymbol{s}, \boldsymbol{a}), \boldsymbol{\sigma}^2_{\Theta}(\boldsymbol{s}, \boldsymbol{a}))\). The calibration error compares the predictive quantiles of this Normal distribution with the corresponding empirical frequencies of data points, that fall below the predicted quantiles. Formally, we define \(\Phi_{\boldsymbol{\tau}}^{-1}(\alpha; \boldsymbol{s}, \boldsymbol{a}): [0,1]^{d_s} \rightarrow \mathcal{S}\) as the quantile function (inverse cumulative density function) of \(\mathcal{N}(\boldsymbol{\mu}_\Theta(\boldsymbol{s}, \boldsymbol{a}), \boldsymbol{\tau}^2 \boldsymbol{\sigma}^2_{\Theta}(\boldsymbol{s}, \boldsymbol{a}))\) where \(\boldsymbol{\tau} \in \mathbb{R}^{d_s}\) is the temperature scaling vector. Given a calibration dataset \(\mathcal{D}_c = \{(\boldsymbol{s}, \boldsymbol{a}, \boldsymbol{s}') \}\), the calibration error [17] for multivariate distributions follows as \[\mathrm{CalErr}(\boldsymbol{\tau}) \mathrel{\vcenter{:}}= \frac{1}{d_s} \sum_{j=1}^{d_s} \frac{1}{|A|} \sum_{\alpha \in A} \left(\mathrm{EmpFreq}(\alpha, \boldsymbol{\tau})_j - \alpha \right)^2 ~,\] where \(A = \{0.1, \cdots, 0.9, 0.99\}\) is a set of confidence levels and \[\mathrm{EmpFreq}(\alpha; \boldsymbol{\tau}) \mathrel{\vcenter{:}}= \frac{1}{|\mathcal{D}_c|} \sum_{(\boldsymbol{s}, \boldsymbol{a}, \boldsymbol{s}') \in \mathcal{D}_c} \mathbf{1}\{ \boldsymbol{s}' \leq \Phi_{\boldsymbol{\tau}}^{-1}(\alpha; \boldsymbol{s}, \boldsymbol{a}) \}\] is a vector-valued function of the (per dimension) empirical frequencies of the prediction targets that fall below the \(\alpha\) quantile. Finally, we recalibrate the BNN predictions, by choosing the variance scaling vector \(\boldsymbol{\tau}\) such that the calibration error is minimized, i.e., we choose \[\boldsymbol{\tau}^* = \mathop{\mathrm{arg\,min}}_{\boldsymbol{\tau}} \mathrm{CalErr}(\boldsymbol{\tau}) ~.\] Algorithm 6 summarizes this BNN re-calibration procedure:

8.4 The NN-Based HAMBO variants↩︎

Here, we provide algorithmic descriptions of the NN-Based HAMBO variants from Section [sec:hambo95nn] as well as details about their implementation and how the corresponding experiments were conducted.

8.4.1 HAMBO with a Continuous Adversary (HAMBO-CA)↩︎

HAMBO-CA directly reflects the hallucinated adversarial transition model, introduced in Section [section:hambo95general95idea]. The adversary \(\boldsymbol{\eta}(\boldsymbol{s}, \boldsymbol{a}) \in [-1, 1]^{d_s}\) chooses the mean of the Gaussian transition probability from the epistemic confidence set, i.e., \[\begin{align} \tilde{p}_{\boldsymbol{\eta}}(\boldsymbol{s}'| \boldsymbol{s}, \boldsymbol{a}) \mathrel{\vcenter{:}}= \mathcal{N}\big(\boldsymbol{s}'; \boldsymbol{\mu}_\Theta(\boldsymbol{s}, \boldsymbol{a}) + \tau^2 \boldsymbol{\eta}(\boldsymbol{s}, \boldsymbol{a}) \boldsymbol{\sigma}^2_{\Theta, e} , \boldsymbol{\sigma}^2_{\Theta, a}(\boldsymbol{s}, \boldsymbol{a}) \big) \end{align}\] For obtaining the corresponding conservative value estimate \(\tilde{J}(\pi_e) = \min_{\boldsymbol{\eta}} J_{\tilde{p}_{\boldsymbol{\eta}}}(\pi_e)\), we need to find the adversary \(\boldsymbol{\eta}^\star\) that minimizes the expected return. For this, we parameterize the adversary \(\boldsymbol{\eta}\) as a neural network policy with two hidden layers of size 256 with ReLU activations and a tanh squashed Gaussian conditional distribution over the adversary actions in \([-1, 1]^{d_s}\). We use SAC [18] to maximize the negative expected return of the adversary policy. As usual, to stabilize the SAC training and avoid Q-value overestimation, we use double critics and trailing target critics. The SAC training is conducted in rounds consisting of rollouts of 1000 episodes under the hallucinated transition model where actions are chosen by \(\pi_e\), followed by 1000 gradient steps on the SAC objectives. For the gradient steps, we use a batch size of 1024 and the Adam optimizer with a learning rate of \(10^{-3}\) for critic and policy and \(5 * 10^{-5}\) for the SAC entropy parameter. After SAC has converged, we take the adversary policy \(\boldsymbol{\eta}^\star\) and estimate the expected return \(\hat{J}_{\tilde{p}_{\boldsymbol{\eta}^\star}}(\pi_e)\) of \(\pi_e\) under the adversary transition model, induced by \(\boldsymbol{\eta}^\star\) with \(L = 10^4\) trajectories (see Eq. 5 ). The HAMBO-CA method is summarized in Algorithm 7.

8.4.2 HAMBO with a Discrete Adversary (HAMBO-DA1 and HAMBO-DAinf)↩︎

In the case of HAMBO-DA1 the adversary \(\vartheta\) has discrete action \(\{1,..., K\}\), i.e. picking one of the \(K\) particles. We parameterize the adversary policy as a neural network with two hidden layers of size 256 with ReLU activations and softmax-categorical distribution over the \(K\) discrete actions. To train this adversary policy, we use clipped double DQN [19]. The double DQN training is conducted in rounds consisting of rollouts of 1000 episodes under the hallucinated transition model where actions are chosen by \(\pi_e\), followed by 1000 gradient steps on the DQN objectives. For the gradient steps, we use a batch size of 1024 and the Adam optimizer with a learning rate of \(10^{-3}\). Once double DQN has converged, we take the adversary policy \(\vartheta^\star\) and estimate the expected return \(\hat{J}_{\tilde{p}_{\vartheta^\star}}(\pi_e)\) of \(\pi_e\) under the adversary transition model, induced by \(\vartheta^\star\) with \(L = 10^4\) trajectories (see Eq. 5 ). The overall HAMBO-DA1 method is summarized in Algorithm 8.

In contrast to HAMBO-DA1, HAMBO-DAinf uses a weaker adversary that has to commit to one of the BNN particles for the entire trajectory. As a result, the corresponding pessimistic HAMBO estimate can simply be chosen as the minimum of the expected evaluation policy return under each of the NN models in the particle approximation, i.e. \(J(\pi_e) = \min_{k \in \{1, \dots, K\}} J_{p_{\boldsymbol{\theta}_k}}(\pi_e)\). The HAMBO-DAinf method is summarized in Algorithm 8.

9 Proofs and Derivations↩︎

9.1 Proof of Theorem 2↩︎

The following lemmata will be used to prove the theorem.

In the following, we bound the difference between the pessimistic and true return with the distance between the true and pessimistic trajectory using the Lipschitz continuity of the reward function and the policy’s Wasserstein-one distance.

Next, we bound the distance between the true and pessimistic trajectory with the epistemic uncertainty around the true trajectory.

Using the above lemmas, we present the proof to the main theorem.

In summary, the deviation between the true and pessimistic return is proportional to the expected epistemic uncertainty of the evaluation policy state-occupancy measure \(\rho^{\pi_e}\), and the constant \(C_n\) defined as \[C_n \mathrel{\vcenter{:}}= \Bar{L}_r \left(1 + \sqrt{d_s}\right)\beta_n T^2\left(1+\Bar{L}_f + \left(1 + \sqrt{d_s}\right)\beta_n\Bar{L}_{\sigma}\right)^{T-1}.\] In Appendix 9.3, we provide consistency guarantees for our method. In particular, we prove under further assumptions on the true dynamics function \(f\), that \(\left|J(\pi_e) - \tilde{J}(\pi_e)\right| \to 0\), for \(n \to \infty\).

9.2 Proof of known results for kernel methods↩︎

We first recall the notion of maximum mutual information [41], [42]. The mutual information \(I({\boldsymbol{x}}_{1:n}; k)\) quantifies the reduction in uncertainty due to the observations \({\boldsymbol{x}}_{1:n}\). Given a GP model \(\mathcal{GP}(0, k(\cdot, \cdot))\) and gaussian noise assumption, mutual information is equal to \[I({\boldsymbol{x}}_{1:n}) = \frac{1}{2}\log\det({\boldsymbol{I}}+ \sigma_\epsilon^{-2} {\boldsymbol{K}})\] with the kernel matrix \({\boldsymbol{K}}= [k({\boldsymbol{x}}_i, {\boldsymbol{x}}_j)]_{i,j \leq n}\). The maximum information capacity or maximum mutual information of a kernel \(k\) is an upper bound on the mutual information, and is defined as \[\gamma_n = \max_{{\boldsymbol{x}}_{1:n}} I({\boldsymbol{x}}_{1:n}).\] Table 1 shows the growth rate of \(\gamma_n\) with \(n\) for multiple different kernels.

9.3 Proof of Theorem 3↩︎

For showing consistency of our lower bound in Theorem 2 for the GP case, we first prove that the uncertainty with respect to an i.i.d., data sampling distribution \(p(x)\) shrinks in expectation.

10 HAMBO for Offline Reinforcement Learning↩︎

OPE methods are commonly used in offline reinforcement learning (ORL) [31] to recommend/learn an optimal policy. Moreover, ORL methods also suffer from distribution shifts and are susceptible to overestimation, i.e., overestimating the performance of the recommended policy. Therefore, in principle, a good COPE method can be applied for ORL applications. To this end, we propose a natural modification of HAMBO-CA for ORL. \[\label{eq:pessimistic95value95orl}\tilde{J}(\pi^*) := \max_{\pi} \min_{\boldsymbol{\eta}} J_{\tilde{p}_{\boldsymbol{\eta}}}(\pi) ~. \tag{8}\] Our proposed method induces pessimism with respect to the epistemic uncertainty of the learned transition model to tackle distribution shifts. Similar, to HAMBO-CA, we can also use the HAMBO-DS1 variant to induce pessimsm.

We compare our HAMBO-based ORL variants to other ORL algorithms on the OpenAI Gym tasks from the D4RL benchmark [44]. Specifically, we consider the HalfCheetah environment with data sets generated with a random and a medicore-performing policy. Our results are presented in table 2.

The max-min optimization in eq (8 ) is typically very challenging. For our experiments we use the soft actor critic algorithm to train the policy and adversary together (DQN algorithm is used for the HAMBO-DA1 variant).

Note, our proposed ORL algorithms recommend the policy with the best lower bound and not the best expected return (see eq 8 ). Therefore, in general, they may fail to recommend the optimal policy. This is the price we pay for inducing robustness in our ORL methods. However, in practice (see table 2) we observer that the HAMBO based ORL methods perform competitively to the start of the art in the field.

References↩︎

---

1. Equal contribution.↩︎