1 Introduction↩︎

Offline reinforcement learning (RL) aims to learn decision-making strategies from static logged datasets [1], [2]. It has attracted increased interest in recent years, because the availability of large offline datasets are on the rise and online exploration required by alternative approaches such as online RL [3] remains expensive and risky in many real-world applications, such as robotics and healthcare.

Among offline RL algorithms, we focus on model-free approaches using dynamic programming with value bootstrapping. These algorithms, including the commonly used CQL [4], TD3+BC [5], IQL [6], and ATAC [7], have demonstrated strong performance in offline RL benchmarks [8], [9]. They follow the actor-critic scheme and adopt the principle of pessimism in the face of uncertainty to optimize an agent via a performance lower bound that penalizes taking unfamiliar actions. Despite their strengths, existing model-free offline RL methods also have a major weakness: they do not perform consistently. An algorithm that does well on one dataset may struggle on another, sometimes even underperforming behavior cloning (see [10] and Appendix 11.2). These performance fluctuations stand in the way of applying even the strongest offline RL approaches to practical problems.

In this work, we propose Heuristic Blending (HUBL), an easy-to-implement technique to address offline RL’s performance inconsistency. HUBL is an “aide” that operates in combination with a bootstrapping-based offline RL algorithm by using heuristic state value estimates¹ to modify the rewards and discounts in the dataset that the base offline RL algorithm consumes. Effectively, this modification blends heuristic values into dynamic programming to partially replace bootstrapping. Relying less on bootstrapping alleviates potential issues that bootstrapping causes and helps achieve more stable performance.

Combining HUBL with an offline RL method is very simple, as summarized in 1, and amounts to running this method on a version of the original dataset with modified rewards \(\tilde{r}\) and discounts \(\tilde{\gamma}\): \(\tilde{r} = r + \gamma \lambda h\) and \(\tilde{\gamma} = \gamma(1-\lambda)\), where \(r\) is blended with a heuristic \(h\), \(\tilde{\gamma}\) is the reduced discount, and \(\lambda\in[0,1]\) is a blending factor representing the degree of trust towards the heuristic. We propose to set \(h\) as the Monte-Carlo returns of the trajectories in the dataset for offline RL. Such heuristics are efficient and stable to compute, unlike bootstrapped \(Q\)-value estimates. The blending factor \(\lambda\) in HUBL can be trajectory-dependent. Intuitively, we want \(\lambda\) to be large (relying more on the heuristic) at trajectories where the behavior policy that collected the dataset performs well, and small (relying more on bootstrapped \(Q\)-values) otherwise. We provide three practical designs for \(\lambda\); they use only one hyperparameter, which, empirically, does not need active tuning.

We analyze HUBL’s performance both theoretically and empirically. Theoretically, we provide a finite-sample performance bound for a tabular offline RL with HUBL by framing it as solving a reshaped Markov decision process (MDP). To our knowledge, this is the first theoretical result for RL with heuristics in the offline setting. Our analysis shows that HUBL performs a bias-regret trade-off. On the one hand, solving the reshaped MDP with a smaller discount factor requires less bootstrapping and is relatively “easier”, so the regret is smaller. On the other hand, HUBL induces bias due to reshaping the original MDP. Nonetheless, we demonstrate that the bias can be controlled by setting the \(\lambda\) factor based on the above intuition, allowing HUBL to improve the performance of the base offline RL method.

Empirically, we run HUBL with the four aforementioned offline RL methods – CQL, TD3+BC, IQL, and ATAC – and show that enhancing these SoTA algorithms with HUBL can improve their performance by \(9\%\) on average across 27 datasets of D4RL [8] and Meta-World [12]. Notably, in some datasets where the base offline RL method shows inconsistent performance, HUBL can achieve more than 50% relative performance improvement.

2 Related Work↩︎

Bootstrapping-based offline RL \(\,\) A fundamental challenge of bootstrapping-based offline RL is the deadly triad [3]: a negative interference between

Modern offline RL algorithms such as CQL [4], TD3+BC [5], IQL [6], ATAC [7], PEVI [13], MOReL [14], and VI-LCB [15] employ pessimism to discourage the agent from taking actions unsupported by the data, which has proved to be an effective strategy to address the issue of limited support. However, they still suffer from a combination of errors in bootstrapping and function approximation. HUBL aims to address this with stable-to-compute Monte-Carlo return heuristics and thus is a complementary technique to pessimism.

RL by blending multi-step returns \(\,\) The idea of blending multi-step Monte-Carlo returns into the bootstrapping operator to reduce the degree of bootstrapping and thereby increase its performance has a long history in RL. This technique has been widely used in temporal difference methods [3], where the blending is achieved by modifying gradient updates [16]–[18] and reweighting observations [19]. It has also been applied to improve Q function estimation [20], online exploration [21], [22] and the sensitivity to model misspecification [23], and is especially effective for sparse-reward problems [24]. In contrast to most of the aforementioned works, which focus on blending Monte-Carlo returns as part of an RL algorithm’s online operation, HUBL is designed for the offline setting and acts as a simple data relabeling step. Recently, [24] have also proposed the idea of data relabeling, but their design takes a max of multi-step returns and bootstrapped values and as a result tends to overestimate \(Q\)-functions. We observed this to be detrimental when data has a limited support.

RL with heuristics \(\,\) More generally, HUBL relates to the framework of blending heuristics (which might be estimating quantities other than a policy’s value) into bootstrapping [25]–[28]. However, the existing results focus only on the online case and do not conclusively show whether blending heuristics is valid in the offline case. For instance, the theoretical analysis in [25] breaks when applied to the offline setting as we will demonstrate in Section 5. The major difference is that online approaches rely heavily on collecting new data with the learned policy, which is impossible in the offline case. In this paper, we employ a novel analysis technique to demonstrate that blending heuristics is effective even in offline RL. To the best of our knowledge, ours is the first work to extend heuristic blending to the offline setting with both rigorous theoretical analysis and empirical results demonstrating performance improvement. One key insight, inspired by transition-dependent discount factor from [29], is the adoption of a trajectory-dependent \(\lambda\) blending factor, which is both a performance-improving design as well as a novel analysis technicality.

Discount regularization \(\,\) HUBL modifies the reward with a heuristic and reduces the discount. Discount regularization, on the other hand, is a complexity reduction technique that reduces only the discount. The idea of simplifying decision-making problems by reducing discount factors can be traced back to Blackwell optimality in the known MDP setting [30]. Most existing results on discount regularization [31], [32] study the MDP setting or the online RL setting [33]. Recently, [34] has shown that discount regularization can also reduce the complexity of offline RL and serve as an extra source of pessimism. However, as we will show, simply reducing the discount without the compensation of blending heuristics in the offline setting can be excessively pessimistic and introduce large bias that hurts performance. In Section 6.1, we rigorously analyze this bias and empirically demonstrate the advantages of HUBL over discount regularization.

3 Background↩︎

In this section, we define the problem setup of offline RL (Section 3.1), review dynamic programming with value bootstrapping, and briefly survey methods using heuristics (Section 3.2).

3.1 Offline RL and Notation↩︎

We consider offline RL in a Markov decision process (MDP) \({\mathcal{M}}= ({\mathcal{S}}, {\mathcal{A}}, {\mathcal{P}}, r, \gamma)\), where \({\mathcal{S}}\) denotes the state space, \({\mathcal{A}}\) denotes the action space, \({\mathcal{P}}\) denotes the transition function, \(r: {\mathcal{S}}\times {\mathcal{A}}\to [0,1]\) denotes the reward function, and \(\gamma\in [0,1)\) denotes the discount factor. A decision-making policy \(\pi\) is a mapping from \({\mathcal{S}}\) to \({\mathcal{A}}\), and its value function is defined as \(V^{\pi}(s) = {\mathbb{E}}[\sum_{t=0}^\infty\gamma^t r(s_t,a_t) |s_0=s, a_t\sim\pi(\cdot|s_t)]\). In addition, we define \(Q^\pi(s,a) \mathrel{\vcenter{:}}= r(s,a) + \gamma {\mathbb{E}}_{s'\sim{\mathcal{P}}|s,a}[V^\pi(s')]\) as its state-action value function (i.e., Q function). We use \(V^*\) to denote the value function of the optimal policy \(\pi^*\), which is our performance target.

In addition, we introduce several definitions of distributions. First, we define the average state distribution of a policy \(\pi\) starting from an initial state \(s_0\) as \(d^\pi(s,a; s_0) := (1-\gamma)\sum_{t=0}^\infty \gamma^td_t^\pi(s,a; s_0)\), where \(d_t^\pi(s,a;s_0)\) is the state-action distribution at time \(t\) generated by running policy \(\pi\) from an initial state \(s_0\). We assume that the MDP starts with a fixed initial state distribution \(d_0\). With slight abuse of notation, we define the average state distribution starting from \(d_0\) as \(d^\pi(s,a):=d^\pi(s,a; d_0)\), and \(d^\pi(s,a,s') := d^\pi(s,a) \mathcal{P}(s'|s,a)\).

The objective of offline RL is to learn a well-performing policy \(\hat{\pi}\) while using a pre-collected offline dataset \(\mathbb{D}\). The agent has no knowledge of the MDP \({\mathcal{M}}\) except information contained in \(\mathbb{D}\), and it cannot perform online interactions with environment to collect more data. We assume \(\mathbb{D} := \{\tau\}\) contains multiple trajectories collected by a behavior policy, where each \(\tau=\{(s_t,a_t,r_t)\}_{t=1}^{T_\tau}\) denotes a trajectory with length \(T_\tau\). Suppose these trajectories contain \(N\) transition tuples in total. With abuse of notation, we also write \(\mathbb{D} := \{(s,a,s', r, \gamma)\}\), where states \(s\) and action \(a\) follow a distribution \(\mu(s,a)\) induced by the behavior policy, \(s'\) is the state after each transition, \(r\) is the reward at \(s,a\), and \(\gamma\) is the discount factor of the MDP. Note that the value of discount \(\gamma\) is the same in each tuple. We use \(\Omega\) to denote the support of \(\mu(s,a)\). We do not make the full support assumption, in the sense that the dataset \(\mathbb{D}\) may not contain a tuple for every \(s,a,s'\) transition in the MDP.

3.2 Dynamic Programming with Bootstrapping and Heuristics↩︎

Bootstrapping \(\,\) Many offline RL methods leverage dynamic programming with value bootstrapping. Given a policy \(\pi\), we recall \(Q^{\pi}\) and \(V^{\pi}\) satisfy the Bellman equation: \[\label{eq:dpe} Q^{\pi}(s,a) = r(s,a) + \overbrace{\gamma {\mathbb{E}}_{s' \sim {\mathcal{P}}(\cdot|s,a)}[ V^{\pi}(s') ]}^{\textrm{(bootstrapping)}}.\tag{1}\] These bootstrapping-based methods compute \(Q^{\pi}(s,a)\) using an approximated version of 1 : given sampled tuples \((s, a, s', \gamma, r)\), such methods minimize the difference between the two sides of 1 with both \(Q^\pi\) and \(V^\pi\) replaced with function approximators ². With limited offline data, learning the function approximator using bootstrapping can be challenging and yields inconsistent performance across different datasets [3], [35], [36], as we will also later see in the experiment section (6).

Heuristics \(\,\) A heuristic is a value function \(h: {\mathcal{S}}\rightarrow \mathbb{R}\) calculated using domain knowledge, Monte-Carlo averages, or pre-training. Heuristics are widely used in online RL, planning, and control to improve the performance of decision-making [25]–[28], [37]. In this paper, we focus on the offline setting, and consider heuristics \(h\) that approximate the value function of the behavior policy. They can be estimated from \(\mathbb{D}\) via Monte-Carlo methods.

4 Heuristic Blending(HUBL)↩︎

In this section we describe our main contribution — Heuristic Blending (HUBL), an algorithm that works in combination with bootstrapping-based offline RL methods and improves their performance.

4.1 Motivation↩︎

HUBL uses a heuristic computed as the Monte-Carlo return of the behavior policy in the training dataset \(\mathbb{D}\) to improve offline RL. It reduces an offline RL algorithm’s bootstrapping at trajectories where the behavior policy performs well, i.e., where the value of the behavior policy (i.e. the heuristic value) is high. With less amount of bootstrapping, it mitigates bootstrapping-induced issues on convergence stability and performance. In addition, since the extent of blending between the heuristic and bootstrapping is trajectory-dependent, HUBL introduces only limited performance bias to the base algorithm, and therefore can improve its performance overall.

4.2 Algorithm↩︎

As summarized in Algorithm 2, HUBL is easy to implement: first, relabel a base offline RL algorithm’s training dataset \(\mathbb{D}=\left\{(s,a,s',r,\gamma)\right\}\) with modified rewards \(\tilde{r}\) and discount factors \(\tilde{\gamma}\), creating a new dataset \(\tilde{\mathbb{D}}:=\left\{(s,a,s',\tilde{r},\tilde{\gamma})\right\}\); next, run the base algorithm on \(\tilde{\mathbb{D}}\).

The data relabeling is done in three steps:

Step 0: As a preparation step, we convert the data tuples in \(\mathbb{D}\) back to trajectories like \(\tau = \{(s_t,a_t, r_t)\}_{t=1}^{T_\tau}\). For the next two steps, we work on data trajectories instead of data tuples to compute heuristics and blending factors.

Step 1: Computing heuristic \(h_t\) \(\,\) We compute heuristics by Monte-Carlo returns. For each \(\tau =\{(s_t,a_t, r_t)\}_{t=1}^{T_\tau} \in \mathbb{D}\), we calculate the heuristics as³ \(h_t=\sum_{k=t}^{T_{\tau}}\gamma^{k-t}r_{k}\) and update the data trajectory as \(\tau \gets \left\{(s_t,a_t, r_t, h_t)\right\}_{t=1}^{T_\tau}\).

Step 2: Computing blending factor \(\lambda_t\) \(\,\) We append a scalar \(\lambda_t = \lambda(\tau) \in[0,1]\) at each time point \(t\) of each trajectory as the blending factor, leading to \(\tau \gets \{(s_t,a_t, r_t,h_t, \lambda_t)\}_{t=1}^{T_\tau}\). \(\lambda_t\) indicates the confidence in the heuristics on the trajectory. Intuitively, \(\lambda_t\) decides the contribution of heuristics over bootstrapped values in dynamic programming to update the \(Q\)-function. We desire \(\lambda_t\) to be closer to 1 when the heuristic value \(h_t\) is higher (i.e., at states where the heuristic is closer to the optimal \(Q\)-value) to make offline RL rely more on the heuristic, and \(\lambda_t\) closer to zero when heuristic is lower to make offline RL to use more bootstrapping. We experiment with three different designs of \(\lambda(\tau)\):

• Constant: As a baseline, we consider \(\lambda(\tau) =\alpha\in[0,1]\) for all \(s\). We show that, despite forcing the same heuristic weight for every state, this formulation already provides performance improvements.

• Sigmoid: As an alternative, we use the sigmoid function to construct a trajectory-dependent blending function \(\lambda(\tau) = \alpha\sigma(\sum_{t=1}^{T_\tau}h(s_t)/T_{\tau})\), where \(\alpha\in[0,1]\) is a tunable constant and \(\sigma\) is the sigmoid function. Thus, \(\lambda(\tau)\) varies with the performance of the behavior policy over data.

• Rank: Similar to the Sigmoid labeling function, we provide a rank labeling function \(\lambda(\tau) = \alpha \sum_{\tau' \in \mathbb{D}} \mathbb{1}_{\bar{h}(\tau') \leq \bar{h}(\tau)}/n\) where \(n\) is the number of trajectories in \(\mathbb{D}\), and \(\bar{h}(\tau) = \frac{1}{T}\sum_{h_t\in\tau} h_t\).

Step 3: Relabeling \(r\) and \(\gamma\) \(\,\) Finally, we relabel the reward as \(\tilde{r}\) and the discount factor as \(\tilde{\gamma}\) in each tuple of \(\mathbb{D}\). To this end, we first convert the updated data trajectories \(\{ \{(s_t,a_t, r_t,h_t, \lambda_t)\}_{t=1}^{T_\tau}\}\) back into data tuples \(\{(s,a,s',r,\gamma,h',\lambda')\}\), where \(h'\) and \(\lambda'\) denote the next-step heuristic and blending factor. Then, for each data tuple, we compute \[\begin{align} \label{eq:hubo32rule} \tilde{r} = r + \gamma \lambda' h' \textrm{\qquad and \qquad} \tilde{\gamma} = \gamma(1-\lambda'), \end{align}\tag{2}\] to form a new dataset \(\tilde{\mathbb{D}} \mathrel{\vcenter{:}}=\left\{(s,a,s',\tilde{r},\tilde{\gamma})\right\}\). Intuitively, one can interpret \(\tilde{r}\) as injecting a heuristic-dependent quantity \(\gamma \lambda' h'\) into the original reward, and \(\tilde{\gamma}\) as reducing bootstrapping by shrinking the original discount factor by a factor of \(1-\lambda'\). We formally justify the design of \(\tilde{r}\) and \(\tilde{\gamma}\) in Section 5.

5 Understanding HUBL↩︎

In this section, we take a deeper look into how HUBL works. At a high level, our theoretical analysis explains that the modification made by HUBL introduces a trade-off between bias and regret (which is similar to the variance of policy learning) into the base offline RL algorithm. While an important part of our analysis is for tabular settings (5.3), we believe it still provides valuable intuitions as to why reshaping of rewards and discounts per 2 gives a performance boost to state-of-the-art offline RL algorithms in the continuous-state-space benchmarks in Section 6.

5.1 HUBL as MDP Reshaping↩︎

We analyze HUBL by viewing it as solving a reshaped MDP \(\tilde{\mathcal{M}}:= ({\mathcal{S}}, {\mathcal{A}}, {\mathcal{P}}, {\tilde{r}}, \tilde{\gamma})\) constructed by blending heuristics into the original MDP. To this end, we make a simplification by assuming that both the heuristic and blending factor are functions of states. Specifically, let \(\Omega\) denote the support of the data distribution; we suppose that some functions \(h(\cdot):\Omega \to \mathbb{R}\) and \(\lambda(\cdot):\Omega \to [0,1]\) are given to HUBL. For analysis, we extend them to out-of-\(\Omega\) states as below: \[\begin{align} \label{eq:extended32h32and32lambda} h(s) = \begin{cases} h(s) &\textrm{ for }s\in\Omega \\ 0 &\textrm{ otherwise} \end{cases} \,\quad \textrm{and } \quad \lambda(s,s') = \begin{cases} \lambda(s') &\text{for } s,s' \in \Omega \\ 0 & \text{otherwise} \end{cases} \end{align}\tag{3}\] We note that this extension is only for the purpose of analysis, since HUBL never uses the values \(h(\cdot)\) and \(\lambda(\cdot)\) outside \(\Omega\); the given \(h\) on out-of-\(\Omega\) can have any value and the following theorems still hold.

We define the reshaped MDP \(\tilde{\mathcal{M}}\) with redefined reward function and discount factor as \({\tilde{r}}(s,a) := r(s, a) + \gamma {\mathbb{E}}_{s' \sim {\mathcal{P}}(\cdot|s,a)} \big[ \lambda(s,s') h(s') \big] \textrm{, and } \tilde{\gamma}(s,s'):= \gamma(1- \lambda(s,s'))\) respectively. Note that we blend the original reward function with the expected heuristic and blending factor while adjusting the original discount factor correspondingly. The extent of blending is determined by the function \(\lambda(\cdot)\). Notice that this reshaped MDP has a transition-dependent⁴ discount factor \(\tilde{\gamma}(s,s')\) which is compatible with generic unifying task specification of [29]. This novel definition of transition-dependent discount factor is a key analysis technique to show that blending heuristics is effective in the offline setting.

Reshaped Dynamic Programming \(\,\) When solving this reshaped MDP by dynamic programming, the Bellman equation changes from 1 accordingly into \[\begin{align} \label{eq:shaped-dpe} \begin{aligned} \tilde{Q}^{\pi}(s,a) &= \tilde{r}(s,a) + {\mathbb{E}}_{{s' \sim {\mathcal{P}}(\cdot|s,a)}}[\tilde{\gamma}(s,s')\tilde{V}^\pi(s')] \\ &= r(s,a) + \gamma \overbrace{{\mathbb{E}}_{s' \sim {\mathcal{P}}(\cdot|s,a)}[ (1-\lambda(s,s'))\tilde{V}^{\pi}(s') ]}^{\textrm{bootstrapping}} + \gamma \overbrace{{\mathbb{E}}_{s' \sim {\mathcal{P}}(\cdot|s,a)}[ \lambda(s,s')h(s') ]}^{\textrm{heuristic}}. \end{aligned} \end{align}\tag{4}\] Here \(\tilde{Q}^{\pi}\) denotes the \(Q\)-function of policy \(\pi\) in \(\tilde{\mathcal{M}}\), and \(\tilde{V}^{\pi}\) denotes \(\pi\)’s value function. Compared to the original Bellman equation 1 , it can be seen that \(\lambda(\cdot)\) blends the heuristic with bootstrapping: the bigger \(\lambda\)’s values, the more bootstrapping is replaced by the heuristic. The effect of solving HUBL’s reshaped MDP \(\tilde{\mathcal{M}}\) is twofold. On the one hand, \(\tilde{\mathcal{M}}\) is different from the original MDP \({\mathcal{M}}\): the optimal policy for \(\tilde{\mathcal{M}}\) may not be optimal for \({\mathcal{M}}\), so solving for \(\tilde{\mathcal{M}}\) could potentially lead to performance bias. One the other hand, \(\tilde{\mathcal{M}}\) has a smaller discount factor and thus is easier to solve than \({\mathcal{M}}\), as the agent needs to plan for a smaller horizon. Therefore, we can think of applying HUBL to offline RL problems as performing a bias-variance trade-off, which reduces the learning variance due to bootstrapping at the cost of the bias due to using a suboptimal heuristic. We will explain this more concretely next.

5.2 Bias-Regret Decomposition↩︎

The insight that HUBL reshapes the Bellman equation that the offline RL algorithm uses allows us to characterize HUBL’s effects on policy learning. Namely, we use the modified Bellman equation from 4 to decompose the performance of the learned policy into bias and regret terms.

The performance of \(\pi\) depends on both bias and regret. The bias term describes the discrepancy caused by solving the reshaped MDP with \(\lambda(\cdot)\). When \(\lambda(s') = 0\), the bias becomes zero. The regret term describes the performance of the learned policy in the reshaped MDP. Intuitively, at states whose successors have bigger \(\lambda(s')\) values, the reshaped MDP has a smaller discount factor and thus is easier to solve, which leads to smaller regret (i.e., smaller values for \(\tilde{V}^{\pi^*}(d_0) - \tilde{V}^{\hat{\pi}}(d_0)\)). Therefore, solving a HUBL-reshaped MDP induces a bias term but may generate a smaller regret.

Remark \(\,\) The critical difference – and novelty – of Theorem 1 compared to existing theoretical results for RL with heuristics in the offline setting is that both the bias and regret in Theorem 1 depend only on states in the data distribution support \(\Omega\). This is crucial, because in the offline setting we have no access to observations beyond \(\Omega\). In contrast, if in the preceding analysis we replace Theorem 1 by, for example, Lemma A.1 from [25] with a constant \(\lambda\), we will get a performance decomposition \(V^*(d_0)-V^{\pi}(d_0) = (V^*(d_0) - \tilde{V}^{*}(d_0))+ \frac{\gamma\lambda}{1-\gamma} {\mathbb{E}}_{s,a\sim d^{\pi}}{\mathbb{E}}_{s'|s,a} [h(s') - \tilde{V}^*(s')] \\ + (1-\lambda)(\tilde{V}^*(d_0) - \tilde{V}^{\pi}(d_0)) + \frac{\lambda}{1-\gamma} (\tilde{V}^*(d^{\pi}) - \tilde{V}^{\pi}(d^\pi)).\) The decomposition, however, suggests that the out-of-\(\Omega\) values of \(\lambda(s)\) or \(h(s)\) are important to the performance of HUBL.

5.3 Finite-Sample Analysis for Bias and Regret↩︎

The finite-sample analysis of policy learning that we provide next illustrates more concretely how HUBL trades off bias and regret. Our analysis uses offline value iteration with lower confidence bound (VI-LCB) [15] as the base offline RL method. Following its original convention, we make some technical simplifications to make the presentation cleaner. Specifically, we consider a tabular setting with \(\lambda(s)=\alpha \in [0,1]\) for \(s\in\Omega\) as a constant value, which can be interpreted as the quality of the behavior policy averaged across states. Note that although \(\lambda(s) = \alpha\) is a constant on \(\Omega\), the reshaped MDP is still defined by \(\lambda(s,s')\) in 3 (i.e., \(\lambda(s,s')=\alpha\) if \(s,s'\in\Omega\) and zero otherwise). The details of VI-LCB with HUBL are in Appendix 10 due to space limits.

Theorem 2 summarizes our finite-sample results. The proof of Theorem 2 can be found in Appendix 9.

For the bias bound, the assumption \(h(s) = V^{\mu}(s)\) for any \(s\in \Omega\) is made for the ease of presentation. If it does not hold, an additive error term can be introduced in the bias bound to capture that. For the regret bound, \(\max_{s,a}\frac{ d^{\pi^*}(s,a;d_{0})}{\mu(s,a)}\) can be infinite, whereby the best regret bound is just \(V_{max}\). But when it is bounded as assumed by existing works [15], our results demonstrate how \(N\), \(\gamma\) and \(\lambda\) affect the regret bound.

The implications of Theorem 2 are threefold. First of all, it provides a finite-sample performance guarantee for HUBL with VI-LCB under the tabular setting. Compared with the performance bound of the original VI-LCB, \(\min\left( V_{max},\sqrt{\frac{V_{max}^2 \@ifstar{\abs}{\abs*}{\mathcal{S}}}{(1-\gamma)^3N }\max_{s,a} \frac{ d^{\pi^*}(s,a)}{\mu(s,a)} } \right)\), HUBL shrinks the discount factor by \(1-\lambda\) and thus potentially improves the performance while inducing bias. Second, Theorem 2 hints at the source of HUBL’s bias and regret of HUBL. The bias is related to the the performance of the behavior (data-collection) policy, characterized by \(V^*(s) - V^{\mu}(s)\). In the extreme case of data being collected by an expert policy, the bias induced by HUBL is 0. The regret is affected by \(\frac{ d^{\pi^*}(s,a)}{\mu(s,a)}\), which describes the deviation of the optimal policy from the data distribution. Finally, Theorem 2 also provides guidance on how to construct a blending factor function \(\lambda(\cdot)\). To reduce bias, \(\lambda(s)\) should be small at states where \(V^*(s)-V^{\mu}(s)\) is small. To reduce regret, \(\lambda(\cdot)\) should be generally large but small at states where the learned policy is likely to deviate from the behavior policy. Therefore, an ideal \(\lambda(\cdot)\) should be large when the behavior policy is close to optimal but small when it deviates from the optimal policy. This is consistent with our design principle in Section 4.2.

Extension to Model-based Offline RL Conceptually, we can implement HUBL with model-based offline RL methods by directly relabeling the data, learning the heuristic-modified reward model, and then doing model-based planning with a smaller discount, following 2 . Theorem 1 still applies to model-based algorithms. For Theorem 2, we expect that a similar analysis can be applied to model-based algorithms like MOReL. We defer this direction to future work and focus on improving the stability of a wide range of model-free offline RL methods.

6 Experiments↩︎

We study 27 benchmark datasets in D4RL and Meta-World. We show that HUBL improves the performance of existing offline RL methods by \(9\%\) with easy modifications and simple hyperparameter tuning. Remarkably, in some datasets where the base offline RL performs inconsistently or poorly, HUBL can achieve more than 50% performance improvement. When the dataset is simple and the base offline RL is already near-optimal, HUBL does not significantly harm the performance.

HUBL variants and base offline RL methods \(\,\) We implement HUBL with four state-of-the-art offline RL algorithms as base methods: CQL [4], TD3+BC [5], IQL [6], and ATAC [7]. For each base method, we compare the performance of its original version to its performance with HUBL running three different blending strategies discussed in Section 4.2: constant, sigmoid and rank. Thus, we experiment with 16 different methods in total. The implementation details of the base methods are in Appendix 11.1.

Metrics \(\,\) We use relative normalized score improvement, abbreviated as relative improvement, as a measure of HUBL’s performance improvement. Specifically, for a given task and base method, we first compute the normalized score \(r_{base}\) achieved by the base method, and then the normalized score \(r_{HUBL}\) of HUBL. The relative normalized score improvement is defined as \(\frac{r_{HUBL}-r_{base}}{\@ifstar{\abs}{\abs*}{r_{base}}}\). We report the relative improvement of HUBL averaged over three seeds \(\{0, 1, 10\}\) in this section, with standard deviations, base method performance, behavior cloning performance and absolute normalized scores provided in Appendix 11.2, 11.5, and 11.7.

Hyperparameter Tuning \(\,\) For each dataset, the hyperparameters of the base methods are tuned over six different configurations suggested by the original papers. HUBL has one extra hyperparameter, \(\alpha\), which is fixed for all the datasets but different for each base method. Specifically, \(\alpha\) is selected from \(\left\{0.001, 0.01, 0.1\right\}\) according to the relative improvement averaged over all the datasets. In practice, we notice that a single choice of \(\alpha\) around \(0.1\) is sufficient for good performance across most base offline RL methods and datasets as demonstrated by the sensitivity analysis in Appendix 11.3.

6.1 Performance Improvement of HUBL↩︎

We study 9 benchmark datasets in D4RL and 18 datasets for tasks in Meta-World, collected following the procedure detailed in Appendix 11.4. The relative improvement of HUBL with the rank labeling method is reported in 3 and 4. First, despite being simple and needing little hyperparameter tuning, HUBL improves the performance of base methods in most settings and only slightly hurts in some expert datasets where base offline RL methods are already performing very well with little space for improvement. Second, there are cases where HUBL achieve very significant relative improvement—more than 50%. Such big improvement happens in the datasets where the base method shows inconsistent performance and underperforms other offline RL algorithms. HUBL solves this inconsistent performance issue by simply relabeling the data. Further, HUBL improves performance even on data with few expert trajectories like hopper-med-rep-v2, walker2d-med-rep-v2, and hopper-med-v2, because HUBL conducts more bootstrapping and relies less on the heuristic on suboptimal trajectories (see Section 4.2).

r0.4

6.2 Ablation Studies↩︎

Discount Relabeling \(\,\) HUBL relabels both the reward and the discount factor, per 2 . In contrast, existing methods like [34] suggest that a lower discount factor alone, without blending heuristics into rewards, can in general improve offline RL. To assess the need for modifying both the discount factor and rewards like 2 , we consider ablation methods which shrink only the discount factor as \(\tilde{\gamma}\) without \(\tilde{r}\). The achieved average relative improvement of these ablations is reported and compared with that of HUBL in Table [tbl:tab:ablation]. HUBL consistently outperforms these ablations, which justifies HUBL’s coordinated modifications in both the reward and the discount factor. The advantage of HUBL is also consistent with what our theoretical analysis predicts. The considered ablations do not modify the rewards and thus are equivalent to solving \(\tilde{Q}^{\pi}(s,a) = r(s,a) + \gamma {\mathbb{E}}_{s' \sim {\mathcal{P}}(\cdot|s,a)}[ (1-\lambda(s'))\tilde{V}^{\pi}(s') ]\). Comparing it with 1 , the solution will be consistently smaller than the true \(Q\)-function, inducing a pessimistic bias. Crucially, this bias is much more challenging to tackle than the bias induced by HUBL, because the former is inevitable even when the data is from an expert policy.

Comparison of Blending Strategies \(\,\) We present results for HUBL with rank blending, while the results of other blending strategies (sigmoid and constant) are provided in Appendix  11.2 and 11.5. We notice that the rank blending outperforms the other two. With 27 datasets and 4 base methods, we have 108 cases. In Table 1 we report the number of cases where a given blending strategy provides the best performance among the three. We can see that rank is favored on average.

We also experiment HUBL in other different settings, including on datasets collected by expert policies (Appendix 11.6.1), with online RL methods (Appendix 11.6.2), and in stochastic environments (Appendix 11.6.3). HUBL shows consistent performance improvements in these settings.

7 Conclusion↩︎

In this work, we propose HUBL, a method for improving the performance of offline RL methods by blending heuristics with bootstrapping. HUBL is easy to implement and is generally applicable to various of offline RL methods. Empirically, we demonstrate the performance improvement of HUBL on 27 datasets in D4RL and Meta-World. We also provide a theoretic finite-sample performance bound for HUBL, which sheds lights on HUBL’s bias-regret trade-off and blending factor designs.

8 Extended Results of Theorem 1↩︎

Below we prove the statement which is a restatement of 1.

Specifically, the performance of \(\pi\) depends on both bias and regret. The bias term describes the discrepancy caused by solving the reshaped MDP with \(\lambda(\cdot)\). The regret term describes the performance of the learned policy in the reshaped MDP.

To prove Theorem 3, we first conduct a regret decomposition as \[\begin{align} V^*(s_0) - V^{\hat{\pi}}(s_0) &= \left( V^*(s_0) - \tilde{V}^{\pi^*}(s_0) \right) + \left( \tilde{V}^{\pi^*}(s_0) - \tilde{V}^{\hat{\pi}}(s_0) \right) + \left( \tilde{V}^{\hat{\pi}}(s_0) - V^{\hat{\pi}}(s_0) \right). \end{align}\] Then, we rewrite both \(V^*(s_0) - \tilde{V}^{\pi^*}(s_0)\) and \(\tilde{V}^{\hat{\pi}}(s_0) - V^{\hat{\pi}}(s_0)\) using the heuristics \(h(s)\).

8.1 Technical Lemmas↩︎

Before proceeding to the proof details of Theorem 3, we first prove a lemma on \(V^\pi(s_0) - \tilde{V}^{\pi}(s_0)\). For a policy \(\pi\) under the original MDP \({\mathcal{M}}\) and the reshaped MDP \(\tilde{{\mathcal{M}}}\), we aim to quantify the value difference using the heuristics \(h(s)\).

8.2 Proof of Theorem 3↩︎

To prove the theorem, we decompose the regret into the following terms: \[\begin{align} V^*(s_0) - V^{\hat{\pi}}(s_0) &= \left( V^*(s_0) - \tilde{V}^{\pi^*}(s_0) \right) + \left( \tilde{V}^{\pi^*}(s_0) - \tilde{V}^{\hat{\pi}}(s_0) \right) + \left( \tilde{V}^{\hat{\pi}}(s_0) - V^{\hat{\pi}}(s_0) \right) \end{align}\] We apply Lemma 1 to rewrite the first and the last terms as \[\begin{align} V^*(s_0) - \tilde{V}^{\pi^*}(s_0) &= \frac{\gamma}{1-\gamma} {\mathbb{E}}_{(s,a,s')\sim d^{\pi^*}}[ \lambda(s')( \tilde{V}^{\pi^*}(s') - h(s') ) | s, s' \in \Omega] \\ \tilde{V}^{\hat{\pi}}(s_0) - V^{\hat{\pi}}(s_0) &= \frac{\gamma}{1-\gamma}{\mathbb{E}}_{(s,a,s')\sim d^\pi}[\lambda(s') (h(s') - \tilde{V}^{\hat{\pi}}(s')) | s,s'\in\Omega]. \end{align}\]

Combining the two completes the proof.

9 Extended Results for Theorem 2↩︎

To prove Theorem 2, we separately bound the bias term \[\begin{align} \begin{aligned} \textrm{Bias}(\hat{\pi}, \lambda):= \frac{\gamma}{1-\gamma} {\mathbb{E}}_{(s,a,s')\sim d^{\pi^*}}[ \lambda(s')( \tilde{V}^{\pi^*}(s') - h(s') ) | s, s' \in \Omega ], \end{aligned} \end{align}\] and the regret term \[\begin{align} \begin{aligned} \textrm{Regret}(\pi, h, \lambda)&:= \tilde{V}^{\pi^*}(d_0) - \tilde{V}^{{\pi}}(d_0) + \frac{\gamma}{1-\gamma}{\mathbb{E}}_{(s,a,s')\sim d^\pi}[\lambda(s') ( h(s') - \tilde{V}^{{\pi}}(s')) | s, s'\in\Omega ]. \end{aligned} \end{align}\] Specifically the bias is bounded by Lemma 3 and the regret is bounded by Lemma 6.

9.1 Technical Lemmas↩︎

Now, we provide some technical lemmas for the proof of Theorem 2.

9.1.1 Bias Component↩︎

We first prove the upper bound for the bias component.

9.1.2 Regret Component↩︎

Next, we focus on the regret component.

9.2 Proof of Theorem 2↩︎

The proof follows by combining Lemma 3 and 6.

10 VI-LCB with HUBL↩︎

We use the offline value iteration with lower confidence bound (VI-LCB) [15] as the base algorithm to analyze concretely the effects of HUBL with finite samples for the tabular case.

10.1 Algorithm↩︎

We detail the procedure of HUBL when implemented with VI-LCB for the tabular setting. To start with, we introduce several definitions which will be used in the following algorithm and theoretical analysis. Without loss of generality, we assume that the sates take values in \(\left\{1, 2, \cdots, \@ifstar{\abs}{\abs*}{{\mathcal{S}}}\right\}\) and that the actions take values in \(\left\{1, 2, \cdots, \@ifstar{\abs}{\abs*}{{\mathcal{A}}}\right\}\). Then, let \(h\) be a \(\@ifstar{\abs}{\abs*}{{\mathcal{S}}}\times 1\) vector which denotes the heuristic function. We assume each component satisfies \(h_s = V^{\mu}(s)\) for \(s\in\Omega\). Let \(\Lambda\) be a \(\@ifstar{\abs}{\abs*}{{\mathcal{S}}}\times \@ifstar{\abs}{\abs*}{{\mathcal{S}}}\) matrix with \(\Lambda_{s,s'}=\lambda\) if \(s,s'\in\Omega\) and \(\Lambda_{s,s'}=0\) if \(s\) or \(s'\in\Omega\). We use \(\odot\) to denote the component-wise multiplication, and \(\Lambda_{s,:}\) to denote a row of \(\Lambda\) as a \(\@ifstar{\abs}{\abs*}{{\mathcal{S}}}\times 1\) vector. With slight abuse of notation, we use \(t\) to denote the index of iteration in this section, with \(T\) as the total number of iterations.

To conduct VI-LCB with HUBL, we follow Algorithm 3 in [15] but with a modified updating rule. The procedure is summarized Algorithm 5. Compared with the original VI-LCB, we highlight the key modification in Line 14 with blue color. Specifically, at the \(t^{\textrm{th}}\) iteration, based on 2 , we modify updating rule of Algorithm 3 in [15] into \[Q_{t}(s,a) \leftarrow r_t(s,a)-b_t(s,a) + \gamma P_{s,a}^t\odot(I -\Lambda_{s,:})\cdot V_{t-1} + \gamma P_{s,a}^t\odot\Lambda_{s,:} \cdot h.\] Note that we introduce heuristics by \(\gamma P_{s,a}^t\odot\Lambda_{s,:} \cdot h\), while reducing the bootstrapping by \(\gamma P_{s,a}^t\odot(I -\Lambda_{s,:})\cdot V_{t-1}\).

10.2 Regret Analysis↩︎

In this section, we study the regret under the reshaped MDP constructed by HUBL. Specifically, we bound the regret of Algorithm 5 in Lemma 7.

10.3 Notations↩︎

We first provide some matrix notations for MDPs. We use \(P^{\pi} \in \mathbb{R}^{\@ifstar{\abs}{\abs*}{{\mathcal{S}}}\@ifstar{\abs}{\abs*}{{\mathcal{A}}} \times \@ifstar{\abs}{\abs*}{{\mathcal{S}}} \@ifstar{\abs}{\abs*}{{\mathcal{A}}}}\) to denote the transition matrix induced by policy \(\pi\) whose \((s,a)\times(s,a')\) element is equal to \(P(s'|s,a)\pi(a'|s')\), and \(d^{\pi} \in \mathbb{R}^{\@ifstar{\abs}{\abs*}{{\mathcal{S}}}\@ifstar{\abs}{\abs*}{{\mathcal{A}}}}\) to denote a state-action distribution induced by policy \(\pi\) whose \((s,a)\) element is equal to \(d(s)\pi(a|s)\). Similarly, we use \(\Lambda_Q\in \mathbb{R}^{\@ifstar{\abs}{\abs*}{{\mathcal{S}}}\@ifstar{\abs}{\abs*}{{\mathcal{A}}} \times \@ifstar{\abs}{\abs*}{{\mathcal{S}}} \@ifstar{\abs}{\abs*}{{\mathcal{A}}}}\) to denote a extended matrix version of \(\Lambda\).

Further, we focus on the policies which stay in the data distribution support \(\Omega\). Such polices are formally defined as \(\left\{\pi|d^{\pi}(s,a;s_0) = 0 \textrm{ for any } (s,a)\notin\Omega \textrm{ and } s_0\in\Omega\right\}\). We also define a clean event: \[\mathcal{E}_{MDP} := \left\{\forall s,a,t, \@ifstar{\abs}{\abs*}{r(s,a)-r_t(s,a)+\gamma(P_{s,a}-P_{s,a}^t)\odot(I-\Lambda)\cdot V_{t-1} } \leq b_t(s,a) \right\}.\]

10.4 Technical Lemmas↩︎

With the aforementioned definitions and assumptions, we provide the following lemmas.

10.5 Proof of Lemma 7↩︎

By the event \(\mathcal{E}_{cover}\), we can decompose the regret into \[\begin{align} \label{eq:in-support-regret} \begin{aligned} & {\mathbb{E}}_{\mathbb{D}}[\tilde{V}^{\pi^*}(d_{init}) - \tilde{V}^{\hat{\pi}}(d_{init})] \\&\leq {\mathbb{E}}_{\mathbb{D}}[\tilde{V}^{\pi^*}(d_{init}) - \tilde{V}^{\hat{\pi}}(d_{init})\mathbb{1}\left\{\mathcal{E}_{cover}\right\}] + {\mathbb{E}}_{\mathbb{D}}[\tilde{V}^{\pi^*}(d_{init}) - \tilde{V}^{\hat{\pi}}(d_{init})\mathbb{1}\left\{\mathcal{E}_{cover}^c\right\}] \\ &\leq {\mathbb{E}}_{\mathbb{D}}[\tilde{V}^{\pi^*}(d_{init}) - \tilde{V}^{\hat{\pi}}(d_{init})\mathbb{1}\left\{\mathcal{E}_{cover}\right\}] + V_{max} \@ifstar{\abs}{\abs*}{{\mathcal{S}}}\bigg(\frac{\@ifstar{\abs}{\abs*}{{\mathcal{S}}}-1}{\@ifstar{\abs}{\abs*}{{\mathcal{S}}}}\bigg)^\frac{N}{2}, \end{aligned} \end{align}\tag{9}\] where the second inequality is by Lemma 8. Next, we focus on \({\mathbb{E}}_{\mathbb{D}}[\tilde{V}^{\pi^*}(d_{init}) - \tilde{V}^{\hat{\pi}}(d_{init})\mathbb{1}\left\{\mathcal{E}_{cover}\right\}]\). Following the analysis in C.5 of [15], we can derive \[\begin{align} \begin{aligned} &{\mathbb{E}}_{\mathbb{D}}[(\tilde{V}^{\pi^*}(d_{init}) - \tilde{V}^{\hat{\pi}}(d_{init}))\mathbb{1}\left\{\mathcal{E}_{cover}\right\}] \\ \leq &{\mathbb{E}}_{\mathbb{D}}[(\tilde{V}^{\pi^*}(d_{init}) - \tilde{V}^{\hat{\pi}}(d_0))\mathbb{1}\left\{\mathcal{E}_{cover}\right\}] \\ \leq& {\mathbb{E}}_{\mathbb{D}}[{\mathbb{E}}_{s\sim d_{init}}[(\tilde{V}^{\pi^*}(d_{init}) - \tilde{V}^{\hat{\pi}}(d_{init})) \mathbb{1}\left\{\exists t\leq T,m_t(s,\pi^*(s))=0 \right\}\mathbb{1}\left\{\mathcal{E}_{cover}\right\} ]]:=T_1 \\&+ {\mathbb{E}}_{\mathbb{D}}[{\mathbb{E}}_{s\sim d_{init}}[(\tilde{V}^{\pi^*}(d_{init}) - \tilde{V}^{\hat{\pi}}(d_{init})) \\&\mathbb{1}\left\{\forall t\leq T,m_t(s,\pi^*(s))\geq 1 \right\}\mathbb{1}\left\{\mathcal{E}_{MDP}\right\}\mathbb{1}\left\{\mathcal{E}_{cover}\right\} ]]:=T_2 \\&+ {\mathbb{E}}_{\mathbb{D}}[{\mathbb{E}}_{s\sim d_{init}}[(\tilde{V}^{\pi^*}(d_{init}) - \tilde{V}^{\hat{\pi}}(d_{init})) \\&\mathbb{1}\left\{\forall t\leq T,m_t(s,\pi^*(s))\geq 1 \right\}\mathbb{1}\left\{\mathcal{E}^c_{MDP}\right\}\mathbb{1}\left\{\mathcal{E}_{cover}\right\} ]]:=T_3. \end{aligned} \end{align}\] By Lemma 8, under the event \(\mathcal{E}_{cover}\), \(\hat{\pi}\) stays in \(\Omega\). In other words, Lemma 9, 10 and 11 are all applicable to \(\hat{\pi}\).

Next, for the following analysis, we consider the case that \(\pi^*\) stays in \(\Omega\). We will discuss the case that \(\Omega\) does not cover \(\pi^*\) at the end of the proof.

By Lemma 9 and C.5.1 of [15], \[T_1 \leq \frac{8V_{max}\@ifstar{\abs}{\abs*}{{\mathcal{S}}}T^2 \max_{s,a}\frac{ \tilde{d}^{\pi^*}(s,a; d_{init})}{\mu(s,a)}}{9(1-(1-\lambda)\gamma)N}.\] By Lemma 11 and C.5.2 of [15], \[T_2\leq V_{max}\gamma^T (1-\lambda)^T + 32\frac{V_{max}}{1-\gamma(1-\lambda)}\sqrt{\frac{2L\@ifstar{\abs}{\abs*}{{\mathcal{S}}}T\max_{s,a}\frac{ \tilde{d}^{\pi^*}(s,a; d_{init})}{\mu(s,a)}}{N}}.\] By C.5.2 of [15], \[T_3 \leq \frac{V_{max}}{N}.\] Combining \(T_1\), \(T_2\) \(T_3\), and 9 , with \(T=\log N/(1-\gamma(1-\lambda))\), \[{\mathbb{E}}_{\mathbb{D}}[\tilde{V}^{\pi^*}(d_{init}) - \tilde{V}^{\hat{\pi}}(d_{init})] \lesssim \min\bigg(V_{max}, V_{max}\sqrt{\frac{\@ifstar{\abs}{\abs*}{{\mathcal{S}}}\max_{s,a}\frac{ \tilde{d}^{\pi^*}(s,a; d_{init})}{\mu(s,a)}}{N(1-\gamma(1-\lambda))^3}}\bigg).\] Finally, by Lemma 12, we finish the proof: \[\label{eq:regret-final} {\mathbb{E}}_{\mathbb{D}}[\tilde{V}^{\pi^*}(d_{init}) - \tilde{V}^{\hat{\pi}}(d_{init})] \lesssim \min\bigg(V_{max}, V_{max}\sqrt{\frac{(1-\gamma)\@ifstar{\abs}{\abs*}{{\mathcal{S}}}\max_{s,a}\frac{ d^{\pi^*}(s,a;d_{init})}{\mu(s,a)}}{N(1-\gamma(1-\lambda))^4}}\bigg).\tag{10}\] Note that 10 is derived under the assumption that \(\pi^*\) stays in \(\Omega\). However, it also holds when \(\pi^*\) is not covered by \(\Omega\). In that case, \(\max_{s,a}\frac{ d^{\pi^*}(s,a;d_{init})}{\mu(s,a)} = \infty\) and \({\mathbb{E}}_{\mathbb{D}}[\tilde{V}^{\pi^*}(d_{init}) - \tilde{V}^{\hat{\pi}}(d_{init})] \leq V_{max}\).

11 Extended Results for Experiments↩︎

We conduct HUBL with four offline RL base methods on 27 datasets in D4RL and Meta-World. By blending heuristics with bootstrapping, HUBL reduces the complexity of decision-making and provides smaller regret while generating limited bias. We demonstrate that HUBL is able to improve the performance of offline RL methods.

11.1 Implementation Details↩︎

We implement base offline RL methods with code sources provided in Table 2.

Experiments with ATAC, IQL, and TD3+BC are ran on Standard_F4S_V2 nodes of Azure, and experiments with CQL are ran on NC6S_V2 nodes of Azure. As suggested by the original implementation of the considered base offline RL methods, we use 3-layer fully connected neural networks for policy critics and value networks, where each hidden layer has 256 neurons and ReLU activation and the output layer is linear. The first-order optimization is implemented by ADAM [38] with a minibatch size as 256. The learning rates are selected following the original implementation and are reported in Table 3.

For each dataset, the hyperparameters of base methods are tuned from six different configurations suggested by the original papers. Such configurations are summarized in Table 4.

Meanwhile, HUBL has one extra hyperparameter, \(\alpha\), which is fixed for all the datasets but different for each base method. Specifically, \(\alpha\) is selected from \(\left\{0.001, 0.01, 0.1\right\}\) according to the relative improvement averaged over all the datasets in one task. Later in the robustness analysis, we show that the performance of HUBL is insensitive to the selection of \(\alpha\). For each configuration and each base method, we repeat experiments three times with seeds in \(\left\{0, 1, 10\right\}\).

11.2 Base Performance and Standard Deviations for D4RL Datasets↩︎

We provide the performance of base offline RL methods and standarde deviations in Table 5.

Next, we study how the performance of HUBL vary over different epochs. Specifically, we provide a plot of the achieved normalized average returns over different epochs for HUBL with TD3+BC on hopper-medium-v2 in Figure 6.

11.3 Robustness of HUBL to \(\alpha\)↩︎

In this section, we demonstrate the robustness of HUBL to \(\alpha\). Specifically, we provide the average relative improvements of HUBL on D4RL datasets in Table ¿tbl:tab:robustness?. Notice that most average relative improvements are positive, which shows that HUBL can improve the performance with different values.

The selected \(\alpha\)’s are reported in Table 6:

11.4 Data Collection for Meta-World↩︎

We collect data for Meta-World tasks using normalized rewards with goal-oriented stopping. Specifically, given that the original rewards of Meta-World are in \([0,10]\), we shift them by \(-10\) and divide by \(10\), so that the normalized rewards take values in \([-1,0]\). Then, we use the hand scripted policy given in Meta-World with different Gaussian noise levels in \(\left\{0.1, 0.5, 1\right\}\) to collect 100 trajectory for each task. In this process, a trajectory ends if

Note that we follow the same rule when testing the performance of a learned policy.

11.5 Base Performance and Standard Deviations for Meta-World Datasets↩︎

We provide the performance of base offline RL methods and standard deviations in Table 7 and 8. We also consider behavior cloning (BC) as as a baseline, and also a representative for imitation learning and inverse reinforcement learning methods [39]–[41]. Since the behavior policy is the scripted policy with Gaussian noise, BC can effective recover the scripted policy and thus is especially competitive.

11.6 Extended Experiments↩︎

In this section, we study the performance of HUBL in various of scenarios.

11.6.1 HUBL on Expert Data↩︎

We study the performance of HUBL on expert collected data. Specifically, we conduct experiments of HUBL with TD3+BC on expert datasets in D4RL. The results are reported in Table 9. Note that HUBL is able to provide performance improvements in most of the cases. But the room for improvement on such datasets is small – for any offline learning technique. Therefore, due to finite sample error, HUBL may occasionally hurt performance empirically

11.6.2 HUBL with Online RL Methods↩︎

We provide experiments where we combine HUBL with DDPG on D4RL datasets, without any additional designs for the offline RL setting. The results are provided in Table 10.

Note that HUBL is able to significantly improve DDPG’s performance in relative terms. However, the absolute performance of DDPG+HuBL is much worse than other methods we considered in the main paper. This is expected, as HUBL is designed to augment offline RL algorithms, as opposed to making a general online RL algorithm (like DDPG) work also offline.

To see this more clearly, we consider a toy example with three actions \(a_1\), \(a_2\), \(a_3\) at state \(s_0\). Suppose that and \(a_1\) and \(a_2\) are all out-of-support. As a result, without additional designs (such as pessimism) for offline RL, the policy may take \(a_1\) and \(a_2\) regardless how we set the heuristic, since the heuristic here would only affect the value for in-support actions and not the value for out-of-support actions. Thus, HUBL cannot turn an online RL algorithm to an offline RL algorithm; it can only enhance an algorithm that is designed for offline RL already.

11.6.3 HUBL in Stochastic Environments↩︎

To test the performance of HUBL in stochastic environments, we construct a random version of walker2d-v2 by adding \(\beta \epsilon\) to the action passed to the environment, where \(\epsilon\) is a uniform random variable taking values in \([-1,1]\), and \(\beta\) is a scalar controlling the amount of randomness. Note that we keep the scale of noise in \([-1,1]\), and the action variable itself is normalized to \([-1,1]\) in D4RL. With the new environment, we regenerate the data using the original walker2d-meidum-v2 policy and apply TD3+BC with HUBL. The results are presented in Table 11.

The base performance decreases as the noise increases, as expected. However, HUBL is still able to provide some performance improvement under this stochastic setting. We also notice that the constant labeling function is more robust to the noise. This makes sense as the constant function is not sensitive to the heuristic estimation. But if there is more noise, the performance of HUBL will further degenerate, which is a limitation of HUBL.

11.7 Extended Results for Absolute Improvement↩︎

We report the experiment results in absolute improvement of normalized score for D4RL and Meta-World.

12 Limitations and Future Work↩︎

In our current framework, HUBL calculates heuristic values as Monte-Carlo returns. This is feasible in practical scenarios where data comes in the form of trajectories, but much of offline RL literature experiments with batch datasets that consist of disconnected transition tuples. For such datasets, HUBL requires other heuristic calculation strategies.

Empirically, we have showed HUBL’s utility on benchmarks where observations are full-information low-dimensional state vectors and state transitions are deterministic. While these benchmarks are standard in offline RL, evaluation in more stochastic domains where observations are partial and high-dimensional, such as robotics, will give a more complete picture of HUBL’s utility. So would evaluating HUBL with model-based offline RL such as MOReL [14]. Also, HUBL is not applicable to bootstrapping-free offline RL methods [42]–[46].

On the theory side, our analysis focuses on state-dependent heuristics. In practice, though, heuristic estimates commonly depend on full trajectories, and our theory needs adjustments to account for this. Same goes for applying our theory to model-based offline RL such as MOReL.

For future direction, we aim to develop better heuristics and blending strategies for batch datasets and sparse-reward problems. We also plan to apply HUBL to other tasks in healthcare and finance [47]–[50] where collecting data is especially expensive.

References↩︎