1 Introduction↩︎

Off-policy evaluation (OPE) [1], [2] is used to estimate the value of a new (target) reinforcement learning (RL) policy prior to deployment using a historical (behavior) dataset from a distinct behavior policy. OPE is particularly important in high-stakes domains [3]–[5], where directly deploying new policies without prior evaluation can be costly or even harmful to participants.

However, standard OPE methods frequently struggle when the target policy is very different from the behavior policy. This is due to limited dataset coverage [6]. To address this, several recent works have proposed using synthetic auxiliary data to improve the coverage of the behavior dataset and subsequently the accuracy of OPE methods [7]–[9]. However, such approaches have either focused on the contextual bandit setting, or focused on promising empirical success in sequential settings; they lack formal assurances on the quality of the proposed estimates.

In high stakes, multi-step domains, it is often important to have confidence intervals (CIs) over the proposed policy value estimates. Such intervals support safer, more informed policy selection and deployment. Therefore, we argue that principled uncertainty quantification is necessary for OPE in RL in the emerging regime where both real and synthetic trajectories are used. While there is a notable body of prior work that developed CIs using only offline (real) data for OPE in RL [10]–[13], to our knowledge, none provides guarantees in settings that combine offline and synthetic trajectories. In this paper we take steps towards addressing this gap.

We formalize uncertainty quantification for OPE with mixed (real and synthetic) behavior data and identify two settings that require uncertainty-aware OPE. First, in domains like healthcare, it is common for stakeholders to deliberate between decision-making policies to use for individuals that start in the same state: for example, a clinician may use the same treatment policy on all patients in the same stage of a disease. Estimating CIs for initial-state-conditioned policy performance is thus an important task that can benefit substantially from data augmentation. Our first method, CP-Gen, provides conformal prediction intervals for such state-conditioned values.

Second, we address evaluation of the target policy’s expected value averaged over the distribution of initial states. This is relevant in settings where a single policy may be selected for the whole population, and a stakeholder wants to choose among different policies. We introduce a second method DR-PPI, which leverages techniques from doubly robust estimation and prediction-powered inference [14] to correct biases from synthetically generated trajectories and produce valid CIs.

Our empirical studies across inventory control, sepsis treatment, robotic control, and the MIMIC-IV electronic health record (EHR) dataset demonstrate that our methods, which can leverage synthetic data, can match or improve over state-of-the-art baselines that provide correct CIs using only real data. Our contributions follow.

1. We formalize the problem of uncertainty quantification for OPE in MDPs that leverage synthetically generated trajectories and introduce CP-Gen and DR-PPI (3) for two natural settings where CIs are important.

2. We prove that both methods yield valid CIs and achieve the desired coverage probability either asymptotically or within a small margin of error for finite sample sizes (4).

3. We empirically evaluate the estimators in four domains including a real-world EHR dataset, showing that our estimates, which leverage auxiliary synthetic data, produce CIs with the correct coverage that match or are tighter than baselines that do not use the auxiliary data. (5).

2 Background↩︎

2.1 Problem setting↩︎

We consider a decision-making setting defined by the MDP \(\mathcal{M}=(\mathcal{S},\mathcal{A},P,R,d_{0},\gamma,H)\). \(\mathcal{S}, \mathcal{A}\) denote the possibly infinite state and action spaces respectively. \(P: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})\) represents the transition dynamics, \(R: \mathcal{S} \times \mathcal{A} \to \Delta(\mathbb{R})\) is the reward function, and \(d_0 \in \Delta(\mathcal{S})\) is the initial state distribution. \(\gamma\) is the discount factor and \(H\) is the fixed horizon. A trajectory \(\tau\) is defined as \(\tau: \{s_t, a_t, r_t\}_{t=0}^H\) where \(s_t, a_t, r_t\) are the state, action, and instantaneous reward observed at timestep \(t\). The return of a trajectory \(\tau\) is \(J(\tau) = \sum_{t=1}^H \gamma^{t-1} r_t\) where \(\tau \sim \pi\) and \(\pi\) is the policy used to generate the trajectory. The value of the policy \(V^{\pi} = \mathbb{E}_{\tau \sim \pi}[J(\tau)]\) is calculated as an expectation over the possible trajectories that could arise from \(\pi\). The value of a policy conditioned on an initial starting state \(s\) is \(V^{\pi}(s) = \mathbb{E}_{\tau \sim \pi}[J(\tau) | s_0 = s]\).

2.2 Off-policy evaluation (OPE)↩︎

Our work focuses on the task of OPE, where the goal is to estimate the value of a target policy \(\pi_e\) given a dataset of behavior trajectories \(D_{\pi_b}\) that arise from a distinct behavior policy \(\pi_b\). In a typical OPE setup, we assume access to \(\pi_e\). In our work, we also assume \(\pi_b\) is known similar to prior work [15], [16], though we apply our methods empirically in settings where \(\pi_b\) must be estimated.

There are several standard approaches for OPE, including importance sampling (IS)  [1], direct method (DM) [17]–[22], and doubly robust (DR) approaches  [6], [16], [23]. IS-based estimators re-weigh each trajectory in the \(D_{\pi_b}\) using an inverse propensity score (IPS) \(\rho(\tau) = \prod_{t=1}^H \frac{\pi_e(a_t|s_t)}{\pi_b(a_t|s_t)}\). DM estimators learn a reward model using the behavior trajectories to directly estimate the value of the target policy. DR methods combine the advantages of IS and DM estimators and provide favorable guarantees when either the IPS ratio or the reward model is inaccurate.

2.3 Conformal Prediction↩︎

One strategy we consider for uncertainty quantification is conformal prediction, which is a framework for constructing prediction intervals with finite-sample coverage guarantees under minimal assumptions [24]. Given a dataset of i.i.d.samples \(\{Z_i\}_{i=1}^n\) and a nonconformity score function \(V(Z)\), CP constructs a prediction set \(\mathcal{C}_{n,\alpha}\) such that \[\mathbb{P}(Z_{n+1} \in \mathcal{C}_{n,\alpha}) \ge 1 - \alpha,\] without requiring distributional assumptions beyond exchangeability.

In its simplest form, split conformal prediction partitions the data into a training set and a calibration set. A model is fit on the training set, and nonconformity scores \(\{V_i\}_{i=1}^n\) are computed on the calibration set. The prediction interval is then constructed using empirical quantiles: \[\mathcal{C}_{n,\alpha} = \{z : V(z) \le Q_{1-\alpha}(\{V_i\}_{i=1}^n)\}.\]

In settings with distribution shift, such as OPE, the calibration data are not drawn from the target distribution. Weighted conformal prediction addresses this by assigning importance weights \(w_i\) to each calibration sample [12], [25]. The empirical distribution is replaced by a weighted empirical distribution \[F_n(v) = \sum_{i=1}^n p_i \mathbf{1}\{V_i \le v\}, \quad p_i = \frac{w_i}{\sum_{j=1}^n w_j + w_{n+1}},\] where \(w_{n+1}\) corresponds to the test point.

The resulting prediction set uses weighted quantiles \[\mathcal{C}_{n,\alpha} = \{z : Q_{\alpha/2}(F_n) \le V(z) \le Q_{1-\alpha/2}(F_n)\}.\]

2.4 Prediction-Powered Inference↩︎

A related line of work addresses the problem of drawing valid statistical conclusions when labeled data is scarce but model predictions can be used to add dataset coverage. Prediction-powered inference (PPI) [14] is a framework for constructing valid confidence intervals by combining a small labeled dataset with a larger unlabeled dataset augmented with model predictions.

Let \(\{(X_i, Y_i)\}_{i=1}^n\) denote a labeled dataset, where \(X_i \in \mathcal{X}\) are inputs and \(Y_i \in \mathbb{R}\) are observed outcomes. In addition, suppose we have access to a larger unlabeled dataset \(\{X_j\}_{j=1}^N\), for which the corresponding outcomes are not observed. Let \(\hat{f}(X)\) denote a prediction model trained to estimate \(Y\) from \(X\).

The goal is to estimate a population parameter of the form \[\theta = \mathbb{E}[Y],\] or more generally, \(\theta = \mathbb{E}[g(X, Y)]\) for some function \(g\). A prediction-powered estimator takes the form \[\hat{\theta}_{\text{PPI}} = \frac{1}{N} \sum_{j=1}^N \hat{f}(X_j) + \frac{1}{n} \sum_{i=1}^n \big(Y_i - \hat{f}(X_i)\big),\] where the first term uses model predictions over the large unlabeled dataset for efficiency, and the second term corrects for bias using the labeled data. The PPI estimator can be interpreted as a bias-corrected plug-in estimator: the model-based term provides a low-variance estimate, while the residual correction ensures consistency even if the model \(\hat{f}\) is misspecified. Under mild conditions, \(\hat{\theta}_{\text{PPI}}\) is asymptotically normal and admits valid confidence intervals.

In our work, we take inspiration from the construction of the standard PPI estimator, which yields an asymptotic CI. However, the problem setup in PPI is distinct from ours. PPI assumes that we have access to a large dataset of observations that are unlabeled; the role of the ML model is to label the observations. In contrast, in our setting, we must both generate synthetic samples (i.e., trajectories) and their corresponding labels (i.e., returns); this setting necessitates a distinct methodology.

2.5 Related Literature↩︎

OPE with data augmentation. As discussed in 1, standard OPE methods suffer when the behavior dataset has limited coverage. Because OPE methods are typically used with finite sample sizes, OPE estimates can be biased or have high variance [1], [6], [15]. To address this concern, several works have proposed using auxiliary information to enhance OPE estimators, using data augmentation either from a secondary dataset [7], [9] or by generating synthetic trajectories based on historical data [8], [26], [27].

Within model-based approaches, different classes of generative models have been considered. Some works learn transition models (e.g., Neural Networks [28] or Variational Autoencoders (VAEs) [8]) that enable step-wise rollout under a policy, while others employ trajectory-level generative models such as diffusion-based models [26] to generate entire trajectories. These approaches differ in how they model dynamics and generate samples, but share the common goal of improving coverage of the state-action space. These works find that leveraging auxiliary data can improve OPE estimates in some domains such as robotic control. However, these methods may introduce additional bias due to errors in the auxiliary data, and lack theoretical guarantees or rigorous uncertainty quantification for the MDP setting. In contrast, our work emphasizes uncertainty quantification, which can provide more information to effectively compare between policies.

Conformal prediction for OPE. There are several strategies to perform uncertainty quantification, including conformal prediction. [12] first applied weighted conformal prediction to the OPE setting for contextual bandits. [13] later extended this work to create conformal intervals for OPE in the multi-step setting. Crucially, their approximation relies on an integral that is difficult to compute and implement in setting with continuous state spaces. One of our proposed methods, CP-Gen, is inspired by the last approach, but uses a novel technique to compute the weights needed for conformal prediction, allowing us to use OPE in settings with continuous and higher-dimensional state spaces than prior work. In addition, prior work did not consider data augmentation. Our new approach achieves tighter CIs through careful use of auxiliary synthetic datasets.

3 Methods↩︎

In this work, we study a setting in which we have access to a real, offline dataset, and synthetically generated trajectories. In general, trajectories produced by generative models may be biased or drawn from a distribution distinct from \(\pi_b\), which can introduce error and/or variance into the resulting OPE estimate for sequential decision processes. We propose two new methods for computing CIs for OPE in RL for two common settings where CIs would be beneficial. For clarity, we summarize key notation in a table that can be referenced throughout the paper (1).

3.1 CP-Gen: Confidence Intervals for OPE from a Starting State↩︎

First, we consider a setting in which our goal is to estimate an initial-state-conditioned policy value \(V^{\pi_e}(s)\). Estimating state-conditioned policy values has been understudied in the OPE literature, which tends to focus on estimators that average performance over the full population of initial states. The task of estimating state-conditioned policy values can benefit from data augmentation, since data from individual starting states is sparse. To address this, we propose CP-Gen, a new conformal prediction method for OPE.

Given an initial state \(s\), we estimate \(V^{\pi_e}(s)\) by carefully manipulating the definition of the policy value as follows. \[\begin{align} V^{\pi_e}(s) &= \mathbb{E}_{\tau \sim p^{\pi_e}|s_0=s }\left[J(\tau)\right]\\ &=\sum_{\tau \sim p^{\pi_e}|s_0=s}p^{\pi_e}(\tau)J(\tau) \\ &=\underbrace{ \sum_{\tilde{\tau} \sim \tilde{p}^{\pi_e}|s_0=s}\tilde{p}^{\pi_e}(\tilde{\tau})J(\tilde{\tau})}_{\text{simulator estimate}} + \underbrace{\left[\sum_{\substack{\tau \sim p^{\pi_e}|s_0=s}}p^{\pi_e}(\tau)J(\tau) - \sum_{\tilde{\tau} \sim \tilde{p}^{\pi_e}|s_0=s}\tilde{p}^{\pi_e}(\tilde{\tau})J(\tilde{\tau})\right]}_{\text{model bias / return discrepancy}} \label{eq:add95subtract} \end{align}\tag{1}\] Here, \(\tau \sim p^{\pi_e}\) is a trajectory drawn from the dynamics distribution associated with \(\pi_e\) and \(p^{\pi_e}(\tau)\) is the probability of observing trajectory \(\tau\) under the policy \(\pi_e\). In 1 , we add and subtract the simulator estimate of the state-conditioned policy value \(\sum_{\tilde{\tau} \sim \tilde{p}^{\pi_e}|s_0=s}\tilde{p}^{\pi_e}(\tilde{\tau})J(\tilde{\tau})\), where \(\tilde{p}\) is the dynamics distribution induced by the generative model and \(\tilde{\tau}\) is a synthetic trajectory sampled from a generative model that approximates the dynamics distribution of the target policy, \(\tilde{p}^{\pi_e}\).

We can now approximate the “model bias/return discrepancy” term using empirical averages. We note that the empirical averages require access to trajectories that are sampled from the target policy distribution, \(p^{\pi_e}\). The “model bias/return discrepancy” term can be approximated as \[\begin{align} & \approx \sum_{\tilde{\tau} \sim \tilde{p}^{\pi_e}|s_0=s}\tilde{p}^{\pi_e}(\tilde{\tau})J(\tilde{\tau}) + \underbrace{\frac{1}{n}\sum_{i=1}^n J(\tau_i|s_0=s) - \frac{1}{nM}\sum_{j=1}^{nM}J(\tilde{\tau}_j|s_0=s)}_{\text{approximate the expected value by empirical average}}\\ &= \sum_{\tilde{\tau} \sim \tilde{p}^{\pi_e}|s_0=s}\tilde{p}^{\pi_e}(\tilde{\tau})J(\tilde{\tau}) + \frac{1}{nM}\sum_{i=1}^n \sum_{j=1}^{M}\underbrace{(J(\tau_i|s_0=s) - J(\tilde{\tau}_{ij}|s_0=s))}_{\text{return difference of a pair of trajectories}} \label{eqn:return95difference}, \end{align}\tag{2}\]

where \(n/M\) is the number of behavior/synthetic trajectories, and \(J(\tau|s_0=s)\) is the return of trajectory \(\tau\) given initial state \(s\).

Inspired by conformal prediction for regression, our goal is to produce an interval \(\hat{C}_{n,\alpha}(s)\), which is specific to initial state \(s\) and the number of offline behavior trajectories \(n\). This interval defines a band such that, with high probability, the return difference between any offline trajectory and its corresponding generated trajectory that starts from the same initial state \(s\) (“return difference of a pair of trajectories” in 2 ) lies within the band. More specifically, \[P^{\pi_e}\left(\underbrace{J(\tau|s_0=s) - J(\tilde{\tau}|s_0=s)}_{\text{return difference of a pair of trajectories}} \in \hat{C}_{n,\alpha}(s)\right) \ge 1-\alpha,\] where \(P^{\pi_e}\) is the probability measure induced by the target policy \(\pi_e\) and \(\alpha\) is the confidence level. Given this goal, the final conformal prediction interval for the value of the initial state \(s\), \(V^{\pi_e}(s)\), is \[\label{eqn:cp95final95interval} \sum_{\tilde{\tau}\sim \tilde{p}^{\pi_e},s_0=s}\tilde{p}^{\pi_e}(\tilde{\tau})J(\tilde{\tau}) + \hat{C}_{n,\alpha}(s).\tag{3}\]

The derivation above assumes access to trajectory pairs \((\tau, \tilde{\tau})\) drawn under the target policy \(\pi_e\). However, in practice, we only observe real trajectories from the behavior policy \(\pi_b\). To construct comparable pairs, we generate synthetic trajectories \(\tilde{\tau}\) using the learned model while conditioning on initial states observed under \(\pi_b\), and rolling out using \(\pi_b\). As a result, the empirical return differences \(J(\tau \mid s_0 = s) - J(\tilde{\tau} \mid s_0 = s)\) are drawn from a distribution induced by \(\pi_b\), rather than \(\pi_e\). To bridge this gap, we apply weighted conformal prediction, which reweights these samples to approximate the distribution of return differences under \(\pi_e\), enabling valid inference despite the distribution shift.

Now, we use conformal prediction to calculate the band. Unlike standard conformal prediction, we must address the distribution shift induced by the difference between the behavior and target policies. To do so, prior work [13], which builds on related work [12], [25], proposed CP methods for MDPs that weigh the calibration scores using estimates of the likelihood ratio.

However, this prior work does not consider the use of generated trajectories. Therefore we introduce a new sample reweighting technique that accounts for the distribution shift (between \(\pi_b\) and \(\pi_e\)) in both the real and generated trajectories (see full derivation in Appendix 11). To simplify notation, let \(s \in S\) be the initial state and \(\delta_{rr'} \in \Delta_{rr'}\) be the return difference of a pair of trajectories (one from the original behavior dataset, and one generated). Then, the weight for a given sample is \[\begin{align} \tag{4} w(s,\delta_{rr'})&:= \mathbb{P}^{\pi_e}_{(S,\Delta_{rr'})}(s,\delta_{rr'}) /\mathbb{P}^{\pi_b}_{(S,\Delta_{rr'})}(s,\delta_{rr'})\\ & = \mathbb{E}_{\tau \sim p^{\pi_b}, \tilde{\tau} \sim \tilde{p}^{\pi_b}}\left[\underbrace{\frac{\prod_{t=1}^H\pi_e(a_t|s_t)\pi_e(\tilde{a}_t|\tilde{s}_t)}{\prod_{t=1}^H\pi_b(a_t|s_t)\pi_b(\tilde{a}_t|\tilde{s}_t)}}_\text{\shortstack{ product of IPS ratios\\ of real and generated trajectories }}| \underbrace{s_0 = s, \delta_{J(\tau)J(\tilde{\tau})}=\delta_{rr'}}_\text{\shortstack{ conditioned on same initial state\\ and reward difference }}\right]. \tag{5} \end{align}\] This weight is an expectation of the IPS ratio over all observations that share the same input (\(s\)) and score (\(\delta_{rr'}\)). However, calculating \(w(s, \delta_{r r'})\) will become intractable as the size of the MDP increases, because we may not have access to many trajectories that share \(s\) and \(\delta_{rr'}\).

To mitigate this, and allow us to compute valid conformal prediction intervals in continuous state and action spaces, we use something we refer to as “\(\epsilon-\)approximation” to estimate the weight for a given sample. With \(\epsilon\)-approximation, we can approximate the weight as \[\label{eqn:cpgen95weight} w_\epsilon(s, \delta_{rr'}) = \mathbb{E}_{\tau \sim p^{\pi_b}, \tilde{\tau} \sim \tilde{p}^{\pi_b}}\left[\frac{\prod_{t=1}^H\pi_e(a_t|s_t)\pi_e(\tilde{a}_t|\tilde{s}_t)}{\prod_{t=1}^H\pi_b(a_t|s_t)\pi_b(\tilde{a}_t|\tilde{s}_t)}| \underbrace{s_0\in B(s,\epsilon_s), \delta_{J(\tau)J(\tilde{\tau})}\in B(\delta_{rr'},\epsilon_r)}_\text{\epsilon-balls around s and \delta_{rr'}}\right],\tag{6}\] where \(B(s, \epsilon_s)\) represents a ball around the input \(s\) of radius \(\epsilon_s\) and \(B(\delta_{rr'}, \epsilon_r)\) is a ball around the output \(\delta_{rr'}\) of radius \(\epsilon_r\). This setup allows for small perturbations around \(s\) and \(\delta_{rr'}\) when aggregating samples. In particular, \(B(s, \epsilon_s)\) captures any input \(s\) that is within a small distance \(\epsilon_s\) of \(s\), and likewise for \(B(\delta_{rr'}, \epsilon_r)\).

Now, using the weights \(w_\epsilon\), the conformal band is \[\begin{align} \label{eq:cp-ppi-hatC} \hat{C}_{n,\alpha}(s) &= \{ \delta_{rr'}: \underbrace{Q(\frac{\alpha}{2}, F_n^{(s,\delta_{rr'})})}_\text{\frac{\alpha}{2} quantile of the CDF F_n} \le \underbrace{\text{score}_{n+1}^{(s,\delta_{rr'})}}_\text{score of this pair of trajectories} \le \underbrace{Q(1-\frac{\alpha}{2}, F_n^{(s,\delta_{rr'})})}_\text{(1- \frac{\alpha}{2}) quantile of the CDF F_n} \}, \end{align}\tag{7}\] where \[\quad F_n^{(s,\delta_{rr'})}(v) = \sum_{i=1}^n \underbrace{p_i^w(s,\delta_{rr'})}_\text{weighted quantile}\mathbb{1}\{\text{score}_i \le v\} + \underbrace{p_{n+1}^w(s,\delta_{rr'})}_\text{weighted quantile}\mathbb{1}\{\infty \le v\}, \label{eq:cp-ppi-F}\tag{8}\] \[p_i^w(s,\delta_{rr'}) = \begin{cases} \frac{w(S_i,\Delta_{rr',i})}{\sum_{j=1}^nw(S_j,\Delta_{rr',j})+w(s,\delta_{rr'})} & \text{if i \le n,}\\ \frac{w(s, \delta_{rr'})}{\sum_{j=1}^nw(S_j,\Delta_{rr',j})+w(s,\delta_{rr'})} & \text{if i = n+1.} \end{cases} \label{eq:cp-ppi-p}\tag{9}\] Additionally, \(Q\) is a quantile function and \(\text{score}_i^{(S_i, \Delta_{rr',i})} = \Delta_{rr',i}\). We also note that typical conformal prediction methods do not provide coverage guarantees for individual samples. In our setting, however, the target of interest is \(V^{\pi}(s)\), which is itself an expectation, so marginal coverage is sufficient.

3.2 DR-PPI: Confidence Intervals for Unconditional OPE Value Estimation↩︎

In 3.1, we derived a valid confidence interval for the initial-state conditioned policy value. Now, we study the more common task in OPE, which is to estimate the policy value averaged over the initial state distribution. A natural approach would be to aggregate the CP-Gen estimates across initial states, for example by applying a union bound. Unfortunately, this approach would result in confidence intervals that are impractically wide. Thus, we introduce a second estimator tailored to this setting, DR-PPI, which builds on ideas from doubly robust estimation and prediction-powered inference. Our goal is to construct an estimator of \(V^{\pi_e} = \mathbb{E}_{s_0}[V^{\pi_e}(s_0)]\) with valid confidence intervals.

First, we assume that the initial state distribution \(d_0\) is known (though our results extend to settings in which \(d_0\) must be estimated). Now, we construct a cross‐fitted, doubly‐robust estimate of the policy value \(V^{\pi_e}\) as follows. First, we split the behavior dataset \(D_{\pi_b}\) into two equal parts, which we refer to as \(D_1\) and \(D_2\). We first use \(D_1\) to fit a generative model \(f_1\); this procedure is agnostic to the generative model used, and reasonable approaches include a diffusion model or a variational auto-encoder (VAE). Then, we use \(f_1\) to generate \(N_f\) rollouts \(\{\tilde{\tau}_i\}_{i=1}^{N_f}\) where each rollout \(\tilde{\tau}_i\) contains actions sampled from the target policy \(\pi_e\). The rollouts are then used to calculate the model‐based return; however since we expect this return to be biased, we add a correction term using the trajectories observed in \(D_2\) as follows: \[\label{eq:dr-ppi-hatV1} \widehat V_{\text{\boldsymbol{\texttt{\textcolor{BurntOrange}{DR-PPI}}}}:1}^{\pi_e} = \underbrace{\frac{1}{N_f}\sum_{i=1}^{N_f}J(\tilde{\tau}_i) \;}_\text{model-based term}+\;\underbrace{\frac{1}{n/2}\sum_{j\in D_2}\Bigl(\tilde{J}(\tau_j) \;-\;\frac{1}{M}\sum_{m=1}^M J(\tilde{\tau}_{m,j}\mid s_{0,j})\Bigr)}_\text{correction term}.\tag{10}\] The correction term also uses synthetic trajectories. For each behavior trajectory \(\tau_j\), we generate \(M\) synthetic trajectories \(\{\tilde{\tau}_{m,j}\}_{m=1}^M\), where each \(\tilde{\tau}_{m,j}\) starts from the same initial state \(s_{0,j}\) as \(\tau_j\) and is generated using \(f_1\), with actions sampled from \(\pi_e\).

In the DR-PPI estimator, \(n\) is the number of original behavior trajectories, and \(\tilde{J}(\tau_i)\) is the re-weighted return of the behavior trajectory \(\tau_i\). There are several possible ways to perform this re-weighting: IS (e.g., \(\tilde{J}(\tau_i) = \rho(\tau_i) J(\tau_i)\)), weighted IS (WIS), and per-decision IS (PDIS). Because the trajectory \(\tau_i\) arises from the behavior policy, the re-weighting technique allows us to estimate its value as if it was generated from the target policy. Each re-weighting technique involves different bias-variance tradeoffs that are well-studied in the literature, and the preferred choice will depend on the horizon length and dataset size of the specific application. Regardless of the re-weighting technique, our asymptotic theoretical results hold.

To ensure that the data is used efficiently, we use cross-fitting [29] with two splits of the data. \(\widehat{V}_{\text{\boldsymbol{\texttt{\textcolor{BurntOrange}{DR-PPI}}}}:1}\) uses \(D_1\) to fit the generative model \(f_1\) and uses \(D_2\) to provide the correction. Similarly, we fit the generative model on \(D_2\) to produce \(f_2\) and correct the estimator using \(D_1\), which yields \(\widehat{V}_{\text{\boldsymbol{\texttt{\textcolor{BurntOrange}{DR-PPI}}}}:2}\). The final estimate (2) is the average of \(\widehat{V}_{\text{\boldsymbol{\texttt{\textcolor{BurntOrange}{DR-PPI}}}}:1}\) and \(\widehat{V}_{\text{\boldsymbol{\texttt{\textcolor{BurntOrange}{DR-PPI}}}}:2}\). The variance of the estimator can then be calculated by combining plug-in estimates of the variance of the model-based term and the correction term for each dataset split.

In particular, we average the outcomes of \(\widehat{V}_{\text{\boldsymbol{\texttt{\textcolor{BurntOrange}{DR-PPI}}}}:1}\) and \(\widehat{V}_{\text{\boldsymbol{\texttt{\textcolor{BurntOrange}{DR-PPI}}}}:2}\) as follows, \[\label{eq:dr-ppi-hatV} \widehat{V}_{\text{\boldsymbol{\texttt{\textcolor{BurntOrange}{DR-PPI}}}}} = (\widehat{V}_{\text{\boldsymbol{\texttt{\textcolor{BurntOrange}{DR-PPI}}}}:1} + \widehat{V}_{\text{\boldsymbol{\texttt{\textcolor{BurntOrange}{DR-PPI}}}}:2}) / 2.\tag{11}\]

The variance of the estimator can be calculated using plug-in estimates as follows, \[\label{eq:dr-ppi-varhatV} \mathbb{V}\left[\widehat{V}_{\text{\boldsymbol{\texttt{\textcolor{BurntOrange}{DR-PPI}}}}}\right] =\frac{1}{4}\Bigl(\frac{\widehat\sigma_{f_1}^2}{N_f}+\frac{\widehat\sigma_{b_1}^2}{n/2} +\frac{\widehat\sigma_{f_2}^2}{N_f}+\frac{\widehat\sigma_{b_2}^2}{n/2}\Bigr),\tag{12}\] where \(\sigma_f^2 = \mathbb{V}_{\tilde{\tau}\sim \tilde{p}^{\pi_e}}\left[J(\tilde{\tau})\right]\), and \(\sigma_b^2 = \mathbb{V}_{\tau \sim p^{\pi_b}, \tilde{\tau} \sim \tilde{p}^{\pi_e}}\left[\tilde{J}(\tau) - \frac{1}{M}\sum_{m=1}^M J(\tilde{\tau}_m|s_{0}(\tau))\right]\).

Using this variance, an approximate CI for a given choice of coverage \((1-\alpha)\) is \[\label{eq:dr-ppi-hatC} \widehat C_\alpha =\widehat V^{\pi_e}_{\text{\boldsymbol{\texttt{\textcolor{BurntOrange}{DR-PPI}}}}}\pm z_{1-\alpha/2}\,\sqrt{\mathbb{V}\left[\widehat{V}^{\pi_e}_{\text{\boldsymbol{\texttt{\textcolor{BurntOrange}{DR-PPI}}}}}\right]}\,\tag{13}\] where \(\mathbb{V}\left[\widehat{V}^{\pi_e}_{\text{\boldsymbol{\texttt{\textcolor{BurntOrange}{DR-PPI}}}}}\right]\) is the variance of the OPE estimate learned by DR-PPI, and \(z_{1-\alpha/2}\) is the z-score corresponding to the \(1-\alpha/2\) quantile of the standard normal distribution.

We observe that DR-PPI differs from traditional PPI in two key ways. First, in PPI, one typically has access to large quantities of unlabeled data, and an ML model is used to predict labels for these samples. In contrast, in our setting, the ML model (e.g., a generative model) is used to produce new samples, which can then be comparatively easily labeled via a reward function. Second, our setting involves distribution shift; we observe trajectories generated by a behavior policy, while our goal is to infer the value of a target policy that induces a different trajectory distribution. In contrast, standard PPI assumes that labeled and unlabeled samples are drawn from the same underlying distribution.

Before discussing the practical considerations when fitting the estimators, we first compare their constructions. When all trajectories in an environment begin from the same initial state, the point estimates of both methods are identical, differing only in their confidence intervals. The re-weighting schemes, however, are distinct: DR-PPI re-weights only the real behavior trajectories, whereas CP-Gen uses a product of IPS ratios averaged across a set of trajectories. Finally, the return differences used to compute the CI in CP-Gen may exhibit higher variance than subtracting the mean of a set of trajectories from the return of a single trajectory in DR-PPI. However, this effect depends on the stochasticity of the generated trajectories and may vary across domains.

3.3 Practical considerations↩︎

There are several practical considerations to enable OPE in environments with large state and action spaces as well as settings in which \(\pi_b\) and \(\pi_e\) differ substantially. First, it is occasionally necessary to clip the largest IPS ratios to avoid extremely large intervals. [30] shows that using a clip constant set to \(n^{1/2}\) where \(n\) is the number of dataset samples, provides an optimal first order rate in the resulting mean-squared error of the OPE estimator, balancing the bias introduced by the clipping with the variance reduction. This clipping constant also ensures the resulting estimate is still consistent. Following this, we set the clipping constant at a rate of \(n^{1/2}\). We note that clipping typically introduces an additional bias to the theoretical results, which we do not account for in this work and leave to future work.

Additionally, in 2, we propose splitting the behavior dataset into two portions and aggregating the OPE estimate calculated using each portion. If a pre-trained generative model is available, we use the full dataset to construct the CI, and no data splitting for generative model training is necessary. However, if no pre-trained model exists, we divide the data and use one half train the generative model, and the other half to compute the CI. Because these two subsets are independent, this preserves the exchangeability criterion for conformal prediction and the validity of DR-PPI. However, in practice, it may not possible to split the behavior dataset due to its size. For these settings, we choose not to perform cross-fitting, and instead report results without sub-dividing the dataset. As discussed in 5, this can still result in valid, but higher variance CIs.

Finally, for CP-Gen, we must set \(\epsilon_s\) and \(\epsilon_r\) depending on the environment. We view \(\epsilon_s\) and \(\epsilon_r\) as hyperparameters that need. One way to do this is via cross-validation, where we split the behavior dataset into training and validation sets, and choose the \(\epsilon_r, \epsilon_s\) that yields the most accurate estimate of the value function \(V^{\pi_b}(s)\) on the validation set.

4 Theoretical Results↩︎

Now, we discuss the theoretical guarantees of our approaches. As is standard in prior OPE literature, we assume that the target and behavior policies share common support, and that the instantaneous rewards and IPS ratios are bounded [15], [16].

4.1 CP-Gen produces valid conformal prediction intervals↩︎

We make a few additional assumptions to analyze CP-Gen. These assumptions balance theoretical rigor with practical relevance, allowing us to derive meaningful guarantees settings with continuous state spaces. Importantly, they still encompass a broad class of real-world MDPs.

We consider [asp:lipschitz_policy,asm:lipschitz_transition] mild. In most cases, 1 holds with a sufficiently large Lipschitz constant; in practice, these constants are small when similar states are assigned similar actions, a condition often justified in domains like healthcare, where similar patients receive similar treatments. A comparable assumption has been studied in prior work [31]. Similarly, 2 is a smoothness assumption on the transition dynamics which has been used in prior work [32]. For example, in healthcare, patients with comparable clinical profiles often respond similarly to similar treatments; small changes in dosage or patient characteristics typically produce gradual, not abrupt, differences in outcomes.

3 requires that the return differences between trajectories are smooth in their expected IPS ratios. However, in domains with a large number of samples, where we can use a more fine-grained \(\epsilon_r\), the Lipschitz assumption here (which is multiplied by \(\epsilon_r\) in our theoretical bound) will have much less impact. Overall, our assumptions are used to account for potential errors introduced by \(\epsilon\)-approximation, used in large or continuous state spaces and ensure that the resulting averaging error is bounded.

Under the stated assumptions, we now demonstrate that CP-Gen produces valid conformal prediction intervals within a small margin of error (1, proof in Appendix 11).

4.2 DR-PPI produces asymptotically valid confidnece intervals↩︎

In 3.2, we mention several choices for the importance-sampling correction including IS, WIS, and PDIS. Regardless of the correction style, we achieve asymptotically valid CIs (2, proof in Appendix 11).

5 Empirical Results↩︎

Our theoretical analyses demonstrated that CP-Gen and DR-PPI can calculate valid CIs under mild assumptions. To complement this analysis, we seek to answer the following questions using empirical results: 1) Does the \(\epsilon\)-approximation used in CP-Gen cause the estimated interval to be biased? 2) Do DR-PPI and CP-Gen produce intervals that cover the ground truth policy value? 3) Under what conditions do the DR-PPI estimates outperform baseline approaches?

5.1 Datasets↩︎

To answer our empirical questions, we use the following domains.

Inventory Control [13]: We adapt this simulator to accommodate a continuous state and reward space.
Sepsis [33]: In this popular sepsis simulator, the goal is to successfully discharge a simulated patient. We approximate the dynamics using a feed-forward network.
D4RL HalfCheetah [5]: The HalfCheetah environment is a Mujoco task in the D4RL suite where the goal is to get the cheetah to move forward. Here, we approximate the dynamics using a variational auto-encoder (VAE) [27].
MIMIC-IV [34], [35]: We consider a subset of patients from MIMIC-IV that receive potassium repletion. To emulate a setting in which we have access to both a behavior and target cohort, we construct two sub-cohorts. The behavior sub-cohort consists of patients who receive lower dosages (<20 mEq/L), and the target sub-cohort consists of patients who receive higher dosages (>= 20mEq/L). We use a VAE to generate synthetic trajectories. Our goal is to learn the value of the target policy (i.e., repletion strategy in the higher-dosage cohort).

The domains we consider range in complexity and relevance to real-world settings. This breadth allows us to understand how specific environmental factors, including simulator quality and the size of the state and action space, impact estimator performance.

5.2 Baselines↩︎

In addition to the baseline proposed in  [13], we compare to the following approaches: Importance Sampling (IS): We use standard IS, deriving a bound using central limit theorem (CLT) or bootstrapping.
Augmented Importance Sampling (AugIS): We use both the original dataset and a set of synthetic trajectories to calculate an IS estimate, with bounds estimated using either CLT or boostrapping.
Direct Method (DM): We use rollouts from the learned model and calculate the expectation of the trajectory returns. DM estimates do not produce CIs.
Doubly Robust (DR): Here, we compute a DR estimated using DQL to learn the reward model.
Augmented Doubly Robust (AugDR): Here, we use both offline trajectories and synthetic trajectories to learn a Deep Q-learning (DQL) reward model and then compute a DR estimate.
Q-Bootstrap: Here, we fit a \(Q\)-function using the behavior dataset and use it to learn a bootstrapped estimate of \(V^{\pi_e}(s)\).

3pt

5.3 Results↩︎

****CP-Gen** produces valid CP intervals. As discussed in 3, to scale prior conformal prediction approaches to large MDPs, we use an \(\epsilon\)-approximation strategy. Despite this approximation, we find that CP-Gen** still results in conformal prediction intervals that cover \(V^{\pi_e}(s)\) with the specified confidence level, often with a smaller interval size than baseline approaches (2, full intervals reported in 5). We compare to a DM-style baseline where we average the return of synthetic trajectories that start in the given initial state. We find that the DM baseline can produce a biased result with a poor generative model (e.g., in D4RL, MIMIC-IV). We also evaluate the baseline reported in  [13]. This baseline covers the ground truth value but produces wider intervals than CP-Gen in all environments with continuous state spaces (e.g., all environments except Sepsis). These results suggest that CP-Gen newly enables conformal prediction for OPE in MDP settings with large or continuous state and action spaces.

****DR-PPI** identifies valid confidence intervals that cover \(V^{\pi_e}\) across all domains. Across all domains, DR-PPI** produces valid CIs that cover \(V^{\pi_e}\), as our theoretical results suggest (¿tbl:tab:drppi95perf?). In contrast, most baseline approaches either have wide CIs or have intervals that do not cover \(V^{\pi_e}\). In fact, any baseline approach that uses generated trajectories (e.g., AugIS, AugDR) produces a biased interval, which suggests that naively incorporating auxiliary synthetic trajectories results in a biased estimator. Furthermore, we find that in the D4RL and MIMIC-IV settings, DQL is unable to identify an accurate Q function; as a result, the CIs become exponentially large.

****DR-PPI** performs best in stochastic domains with high quality generative models. Finally, we clarify the settings in which DR-PPI** outperforms baselines. When the environment is deterministic (e.g., D4RL HalfCheetah), or the generative model is of poor quality (e.g., MIMIC-IV) DR-PPI performs similarly to the IS baselines. In such settings we do not get a favorable variance reduction from the synthetic trajectories because they are often highly deterministic. In contrast, in settings where the environment is stochastic and our learned generative model is good (e.g., Inventory, Sepsis), DR-PPI has tighter CIs. Given that both IS with bootstrapping and DR-PPI produce valid CIs, we recommend a simple rule: use the estimator with the narrower interval. We defer a rigorous selection criterion to future work.

Empirical coverage rates. Finally, we investigate empirical coverage rates for both methods in the all settings (3). We note that in all settings, DR-PPI covers the ground truth value of the policy, and that CP-Gen achieves the requested coverage in Inventory. We believe that the slight loss of coverage for CP-Gen in the Sepsis, D4RL, and MIMIC-IV settings is due to a higher \(\Delta_w\). For example, the Sepsis environment, due to its discrete state and reward space, exhibits weak Lipschitz continuity, with a large Lipschitz constant. Furthermore, in this setting, \(C_{ips}\), the upper bound of the IPS ratios, is large given that the target and behavior policies are quite distinct. As suggested in Theorem 1, these two factors contribute to a higher \(\Delta_w\), which results in a small loss of coverage.

6 Conclusion↩︎

Here, we take steps toward uncertainty-aware OPE in settings that combine real and synthetic trajectories. We present two complementary approaches, CP-Gen and DR-PPI, that use auxiliary data to construct CIs for OPE. CP-Gen calculates state-conditioned policy values, while DR-PPI estimates unconditional policy values. We provide theoretical guarantees (4) and examine behavior in four empirical domains(5). Our results illustrate that obtaining valid CIs for OPE with auxiliary data is feasible across a variety of domains, from fully synthetic settings to real-world EHR data.

Limitations and future work. Our work considers settings with continuous state spaces of moderate dimension. When applied to higher-dimensional settings, the choice of \(\epsilon_s, \epsilon_r\) becomes increasingly consequential as poor choices can inflate \(\Delta_w\), thus reducing coverage rates. Guidance on selecting these parameters is provided in 3.3. Future work can explore developing more principled procedures for setting \(\epsilon_s\) and \(\epsilon_r\), alternative classes of generative models such as diffusion models, and investigate strategies to mitigate the impact of poor-quality generated trajectories. More broadly, we see value in analyzing these approaches under distribution shift and partial observability.

Broader Impact Statement↩︎

In this work, we propose two strategies to estimate confidence intervals for off-policy evaluation when used with both real and synthetic data. We believe this work is foundational and acknowledge that it has the potential to improve the downstream application of RL policies in high-stakes domains.

Acknowledgments↩︎

The authors would like to thank Matthew Jörke, Shengpu Tang, John Duchi, and Alex Nam for feedback on early versions of this manuscript. AM was funded in part by a Stanford Data Science fellowship. BEE was funded in part by grants from the Parker Institute for Cancer Immunology (PICI), the Chan-Zuckerberg Institute (CZI), the Biswas Family Foundation, NIH NHGRI R01 HG012967, and NIH NHGRI R01 HG013736. BEE is a CIFAR Fellow in the Multiscale Human Program. BEE is on the Scientific Advisory Board for ArrePath Inc, GSK AI for Cancer, and Freenome; she consults for Neumora.

7 Additional Empirical Results↩︎

First, we discuss additional empirical results. In the main text, we report CP-Gen and DR-PPI results across all domains. First, we include results in the Inventory setting that investigate the performance of our methods if the behavior policy \(\pi_b\) is unknown. We report coverage rates across 50 trials, and define the estimation error in terms of \(\epsilon_x\). Here, the true behavior policy is static and uniform across all actions \([1/11 \text{ for - in range(11)}]\). The estimated behavior policy is defined as \([1/11 + \epsilon_x \text{ for - in range(5)}] + [1/11- \epsilon_x \text{ for - in range(5)}] + [1/11]\). That is, the perturbation by increases the probability of the first 5 actions, decreases the probability of the next 5 actions, and retains the probability of the last action. Our results (4) support the claim that our methods are robust to small errors in the estimation of the behavior policy, with coverage dropping at \(\epsilon_x \geq 0.01\). We observe that the drop in coverage is more pronounced for DR-PPI, though we also note that an \(\epsilon_x\) of 0.03 refers to a very large degree of misspecification.

Additionally, we discuss the performance of our estimators as a function of the number of generated synthetic trajectories and the noise of the synthetic trajectories (3). We find that in the majority of settings, DR-PPI and CP-Gen achieve the requested coverage (\(\alpha=0.05\)). When DR-PPI does not achieve the requested coverage, we believe there are too few generated trajectories (i.e., non-asymptotic result). When CP-Gen does not achieve the requested coverage, we believe this is due to a higher \(\Delta_w\) (similar to the empirical coverage experiment in 3). We also study the effect of the trajectory quality on the performance of our estimators. For the Inventory setting, we find that the coverage rate only slightly decreases at higher degrees of noise. For the sepsis setting, we find that coverage does not change in comparison to perfect generated trajectories, which suggests that our methods can correct for noisy synthetic trajectories in these settings.

3pt

8 Code and Synthetic Trajectory Generation Procedure↩︎

The code to reproduce all experiments is provided on Github.

Synthetic trajectories are generated by rolling out a learned dynamics model under a specified policy. For the Inventory and Sepsis settings, we use a feedforward neural network for the dynamics model, and for the D4RL and MIMIC settings, we use a VAE. The feedforward neural network has 2 hidden layers trained on behavior data to predict the next state and reward for the dynamics model.

For the D4RL and MIMIC-IV settings, we use a VAE. We adopt a training procedure similar to [27]. In particular, the VAE encodes consecutive observations into a 16-dimensional latent space via a two-layer MLP and an LSTM (hidden size 64), then decodes using an ensemble of 10 branches, each with independently parameterized MLPs for state reconstruction, reward prediction, and episode termination. Final predictions are a learned weighted average across branches. The model is trained by maximizing an ELBO that balances state/reward/termination likelihoods against KL regularization of the latent dynamics, with an additional loss aligning encoder and decoder LSTM hidden states. At inference time, we sample an initial state from the empirical state distribution, sample an action according to the target policy, and iteratively sample next states from the learned dynamics model for a fixed horizon of H steps.

9 Empirical Settings↩︎

In the main text, we consider empirical results using four datasets. Here, we expand the description of each dataset.

Inventory Control:
The inventory control simulator is adapted from a version featured in  [13]. The state is the current inventory, actions are the number of units purchased, and reward is the end-of-day earnings. We make several adaptations to make this inventory control environment more suitable for our work. First, the distribution of the stochastic demand in the inventory is \(o\) is changed from Poisson to normal \(\mathcal{N}(\mu, \sigma)\). Additionally, the cost of buying items is \(k \times \mathbb{1}_{\{a>0\}} + c(\min(N, x_t + a) - x_t)\), where \(k > 0\) is the fixed cost for a single order, \(c > 0\) is the cost of a single unit bought, \(N\) is the inventory upper-bound, \(x_t\) is the state at timestep \(t\), and \(a_t\) is the action at timestep \(t\). The next state \(x_{t+1}\) is calculated as \(x_{t+1} = \max(0, \min(N, x_t + a_t) - o_{t+1})\). The instantaneous reward observed at the end of the day is the sum of the costs and earnings listed above (e.g., \(r(x_t, a_t, x_{t+1}) = 10^2 \times(-k 1_{\{a_t>0\}} - z x_t - c(\min(N, x_t+a_t) - x_t) + p \max(0, \min(N, x_t + a_t) - x_{t+1}))\)). When testing our algorithm, we chose the following parameters: \(N = 10, k = 1, c = 2, z = 2, p = 4, \mu=5, \sigma=10, H=20\). We approximate the dynamics in this setting using a feed-forward network.

Sepsis:
The Sepsis simulator is taken directly from [33], which models a synthetic Sepsis treatment setting. The state is an 8-dimensional vector which contains information about relevant vitals and labs, indicators of ongoing treatment (e.g., antibiotics, vasopressors, ventilator), and an indication of whether the patient is diabetic. There are 8 possible actions, each corresponding to a different combination of 3 binary treatments (e.g., antibiotics, vasopressors, ventilator). The reward is \(+1\) if the synthetic patient is off of treatment and has stable vitals, \(-1\) if the patient has unstable vitals, and \(+0\) otherwise. We do not alter any environment details, and report results with a maximum horizon of \(H=20\).

D4RL HalfCheetah:
The HalfCheetah environment is a Mujoco task in the D4RL suite [5]. The cheetah is a two-dimensional robot that has 9 body parts and 8 joints connecting the body parts. Each state is represented as a 17-dimensional vector that contains information about the position and velocity of each of the joints. Each action is represented as a 5-dimensional vector, and applies torque to a subset of the joints, and the goal of the environment is to get the cheetah to move forward as quickly as possible. The reward corresponds to how far the cheetah traveled, with negative reward indicating that the cheetah moved backward. We report results using a maximum horizon of \(H=1000\).
MIMIC-IV:
MIMIC-IV is an electronic health records (EHR) dataset collected from patients admitted to the Beth Israel Deaconess Medical Center in Boston, MA [34], [35]. We consider a subset of patients that receive potassium repletion through an intravenous (IV) line. We represent the patient state as a 20-dimensional vector containing information about important labs, administered medicines, and static covariates such as age and gender; each state represents a 4-hour interval in a patient’s hospital stay. There are five actions, each corresponding to a dosage of potassium delivered through an IV. The reward function is a binary indicator of whether the patient’s potassium lab value is within the potassium reference range (3.5-4.5 mmol/L), within 2 hours of receiving the administered potassium.

In the OPE task, we assume access to a behavior policy \(\pi_b\) and a target policy \(\pi_e\), and evaluate using RMSE. It is not immediately obvious how to create this setup within MIMIC-IV. To emulate a setting in which we have access to both a behavior and target cohort, we construct two sub-cohorts from the patients that receive potassium repletion. The behavior sub-cohort consists of patients who receive lower dosages (<20 mEq/L) of potassium, and the target sub-cohort consists of the patients who receive higher dosages (>= 20mEq/L) of potassium. The behavior policy corresponds to the repletion policy in the behavior cohort, and the target policy corresponds to the repletion policy in the target cohort. Both policies are inferred using behavior cloning. Our goal is to learn the value of the target policy, and a ground truth calculation of this value is calculated by averaging the returns of the target trajectories. We observe that the maximum horizon length is \(H=189\), though most patients have trajectories that are less than 20 timesteps.

10 Computational cost of experiments↩︎

All experiments were conducted on an internally hosted cluster equipped with an NVIDIA RTX A6000 GPU featuring 48 GB of memory. In total, our experiments consumed approximately 250 compute hours, primarily driven by VAE training and Q-function learning on large datasets.

11 Proofs for Theoretical Results↩︎

11.1 Proof of 5↩︎

11.2 Additional Assumptions↩︎

First, we formally state the assumptions used in prior literature [15], [16] to support our theoretical results.

These assumptions are standard in the literature and minimally restrictive, thus enabling the analysis of CP-Gen’s performance under realistic conditions. We also consider 7, a mild regularity condition that holds in a wide variety of real-world MDPs, including those with heterogeneous populations and varied outcomes, such as clinical settings with diverse patient cohorts. Prior work in bandit and reinforcement learning has used similar assumptions [37]–[39].

11.3 Proof of 1↩︎

We now start the main proof of 1.

11.4 Proof of 2↩︎

We use [asm:bounded_ips_weights,asm:bounded_reward,asm:common-support], which are standard in prior OPE literature.

11.5 Variance of DR-PPI↩︎

The estimator for the first fold is \[\label{eq:estimator} \widehat{V}_{\mathrm{DR\text{-}PPI}:1}^{\pi_e} = \underbrace{\frac{1}{N_f}\sum_{i=1}^{N_f} J(\tilde{\tau}_i)}_{A} + \underbrace{\frac{1}{n/2}\sum_{j \in D_2} \tilde{J}(\tau_j)}_{B} - \underbrace{\frac{1}{n/2}\sum_{j \in D_2} \frac{1}{M}\sum_{m=1}^{M} J(\tilde{\tau}_{m,j} \mid s_{0,j})}_{C}.\tag{14}\]

Throughout, we condition on the generative model \(f_1\) learned from \(D_1\), treating it as fixed. We use the following notation: \[\begin{align} \sigma_{f_1}^2 &:= \mathrm{Var}\!\bigl(J(\tilde{\tau})\bigr), &&\text{marginal variance of synthetic returns (e.g., averaged across all initial states),}\\ \sigma_{f_1}^2(s_0) &:= \mathrm{Var}\!\bigl(J(\tilde{\tau})\mid s_0\bigr), &&\text{conditional variance of synthetic returns,}\\ \sigma_r^2(s_0) &:= \mathrm{Var}\!\bigl(\tilde{J}(\tau)\mid s_0\bigr), &&\text{conditional variance of IS-weighted real returns,}\\ \mu_{f_1}(s_0) &:= \mathbb{E}\!\bigl[J(\tilde{\tau})\mid s_0\bigr], &&\text{conditional mean synthetic return (expectation taken over trajectories from f_1),}\\ \mu_r(s_0) &:= \mathbb{E}\!\bigl[\tilde{J}(\tau)\mid s_0\bigr], &&\text{conditional mean IS-weighted return.} \end{align}\]

References↩︎