Keywords: Causal Inference; Dynamic Treatment Regime; Highly Adaptive Lasso; Micro-Randomized Trial; Stochastic Intervention; Undersmoothing; Just-In-Time Adaptive Intervention

1 Introduction↩︎

Modern mobile and wearable technologies provide unprecedented opportunities to deliver just-in-time adaptive interventions (JITAIs; [1])—intervention designs that adapt to an individual’s rapidly changing internal state in real-time, real-world settings. These interventions are especially attractive in behavioral settings, where treatment effectiveness may depend strongly on momentary vulnerability, receptivity, and burden. Recent advances in micro-randomized trials (MRTs; [2]) have made it possible to collect intensive longitudinal data that can empirically inform the construction of JITAIs by repeatedly randomizing treatment assignment across many decision points [3], [4]. A motivating example is Mobile Assistance for Regulating Smoking (MARS; [5]), a ten-day MRT conducted among people attempting to quit smoking that was designed to inform the development of a mobile health intervention for promoting real-time, real-world engagement in evidence-based self-regulatory strategies. There, participants could be randomized up to six times per day to receive no prompt, a prompt recommending a low-effort self-regulatory strategy, or a prompt recommending a higher-effort strategy. This type of high-frequency randomization yields the intensive longitudinal data needed to develop empirically grounded JITAIs.

Despite this promise, much of the existing literature analyzing MRT data has focused on proximal outcomes measured shortly after each decision point, such as short-term engagement in a target behavior or immediate response to treatment [6], [7]. However, in many behavioral health applications, outcomes measured farther in the future are also of substantive interest—particularly when the relevant processes involve habit formation or delayed treatment effects. In such settings, proximal responses may not fully capture long-term success, even when they remain important intermediate targets.

More broadly, using MRT data to construct JITAIs is a special case of learning an optimal treatment regime from longitudinal data. Classical work in longitudinal causal inference formalized counterfactual outcomes under time-varying interventions and established identification strategies for regime-specific mean outcomes from observed data [8]–[10]. Subsequent developments based on marginal structural models (MSMs) and related dynamic regime formulations extended these ideas to estimation and comparison of treatment rules over classes of candidate regimes [11], [12]. These approaches provide a natural starting point for constructing JITAIs from MRT data, but the large number of decision points in many MRTs creates methodological challenges that are not well handled by standard longitudinal methods: inverse-probability weights can become highly variable, estimators may become irregular, and finite-sample bias can be substantial [13].

Beyond MSMs and related approaches, the policy-learning literature seeks to directly estimate an optimal rule rather than first estimating the expected outcome under each candidate rule, including outcome- and residual-weighted learning [14], [15] and observational policy-learning methods [16]. While these approaches are appealing alternatives to the previous estimation-reliant methods, they were not developed for intensive longitudinal settings in which individuals repeatedly face treatment decisions over time. Reinforcement-learning methods are explicitly designed for sequential decision problems over long horizons, but they are commonly formulated to optimize cumulative or discounted rewards constructed from proximal outcomes observed after each decision point [17], [18]. Such objectives are not well suited to our setting for two important reasons. First, from scientific and clinical perspectives, distal outcomes, such as substance use at six months, are often more relevant than proximal outcomes, such as next-day substance use. Second, choosing an appropriate discount rate, \(\gamma\), which governs the relative importance of near-term versus long-term outcomes, can be challenging in mobile health settings because of limited empirical or practical guidance for this choice [3].

Motivated by these limitations, we propose new methods to construct an optimal JITAI with respect to a clinically meaningful distal outcome, without the need to pre-specify candidate decision rules. We focus on a class of stochastic policies parameterized by a finite-dimensional vector. From a practical perspective, stochastic policies provide a useful alternative to dichotomized deterministic rules by allowing treatment delivery to remain flexible and by permitting decision-makers to incorporate additional considerations not explicitly modeled, such as side effects or insurance constraints. In behavioral settings, stochastic policies may allow occasional intervention in states where a deterministic rule would not intervene, helping to sustain participant engagement, reinforce healthy behaviors, and prevent disengagement over time. Moreover, stochastic policies help mitigate finite-sample positivity problems that arise when many decision points make deterministic or near-deterministic treatment paths poorly supported in the observed data [19]–[23]. However, with many decision points, they may still lead to unstable estimation. To address this, we introduce a tilting approach that forms a weighted combination of the estimated optimal stochastic policy and the behavioral policy, that is, the policy generating the data.

This paper makes four main contributions. First, we develop a nonparametrically efficient estimator for the regimen–response curve, which characterizes the expected distal outcome under a class of stochastic policies. Our estimator is constructed to avoid estimating the large collection of augmentation terms that would otherwise arise in efficient estimation, thereby extending [24] to time-varying settings with stochastic policy learning. This construction also avoids explicit derivation of the efficient influence function and mitigates the irregularity issues that can arise with doubly robust estimators [13]. Second, we introduce a data-adaptive tilting procedure that addresses finite-sample instability by balancing bias and variance, and we establish conditions under which tilting is asymptotically negligible. Third, we show that the estimated regimen-response curve converges weakly to a Gaussian process, which enables the creation of simultaneous confidence bands for the curve. Finally, we derive the asymptotic properties of the optimizer of the regimen-response curve, corresponding to the optimal stochastic policy.

2 Notation and Preliminaries↩︎

2.1 Observed Data and Potential Outcomes↩︎

Suppose that we observe \(n\) i.i.d. observations \(O=(S_{1},A_{1},\ldots,S_{T},A_{T},Y)\sim \mathbb{P}_0\in \mathcal{M}\), where \(\mathcal{M}\) is some nonparametric model and \(T\) denotes the number of decision points. At time \(t \in \{1,\ldots,T\}\), \(S_t\in \mathbb{R}^{d}\) denotes time-varying covariates, \(A_t\in\{0,1\}\) is a binary treatment indicator, and \(Y\) is the distal (end-of-study) outcome. We use overbar notation to denote complete histories: \(\bar{S}_t\coloneq (S_1,\ldots,S_t)\) and \(\bar{A}_t\coloneq(A_1,\ldots,A_t)\); additionally, let \(\bar{A}\coloneq\bar{A}_T\) and \(\bar{S}\coloneq\bar{S}_T\) be the full covariate and treatment histories. Let \(H_t\coloneq(\bar{S}_t,\bar{A}_{t-1})\) denote all information observed prior to assigning \(A_t\).

Let \(Y^{\bar{a}}\) denote the potential outcome that would have been observed if, possibly contrary to reality, treatment were assigned as \(\bar{a}\). Similarly, let \(S_t^{\bar{a}_{t-1}}\) represent the covariates at time \(t\) that would have been observed if treatment was assigned as \(\bar{a}_{t-1}\) up until time \(t-1\). Implicit in this notation, we assume that these covariates are not affected by future treatment decisions \(a_t,\ldots,a_T\); i.e., \(S_t^{\bar{a}_{t-1}}=S_t^{\bar{a}_T}\) for any \(\bar{a}_T\in\{0,1\}^T\). We define full data as \(X \coloneq (S_1,\,\{S_t^{\bar a_{t-1}}: \bar a_{t-1}\in\{0,1\}^{t-1},\;t=2,\ldots,T\},\, \{Y^{\bar a}: \bar a\in\{0,1\}^T\}) \sim \mathbb{P}_X \in \mathcal{M}^F\) where \(\mathcal{M}^F\) is the nonparametric model for the full data \(X\).

Denote the empirical distribution function as \(\mathbb{P}_n\), and let \(\require{physics} \mathbb{P}f = \int f(o)\dd\mathbb{P}(o)\) for any measurable function \(f\) and distribution \(\mathbb{P}\). For any random variable, \(X\), we let the calligraphic \(\mathcal{X}\) denote its support.

2.2 Stochastic Adaptive Interventions and Our Causal Estimand↩︎

A stochastic adaptive intervention assigns \(A_t\), at each time \(t\), by drawing from a multinomial distribution in which the probabilities are functions of the current history \(H_t\). To operationalize policy optimization and avoid the curse of dimensionality, we focus on policies that only depend on the current state \(S_t\). Henceforth, we consider the parametric class \(\{q_t^\theta:\theta\in\Theta\}\) for some compact index set \(\Theta\subset \mathbb{R}^p\), in which, for \(a\in\{0,1\}\) \[\label{equation:q-theta} q_t^\theta(a,s)\coloneq 1 - \lvert \mathop{\mathrm{expit}}\{b(s)^{\top}\theta\}- a\rvert\,\text{,}\tag{1}\] for some fixed basis function \(b: \mathbb{R}^d \to \mathbb{R}^p\). This gives the familiar logistic model: \(q_t^\theta(1,s)=\mathop{\mathrm{expit}}\{b(s)^{\top}\theta\}\) and \(q_t^\theta(0,s)=1-\mathop{\mathrm{expit}}\{b(s)^{\top}\theta\}\). Other smooth parametric forms are also admissible provided that they satisfy suitable regularity conditions. We restrict attention to genuinely stochastic policies whose treatment probabilities are bounded away from \(0\) and \(1\); 9.1 provides a practical construction of \(\Theta\) from investigator-specified probability bounds.

For each \(\theta\in\Theta\), let \(Y^\theta\) denote the potential distal outcome that would have been observed had the policy \(\{q_t^\theta\}_{t=1}^T\) been followed throughout. Our primary causal estimand is the regimen-response curve \[\psi_0(\theta)=\mathbb{P}_0(Y^\theta), \qquad \theta\in\Theta.\] This curve has two roles. For a fixed \(\theta\), it gives the mean distal outcome under the corresponding stochastic intervention. As a function of \(\theta\), it provides the basis for policy learning: an optimal stochastic policy can be defined as any maximizer of \(\psi_0(\theta)\) over \(\Theta\), or of a penalized version of this criterion when additional regularization is imposed. Thus, policy learning is reduced to estimation and maximization of a smooth finite-dimensional regimen-response curve. Identification of \(\psi_0(\theta)\) from the observed data relies on the standard longitudinal causal assumptions stated next.

1 consists of standard conditions required to identify counterfactual outcomes under longitudinal interventions [8]–[10], [12]. Consistency links the observed data to the corresponding potential outcomes under the realized treatment history. This assumption may be violated, even in MRT settings, if one individual’s treatment assignments affect the outcomes of another individual. Sequential randomization asserts that, conditional on the observed history \(H_t\), treatment assignment at \(A_t\) is effectively randomized and does not depend on future potential outcomes; this is satisfied by design in MRT settings. Positivity ensures that each treatment option has a nonzero probability at every decision point and guarantees absolute continuity of the counterfactual law under the target policy with respect to the observed data law. We note that only weak positivity is required for identification; however, later theoretical results will leverage strong positivity. Although strong positivity holds by design in MRTs, the large number of decision points can still induce finite-sample instability in inverse probability weights, motivating the use of stochastic interventions and the policy-tilting approach introduced in 3.1. We note that protocols that deterministically restrict treatment at some decision points are handled in 9.2. Together, these assumptions ensure that the mean counterfactual outcome under any stochastic policy is nonparametrically identifiable using a change-of-measure argument.

3 Proposed Approach↩︎

We first introduce an \(\alpha\)-tilting device that stabilizes longitudinal products of inverse probability weights by shrinking early decision rules toward the observed treatment mechanism (3.1). We then formalize identification and the semiparametric structure of the resulting (tilted) regimen-response curve, highlighting the key nuisance quantities that arise in efficient estimation (3.2). Building on this structure, we propose an undersmoothed inverse probability weighted estimator that attains nonparametric efficiency while avoiding explicit estimation of the large collection of time- and policy-indexed outcome regressions (3.3). We conclude with practical implementation components: data-adaptive selection of the tilting level \(\alpha_n\), investigator-driven elicitation of the feasible policy set \(\Theta\), and a modification to accommodate protocol-driven randomization ineligibility ([sec:adaptive-tilting] [sec:index-set] [sec:ineligible]).

3.1 Tilted Policies↩︎

Common inverse probability weighted (IPW) or doubly robust estimators of the regimen-response curve typically involve products of time-specific weights: \(\prod_{t=1}^T q_t^{\theta,\alpha}/\pi_{t,n}\). When \(T\) is large and the policy of interest substantially differs from the observed treatment mechanism, these products can be numerically unstable, resulting in bias and large variances in finite samples even when strong positivity holds by design. To mitigate the effect of such finite-sample positivity violations, we propose a policy-tilting device that shrinks earlier weights towards \(1\) by interpolating between the target policy and the treatment assignment process observed in the data. For \(\alpha\in[0,\infty)\), define the \(\alpha\)-tilted policy \[\label{equation:q-theta-alpha} q_t^{\theta,\alpha}(A_t, H_t) \coloneq (t/T)^\alpha q_t^\theta(A_t,S_t) + \{1-(t/T)^\alpha\}\pi_{t,0}(A_t\mid H_t)\,\text{.}\tag{2}\] For \(\alpha=0\), the tilted policy reduces to the original policy of interest, \(q_t^{\theta,0}=q_t^\theta\), and the corresponding estimand coincides with the scientific target. For \(\alpha>0\), tilting defines a distinct stochastic intervention and hence a different counterfactual mean; in 3.4 we select \(\alpha\) to balance this estimand drift against the finite-sample variance reduction gained by stabilizing weights, and later results show that sufficiently small \(\alpha\) yields asymptotically negligible drift. We note that although the scientific policy \(q_t^\theta\) is restricted to depend only on the current state \(S_t\), the tilted rule \(q_t^{\theta,\alpha}\) depends on the full history \(H_t\) through the observed assignment mechanism \(\pi_{t,0}(\cdot\mid H_t)\).

To see the stabilization effect, we can write the time-\(t\) weight contribution for a generic observation as \[W_{t,0}^{\theta,\alpha}(O)\coloneq \frac{q_t^{\theta,\alpha}(A_t, H_t)}{\pi_{t,0}(A_t\mid H_t)} = (t/T)^\alpha\frac{q_t^\theta(A_t,S_t)}{\pi_{t,0}(A_t\mid H_t)} + \{1-(t/T)^\alpha\}\,\text{,}\] which is a convex combination of the usual IPW factor \(q_t^\theta/\pi_{t,0}\) and \(1\). Because \((t/T)^\alpha\to0\) as \(\alpha\to\infty\) for each \(t<T\), we have \(W_{t,0}^{\theta,\alpha}(O)\to1\) for all early decision points \(t<T\). Hence, the product weight \(\prod_{t=1}^T W_{t,0}^{\theta,\alpha}(O)\) shrinks toward a product with fewer volatile factors.

Let \(Y^{\theta,\alpha}\) denote the potential outcome under regimen \(\{q_t^{\theta,\alpha}\}_{t=1}^T\) and denote the \(\alpha\)-tilted regimen-response curve by the mapping \((\theta,\alpha)\mapsto \mathbb{P}_0(Y^{\theta,\alpha})\). By the previous logic, we have \(\mathbb{P}_0(Y^{\theta,\alpha=0})=\mathbb{P}_0(Y^{\theta})\).

3.2 Inverse Probability Weighting Mapping↩︎

Under the full data \(X\), the mean counterfactual outcome under the tilted policy \(q_t^{\theta,\alpha}\) is defined as \(\Psi^F_{\theta,\alpha}(P_X) = P_X (Y^{\theta,\alpha})\) where \(\Psi^F: \mathcal{M}^F \rightarrow \mathbb{R}\). For a fixed policy parameter \(\theta\in\Theta\) and tilting parameter \(\alpha\ge 0\), define the corresponding full data canonical gradient \(D^F_{\theta,\alpha}(X,\Psi^F) = Y^{\theta,\alpha} - \Psi^F_{\theta,\alpha}(P_X)\). A longitudinal inverse probability weighting mapping of \(D^F_{\theta,\alpha}(X,\Psi^F)\) to the observed data is given by \(U(O;\theta,\alpha,\Psi)=W^{\theta,\alpha}(P)(O)Y-\Psi(P)\) where \[W^{\theta,\alpha}(P)(O) = \prod_{t=1}^T \frac{(t/T)^\alpha q_t^\theta(A_t,S_t)+\{1-(t/T)^\alpha\}P(A_t\mid H_t)}{P(A_t\mid H_t)}\,\text{.}\] Under Assumption 1, the target parameter \(\Psi^F_{\theta,\alpha}(\mathbb{P}_X)\) is identified by \[\Psi_{\theta,\alpha}(P)\coloneq P\{W^{\theta,\alpha}(P)(O)Y\}\,\text{,}\] and \(P\{U(O;\theta,\alpha,\Psi) \mid X \}= D^F_{\theta,\alpha}(X,\Psi^F)\). Let \(\mathcal{T}_{\mathrm{CAR}}\) denote the nuisance tangent space corresponding to the sequential treatment mechanism. Because the treatment mechanism factorizes over time, this space decomposes as \(\mathcal{T}_{\mathrm{CAR}} = \mathcal{T}_{\mathrm{CAR},1} \oplus \cdots \oplus \mathcal{T}_{\mathrm{CAR},T}\) where \(T_{\mathrm{CAR},t}\) is the nuisance tangent space generated by the treatment assignment mechanism at time \(t\), for \(t=1,2,\ldots,T\). The canonical gradient of \(\Psi(P)\) is given by \(D^\star(P)(O;\theta,\alpha)=U_\pi(O;\theta,\alpha,\psi_0)-D_{\mathrm{CAR}}(P)\), where \(D_{\mathrm{CAR}}(P)\) is the projection of \(U_\pi\) onto \(\mathcal{T}_{\mathrm{CAR}}\). Owing to the above decomposition, \(D_{\mathrm{CAR}}(P)=\sum_{t=1}^T D_{\mathrm{CAR},t}(O;\eta_0,\theta,\alpha)\) for nuisance parameters \(\eta_0\). As shown in 2, the nuisance parameter \(\eta_0\) includes the regression functions \(\mathbb{P}_0\{(\prod_{j>t}W^{\theta,\alpha}_{j,0})Y \mid A_t,H_t\}\). This representation yields the efficient influence function for the regimen-response curve in the longitudinal stochastic intervention model.

In observational settings under a nonparametric model, both the propensity scores and the regression functions generally must be estimated. When these nuisance functions are estimated nonparametrically, inconsistency in any of them can lead to an irregular estimator with substantial bias. This difficulty becomes especially severe when the number of decision points is large [13], [25]. This is particularly challenging for the regression functions, which must be estimated for each \(\theta\in\Theta\) whenever \(t<T\). In MRTs, the propensity scores are known by design, so the irregularity issue does not arise. However, inconsistent estimation of the regression functions can still lead to inefficiency. This motivates our proposed undersmoothed inverse probability weighted estimator, which only requires estimation of \(T\) propensity scores but still is \(\sqrt{n}\)-consistent and nonparametric efficient.

3.3 Undersmoothed Inverse Probability Weighting Estimator↩︎

The efficient influence function characterization in 3.2 suggests that nonparametric efficient estimation of the regimen-response curve generally involves a large family of nuisance functions indexed by both time and policy, \(\{\mu_{t,0}^{\theta,\alpha}: t<T, \theta\in\Theta\}\), which becomes impractical when \(T\) is large or when \(\Theta\) is not discrete. We therefore construct an undersmoothed inverse probability weighted (UIPW) estimator that achieves nonparametric efficiency without explicitly estimating these policy-indexed outcome regressions, thereby offering a practically feasible approach for intensive longitudinal settings. The key idea is to estimate the time-specific propensity scores flexibly via the highly adaptive lasso (HAL; [26]) with deliberate undersmoothing so that the resulting fits approximately solve a rich collection of empirical score equations. This renders the bias asymptotically negligible while preserving the simplicity of a weighted-mean estimator. We proceed by summarizing the estimator and then describe an implementable procedure to select the undersmoothing level.

The HAL is a nonparametric regression technique that estimates functions via a sparse linear combination of indicator basis functions indexed by observed covariate values. The estimator minimizes an empirical risk while constraining the \(L_1\)-norm of its coefficients by \(\lambda\), which controls the total sectional variational norm of the fitted function. Under mild conditions, HAL estimators achieve fast convergence rates (roughly \(n^{-1/3}\)) for a wide class of càdlàg functions without requiring smoothness assumptions. Full technical details, including the precise definition of the basis and norm, are provided in 8.1.

When the regimen response curve is estimated using IPW, the asymptotic expansion admits a remainder of the form \[\label{eqn:ipw-remainder-generic} \sum_{t=1}^T \mathbb{P}_n\big[f_{t}(O;\theta,\alpha,\bar\pi_{t,n},\mu_{t,0}^{\theta,\alpha})\{A_t-\pi_{t,n}(1\mid H_t)\}\big]\,\text{,}\tag{3}\] where the function \(f_{t}\) involves the estimated weights from previous decision points \(i<t\) and the true outcome regression function looking forward following a particular stochastic intervention at \(j\ge t\). Each term in the sum resembles \(\mathbb{P}_n D_{\mathrm{CAR},t}\) with the estimated propensity score plugged in place of the true propensity score for all \(s\le t\). If the treatment model is estimated using a correctly specified parametric model, this term is \(O_p(n^{-1/2})\) and is absorbed into the estimator’s influence function under the semiparametric model. However, if nonparametric methods are used, this term is generally not even \(O_p(n^{-1/2})\), preventing valid inference.

[24] showed that undersmoothing the HAL—selecting a sufficiently large \(L_1\) bound that is typically greater than the cross-validated bound—ensures that the fitted model is rich enough to approximately solve these empirical score equations, rendering the term asymptotically negligible. Intuitively, this occurs because the HAL approximately solves score functions of the form \(\mathbb{P}_n[\varphi(O)\{A_{t,n}-\pi_{t,n}(1\mid H_t)\}]\) for the included basis functions \(\varphi\). If the model is sufficiently undersmoothed, a rich enough collection of basis functions is retained. Hence, if \(f_{t,n}\) can be approximated by these basis functions within an \(n^{-1/4}\) neighborhood, the HAL will approximately solve the bias term, rendering it asymptotically negligible. 8.2 contains full details regarding undersmoothing and formal statements of the previous conditions.

While the undersmoothing condition provides a theoretical target for eliminating the leading remainder term, it leaves open the practical choice of tuning parameters, which must balance bias control against weight instability. Selecting too large of a penalty can induce oversmoothing and induce bias, whereas too small of a penalty can inflate finite-sample variance, cause numeric instability, and prevent convergence at the \(n^{-1/4}\) rate.

One approach to selection is to directly target a cross-validated estimate of the leading IPW remainder (3 ). However, this would require estimating the large set of nuisance outcome regression functions \(\{\mu_{t,0}^{\theta,\alpha}:t<T,\theta\in\Theta\}\), and doing so fully nonparametrically is generally impractical. To retain this remainder-targeting motivation while avoiding this computational burden, we consider a pragmatic selector that plugs in a low-dimensional working approximation, \(\mu_{t,n}^{\theta,\alpha}\). The bias is then approximated by \[\label{eqn:dcar-bias-cv-approx} R_{t,n,v}(\lambda;\theta)\coloneq \mathbb{P}_{n,v}^1 f_t(O;\theta,\alpha,\bar\pi^{(v)}_{t-1,n,\lambda_{t,n,\text{CV}}},\mu_{t,n}^{(v)},\pi_{t,n,\lambda_{t,v,\text{CV}}}^{(v)}) \{A_t-\pi_{t,n,\lambda}^{(v)}(A_t\mid H_t)\}\,\text{,}\tag{4}\] where \(v=1,\ldots,V\) indexes folds, \(\mathbb{P}_{n,v}^1\) denotes the empirical measure on the testing data from the \(v\)th fold. We then consider two selectors that minimize different norms of this bias approximation: \[\label{eqn:linfty-dcar-criterion-body} \lambda_{t,n,L_\infty}\coloneq \arg\min_{\lambda} V^{-1} \sum_{v=1}^V \max_{\theta\in\Theta_\text{test}} \lvert R_{t,n,v}(\lambda;\theta)\rvert\,\text{,}\tag{5}\] and \[\label{eqn:l2-dcar-criterion-body} \lambda_{t,n,L_2}\coloneq \arg\min_{\lambda} V^{-1} \lvert\Theta_\text{test}\rvert^{-1} \sum_{v=1}^V \sum_{\theta\in\Theta_\text{test}} R_{t,n,v} (\lambda;\theta)^2\,\text{,}\tag{6}\] where \(\Theta_\text{test}\subset\Theta\) is a holdout set of \(\theta\) not used to train the working model \(\mu_{t,n}\). We refer to both \(\lambda_{t,n,L_\infty}\) and \(\lambda_{t,n,L_2}\) as \(D_\mathrm{CAR}\)-based selectors because they minimize the norm of a term that resembles \(D_\mathrm{CAR}\). The \(L_\infty\) criterion is motivated by a desire to uniformly control 3 over \(\Theta\) to support curve-level inference and the construction of simultaneous confidence bands. The \(L_2\) criterion averages the remainder over \(\Theta\) and is typically less sensitive to isolated, high-variance points, often yielding improved finite-sample stability.

Additionally, we consider a score-based selector that depends only on the propensity score. That is, we pick \(\lambda\) by minimizing a cross-validated aggregated standardized score residual over the set of active HAL basis functions. For each \(t\) and \(v\), let \(\mathcal{J}_t^{(v)}(\lambda)\) denote the set of HAL basis functions retained (i.e., with nonzero basis coefficients) in the fitted \(\pi_{t,n,\lambda}^{(v)}\). We select \(\lambda\) by \[\label{eqn:score-criterion-body} \lambda_{t,n,\text{score}} = \arg\min_{\lambda} \frac{1}{V} \sum_{v=1}^V \frac{1}{\lVert \beta^{(v)}_{t,n,\lambda}\rVert_1} \sum_{\varphi\in\mathcal{J}_t^{(v)}(\lambda)}\Bigg\lvert \mathbb{P}_{n,v}^{1} \Bigg\{\varphi(H_t) \frac{A_t-\pi_{t,n,\lambda}^{(v)}(1\mid H_t)}{\pi_{t,n,\lambda}^{(v)}(1\mid H_t)} \Bigg\} \Bigg\rvert\,\text{,}\tag{7}\] where \(\lVert \beta_{t,n,\lambda}\rVert_1\) is the fitted HAL coefficient \(L_1\) norm. This criterion is a computationally tractable proxy to targeting the remainder term that avoids estimating \(\mu_{t,n}^{\theta,\alpha}\). It leverages the fact that the propensity scores enter the remainder terms through inverse-weight factors; hence, minimizing aggregated standardized score residuals balances remainder control against weight instability.

Finally, following [25], we additionally impose a mild cap on oversmoothing to control the rate at which the basis grows and ensure that the HAL converges at the \(n^{-1/4}\) rate. Full details of the \(D_\mathrm{CAR}\) and score based undersmoothing selectors are provided in 8.4.

Under this framework, let \(\{\pi_{t,n}\}_{t=1}^T\) be the undersmoothed propensity score estimates and let \[W^{\theta,\alpha}_n=\prod_{t=1}^T\frac{(t/T)^\alpha q_t^\theta(A_t,S_t)+\{1-(t/T)^\alpha\}\pi_{t,n}(A_t\mid H_t)}{\pi_{t,n}(A_t\mid H_t)}.\] The UIPW regimen-response curve estimator is then \(\psi_n(\theta;\alpha) \coloneq \mathbb{P}_n\{ (\prod_{t=1}^T W^{\theta,\alpha}_{t,n})Y\}\). Let \(\psi_n(\theta)\coloneq\psi_n(\theta;\alpha=0)\) denote the untilted estimator.

Estimation of \(\psi_0(\theta;\alpha)\) enables the construction of policies by searching over \(\Theta\) to identify a \(\theta\) that maximizes the expected distal outcome. However, we note that global maximization over the unrestricted policy class \(\mathbb{R}^p\) typically favors near-deterministic interventions that push treatment probabilities toward \(0\) or \(1\) in regions of the state space where treatment is beneficial or harmful. Consequently, under a bounded-probability feasible set \(\Theta\), the optimizer can behave like a constrained projection: along many directions in the parameter space the estimated mean function is approximately monotone, so the maximizer is attained at the feasible point “closest” to the unconstrained maximizer, which generally lies on \(\partial\Theta\). Because boundary solutions can be numerically unstable and complicate optimizer-based inference, we maximize a penalized value function [18], [27]: \[V_n(\theta;\alpha) \coloneq \psi_n(\theta;\alpha)-\mathcal{P}(\theta)\,\text{.}\] In particular, we will focus on an inverse-distance barrier-weighted ridge penalty: \[\mathcal{P}(\theta)= \frac{\gamma}{\inf_{\theta' \in \partial\Theta} d(\theta,\theta')}\|\tilde{\theta}\|_2^2\,\text{,}\] for some fixed ridge penalty \(\gamma>0\) and distance function \(d\). Here, \(\theta\) is partitioned as \((\theta_0,\tilde{\theta})\) when \(b(s)\) includes an intercept in the first coordinate (and \(\tilde{\theta}=\theta\) otherwise) so as to avoid penalizing the intercept, and \(\partial\Theta\) denotes the boundary of \(\Theta\). This penalty diverges to \(+\infty\) as \(\theta\) approaches \(\partial\Theta\), so the penalized value function assigns value \(-\infty\) on the boundary, thereby ensuring an interior maximizer. We note that this penalty works in parallel with the index set \(\Theta\): careful elicitation of \(\Theta\) ensures a truly stochastic set of admissible policies, \(\mathcal{P}\) prevents a boundary maximizer within this set, and supports inference.

The corresponding population value function is \(V_0(\theta;\alpha)\coloneq \psi_0(\theta;\alpha)-\mathcal{P}(\theta)\). We denote the optimal policy by \(\theta_0^\star(\alpha)\coloneq\mathop{\mathrm{arg\,max}}_{\theta\in\Theta} V_0(\theta;\alpha)\), and consider the estimator \(\theta_n^\star(\alpha)\coloneq\mathop{\mathrm{arg\,max}}_{\theta\in\Theta}V_n(\theta;\alpha)\). Let \(\theta_0^\star=\theta_0^\star(\alpha=0)\) and \(\theta_n^\star=\theta_n^\star(\alpha=0)\) denote the untilted optimizers.

3.4 Data-Adaptive Tilting↩︎

The optimal amount of tilting will depend on a variety of conditions including the sample size, number of decisions, and distance from the optimal policy to the observed policy. If \(T\) is small, \(n\) is large, and the optimal policy \(q_t^{\theta_0^\star(0)}(A_t,S_t)\) approximates the observational policy \(\pi_{t,0}(A_t\mid H_t)\), the weights will be relatively stable near \(\theta_0^\star(0)\) without any tilting. Instead, if the optimal policy is far from the observed policy, the untilted weights may be volatile at \(\theta_0^\star(0)\).

As \(\alpha\) varies, tilting trades off two competing goals. For \(\alpha\) near zero, the tilted policy remains close to the untilted scientific intervention, so the corresponding target value \(V_0(\cdot;\alpha)\) is expected to be close to \(V_0(\cdot;0)\). This limits estimand drift and helps keep the learned optimizer \(\theta_n^\star(\alpha)\) close to the policy that is optimal for the untilted target. On the other hand, increasing \(\alpha\) shrinks early weight factors toward one, reducing the variability and numerical instability induced by longitudinal products of inverse probability weights and thereby stabilizes \(\psi_n(\cdot;\alpha)\) and, in turn, \(V_n(\cdot;\alpha)\).

We operationalize this tradeoff by selecting \(\alpha\) to minimize the corresponding mean squared error (MSE) of the value function at the optimal policy. By setting the derivative of the asymptotic MSE to zero, we obtain \[\require{physics} 2\mathbb{P}_0\bigl\{\bar V_n(\alpha) - \bar V_0(\alpha=0)\bigr\} \mathbb{P}_0\left\{ \pdv\alpha \bar V_n(\alpha)\right\} + 2n^{-1}\bar\sigma_0(\alpha)\pdv\alpha \bar\sigma_0(\alpha)=0\,\text{,}\] where \(\bar V_\cdot(\alpha)\coloneq V_\cdot\{\theta_\cdot^\star(\alpha);\alpha\}\), is the \(\theta\)-profiled value function for \(\cdot\in\{0,n\}\), and \(\bar\sigma^2_0(\alpha)\coloneq \mathbb{P}_0[\phi_\psi\{O;\eta_0, \theta_0^\star(\alpha),\alpha\}^2]\). We know that the optimal (up to a constant) bias-variance trade-off implies \(\mathbb{P}_0\{\bar V_n(\alpha) - \bar V_0(\alpha=0)\} \approx n^{-1/2}\bar\sigma_0(\alpha)\) [28]. Hence, the optimal \(\alpha\) will also solve one of the following equations \(\require{physics} \mathbb{P}_0\left\{\pdv\alpha \bar V_n(\alpha)\right\}\pm c n^{-1/2} \pdv\alpha \bar\sigma_0(\alpha) =0\), where \(c\) is a positive constant. We approximate these expressions via \(\require{physics} \pdv\alpha \bar V_n(\alpha)\pm c n^{-1/2}\pdv\alpha \bar\sigma_n(\alpha)\) where \(\bar\sigma_n(\alpha)\) is a consistent estimator for \(\bar\sigma_0(\alpha)\). The choice of which solution yields the minimizer depends on the sign of \(\require{physics} \pdv\alpha \bar V_n(\alpha)\); because \(\bar V_0(\alpha)\) is generally decreasing in \(\alpha\), we propose selecting \[\label{eqn:alpha-n-def} \alpha_n \coloneq \mathop{\mathrm{arg\,max}}_{\alpha\ge0} \bar{V}_n(\alpha)-\kappa n^{-1/2}\bar\sigma_n(\alpha)\,\text{,}\tag{8}\] for some fixed \(\kappa>0\). This is equivalent to selecting the value at which \(\bar V_n(\alpha)\) attains the highest lower confidence interval bound at level \(\Phi(\kappa)-\Phi(-\kappa)\) where \(\Phi\) is the standard normal distribution function. In practice, the lower-bound criterion can occasionally return very small \(\alpha_n\) corresponding to negligible stabilization relative to \(\alpha=0\). To avoid introducing an unnecessarily tilted estimand in such cases—and to align with asymptotic comparison to the untilted target developed later—we apply a simple clipping rule that sets \(\alpha_n\) to zero whenever it falls below a vanishing threshold: \[\label{eqn:alpha-n-clipped} \tilde{\alpha}_n = \begin{cases}0 & \alpha_n < n^{-1/4} \\ \alpha_n & \alpha_n\ge n^{-1/4}\end{cases}\,\text{.}\tag{9}\] Specifically, in 4 we show that, under the condition that the population profiled value \(\bar V_0(\alpha)\) is maximized at \(\alpha=0\), using \(\tilde{\alpha}_n\) yields the asymptotic limit as the untilted estimator.

4 Theoretical Properties↩︎

We proceed from efficiency characterization to inference for both the regimen-response curve and the induced optimizer. We first establish identification of the tilted counterfactual mean as an observed-data functional expressed through longitudinal inverse probability weights (1). We then derive the canonical gradient of \(\psi_0(\theta;\alpha)\) under the nonparametric model (2). This canonical gradient implies the nonparametric efficiency bound and the nuisance structure associated with efficient estimation.

Building on this characterization, we show that the proposed UIPW estimator is regular and asymptotically linear, and we establish both pointwise and uniform (over \(\Theta\)) limiting results for the estimated regimen-response curve (1). We then show that, for appropriately vanishing tilting sequence \(\alpha_n\), the resulting estimator is asymptotically equivalent to the untilted estimator targeting \(\psi_0(\theta;0)\) (1). We then present analogous results for the learned optimizer \(\theta_n^\star(\alpha)\) and its counterpart under vanishing tilting (2 and 2). Finally, we study the behavior of the data-adaptive tilting selector and provide sufficient conditions for it to attain the requisite rate to support the previous corollaries (3).

4.1 Identification and Canonical Gradient↩︎

This subsection formalizes the observed-data parameter underlying our proposed estimator and characterizes its nonparametric efficiency bound. Under 1, the tilted counterfactual mean \(\mathbb{P}_0(Y^{\theta,\alpha})\) is identifiable from the observed data law. While identification can be expressed via either outcome regression or inverse probability weighting, we focus on an IPW change-of-measure representation because it yields a convenient parameter mapping for deriving the canonical gradient and motivates the efficient IPW construction used throughout.

From this identification, we can obtain the canonical gradient, which characterizes the nonparametric efficiency bound for estimating \(\psi_0(\theta;\alpha)\). Let \[\mu_{t,0}^{\theta,\alpha}(A_t,H_t)\coloneq \mathbb{P}_0\Bigl[ \Bigl\{\prod_{j>t}W_{j,0}^{\theta,\alpha}(A_j,S_j)\Bigr\}Y\mid A_t,H_t\Bigr]\] be the conditional expected outcome if an individual with covariate history \(\{A_t,H_t\}\) were to follow regimen \(q_j^{\theta,\alpha}\) at decision points \(j\in\{t+1,\ldots,T\}\). These regressions admit a standard backward recursion; we record it here for completeness: \(\mu_T^{\theta,\alpha}(A_T,H_T)=\mathbb{P}_0(Y|A_T,H_T)\), and \[\begin{align} \mu_t^{\theta,\alpha}(A_t,H_t) = \mathbb{P}_0\left\{\sum_{a\in\{0,1\}} q^{\theta,\alpha}_{t+1}(A_{t+1}=a, S_{t+1})\mu^{\theta,\alpha}_{t+1}(A_{t+1}=a,H_{t+1}) \mid A_t,H_t \right\}\,\text{,} \end{align}\] for \(t=1,\ldots,T-1\). Let \(\eta_0\coloneq\{\pi_{1,0},\ldots,\pi_{T,0},\mu_{1,0}^{\theta,\alpha},\ldots,\mu_{T,0}^{\theta,\alpha}\}\) denote the set of nuisance functions corresponding to both sequential treatment assignment and outcome regressions.

2 shows that the canonical gradient decomposes into a naïve term, \(W_0^{\theta,\alpha}Y-\psi_0(\theta;\alpha)\) minus an augmentation term \(\sum_{t=1}^T D_{\mathrm{CAR},t}(O;\eta_0,\theta,\alpha)\) that projects the IPW mapping onto the nuisance tangent space associated with the sequential treatment mechanism. Each \(D_{\mathrm{CAR},t}\) term can be read as a weighted, scaled IPW residual at time \(t\), providing a connection to the HAL score equations. This canonical gradient determines the nonparametric efficiency bound for \(\psi_0(\theta;\alpha)\) and will serve as the reference influence function in our asymptotic linearity results.

4.2 Efficient Inference for the Regimen Response Curve↩︎

We state a set of regularity conditions used to control both the pointwise asymptotic linear expansion of \(\psi_n(\theta;\alpha)\) for fixed \((\theta,\alpha)\) and the stronger uniform convergence needed for functional inference over \(\Theta\). 2 is a convenient sufficient condition that ensures the relevant influence functions and remainder terms are uniformly integrable and that the empirical process arguments used for \(\ell^\infty(\Theta)\) convergence apply without additional tail conditions. 3 enforces stochasticity and smoothness of the policy class; [assumption:q-smooth:derivs] is only needed for downstream results on optimizer inference and can be relaxed if one is only interested in curve-level inference.

These conditions are readily verifiable for the logistic model paired with a compact index set \(\Theta\). However, deterministic interventions, given by \(q_t^\theta(1,s)=I\{b(s)^{\top}\theta>0\}\), do not satisfy these conditions. We note that the indicator function can be retrieved from the logistic model by letting \(\lVert\theta\rVert \to\infty\); this is in part why we do not consider the general index set \(\Theta=\mathbb{R}^p\) and require a compact subset.

We now combine the canonical gradient representation from 2 with the undersmoothing condition to obtain an asymptotic linear expansion for \(\psi_n(\theta;\alpha)\).

1 states that, under regularity conditions and sufficient undersmoothing, \(\psi_n(\theta;\alpha)\) is regular and asymptotically linear with influence function equal to the canonical gradient \(\phi_\psi\), and therefore attains the nonparametric efficiency bound for each fixed \((\theta,\alpha)\). The uniform weak convergence result strengthens this to support functional inference over \(\Theta\), providing the basis for simultaneous inference for the regimen-response curve.

The next corollary shows that, for a vanishing tilting sequence \(\alpha_n\), the tilted estimator has the same first-order asymptotic linear representation—and hence limiting law—as the untilted estimator \(\psi_n(\theta;0)\).

4.3 Efficient Policy Optimization↩︎

We now study the learned optimal policy parameter \(\theta_n^\star(\alpha)\), defined as a maximizer of the penalized value function \(V_n(\theta;\alpha)\) over \(\Theta\). Our argument follows a standard argmax/M-estimation strategy: under suitable curvature, the optimizer can be characterized through a first-order condition, and its limiting behavior is governed by the corresponding gradient process.

Assumption 4 imposes standard regularity for argmax functionals: uniqueness and interiority ensure that \(\theta_0^\star(\alpha)\) is characterized by the first-order condition \(\nabla_\theta V_0\{\theta_0^\star(\alpha);\alpha\}=0\), while negative definiteness of \(H_0(\alpha)\) provides local curvature needed to invert the linearization and obtain a \(\sqrt{n}\)-rate expansion. Under this assumption, expanding the first-order condition, \[\theta_n^\star(\alpha)-\theta_0^\star(\alpha)=-H_0(\alpha)^{-1}\nabla_\theta V_n\{\theta_0^\star(\alpha);\alpha\} + o_p(n^{-1/2})\,\text{.}\] Consequently, inference for \(\theta_n^\star(\alpha)\) reduces to establishing regular asymptotic linearity of the gradient \(\nabla_\theta V_n(\theta;\alpha)\) evaluated at \(\theta=\theta_0^\star(\alpha)\).

To that end, we show that, under one additional HAL condition regarding approximation of the relevant gradient multipliers (7), the gradient process is asymptotically linear with influence function equal to the canonical gradient of \(\nabla_\theta V_0(\theta;\alpha)\) (10 12). It follows that the optimizer inherits an asymptotic linear expansion determined by the inverse Hessian, \(H_0(\alpha)^{-1}\), and the canonical gradient of \(\nabla_\theta\psi_0(\theta;\alpha)\). This is formalized in the following theorem.

Theorem 2 shows that, under curvature and undersmoothing conditions, \(\theta_n^\star(\alpha)\) is a regular asymptotically linear estimator of \(\theta_0^\star(\alpha)\) with a \(\sqrt{n}\) limit distribution. Relative to curve-level inference, the optimizer expansion introduces the additional nuisance component \(\eta_0'\coloneq\{\nabla_\theta \mu_{t,0}^{\theta,\alpha} \mid_{\theta=\theta_0^\star(\alpha)}\}_{t=1}^T\), reflecting that inference for an argmax depends on the behavior of the gradient process and therefore on derivatives of the forward regressions with respect to \(\theta\).

Analogous to 1, if \(\alpha_n=o_p(n^{-1/2})\) then \(\theta_n^\star(\alpha_n)\) admits the same first-order asymptotic linear representation as the untilted optimizer estimator \(\theta_n^\star(0)\).

4.4 Data-Adaptive Tilting↩︎

The asymptotic results in 1 2 require \(\alpha_n=o_p(n^{-1/2})\) so that the tilted estimators share the same first-order limit as the untilted analogs. The following lemma provides sufficient conditions for the trimmed lower-confidence-bound selector \(\tilde{\alpha}_n\) to attain the necessary rate.

The first condition of 3 requires that the untilted regimen response curve attains the highest possible value function (as discussed in 2) and that there is linear separation near \(\alpha=0\). The second condition ensures that \(\sup_{\alpha\in[0,\delta]}\lvert\{\bar V_n(\alpha)-\kappa n^{-1/2}\bar \sigma_n(\alpha)\}-\bar V_0(\alpha)\rvert=O_p(n^{-1/2})\). We note that this is a weak assumption that generally holds under [assumption:identification:positivity] 2. Together, these conditions enable an argmax argument for \(\alpha_n\) that accounts for the boundary maximizer.

5 Numerical Simulations↩︎

5.1 Simulation Setup↩︎

The simulation study was designed to generate longitudinal data with features that could plausibly arise in studies of mobile prompts intended to support cigarette quit attempts. In this setting, prompts may be useful when an individual is experiencing elevated negative affect, particularly when cigarettes are available, but may be burdensome when delivered at low-risk moments. We therefore generated repeated measurements of two time-varying state variables \(S_{t,1}\) and \(S_{t,2}\), intended to respectively represent negative affect and cigarette availability at time \(t\). For participant \(i=1,\ldots,n\) we generated the trajectory \(O \coloneq (S_{1},A_{1}, \ldots, S_{T},A_{T},Y)\) where \(S_{t}\coloneq (S_{t,1},S_{t,2})\) contains both state variables. The data generation process is described in 10.

We considered the one-dimensional policy class \(q_t^\theta(A_t,S_t)=\mathop{\mathrm{expit}}(\theta S_{t,1})\) with \(\Theta=[-0.73,0.73]\). This index set, constructed following 9.1, ensures that \(q_t^\theta(a,S_t)\in[0.05,0.95]\) for all \(a\in\{0,1\}\), \(S_t\in\mathcal{S}_t\). Policy optimization was performed using a penalty weight of \(\lambda=0.01\).

We considered five IPW estimators, which differed only in how the treatment mechanism was handled. The oracle estimator used the true treatment probabilities \(\pi_{t,0}\). All other estimators used pooled propensity score models fit to observations from all \(T\) decision points, with decision time included as a covariate. The parametric estimator used a correctly specified logistic regression model with linear terms for \(S_{t,1}\), \(S_{t,2}\), \(t\), and \(S_{t,1}S_{t,2}\). The remaining three estimators used the HAL to estimate the pooled propensity score as a function of \(S_{t,1}\), \(S_{t,2}\), and \(t\), and were cross-fit over two external folds. The CV HAL estimator used the standard HAL fit with the \(L_1\)-norm selected by five-fold cross-validation. The UIPW-\(D_\mathrm{CAR}\) and UIPW-Score estimators selected undersmoothed HAL fits by minimizing, respectively, aggregated \(L_2\) \(D_\mathrm{CAR}\) (6 ) and score (7 ) criteria. Because a pooled propensity model was used, both UIPW selectors targeted sums of the corresponding time-specific criteria across \(t=1,\ldots,T\), matching the decomposition of the longitudinal IPW remainder. For each estimator, we considered both untilted (\(\alpha=0\)) and data-adaptive (3.4) implementations. We implemented truncated product-weights for all estimators, forcing \(\prod_{t=1}^TW_{t,n}^{\theta,\alpha}\le20\).

We evaluated estimator performance under the following factorial design: \(a\in\{0,0.25\}\) (where \(\pi_{t,0}(A_t=1\mid H_t) = \mathop{\mathrm{expit}}(a S_{t,1})\), thereby considering MRT and observational designs), \(n\in\{100,500,1000\}\), and \(T\in\{10,25,50\}\). We report relevant metrics, averaged across \(200\) Monte Carlo replications, as follows. For the regimen response curve, we summarize the pointwise bias, mean squared error (MSE), and coverage as a function of \(\theta\). For the optimal policy, we summarize the square root mean squared error (RMSE) and coverage.

5.2 Simulation Results↩︎

We summarize performance for both regimen-response curve estimation and policy learning under the MRT (\(a=0\)) and observational (\(a=0.25\)) assignment mechanisms.

1 presents the scaled bias of all estimators without tilting (i.e., \(\alpha=0\)) and shows that the bias was generally small in the MRT setting across all estimators. We note that in this setting, CV HAL often selected very small bases of \(1\)–\(2\) functions, and therefore behaved similarly to a low-dimensional or near-parametric estimator. In contrast, in the observational setting, CV HAL shows more pronounced scaled bias as \(n\) increases, especially when \(T=10\). In contrast, the UIPW estimators generally controlled the bias. This pattern supports the role of undersmoothing in reducing the leading IPW bias term when the treatment mechanism must be learned nonparametrically.

2 demonstrates broadly similar pointwise coverage patterns across all untilted estimators, including the oracle IPW estimator. Coverage was generally close to the nominal rate for moderate values of \(\theta\), while undercoverage was observed when the target policy differed more from the observational policy. The fact that undercoverage appears in similar regions for all estimators—including the oracle IPW estimator—suggests that the main driver was not estimator-specific bias control or the variance estimator, but numerical positivity issues associated with the longitudinal weights. 5 compares coverage for the oracle and UIPW estimators using data-adaptive \(\alpha_n\) tilting. There, tilting resulted in decreased coverage for \(\theta\) far from the observational policy; this was more pronounced in the MRT setting, where larger \(\alpha_n\) were generally selected, resulting in larger bias away from the observational policy.

3 presents the relative efficiency of each estimator to the oracle IPW estimator in the absence of tilting. In the MRT setting, most estimators behaved similarly to the oracle IPW estimator. Score-based undersmoothing resulted in larger variances than all other estimators nearly uniformly across settings; this is likely the result of the score-based selector often selecting the maximum amount of undersmoothing allowed, resulting in potential numerical instability. In the observational setting, the relative efficiencies of all nonparametric estimators varied more strongly in \(\theta\). This heterogeneity reflects that the outcome regression functions \(\mu_{t,0}^{\theta,\alpha}\) that appear in the leading bias term, 3 , depend on \(\theta\). Hence, some regions of the policy class appear to induce more stable or easier-to-approximate empirical score functions than others; this is reflected in efficiency gains where these functions are adequately solved, and potential inefficiency otherwise.

4 shows that adaptive tilting reduced the MSE for nearly all estimators and settings. These gains were not restricted to estimators with flexibly estimated propensity scores—oracle and correctly specified parametric IPW estimators also benefited. These gains were often largest in larger-\(T\) settings, consistent with the accumulation of weight variability across repeated decision points.

1 presents the scaled RMSE for each policy selector induced by the corresponding regimen response curve estimator, with and without adaptive tilting. Adaptive tilting generally improved policy RMSE relative to the untilted comparators; the largest gaps between tilted and untilted estimators tended to occur in high-\(T\) settings. Additionally, UIPW using the \(L_2\) \(D_\mathrm{CAR}\) selector generally had similar RMSE and regret to the optimal selector within each setting.

2 displays the coverage of each policy estimator. Most estimators attained or surpassed the nominal coverage rate for moderate \(n\). In some settings, all estimators demonstrated overcoverage.

5.3 Simulation Conclusions↩︎

In our simulations, we observed that correctly specified parametric and oracle IPW estimators can be more efficient than the UIPW estimators in small-to-moderate samples, despite our asymptotic efficiency theory. This suggests that the undersmoothing required to control the leading IPW bias can introduce non-negligible finite-sample variability. This pattern is consistent with prior work showing that practical undersmoothing implementations can have inflated variance at smaller sample sizes, diminishing with \(n\). In the fully randomized setting studied by [24], undersmoothing based on the score criterion yielded larger mean squared errors than both an unadjusted estimator and an IPW estimator using cross-validated HAL propensity scores, while DCAR-based undersmoothing was typically the most stable of the undersmoothing rules considered. Similarly, [25] reported that, at smaller sample sizes, UIPW variants selected by either score- or \(D_\mathrm{CAR}\)-based criteria can exhibit inflated variance compared with the cross-validated HAL IPW and DR estimators, with these gaps diminishing as \(n\) increases. Our simulations show a similar pattern in longitudinal settings: the score-based selector often appeared to undersmooth aggressively and produced larger MSEs, whereas the \(L_2\) \(D_\mathrm{CAR}\) selector was generally more stable and yielded policy RMSE and regret close to the best-performing selector across many settings. Thus, although undersmoothing is essential for first-order remainder control in the nonparametric setting, its finite-sample performance depends strongly on how the undersmoothing level was selected. The pragmatic \(L_2\) \(D_\mathrm{CAR}\) selector used here based on a working parametric model provides one way to target the relevant longitudinal remainder while avoiding full nonparametric estimation of the policy- and time- indexed outcome regressions, but further stabilization may be possible.

More broadly, the empirical efficiency gains available from undersmoothing and other asymptotically efficient estimation approaches may be smaller in the stochastic setting relative to the well-established gains in the deterministic setting. Because stochastic interventions remain probabilistic, they typically yield more stable weights, leaving less variability for efficient augmentation or undersmoothing-based corrections to mitigate. This interpretation is consistent with the additional simulations in 15, which suggest that potential efficiency gains associated with (nonparametric) efficient estimation increase as stochastic interventions approach near-deterministic regimes.

6 Discussion↩︎

In this work, we address estimation and optimization of policies for distal outcomes in settings with many decision points, as arise in MRTs and other JITAI designs. In such settings, standard semiparametric-efficient estimation typically requires estimating a large collection of forward outcome regression functions indexed by both time and policy, which becomes impractical and potentially fragile as \(T\) and the policy space \(\Theta\) grow. Moreover, inverse probability weighted estimators involve products of weights across decision points, and can be numerically unstable when \(T\) is large. We proposed a UIPW approach for estimating the regimen-response curve and optimal policy over a parametric stochastic policy class. Our approach is nonparametrically efficient yet avoids explicit modeling of the policy-indexed outcome regressions. To improve finite-sample stability, we introduced a data-adaptive tilting device that interpolates between the target policy and the observed treatment mechanism. We showed that the estimated regimen-response curve converges to a Gaussian process over \(\Theta\), enabling the construction of simultaneous confidence bands. We also derived an asymptotic linearization for the policy optimizer, allowing inference for the learned optimal policy.

Our approach can be viewed as a saturated estimator of the regimen-response curve, in contrast to MSM analyses that summarize this curve through a lower-dimensional working model. [29] provided a complementary perspective in the single-decision point setting by defining a fully nonparametric MSM in which the covariate-adjusted regimen-response curve is characterized as a risk minimizer over a rich càdlàg function class with bounded sectional variational norm. A natural future direction is to extend this nonparametric MSM framework to our longitudinal stochastic setting by targeting a covariate adjusted curve \(m_0(\theta;V)\) for \(V\subseteq S_1\) to support \(V\)-specific optimal policies, all while using undersmoothing to avoid outcome regression estimation. Because our stochastic policy parameterization already induces substantial smoothness in \(\theta\) through smoothly varying weights, we expect the primary gains of such an MSM formulation to arise from covariate adjustment and structured estimation across \(V\), rather than from additional smoothing across \(\theta\) alone.

We introduced an \(\alpha\)-tilting device that interpolates between the target policy and observed treatment mechanism at each decision point, yielding a stabilized policy family that shrinks early weight contributions toward one. Although our default tilting schedule prioritizes stabilization of earlier weights, one could instead use an approach that preserves the target policy more strongly at earlier decision points if early actions are thought to drive distal outcomes. More flexibly, one could allow the direction of tilting to depend on a signed tuning parameter, selected in a data-adaptive way. This interpolation has a close conceptual parallel to incremental propensity score interventions, which conceptualize shifting the propensity score away from the observed mechanism, rather than deterministically setting treatment [22]. A key contrast is that incremental and related approaches are motivated as positivity-robust estimands whereas our tilted estimand was developed to ease estimation when overlap holds by design but products of weights are numerically volatile; it does not resolve genuine support violations.

In summary, these developments were motivated by the practical goal of constructing JITAIs using intensive longitudinal data when the scientific objective is a distal end-of-study outcome rather than a discounted sum of proximal rewards. By providing functional inference for the regimen–response curve and principled inference for the induced optimizer in a high-\(T\) setting, the proposed framework helps close the methodological gap between MRT data structures and the inferential targets that commonly drive JITAI design.

Appendix↩︎

This appendix does the following:

7

presents supplemental figures.

8

describes the highly adaptive lasso (8.1). It includes theoretical undersmoothing conditions (8.2), a strengthened version of Lemma 1 from [24] to offer supremum norm control of remainder terms (8.3), and practical undersmoothing implementation details (8.4).

9

offers practical implementation details including index set elicitation (9.1) and accounting for randomization ineligibility common in mobile health trials (9.2).

10

describes the full longitudinal data generation process used in the numerical simulations.

11

provides auxiliary statements and proofs used throughout the rest of the appendix.

12

supports theoretical arguments regarding regimen-response curve estimation, proving 1 2 1 1.

13

supports theoretical arguments regarding policy optimization, proving 2 2.

14

supports theoretical results regarding data-adaptive tilting, proving 3.

15

discusses how the efficiency gains of asymptotically efficient estimators beyond conventional IPW estimators may be reduced for stochastic versus deterministic interventions.

7 Supplemental Figures↩︎

8 The Highly Adaptive Lasso and Undersmoothing↩︎

8.1 The Highly Adaptive Lasso↩︎

Let \(\mathbb{D}[0,\tau]\) denote the Banach space of real-valued càdlàg functions from \([0,\tau]\subset \mathbb{R}^d\) to \(\mathbb{R}\). Here, \(\tau<\infty\) is the upper bound of the support of all functions in this space. Consider some subset of the covariates: \(s\subset \{1,\ldots,d\}\) and function \(f\in\mathbb{D}[0,\tau]\), let the \(s\)th section of \(f\) be given by \(f_s(u)=f\{I(1\in s)u_1,\ldots, I(d\in s)u_d\}\), that is, varying the function along the variables in \(s\) but setting all other variables to \(0\). The sectional variational norm of \(f\) is then given by aggregating all subsets’ variational norms: \[\require{physics} \norm{f}_v^\star\coloneq \lvert f(0)\rvert+\sum_{s\subset\{1,\ldots,d\}}\int_{0_s}^{\tau_s}\lvert \dd f_s(u) \rvert\,\text{.}\]

Consider the propensity score at a given \(t \in\{1,\ldots,T\}\), \(\pi_{t,0}\). Given that \(\pi_{t,0}\) is càdlàg with finite sectional variational norm, [30] shows that \[\require{physics} \begin{align} \mathop{\mathrm{logit}}\pi_{t,0}(w) &= \mathop{\mathrm{logit}}\pi_{t,0}(0) + \sum_{s\subset\{1,\ldots,d\}} \int_{0_s}^{w_s}\dd\mathop{\mathrm{logit}}\pi_{t,0,s}(u) \\ &= \mathop{\mathrm{logit}}\pi_{t,0}(0) + \sum_{s\subset\{1,\ldots,d\}} \int_{0_s}^{\tau_s} I(u_s \le w_s)\dd\mathop{\mathrm{logit}}\pi_{t,0,s}(u)\,\text{.}\stepcounter{equation}}\label{eqn:hal-integral} \end{align}\tag{10}\] The HAL approximates the integral from 10 by \[\begin{align} \mathop{\mathrm{logit}}\pi_{t,0}(w;\beta) = \beta_{t,0} + \sum_{s\subset\{1,\ldots,d\}} \sum_{i=1}^n \beta_{t,s,i}\varphi_{t,s,i}(w_s)\,\text{,} \end{align}\] where \(\varphi_{t,s,i}(w_s)=I(w_s\ge u_{t,s,i})\) and \(u_{t,s,i}\) is the \(i\)th observed value of the \(s\)th section of the covariates at decision point \(t\). The HAL estimator \(\beta_{t,n,\lambda}\) is obtained by solving the constrained minimization problem: \(\beta_{t,n,\lambda} = \arg\min \mathbb{P}_n L(\mathop{\mathrm{logit}}\pi_{t,n,\lambda_{n,t}})\) subject to \(\lvert\beta_{t,0}\rvert + \sum_{s\subset\{1,\ldots,d\}} \sum_{i=1}^n \lvert\beta_{t,s,i}\rvert<\lambda\).

8.2 Undersmoothing Details↩︎

Suppose that the set of \(L_1\)-norms for each HAL fit, \(\lambda_n=\{\lambda_{1,n},\ldots,\lambda_{T,n}\}\), is chosen such that, for all \(t=1,\ldots,T\), \[\require{physics} \label{eqn:hal-undersmooth-condition} \min_{(s,j)\in\mathcal{J}_{t,n}} \norm{\mathbb{P}_n \frac{\dd}{\dd \mathop{\mathrm{logit}}\pi_{t,n,\lambda_{n,t}}} L(\mathop{\mathrm{logit}}\pi_{t,n,\lambda_{n,t}})(\varphi_{t,s,j})}=o_p(n^{-1/2})\,\text{,}\tag{11}\] where \(L(\cdot)\) is the log-likelihood loss and \(\mathcal{J}_{t,n}\) is the set of indices for the basis functions where \(\beta_{t,s,j}\neq0\). Here, \(\require{physics} \dd/\dd \mathop{\mathrm{logit}}\pi_{t,n,\lambda_{n,t}}\{ L(\mathop{\mathrm{logit}}\pi_{t,n,\lambda_{n,t}})(\varphi_{t,s,j})\}\) is \(\require{physics} \dd/\dd\epsilon\{L(\mathop{\mathrm{logit}}\pi_{t,n,\lambda_{n,t}} + \epsilon\varphi_{t,sj}\}\), which denotes the directional derivative of the loss along the path \(\mathop{\mathrm{logit}}\pi_{t,n,\lambda_{n,t}}^\epsilon = \mathop{\mathrm{logit}}\pi_{t,n,\lambda_{n,t}} + \epsilon \varphi_{s,j}\). Under the log-likelihood loss, the derivative is \[\require{physics} \frac{\dd}{\dd\epsilon} L(\mathop{\mathrm{logit}}\pi_{t,n,\lambda_{n,t}})(\varphi_{t,s,j})\bigg\rvert _{\epsilon=0} = \varphi_{t,s,j}\{A_t-\pi_{t,n,\lambda_{n,t}}(1\mid H_t)\}\,\text{,}\] the corresponding score function. Hence, an equivalent statement of the supposition is that \[\label{eqn:hal-undersmooth-condition2} \min_{(s,j)\in\mathcal{J}_{t,n}} \norm{\mathbb{P}_n \varphi_{t,s,j}\{A_t-\pi_{t,n,\lambda_{n,t}}(1\mid H_t)\}}=o_p(n^{-1/2})\,\text{.}\tag{12}\] Although unrestricted maximum likelihood estimation exactly solves these score equations, the \(L_1\) restriction generally prevents exact solutions. The assumption that this term is \(o_p(n^{-1/2})\) is hence equivalent to assuming that the \(L_1\) restriction is weakened until one of the score equations is solved to a precision of \(o_p(n^{-1/2})\). We say that the HAL is “sufficiently undersmoothed” if 11 holds for each \(t=1,\ldots,T\).

We make the following assumptions to allow the undersmoothed HAL basis to sufficiently approximate the leading terms in the DCAR remainders (thereby satisfying the premise of 4 throughout various applications). These are stated as pointwise assumptions, but we will often assume they hold uniformly in \((\theta,\alpha)\) over some subset of \(\Theta \times [0,\infty)\).

The following assumption is used to ensure that \(\nabla_\theta\psi_n(\theta;\alpha)\) is also regular and asymptotically linear, which is needed for the asymptotic properties of \(\theta_n^\star(\alpha)\).

8.3 Formal Lemma and Proof↩︎

To control remainders uniformly across \(\Theta\), we require a strengthened version of [24] Lemma 1. The original lemma provides pointwise control of terms of the form \(\mathbb{P}_n \{ f \times (A_t-\pi_n)\}\) when \(\pi_{t,n}\) is a sufficiently undersmoothed HAL estimate of the propensity score. We extend the result to offer uniform control over a class of leading functions \(f_\theta:\theta\in\Theta\). This proof largely follows the argument provided to bound \(\mathbb{P}_n D(\tilde{f},\pi_n)\) pointwise in [24]; the key differences are that the \(\kappa_\theta\) is now bounded uniformly, and that obtaining a bound on \(D(f,\pi_n)\) from the bound on \(D(\tilde{f},\pi_n)\) requires a stronger empirical process argument that leverages uniform convergence of \(\tilde{f}_\theta\) to \(f_\theta\).

8.4 Undersmoothing in Practice↩︎

The original undersmoothing criteria (11 ) ensures that the penalty parameters for each propensity score model, \(\lambda_{1,n},\ldots,\lambda_{T,n}\), are chosen such that the bias term in the asymptotic representation of \(\sqrt{n}\{\psi_n(\theta;\alpha)-\psi_0(\theta;\alpha)\}\) is \(o_p(n^{-1/2})\). However, this does not directly imply a procedure to select \(\lambda_n\).

Given that the goal of the undersmoothing is to ensure that the bias term satisfies \(\mathbb{P}_n\{ D_{\mathrm{CAR},t}(\mu_0^{\theta,\alpha},\pi_n;\theta,\alpha)\}=o_p(n^{-1/2})\) for all \(t=1,\ldots,T\); one approach is to directly minimize an approximation of this quantity. [24], [25], [29] all offer selectors that directly approximate the DCAR term by nonparametrically estimating all nuisance functions and minimize a cross-validated estimate. However, this is computationally prohibitive in our longitudinal policy-index setting because the augmentation term involves the forward regressions \(\{\mu_{t,0}^{\theta,\alpha}(A_t, H_t):t<T,\theta\in\Theta\}\), and hence would require estimating a large family of nuisance functions jointly indexed by time and policy. Instead, we use a pragmatic DCAR-approximated selector that targets the same remainder but replaces \(\mu_{t,0}^{\theta,\alpha}\) with a low-dimensional working approximation fit only for tuning. We target performance for the untilted estimator and therefore fix \(\alpha=0\) henceforth. For each fold \(v\in1\ldots,V\) and time \(t\), we construct pseudo-outcomes on the training sample of the form \(Z_{i,t}^{\theta,\alpha}=\{\prod_{j>t}W_{i,j,\text{ref}}^{\theta,\alpha}\}Y_i\), where \(W_{i,j,\text{ref}}^{\theta,\alpha}\) uses a reference propensity score \(\pi_{t,j,\text{ref}}\) that is equal to either the cross-validated HAL estimate (i.e., no undersmoothing) or the known propensity score in MRT settings. We then fit working regressions \(\mu_{t,n}^{-(v)}(a,h;\theta)\) by pooling pseudo-outcomes across a modest training subset \(\Theta_{\text{train}}\subset\Theta\) and modeling \(Z_{t}^{\theta,\alpha}\) as a function of \((H_t,A_t)\) together with a smooth basis \(g(\theta)\) (e.g., a spline basis) and optional interactions. Then, given \(\mu_{t,n}^{-(v)}\), for each undersmoothing level \(\lambda\) we evaluate \[\begin{gather} R^{(v)}(\lambda;\theta) \coloneq \mathbb{P}_{n,v}^1 \Bigl\{\prod_{i<t} W_{i,n,\lambda}^{\theta,\alpha}\Bigr\} (t/T)^\alpha\;\\ \times \sum_{a\in\{0,1\}}\mu_{t,n}^{-(v)}(a, H_t;\theta)\, q_t^\theta(a,S_t) \frac{I(A_t=a)-\pi_{t,n,\lambda}^{(-v)}(a\mid H_t)}{\pi_{t,n,\lambda}^{(-v)}(a\mid H_t)}\,\text{,} \end{gather}\] where \(\mathbb{P}_{n,v}^1\) denotes the empirical measure on the testing data from the \(v\)th fold. We then select \(\lambda_{t,n}\) by minimizing either an \(L_2\)-integrated criterion \[\lambda_{t,n,L_2} \coloneq \arg \min_{\lambda} V^{-1}\sum_{v=1}^V \lvert \Theta_{\text{test}}\rvert^{-1}\sum_{\theta\in\Theta_{\text{test}}} \{R^{(v)}(\lambda;\theta)\}^2\,\text{,}\] or a uniform criterion \[\lambda_{t,n,L_\infty} \coloneq \arg \min_{\lambda} V^{-1}\sum_{v=1}^V \max_{\theta\in\Theta_{\text{test}}} \lvert R^{(v)}(\lambda;\theta) \rvert\,\text{,}\] where \(\Theta_{\text{test}}\coloneq\Theta\setminus \Theta_\text{train}\).

Alternatively, we propose the alternate selection criteria based solely on the score function for the propensity score model: \[\label{eqn:undersmooth-score} \lambda_{t,n,\text{score}}=\arg\min_{\lambda} V^{-1}\sum_{v=1}^V \frac{1}{\norm{\beta_{n,\lambda,v}}_{1}}\sum_{(s,j)\in\mathcal{J}_n^{(v)}(\lambda)} \lvert \mathbb{P}_{n,v}^1 \tilde{S}_{t,s,j}(\phi, \pi_{t,n,\lambda,v}) \rvert\,\text{,}\tag{13}\] where \(\mathcal{J}_n^{(v)}(\lambda)\) is a set of indices for the active basis functions such that \(\beta_{t,s,j,n,\lambda,v} \neq 0\), and \[\tilde{S}_{t,s,j}(\varphi, \pi_{t,n,\lambda,v})=\varphi_{t,s,j}( H_t)\{A_t-\pi_{t,n,\lambda,v}(1\mid H_t)\}\{\pi_{t,n,\lambda,v}(1\mid H_t)\}^{-1}\] is the component of the score equation corresponding to \(\beta_{t,s,j}\) divided by the estimated propensity score. This criterion only requires estimating the \(T\) propensity scores used in the general estimation procedure.

Finally, to ensure that the undersmoothed HAL fit still converges at the \(o_p(n^{-1/4})\) rate, as needed for our asymptotic theory, we impose a cap such that the basis does not grow too quickly, as proposed by [25]. Let \(\tilde{\lambda}_{t,n}\coloneq \max \{\lambda : V^{-1}\sum_{v=1}^V \lvert \mathcal{J}_n^{(v)}(\lambda)\rvert < n^{1/2}\}\) be the largest \(L_1\) norm that yields an active basis with fewer than \(n^{1/2}\) functions. We then select \(\min\{\lambda_{t,n,\cdot}, \tilde{\lambda}_{t,n}\}\) for \(\cdot\in\{L_2,L_\infty,\text{score}\}\).

9 Practical Implementation Considerations↩︎

9.1 Index Set Elicitation↩︎

Many policy-learning approaches enforce regularity implicitly through generic parameter-norm constraints or penalties (e.g., \(L_1\) or \(L_2\) regularization), which can be difficult to interpret in terms of scientifically meaningful limits on intervention frequency. By contrast, we impose regularity by directly bounding treatment probabilities, which yields an immediately interpretable constraint on intervention frequency.

To ensure that the explored policies are genuinely stochastic at all decision points and covariate values, we fix user-specified constants \(0<\underaccent{\bar}{q} < \bar q < 1\) and restrict attention to policies satisfying \[\label{eqn:q-bound} q_t^\theta(1, S_t) \in [\underaccent{\bar}{q}, \bar q]\quad \text{for all}\quad S_t \in \mathcal{S}_t\,\text{.}\tag{14}\] This constraint directly encodes investigator-specified minimum and maximum tolerable treatment probabilities (e.g., “the policy never treats with probability below \(\underaccent{\bar}{q}\) or above \(\bar q\)”).

Suppose that the intervention is characterized by some monotone link function \(g\) and linear predictor \(b(S_t)\): \(q_t^\theta(1,S_t)=g\{b(S_t)^{\top}\theta\}\). Then, 14 is equivalent to a uniform bound on the linear predictor. Let \(\ell \coloneq g^{-1}(\underaccent{\bar}{q})\) and \(u \coloneq g^{-1}(\bar q)\). Then 14 holds if and only if \[\label{eqn:q-bound-lin-predictor} \ell \le b(s)^{\top}\theta \le u \quad \text{for all}\quad s \in \mathcal{S}\,\text{.}\tag{15}\] Under this framework, given probability bounds \(\underaccent{\bar}{q}\) and \(\bar q\), define \[\label{eqn:def-Theta} \Theta \coloneq \left\{\theta \in \mathbb{R}^p : \inf_{s\in\mathcal{S}} b(s)^{\top}\theta \ge \ell \quad \text{and} \quad \sup_{s\in\mathcal{S}} \le u \right\}\,\text{.}\tag{16}\] This set is closed and convex; under mild conditions on the basis, it is also compact.

In many applications, \(q_t^\theta\) will follow a form based on a generalized linear model with a linear basis. In this setting, \(\Theta\) can be constructed analytically. Suppose for exposition, without loss of generality, that \(b(s) = (1,s_1,\ldots,s_d)^{\top}\), that is, including all covariate main effects but no interactions or higher-order terms. Let \(s_j\in[a_j,b_j]\) denote the bounds for each component. Moreover, suppose that \(\mathcal{S} = \prod_{j=1}^d [a_j,b_j]\) is a hyper-rectangle. Violations of this supposition may result in overly conservative \(\Theta\); however, the desired bound on \(q^\theta\) is still attained. Expressing \(\theta=(\theta_0,\theta_1,\ldots,\theta_d)^{\top}\), \(M_0\coloneq 1\) and \(M_j \coloneq \max\{\lvert a_j\rvert ,\lvert b_j\rvert\}\) for \(j=1,\ldots,d\), we have, for any \(\theta\), \[\begin{align} \sup_{s \in \mathcal{S}} b(s)^{\top}\theta &= \sum_{0=1}^d \lvert\theta_j\rvert M_j \,\text{,}\quad \text{and}\\ \inf_{s \in \mathcal{S}} b(s)^{\top}\theta &= \sum_{0=1}^d \lvert\theta_j\rvert M_j \,\text{.} \end{align}\] Then, the index set can be written as the weighted \(\ell_1\)-ball: \[\Theta = \Bigl\{\theta \in \mathbb{R}^{d+1} : \sum_{j=0}^d \lvert\theta_j\rvert M_j \le \min(-\ell, u) \Bigr\}\,\text{.}\]

9.2 Randomization Ineligibility↩︎

In many mobile health studies, the protocol may prohibit intervention delivery at certain decision points due to practical or ethical constraints (e.g., if the participant is driving). In such cases, the decision rule is partially constrained by design: when randomization is disallowed, the study assigns a default “do nothing” action. This feature is easily accommodated in our framework by a minor refinement of the policy class, with no other changes downstream.

Let \(E_t\in\{0,1\}\) denote an eligibility indicator determined prior to assignment at each decision point \(t\) (i.e., such that the temporal ordering is \(S_t\rightarrow E_t \rightarrow A_t\)). We assume without loss of generality that when \(E_t=0\), the protocol deterministically assigns \(A_t=0\).

To enforce the design constraint, define the corresponding policy \(q_{t,\text{Elig}}^{\theta}\) by \[q_{t,\text{Elig}}^{\theta}(a, S_t,E_t) \coloneq E_t q_t^\theta(a, S_t)+(1-E_t)I(a=0)\,\text{.}\] That is, when \(E_t=1\), the policy collapses to \(q_t^\theta\); otherwise it assigns \(A_t=0\) deterministically. This modified policy both respects the support of the observed data, and prevents the consideration of policies that would intervene on ineligible participants. Similarly, the behavioral assignment mechanism is \[\pi_{t,\cdot,\text{Elig}}(a\mid H_t,E_t) \coloneq E_t \pi_{t,\cdot}(a \mid H_t,E_t=1) + (1-E_t)I(a=0)\,\text{,}\] for \(\cdot\in\{0,n\}\). Importantly, the inner propensity score is now conditioned on eligibility.

Under this construction, all expressions remain valid after replacing \(q_t^\theta\) and \(\pi_{t,\cdot}\) with their new counterparts. In particular, the time-specific weights \(q_{t,\text{Elig}}^{\theta}(A_t, S_t,E_t)/\pi_{t,\cdot,\text{Elig}}(A_t\mid H_t,E_t)\) equal the standard weights \(q_{t}^{\theta}(A_t, S_t)/\pi_{t,\cdot}(A_t\mid H_t,E_t=1)\) when \(E_t=1\), whereas if \(E_t=0\) then \(A_t=0\) and the weight is equal to one. Consequently, ineligible decision points contribute multiplicative factors of one and can equivalently be omitted from the product.

10 Simulation Details↩︎

The first state component (representing negative affect) was initialized as \(S_{1,1}\sim N(0,1)\) and then evolved according to a persistent autoregressive process with prompt effects. Specifically for \(t\ge2\), \[S_{t,1} = \operatorname{clip}\bigl\{ \rho S_{t-1,1} + A_{t-1}\tau(S_{t-1}) + \sigma\epsilon_t, -4, 4 \bigr\}\,\text{,}\] where \(\rho=0.96\) and \(\sigma = (1-\rho^2)^{1/2}\). Clipping restricts states to the closed interval \([-4,4]\), preventing extreme simulated trajectories and aligning with 2 while preserving temporal dependence. Thus, in the absence of treatment, negative affect follows a highly-persistent mean-zero process. Treatment can shift the subsequent affect through prompt effect \(\tau\), which depends on the current state and is detailed in the following paragraph. The second component (representing cigarette availability) was generated as an independent Bernoulli variable: \(S_{t,2}\sim \operatorname{Bernoulli}(p_2)\) with \(p_2=0.25\).

The prompt model encodes two behavioral features. First, randomization to a prompt is helpful when an individual has high negative affect and has a high potential for benefit. Conversely, the prompt carries a burden cost when an individual is doing relatively well. To implement this, we define the smooth vulnerability score \(V_t\coloneq \mathop{\mathrm{expit}}\{k(S_{t,1}-c)\}\) for \(k=3\) and \(c=0.2\). The prompt effect is then given by \[\tau(S_{t-1}) = -\gamma_\text{help} V_{t-1}(1+\gamma_\text{avail}S_{t-1,2})+\gamma_\text{harm}(1-V_{t-1})\{1+\gamma_\text{avail}(1-S_{t-1,2})\}\,\text{,}\] with \(\gamma_\text{help}=0.35\), \(\gamma_\text{harm}=0.18\) and \(\gamma_\text{avail}=0.6\).

Treatment assignments were generated through a logistic model with potential dependence on the current state: \(\pi_{t,0}(A_t=1\mid H_t) = \mathop{\mathrm{expit}}(a S_{t,1})\). We considered \(a\in\{0,0.25\}\). When \(a=0\), treatment is assigned with probability \(0.5\) at every decision point, corresponding to a micro-randomized trial. When \(a=0.25\), participants with higher negative affect are more likely to receive treatment; this represents a generally good observational policy that moves treatment in the same direction as the optimal stochastic policy in our policy class.

The distal outcome was generated as a nonlinear function of the final state, thereby generating nonlinear outcome regression models and challenging bias terms to control. Let \(B\coloneq \log\{1+\exp(S_{T,1}-1)\}\) and \(H=(\max\{0,S_{T,1}-0.8\})^2\). The outcome mean was then \[m = \mathop{\mathrm{expit}}\{\eta_\text{soft}B -\eta_\text{hinge}H + \eta_{S_2}S_{T,2}-\eta_{S_1S_2}S_{T,1}S_{T,2}\}\,\text{,}\] with \(\eta_\text{soft}=0.9\), \(\eta_\text{hinge}=0.9\), \(\eta_{S_2}=0.3\), and \(\eta_{S_1S_2}=-0.2\). Finally, we let \(Y = \operatorname{clip}(m + \epsilon_Y,0,1)\), where \(\epsilon_Y\sim N(0,0.05^2)\). The term \(B\) acts as a softplus function that introduces a smooth penalty for elevated negative affect whereas \(H\) is a squared hinge term that imposes an additional penalty once negative affect exceeds a higher threshold. Thus, moderate increases in negative affect at time \(T\) reduce the outcome gradually, but more extreme affects are penalized more sharply. The terms involving \(S_{T,2}\) allow cigarette availability to directly influence the final outcome and modify the association between negative affect and the outcome.

11 Auxiliary Results↩︎

We present the following result, which is used to control the supremum of several terms throughout the theoretical results.

12 Proofs Regarding Regimen-Response Curve Estimation↩︎

12.1 Identification↩︎

12.2 Canonical Gradient↩︎

12.3 Theorem 1↩︎

We conclude this subsection by providing bounds for the non-leading terms in 17 . The bias term is controlled by 4, which extends the general UIPW result from [24]. We proceed by applying this result (4) to the specific bias terms that arise in the longitudinal setting. Finally, we establish bounds for the remainder terms using standard empirical process theory arguments.

12.4 Corollary 1↩︎

Before proving results as \(\alpha_n=o_p(n^{-1/2})\) we introduce some auxiliary results.

13 Proofs Regarding Policy Optimization↩︎

13.1 Supporting Results↩︎

Before proving 2 and 2, we establish some supporting lemmata regarding the derivatives of the regimen-response curve.

13.2 Theorem 2↩︎

13.3 Corollary 2↩︎

Before proving 2 we provide some auxiliary results analogous to 9. First, we require a stronger uniform version of 4.

14 Proofs Regarding Data-Adaptive Tilting↩︎

15 Potential for Efficiency Gains in Deterministic Versus Stochastic Regimes↩︎

Our proposed UIPW framework targets stochastic policy classes and uses tilting to stabilize longitudinal inverse-probability weights in high–decision-point settings. A recurring empirical pattern in our simulations is that, relative to oracle or correctly specified parametric IPW benchmarks, the incremental efficiency gains from undersmoothing can be modest. A key reason is structural: stochastic intervention targets are typically closer to the observed treatment mechanism, so the corresponding change-of-measure weights are less variable and often bounded, leaving less variance for augmentation to reduce.

To isolate this phenomenon in the simplest possible setting, we conducted a point-exposure simulation comparing a deterministic estimand, \(\mathbb{P}_0\{Y(1)\}\), to a continuum of incremental propensity score (IPS; [22]) stochastic estimands \(\mathbb{P}_0[Y\{Q(\delta)\}]\) indexed by an odds-multiplier \(\delta>1\). IPS interventions provide a natural bridge between stochastic and deterministic regimes: as \(\delta\) increases, the intervention shifts treatment probabilities toward \(1\) and approaches an always-treat intervention wherever \(\pi(x)>0\), so the IPS targets become increasingly near deterministic. This setup therefore directly probes how the potential efficiency gain changes as we move from stochastic to near-deterministic regimes.

We use the [32] design employed in [22]’s simulation study for IPS interventions: \(X=(X_1,\ldots,X_4)\sim N(0,I_4)\), \(\mathbb{P}_0(A=1\mid X) = \mathop{\mathrm{expit}}(-X_1+0.5X_2-0.25X_3-0.1X_4)\), \((Y\mid A,X)\sim N\{\mu(X,A),1\}\), and \(\mu( x,a)=200+a\{10+13.7(2x_1 + x_2 + x_3 + x_4)\}\). This design is known to induce variable propensities and highlight instability in deterministic weighting estimands. Of note, the original design includes an intercept of \(200\), which makes the IPW estimator incredibly variable; we additional perform simulations with an intercept of \(-10\) such that \(\mathbb{P}_0\{\mu(X,1)\}=0\).

We compare two estimands: \(\psi_0^\text{Det} \coloneq \mathbb{P}_0\{Y(1)\}\), and \(\psi_0^{\text{IPS}}(\delta)\coloneq \mathbb{P}_0[Y\{Q(\delta)\}]\), where the IPS intervention replaces \(\pi(X)\) with the odds-shifted treatment probability \(q_\delta(X)=\delta\pi(X)/\{\delta\pi(X)+1-\pi(X)\}\).

To focus on efficiency, we consider correctly specified parametric models for both the propensity score and outcome regression throughout our simulations. We consider the following estimators. For \(\psi_0^{\text{Det}}\) we consider (i) standard IPW using \(\pi_n\), and (ii) augmented IPW (AIPW) that additionally uses \(\mu_n\) to debias. For \(\psi_0^{\text{IPW}}(\delta)\) we consider: (i) IPS-IPW using a weighted mean representation: \(\mathbb{P}_n[(\delta A + 1-A)/\{\delta\pi_n(X)+1-\pi_n(X)\}Y]\), and (ii) an EIF-based estimator based on [22].

We performed 1000 Monte Carlo simulations for each \(\delta\) with \(\delta=2^1,2^{1.5},\ldots,2^{5}\). We summarized the Monte Carlo variance of each estimator and, for each estimand, reported the relative efficiency of the AIPW estimator to the IPW benchmark (10). Our results suggest that there is a larger potential for efficiency gains when targeting deterministic, rather than stochastic, regimes.

At multiple decision points, the same qualitative mechanism typically amplifies: deterministic regime evaluation produces products of inverse propensity terms over time, which can yield highly variable (and often heavy-tailed) weights even when each single-step propensity is reasonable. In contrast, stochastic regimes that remain close to the observed treatment mechanism tend to keep each step’s Radon–Nikodym factor nearer to \(1\) (and often bounded), so the longitudinal weight product is substantially more stable. As a result, baseline IPW estimators under stochastic regimes are already closer to their efficient limits, leaving less variance for EIF/AIPW-style corrections (or undersmoothing refinements) to remove.

References↩︎