1 Introduction↩︎

Goal-Conditioned Reinforcement Learning (GCRL) has emerged as an interesting paradigm for building generalist agents that learn policies capable of reliably reaching a wide range of goals from diverse initial states. Rather than relying on carefully shaped reward functions, GCRL formulates tasks directly in terms of desired outcomes, such as target positions or object configurations. This is appealing for autonomous agents and robotic systems, where specifying goals is often more natural than designing dense reward signals. Hence, most GCRL works rely on binary signals indicating success only upon achieving the goal.

None

While conceptually appealing, sparse rewards pose notable optimization challenges. Standard approaches typically mitigate sparsity via hindsight relabeling [1], [2], which convert each trajectory into multiple goal-labeled transitions by treating achieved states as alternative goals. However, methods that rely on Temporal Difference (TD) learning inherit the fundamental instability of bootstrapping. In long-horizon settings common in GCRL, error propagation can make TD learning slow, unstable, and hyperparameter-sensitive. Recent Monte Carlo (MC) works sought to avoid bootstrapping by exploiting the structure of goal-reaching problems. Prior approaches reformulate GCRL objectives using supervised learning [3]–[5] or contrastive learning [6], enabling more stable optimization.

In this work, we introduce a complementary probabilistic perspective that interprets GCRL through the lens of time‑to‑event modeling, also known as survival analysis [7], [8]. This statistical framework studies the distributions of variables representing the time until a specific event occurs. Our key observation is that in sparse, terminate‑on‑success settings, the expected discounted return is fully determined by the distribution of the goal-hitting time under the policy. Maximizing the return thus corresponds to optimizing a discounted functional of the hitting-time distribution. The discount factor \(\gamma\) induces an implicit risk sensitivity over arrival times. Unlike in standard RL, intermediate states in GCRL settings do not yield additional rewards; the value function is therefore a statistical summary of the arrival-time distribution and can be estimated directly, as illustrated in Fig. 1. Successful trajectories yield exact event times, whereas those truncated by a finite horizon become right‑censored observations. This correspondence naturally aligns policy evaluation in GCRL with the methodological foundations of survival analysis.

In this work, we derive a closed-form expression of the goal-conditioned value function as a discounted sum of survival probabilities associated with the goal-hitting time. We show that the value function is a statistical summary of an underlying temporal distribution. Consequently, rather than directly regressing a scalar value via Bellman backups, we explicitly learn the full distribution of goal-reaching times, naming our approach Survival Value Learning (SVL). We model the instantaneous probability of reaching the goal, namely the hazard function, via maximum likelihood estimation (MLE) over both event and censored trajectories, thereby inheriting standard MLE convergence properties.

SVL incorporates hindsight relabeling to convert each trajectory into multiple goal-labeled survival observations, providing dense supervision without altering the underlying objective. Unlike prior work using relabeling to train value functions [1], [4], [5], [9]–[12] or representations [6], [13], [14], we use it to generate event-time and censoring signals.

Our key contributions are: (i) we introduce SVL, a theoretical framework connecting GCRL with survival analysis. We propose a practical architecture to learn the hazard function and derive several value estimators trained via maximum likelihood, bypassing unstable TD bootstrapping. (ii) By integrating our SVL estimators with hierarchical actors, we introduce Hierarchical Survival Value Learning (HSVL), a practical offline GCRL algorithm. (iii) Through extensive evaluations and ablation studies on offline GCRL benchmarks, we show that survival‑based value estimation is competitive with strong hierarchical TD and MC baselines and delivers notable improvements on long‑horizon tasks, without additional hyperparameter tuning.

The remainder is organized as follows. Sec. 2 reviews related work, and Sec. 3 introduces background on goal‑conditioned reinforcement learning and survival analysis notations. Sec. 4 presents the core contribution, establishing the connection between GCRL and survival learning and introducing the corresponding value estimators inspired by survival analysis. Sec. 5 evaluates the proposed estimators on standard offline RL benchmarks, and Sec. 6 draws conclusions and future work.

2 Related work↩︎

2.1 Goal-conditioned reinforcement learning↩︎

The problem of learning policies for multiple goals was formalized by universal value function approximators [15], which extended value functions to accept goal arguments. To mitigate the reward sparsity of goal‑reaching tasks, Hindsight Experience Replay [1] proposed relabeling failed trajectories as successful ones for the states that were actually reached. Since then, numerous works have studied GCRL primarily through temporal-difference learning and goal-conditioned value estimation [1], [12], [16]–[19].

Beyond TD-based approaches, alternative formulations have been explored. Imitation and behavioral cloning methods leverage goal-conditioned supervision from trajectories [2], [3], [20]–[22]. Model-based approaches incorporate planning or learned dynamics to guide goal-reaching behaviour [23]–[27]. A complementary line of work combines learned value functions with graph search over replay-buffer [28], [29]. Representation learning methods shape exploration and goal achievement by learning goal-conditioned embeddings, successor-style representations, or by modeling goal-conditioned value functions through contrastive learning [6], [30]–[35] and [36] recast offline GCRL as matching a discounted state-visitation distribution. Some works combine multiple approaches; for example, offline GCRL methods often integrate TD learning with hindsight relabeling to address data scarcity [12], [18], while contrastive RL approaches couple representation learning with relabeling to learn goal‑conditioned representations without relying on a TD objective [6]. Supervised and imitation‑based formulations similarly use hindsight relabeling to construct dense training targets from sparse trajectories [3], [37].

Closest to our work is distance-weighted supervised learning (DWSL) [5], which also models the distribution of number of step between state pairs in a Monte-Carlo, non-TD fashion. SVL differs as it handles unreached goals via right censoring, rather than restricting supervision to state pairs from the same trajectory and recovers the goal-conditioned value function in closed form from the learned survival function, instead of relying on a soft-minimum distance surrogate that lower-bounds the value function.

2.2 Survival analysis and time-to-event learning↩︎

Survival analysis [7], also known as time-to-event analysis, is a classical area of statistics concerned with modeling the distribution of event times, typically through survival and hazard functions [38]–[42]. It has been widely applied in medicine and epidemiology (e.g, time to death), reliability engineering (time to failure), and the social sciences (event-history analysis) [43], [44]. In this statistical branch, foundational tools include nonparametric estimators [38], [41], semi-parametric regression models such as the Cox proportional hazards model [39], discrete-time or grouped-time formulations [42], [44], which we leverage in this work. These models come with a well-developed theory for censored likelihoods and asymptotic guarantees [41], [43], [45].

More recently, survival analysis has been extended using deep learning, including neural Cox models and direct hazard or survival function learning [46]–[49]. Our work builds on this literature by interpreting goal-reaching rollouts as survival data. Unlike prior alternative discounting schemes [50], formulated via policy-independent survival terminology [51], SVL uses the standard discounted GCRL objective to model the policy-dependent first-hitting-time distribution.

3 Preliminaries↩︎

This section provides an overview of GCRL and survival analysis, and introduces the main notations of the paper.

3.1 Goal-conditioned reinforcement learning↩︎

The GCRL problem is formulated as an augmented Markov Decision process (MDP), defined by the tuple \(\mathcal{M} = \left(\mathcal{S}, \mathcal{A}, \mathcal{G}, \mathcal{P}, r, p_g, \mu, \phi, \gamma\right)\). Here, \(\mathcal{S}\), \(\mathcal{A}\), and \(\mathcal{G}\) denote the state, action, and goal spaces, respectively. Let \(\Delta(\mathcal{X})\) represent the set of probability distributions over a set \(\mathcal{X}\). \(\mathcal{P}: \mathcal{S}\times \mathcal{A} \to \Delta(\mathcal{S})\) represents the transition dynamic and \(r: \mathcal{S} \times \mathcal{A}\times\mathcal{G} \to \mathbb{R}\) is the goal-conditioned reward function. Desired goals are sampled from the distribution \(p_g \in \Delta(\mathcal{G})\) with initial states drawn from \(\mu : \mathcal{G} \to \Delta(\mathcal{S})\). Finally, \(\gamma \in [0, 1)\) is the discount factor and \(\phi: \mathcal{S} \to \mathcal{G}\) is a tractable mapping that projects the state into goal space. This mapping accounts for settings where goals represent only partial observations or specific features of the state. For example, in robotic manipulation, \(\mathcal{S}\) might represent joint angles while \(\mathcal{G}\) represents end-effector coordinates. In this case, \(\phi\) corresponds to the forward kinematics function.

Following prior work [1], [52], we adopt a sparse reward formulation in which \(r(s_t, a_t,g) = - \mathbf{1}_{\{\|\phi(s_{t+1}) - g\|\ge \epsilon\}}\). That is, the agent receives a penalty of \(-1\) whenever the next state \(s_{t+1}\) fails to satisfy the goal \(g\) (within tolerance \(\epsilon\)). The objective of GCRL is to learn a goal-conditioned policy, denoted by \(\pi: \mathcal{S}\times \mathcal{G} \to \Delta(\mathcal{A})\), that maximizes the expected discounted cumulative return: \[\label{eq:gcrl95objective} J(\pi) = \mathbb{E}_{g\sim p_g,\;\tau\sim \rho^\pi(\cdot|g)} \Bigl[\sum_{t=0}^{\infty}\gamma^t r(s_t,a_t,g)\Bigr],\tag{1}\] where \(\rho^\pi(\tau|g) = \mu(s_0 \!\mid\! g)\prod_{t=0}^\infty \pi(a_t \!\mid\! s_t, g) \mathcal{P}(s_{t+1} \!\mid\! s_t,a_t)\) is the probability of sampling a trajectory \(\tau = (s_0,a_0,s_1,a_1, \dots)\). Intuitively, this objective corresponds to sampling a goal \(g\) and then optimizing the policy to reach that goal as quickly as possible [6]. Accordingly, for a given policy \(\pi\), we denote the goal-conditioned value-function as follows: \[\label{Q95function} V^\pi(s,g) = \mathbb{E}_{\tau \sim \rho^\pi(\cdot |g)} \left[\sum_{t=0}^\infty\gamma^t r(s_t,a_t,g) |\substack{ s_0=s } \right].\tag{2}\]

3.2 Survival analysis notations in the context of GCRL↩︎

First hitting time. In analogy with survival analysis [38], [46], [47], given a state \(s\), a goal \(g\), and a goal-conditioned policy \(\pi\), we define the first hitting time to reach the goal as: \[\small T^\pi(s,g) \!=\! \inf\Bigl\{\, t\ge 0 : \|\phi(s_{t + 1}) - g\|< \epsilon\; \!\Bigm|\; \! \substack{s_0=s,\; a_{t} \sim \pi(\cdot|s_{t},g)\\ s_{t+1}\sim \mathcal{P}(\cdot|s_t,a_t)\\ } \Bigr\}.\] We adopt the convention \(T^\pi(s,g)=\infty\) if the goal \(g\) is never reached from the initial state \(s\). Thus, \(T^\pi(s,g)\) is the (possibly infinite) time of first arrival to \(g\) when starting from \(s\) and following \(\pi\) under dynamics \(\mathcal{P}\). Unlike standard survival analysis, where event times typically start at \(1\), we allow \(t=0\) to account for instant hits, in which the goal is satisfied immediately after the initial state \(s_0\).

Formally, for a fixed goal and policy \((g,\pi)\), the system evolves as a Markov chain on states \((s_t)_{t\ge 0}\), defined by the transition kernel: \[\label{eq:state95kernel} \mathcal{P}^{\pi,g}(s'|s) := \int_{\mathcal{A}} \pi(a|s,g)\,\mathcal{P}(s'|s,a)\,da .\tag{3}\] The goal hitting time \(T^\pi(s,g)\) is a stopping time with respect to the natural filtration of the process \((s_t)_{t\ge 0}\).

Survival and hazard functions. As in standard discrete time-to-event modeling [42] and survival analysis more broadly [41], we study \(T^\pi(s,g)\) through its associated survival and hazard functions, which provide an equivalent characterization of its distribution. The survival function is defined by \[S^\pi(t|s,g) ~=~ \Pr \bigl(T^\pi(s,g)>t\bigr),\] and corresponds to the probability that the goal has not been reached by time \(t\). The discrete-time hazard function is \[\label{hazard95definition} h^\pi(t|s,g) = \Pr \Bigl(T^\pi(s,g)=t \,\Bigm|\, T^\pi(s,g)\ge t\Bigr).\tag{4}\] It measures the likelihood of reaching the goal at time \(t\) given that it has not yet been reached. Note that the hazards fully determine the law of \(T^\pi(s,g)\) as \[S^\pi(t|s,g) ~=~ \prod_{k=0}^{t}\bigl(1-h^\pi(k|s,g)\bigr), \label{recursion}\tag{5}\] this recursion is detailed in Appendix 7.1. This yields the event-time probabilities \[\Pr\!\bigl(T^\pi(s,g)=t\bigr) ~=~ h^\pi(t|s,g) S^\pi(t-1|s,g). \label{eq:hitting95time95law}\tag{6}\] Episode termination. In practice, rollouts are truncated after a finite horizon \(c\) (e.g., due to timeout, or episode termination). If the goal is not reached by time \(c\), we only observe that \(T^\pi(s,g)>c\), providing no information about subsequent steps. This corresponds to right censoring at time \(c\)  [39], [53]–[57]. Mathematically, a censored observation contributes to the likelihood via the survival function \(S^\pi(c|s,g)\), whereas an uncensored observation (goal reached at \(t\le c\)) contributes via the event probability \(\Pr(T^\pi(s,g)=t)\).

4 Goal-Conditioned RL as Survival Learning↩︎

This section draws the connection between GCRL and survival learning, and introduces our Survival Value Learning (SVL) approach. We first establish in Proposition 1 a link between the goal-conditioned value function and the survival function of the goal-hitting time. Then, by exploiting the survival-hazard recursion together with the event-time probabilities in Eq. 6 , we derive a log-likelihood objective for learning a goal-conditioned hazard function. We address practical considerations and present tractable hazard-function parameterizations, along with plug-in estimators that recover goal-conditioned values from the learned hazards. Finally, we derive an offline GCRL algorithm to empirically evaluate the proposed framework.

4.1 Relating goal-conditioned value functions to survival functions↩︎

We formalize the link between GCRL and survival analysis. Under the standard sparse-reward formulation, the goal-conditioned value function admits a closed-form expression in terms of the survival function of the goal-hitting time.

Proof. We refer readers to Appendix 7.2.
Prop. 1 is exact when the return is fully determined by the goal-hitting time \(T^\pi(s,g)\). For arbitrary dense rewards that depend on the full trajectory, the return is not identifiable from the distribution of \(T^\pi(s,g)\) alone.

This identity implies that estimating the survival function (equivalently, the hazard function) yields a direct approximation of the value function. It offers an intuitive interpretation of the objective: maximizing \(V^\pi(s,g)\) amounts to minimizing the discounted cumulative probability of not having reached the goal over time. In other words, this incentivizes the agent to minimize the task’s survival time, with the discount factor giving greater weight to earlier successes.

4.2 Learning the hazard function↩︎

We derive a population objective for learning hazards, deferring modeling and approximation choices to Section 4.3.

4.2.0.1 Observation.

Lets consider a sample \((g,s)\sim p_g \times \mu\) and roll out \(\pi(\cdot| s_t,g)\) under dynamics \(\mathcal{P}\) for a horizon \(c\). Let \(\delta\in\{0,1\}\) indicate whether the goal is reached within \(c\) steps, and let \(\tau\) denote the (first) hitting time when \(\delta=1\). Thus, \(\delta=1\) implies \(\tau\le c\) (uncensored), while \(\delta=0\) corresponds to right censoring at \(c\). The likelihood of an observation \((\tau,c,\delta)\) is then given by \[\label{likelihood95censor} \Pr(\tau,c,\delta| s,g) = \Pr\bigl(T^\pi(s,g)=\tau\bigr)^{\delta}\; S^\pi(c| s,g)^{1-\delta}\tag{7}\]

4.2.0.2 Parametric hazard and negative log-likelihood.

Let \(h_{\theta}^\pi(t| s,g)\) be a parametric hazard model parameterized by \(\theta\). Using the standard discrete-time identities provided in Eq. 5 and Eq. 6 , the maximum likelihood estimation (MLE) amounts to minimizing the population negative log-likelihood \(\mathcal{L}(\theta) = -\mathbb{E}[\log \Pr_\theta(\tau, c, \delta,s, g )]\), expanding to \[\label{nll95hazard95objective} \begin{align} \mathcal{L}(\theta) = -\mathbb{E}_{(s,g,\tau,c,\delta)}\Bigl[\delta\ell_\tau(\theta) & + (1-\delta)\ell_c(\theta) \Bigr] \end{align}\tag{8}\] with \[\begin{align} &\ell_\tau(\theta) = \log h_{\theta}^\pi(\tau| s,g) + \sum_{k=0}^{\tau-1}\log\left(1-h_{\theta}^\pi(k| s,g)\right), \\[-0.2em] &\ell_c(\theta) = \sum_{k=0}^{c}\log\left(1-h_{\theta}^\pi(k| s,g)\right). \end{align}\] It is worth noting that the expectation is taken with respect to the tuple \((\tau, c, \delta, s, g)\), using the identity \(\Pr_\theta(\tau, c, \delta, s, g)=p_g(g)\,\mu(s | g)\,\Pr_\theta(\tau, c, \delta | s, g)\).

4.2.0.3 Censoring and empirical risk minimization.

In GCRL, censoring is induced by the episode horizon \(c\). Since the horizon is determined by the environment configuration (or sampling strategy) and is independent of \(T^\pi(s,g)\), it satisfies the condition of non-informative right censoring; informative terminations such as catastrophic failures fall outside this assumption and instead call for a competing-risks formulation [41], which we leave to future work. As in standard RL, we optimize an empirical objective by sampling tuples \((s_i,g_i,\tau_i,c_i,\delta_i)\) either from an offline dataset or a replay buffer, which we treat as approximately i.i.d.

Lemma 1 implies that, as the number of survival tuples grows, the estimated hazard (and thus the estimated value) concentrates around its true value, with statistical fluctuations that scale as \(\mathcal{O}(n^{-1/2})\) under standard conditions. In Section 5, we highlight this sample scaling property on the long-horizon task.

For standard references regarding Lemma 1 in the presence of right censoring, we refer readers to [41] (Chapter 6),[42] (Chapter 3), together with MLE theory [58]–[60].

4.2.0.4 Hindsight relabeling.

Directly minimizing the empirical risk in Eq. ?? using only desired goals \(g\sim p_g\) can be data-inefficient, since most rollouts may not reach the specified goal and thus provide weak supervision. Following prior work on goal relabeling [1], [12], [13], we incorporate hindsight goal relabeling by treating visited states as additional achieved goals. This transforms each trajectory into many goal-labeled survival tuples \((s_i,g_i,\tau_i,c_i,\delta_i)\), increasing the amount of event-time and right-censoring supervision available for maximum-likelihood hazard learning.

4.2.0.5 Value estimation.

Given \(\hat{\theta}_n\), Proposition 1 yields the plug-in estimator \(V_{\hat{\theta}_n}^\pi(s,g) = -\sum_{t\ge 0}\gamma^t\, S_{\hat{\theta}_n}^\pi(t| s,g)\).

Our SVL approach provides an alternative estimator of the goal-conditioned value function that does not rely on bootstrapping. Instead, it follows from maximum-likelihood estimation of the hazard model and inherits the classical guarantees of MLE under standard regularity conditions (e.g., consistency and asymptotic normality).

4.3 Practical value estimation and hazard function architecture↩︎

In this section, we address two practical challenges to approximate the plug-in estimator \(V_{\hat{\theta}_n}^\pi\) defined in Proposition 1: how to model the hazard function and how to handle the infinite discounted sum.

4.3.0.1 Finite-horizon MDP.

In the finite-horizon setting of length \(H\), the identity in Eq. ?? is a finite sum, yielding: \[V_{\hat{\theta}_n}^{\pi,H}(s,g)= -\sum_{t=0}^{H-1}\gamma^t\,S^\pi_{\hat{\theta}_n}(t| s,g)\] The straightforward approach is to model the hazard function with a neural network predicting an \(H\)-dimensional vector \(\{h^\pi_{\theta}(t| s,g)\}_{t=0}^{H-1}\). Note that \(\hat{\theta}_n\) is an optimal estimator, while \(\theta\) are learned parameters. Each entry is the conditional probability of reaching the goal at time \(t\), with \(t=0\dots H-1\), given that it was not reached earlier. In that case, the training objective Eq. 8 remains unchanged.

Yet, this approach does not extend directly to the infinite-horizon settings, as it ignores any value accumulated beyond the horizon \(H\). Furthermore, it requires fixing \(H\) a priori, and optimization becomes increasingly difficult as the horizon lengthens. To address these limitations while targeting infinite-horizon settings, we propose reducing the output dimensionality by aggregating time steps into intervals.

Geometric-time binning. To reduce complexity while preserving resolution at early time steps, we adopt grouped-time modeling from discrete-time survival analysis [42]. Let \(K\) be the number of bins, \(0=b_0<b_1<\cdots<b_K=H\) be bin edges with intervals \([b_k,b_{k+1})\) and lengths \(L_k=b_{k+1}-b_k\). We use geometric edges \(b_k=\lfloor \rho^k\rfloor , \;\rho=H^{1/K}\) which allocates many short bins early (where \(\gamma^t\) is large) and fewer long bins late (where the discounted tail contributes less). The hazard network is now designed to output \(K+1 < H\) logits.

Based on geometric time binning, we consider two binned infinite-horizon variants. Both model the prefix on \([0,H]\) with \(K\) bins and approximate the discounted tail \(\sum_{t=H}^{\infty}\gamma^t S(t)\) using the final interval \([H,\infty)\).

Piecewise-constant per-step hazard (PCH). We assume the per-step hazard is constant within each bin: \[h^\pi_{\theta}(t| s,g)=\bar h^\pi_{\theta}(k| s,g), \qquad t\in[b_k,b_{k+1}).\] Piecewise-constant survival per bin (PCS). We assume the survival function is constant within each bin: \[S^\pi_{\theta}(t| s,g)=S^\pi_{\theta}(b_k| s,g), \qquad t\in[b_k,b_{k+1}).\] Per variant, we derive a grouped-time log-likelihood objective involving only the \(K\) bin parameters and the corresponding plug-in estimator of the goal-conditioned value function. Their closed-form expressions are detailed in Appendix 7.5.

Hazard function architecture. To efficiently parameterize temporal hazard, we propose a conditional low-rank basis selection that extends standard discrete-time survival frameworks [61], [62] and mixture-of-experts formulations [49], [63]. While traditional models often predict independent logits or rely on fixed parametric distributions [64] or static polynomial bases [65], our architecture, illustrated in Figure 5, dynamically assembles a temporal structure by performing a double contraction between state-goal dependent weights and a learned basis library \(\boldsymbol{\Psi} \in \mathbb{R}^{S \times H \times K}\). Specifically, given a state-goal encoding \(z\), the model produces selection weights \(\mathbf{w}(z) \in \mathbb{R}^S\) and mixing coefficients \(\mathbf{c}(z) \in \mathbb{R}^K\), which are combined with the library to obtain hazard logits via the conditioned low-rank factorization \(\boldsymbol{\ell}_{1:H}(z) = \mathbf{w}(z)^\top \boldsymbol{\Psi} \mathbf{c}(z)\). This approach allows the network to represent diverse temporal structures. We provide a more detailed explanation of the proposed architecture in Appendix 7.4. Note that the hazard representation with a deep neural network remains an active line of research in the survival analysis community [49], [63], [65].

4.4 Hierarchical policy for offline GCRL↩︎

So far, we have derived a survival-based objective for learning goal-conditioned value functions (SVL). While these estimators theoretically capture the task’s underlying temporal structure, their practical utility is best demonstrated by their ability to guide an agent toward goals. To evaluate the quality of our representations in a decision-making context, we instantiate them within a learning algorithm. We propose Hierarchical Survival Value Learning (HSVL), an offline GCRL algorithm that extracts a hierarchical policy from our SVL value estimator. We adopt a two-level architecture, following the HIQL approach [12], in which a high-level policy selects subgoals to maximize the value, and a low-level policy executes actions to achieve subgoals.

Policy extraction. Given the learned survival value function \(V_\theta\), we extract a hierarchical policy using advantage-weighted regression (AWR) [66], though the survival critic is also compatible with other offline RL policy-extraction schemes that exploit high-value actions [67]. We learn a high-level subgoal policy \(\pi_{\theta^h}(s_{t+k}| s_t,g)\) that proposes intermediate goals every \(k\) steps, and a low-level action policy \(\pi_{\theta^\ell}(a_t| s_t,s_{t+k})\) that executes actions to reach these subgoals. The parameters \(\theta^h\) and \(\theta^\ell\) are optimized by maximizing: \[\begin{align} \tag{9} J_h(\theta^h) &= \mathbb{E}_{(s_t,s_{t+k},g)}\!\left[ w^h_t\, \log \pi_{\theta^h}(s_{t+k}| s_t,g) \right], \\ \tag{10} J_\ell(\theta^\ell) &= \mathbb{E}_{(s_t,a_t,s_{t+1},s_{t+k})}\!\left[ w^\ell_t\; \log \pi_{\theta^\ell}(a_t| s_t,s_{t+k}) \right] \end{align}\] The weights \(w\) represent the advantage of the transition, computed using value differences: \[\begin{align} \label{eq:awr95weights} w^h_t &=\exp\!\left(\beta \bigl(V_\theta(s_{t+k},g)- V_\theta(s_t,g)\bigr) \right), \\ w^\ell_t &=\exp\!\bigl(\beta\,V_\theta(s_{t+1},s_{t+k})- V_\theta(s_t,s_{t+k})\bigr), \end{align}\tag{11}\] where \(\beta>0\) is an inverse-temperature parameter and with \(V_\theta\) computed via the survival identity ?? . The complete procedure is summarized in Algorithm 3.

Design choices. We choose this specific architecture for three key reasons. First, it isolates the contribution of our survival-based estimator. Because policy extraction depends only on the plug-in value \(V_\theta\), any performance improvements can be directly attributed to the quality of the value estimation rather than additional algorithmic choices (e.g., target networks, or delicate actor-critic update schedules).

Second, a value-only interface is particularly robust for offline learning. Unlike \(Q\)-learning, which suffers from bootstrapping errors on out-of-distribution actions, our approach relies on value differences computed over in-distribution dataset transitions, providing a stable signal for advantage-weighted regression [12].

Third, this design leverages hindsight relabeling as a standalone proxy for optimality. Prior work indicates that relabeling can drive goal-reaching behavior without explicit Bellman updates [3], [6]. Accordingly, our method focuses on learning the survival value of the dataset trajectories, avoiding the risky action-maximization steps of Q-learning. Empirically, this formulation outperforms complex baselines, providing evidence that hindsight relabeling implicitly produces a value function superior to the behavior policy. This verifies that safe policy extraction is possible through hazard-based evaluation, without the extrapolation risks of typical offline RL.

5 Experiments↩︎

We evaluate our HSVL on the challenging OGBench benchmark [68], which includes high-dimensional state-based and visual offline goal-conditioned RL tasks. Our experiments aim to answer four key questions: (i) Can SVL scale to long-horizon, sparse-reward tasks where standard TD learning typically fails? (ii) Does SVL generalize to high-dimensional visual observations? (iii) How much of HSVL’s performance is attributable to the survival critic itself, as opposed to the hierarchical policy extraction? (iv) How sensitive is the performance to architectural choices (e.g., network depth, hazard function parameterization)?

We compare HSVL against six offline GCRL baselines: goal-conditioned behavioral cloning (GCBC) [3], [20], goal-conditioned implicit V, Q-learning (GCIVL, GCIQL) [12], [69], quasimetric RL (QRL) [70], contrastive RL (CRL) [6], and hierarchical implicit Q-learning (HIQL) [12]. HIQL is the closest algorithmic baseline to HSVL as it shares the same hierarchical value-to-policy extraction template (Algorithm 3). It differs only in how the value function is learned, via TD-style bootstrapping rather than survival value learning. The HIQL vs.HSVL comparison directly isolates the contribution of the survival formulation, holding the hierarchical actor fixed. CRL is the closest non-TD baseline that avoids bootstrapping. To isolate the survival critic’s contribution relative to this strong MC baseline, we introduce a flat variant of SVL (no hierarchy) and re-evaluate CRL with the same flat DDPG+BC actor, so that the two methods differ only in their critics (Sec. 5.2). We report the normalized success rate averaged over 4 seeds. SVL implementation will be released upon paper acceptance, together with the derived HSVL adapted from HIQL.

5.1 Main results↩︎

Tab. 1 summarizes the performance across state-based and visual domains; full results are provided in Tabs. 5, 6, and 7 of the Appendix. HSVL matches or exceeds the performance of prior methods on standard tasks with substantial gains on complex, long-horizon problems. Note that we fix the inverse temperature parameter, \(\beta\), to 3 across all experiments. Likewise, we kept OGBench default values for all hyperparameters not specific to SVL, as reported in Tab. 2.

Scaling to long horizons. We show substantial improvements on tasks requiring extended temporal reasoning. On antmaze-giant-navigate, HSVL outperforms the strongest baseline (HIQL) by a clear margin (\(74\%\) vs. \(65\%\)). This gap widens for the complex humanoidmaze-giant navigation task: HSVL achieves \(\mathbf{81}\%\) success rate, whereas HIQL reaches only \(12\%\), and non-hierarchical methods fail (\(0\)–\(3\%\)). Since HSVL and HIQL share the same hierarchical extraction template and differ only in how the value is learned, we read the widening gap on longer horizons as controlled evidence that survival-based MLE mitigates the compounding bootstrap errors of TD over long horizons, consistent with recent findings that 1-step TD errors grow sharply with horizon [71].

Visual generalization. We further evaluate HSVL on pixel-based benchmarks to test its ability to learn from high-dimensional observations. On visual-antmaze-giant, our method achieves \(\mathbf{61}\%\) success, surpassing the best hierarchical baseline (HIQL, \(6\%\)) and contrastive baseline (CRL, \(47\%\)). Furthermore, on the challenging visual-humanoidmaze-medium, HSVL is the only method to extract meaningful signal (\(\mathbf{22}\%\) and \(\mathbf{15}\%\) resp for navigate and stitch), while all baselines fail (\(0\)–\(1\%\)). This suggests that the hazard-based objective provides a robust learning signal even when combined with convolutional encoders, without requiring the explicit representation-learning losses used by prior visual RL methods.

Why does HSVL perform better on humanoidmaze-giant than on humanoidmaze-large? Our HSVL achieves a higher success rate on humanoidmaze-giant-navigate (\(81\%\)) than on the smaller variant, humanoidmaze-large-navigate (\(66\%\)). These results look counterintuitive, as the giant maze requires up to 3000 steps to reach the goal. We conjecture that this gap is driven primarily by dataset size rather than by the challenge of long-horizon planning. Although both datasets contain 1000 episodes, the giant variant includes \(4\)M transitions, whereas the large one includes \(2\)M transitions, yielding substantially more survival tuples after relabeling and thus more supervision for hazard learning. This interpretation is consistent with the MLE property of Lemma 1: increasing the number of samples reduces value estimation error at the canonical \(\mathcal{O}(n^{-1/2})\) statistical rate.

5.2 Isolating the survival critic↩︎

To address question (iii), we evaluate the contribution of the survival critic without hierarchical policy extraction. The flat variant of SVL introduced above pairs the survival critic with a standard non-hierarchical DDPG+BC actor. This requires learning a state-action value \(Q^\pi(s,a,g)\) rather than only \(V^\pi(s,g)\). The corresponding derivation follows the same survival identity (Prop. 1), see Appendix 7.3.

Comparison protocol. We aim at a minimal and controlled comparison of the critics. Therefore, we re-evaluate CRL with the same DDPG+BC actor as SVL (depth 6), so that the methods differ only in their critics rather than in actor implementations from different codebases. We use CRL as the most directly comparable baseline. Like SVL, it is a non-TD, Monte-Carlo-style method, so contrasting the two critics under a shared actor isolates what the distributional survival formulation adds over scalar contrastive regression.

4pt

max width=

Results. Table [tab:flat-ablation] reports success rates across six navigate-v0 tasks. The flat SVL critic outperforms the re-evaluated CRL on the harder settings, most visibly on humanoidmaze-large (\(66\%\) vs.\(42\%\)) and humanoidmaze-giant (\(45\%\) vs.\(7\%\)), while matching it on the easier ones. This indicates that the distributional, survival-based critic alone provides a stronger learning signal than contrastive regression in an identical flat setup. Adding hierarchical extraction (HSVL) compounds this gain on the giant mazes, while not necessary for medium and large sizes.

5.3 Architectural ablation studies↩︎

We address question (iv) by examining the sensitivity of HSVL to architectural design choices.

Network depth. Fig. 4 reports performance as we vary the MLP depth of both actor and hazard networks from 3 to 12 layers. On AntMaze, performance is remarkably stable across all configurations, indicating that HSVL is robust to network depth in this regime. On the higher-dimensional HumanoidMaze, we observe a benefit from increased capacity, with success rates peaking at depth 6. Overall, the results suggest that HSVL does not require excessive tuning regarding network architecture to achieve strong performance.

Hazard function architecture. We further investigate the impact of the discretization scheme used to model the time-to-goal. We compare the three estimators derived in Section 4.3: the finite-horizon model, the Piecewise-Constant Hazard (PCH), and the Piecewise-Constant Survival (PCS). As shown in Fig. 2, all three variants yield comparable success rates on OGBench [68] antmaze and humanoidmaze tasks. This robustness implies that the performance gains of HSVL stem from the survival formulation itself rather than a specific approximation method. Consequently, we adopt the PCS model for our main results due to its simplicity of implementation and efficiency.

6 Discussion and Conclusion↩︎

We introduced SVL, a probabilistic framework that frames GCRL as survival analysis. By modeling time-to-goal, we derived an identity that expresses the goal-conditioned value function as a discounted sum of survival probabilities. This enables direct value estimation via MLE for both success events and right-censored trajectories, thereby avoiding the instability and sample inefficiency often associated with TD bootstrapping. By combining SVL with hierarchical actors, we proposed HSVL for offline GCRL.

Our empirical evaluation highlights several findings. First, HSVL mostly outperforms the strongest hierarchical TD-based baseline on complex, long-horizon tasks, supporting our central hypothesis that by modeling the full time-to-goal distribution, SVL mitigates the compounding error inherent to Bellman backups. Second, the observed performance improvements with larger dataset sizes, even in the most challenging environments, suggest that the proposed framework scales favorably in offline settings. Finally, ablation studies indicate that HSVL is robust to architectural choices.

We view this work as a foundational step toward reframing sparse-reward GCRL. Our primary objective was to validate the core potential of the survival perspective; as such, we instantiated SVL within a standard offline architecture (HSVL) to isolate the benefits of the estimator itself. Specific design choices, such as the network architecture and hierarchy, were not optimized and remain open to refinement. A central advantage of the proposed framework is that it learns the entire distribution of time-to-goal, rather than only its expectation. In the present work, the policy compresses this rich distributional signal into a scalar value estimate, leaving significant information unexploited. Leveraging this distribution more directly, for exploration, risk-sensitive control or uncertainty-aware planning, is a promising direction for future work.

In this regard, the next step is to formally connect survival-based value estimation with distributional RL [72]. Unlike traditional distributional RL methods, which rely on temporal-difference equations, SVL exploits the structure of the goal-conditioned setting to avoid bootstrapping altogether, positioning it as a structured distributional Monte Carlo alternative for sparse GCRL. While this paper focuses on offline policy evaluation, extending survival-based learning objectives to online settings is another important direction. In such regimes, learned hazard functions could inform exploration strategies by reasoning about uncertainty in time-to-goal predictions, enabling more principled exploration. Likewise, the survival identity is exact for returns determined by the goal-hitting time. Extending SVL to arbitrary dense rewards via a sparse/dense critic decomposition is a natural next step. A complementary direction, particularly relevant to safety-critical robotics, is to extend SVL to a competing-risks model that distinguishes goal achievement from informative failure terminations, rather than treating both as ordinary right censoring.

Overall, treating goal-reaching as a survival event provides a statistically viable alternative to conventional GCRL methods. By replacing Bellman bootstrapping with likelihood-based estimation, SVL offers a robust, scalable path toward long-horizon decision-making.

Impact Statement↩︎

This paper presents work whose goal is to advance the field of reinforcement learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

Acknowledgments↩︎

This work has received support from the French government, managed by the National Research Agency, under the France 2030 program with the references Organic Robotics Program (PEPR O2R) and “PR[AI]RIE-PSAI” (ANR-23-IACL-0008). This research was funded, in part, by l’Agence Nationale de la Recherche (ANR), projects NIMBLE project (ANR-22-CE33-0008) and PEPR O2R - AS2 (ANR-22-EXOD-0006). The European Union also supported this work through the ARTIFACT project (GA no. 101165695) and the AGIMUS project (GA no. 101070165). The Paris Île-de-France Région also supported this work in the frame of the DIM AI4IDF. The authors gratefully acknowledge the support and resources provided by the CLEPS infrastructure at Inria Paris. This work was performed using HPC resources from the GENCI-IDRIS Jean-Zay cluster (Grant 2024-AD010616763). Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the funding agencies.

7 Appendix↩︎

7.1 Link between survival and hazard functions↩︎

The recursive formula expressing \(S\) using \(h\), Eq. 5 , is proved below for completeness. As \(T^\pi(s,g)\) is a discrete random variable: \[\begin{align} S^\pi(t\mid s,g) &= \Pr(T^\pi(s,g)>t)\\ &=\Pr(T^\pi(s,g)\ge t + 1)\\ &=\Pr(T^\pi(s,g) = t + 1) \\ & \qquad + \Pr(T^\pi(s,g)> t + 1) \notag\\ &=h^\pi(t+1\mid s,g) \Pr(T^\pi(s,g) \ge t + 1) \notag\\ & \qquad + \Pr(T^\pi(s,g)> t + 1) \label{set95identity}\\ &=h^\pi(t+1\mid s,g) S^\pi(t\mid s,g)\notag\\ & \qquad + S^\pi(t + 1\mid s,g) \end{align}\tag{12}\]

Equality 12 is the Bayesian rule applied to \(\{T^\pi(s,g) = t + 1\}=\{T^\pi(s,g) = t + 1\} \cap \{T^\pi(s,g) \ge t + 1\}\).

7.2 Proof of Proposition 1↩︎

Consider the GCRL setting where the agent receives a per-step penalty until success, \(r(s_t, a_t,g) = - \mathbf{1}_{\{\|\phi(s_{t+1}) - g\|\ge \epsilon\}}\), and the episode terminates upon goal achievement. \[\begin{align} V^\pi(s,g) &= \mathbb{E}_{\tau \sim \rho^\pi(\cdot\mid g)} \left[\sum_{t=0}^\infty\gamma^t r(s_t,a_t,g) \mid s_0 = s\right] \\ & = - \mathbb{E}_{\tau \sim \rho^\pi(\cdot\mid g)} \left[\sum_{t=0}^\infty\gamma^t \mathbf{1}_{\{\|\phi(s_{t+1}) - g\|\ge \epsilon\}} \mid s_0=s\right] \\ & = - \mathbb{E}_{T^\pi(s,g)} \left[\sum_{t=0}^{T^\pi(s,g)-1}\gamma^t \right] \label{key95eq}\\ & = - \mathbb{E}_{T^\pi(s,g)} \left[\sum_{t=0}^{\infty}\gamma^t \mathbf{1}_{\{T^\pi(s,g) > t\}} \right] \\ & = - \sum_{t=0}^{\infty}\gamma^t \mathbb{E}_{T^\pi(s,g)} \left[\mathbf{1}_{\{T^\pi(s,g) > t\}} \right] \\ & = - \sum_{t=0}^{\infty}\gamma^t \Pr(T^\pi(s,g) > t) \end{align}\tag{13}\] Equality 13 comes from the definition of \(T^\pi(s,g)\).

7.3 State-action survival identity (Q-function)↩︎

We extend the survival identity of Proposition 1 to the state-action value function. Define the first hitting time conditioned on an initial action \(a_0 = a\) as \[\small T^\pi(s,a,g) \!=\! \inf\Bigl\{\, t\ge 0 : \|\phi(s_{t+1}) - g\| < \epsilon \!\Bigm|\; \substack{s_0=s,\; a_0=a,\\ a_{t} \sim \pi(\cdot|s_{t},g)\\ s_{t+1}\sim \mathcal{P}(\cdot|s_t,a_t)} \Bigr\}.\] The associated survival function is \[S^\pi(t|s,a,g) ~=~ \Pr\bigl(T^\pi(s,a,g)>t\bigr).\]

Under the same sparse-reward setting as Proposition 1, the goal-conditioned state-action value function satisfies \[Q^\pi(s,a,g) ~=~ -\sum_{t=0}^{\infty} \gamma^t\, S^\pi(t \mid s,a,g).\]

Proof. The proof follows identically to that of Proposition 1 (Appendix 7.2), replacing \(T^\pi(s,g)\) with \(T^\pi(s,a,g)\) throughout. The only difference is that the first action \(a_0=a\) is fixed rather than drawn from \(\pi(\cdot|s_0,g)\), which does not affect the derivation.

7.4 Explanation hazard architecture↩︎

There is a growing interest in the survival analysis community regarding the optimal deep neural architecture for modeling hazard functions. The emerging consensus suggests that independently predicting logits for each time bin is suboptimal, as it fails to exploit temporal correlations. Consequently, the primary design objective of modern approaches is to encode temporal information while maintaining computational efficiency [63], [65]. For instance, the Dual Mixture-of-Experts framework [63] extracts features and combines them with learnable time embeddings through multiple heads. However, this approach may not scale, as maintaining separate heads for each time bin significantly increases the computational cost. Conversely, frameworks using static polynomial bases (e.g., Bernstein polynomials) [65] offer a more computationally efficient alternative but impose a global, rigid inductive bias, assuming the underlying temporal structure is invariant across different inputs.

Our proposed architecture bridges these two paradigms by using Conditioned Low-Rank Factorization. To help understand our design, we first consider a simplified version with a single learnable basis set. For a given state-goal pair \((s, g)\), we let \(z = f_{\theta}(s, g)\) be the latent encoding. The hazard logit at time bin \(t\) is expressed as: \[\ell(t, s, g) = \mathbf{c}(z)^\top \boldsymbol{\psi}(t) + b_t ,\] where \(\mathbf{c}(z) \in \mathbb{R}^K\) is a vector of predicted coefficients and \(\boldsymbol{\psi}(t) \in \mathbb{R}^K\) is a learned temporal embedding (a basis function) for bin \(t\). In that way, as in  [65], the hazard function (its logits) is a linear combination of a feature and a time embedding basis, but the time embedding is learnable as in the Dual Mixture-of-Experts.

However, a single basis set \((\boldsymbol{\psi})\) forces all \((s, g)\) pairs to share the same underlying temporal subspace, limiting the model’s ability to represent fundamentally different survival behaviors (e.g., immediate success vs. long-term survival). To resolve this, we extend the formulation to include a Basis Library \(\boldsymbol{\Psi} \in \mathbb{R}^{S \times H \times K}\) containing \(S\) distinct basis sets. The model dynamically selects and mixes these sets using selection weights \(w(z) \in \mathbb{R}^S\): \[\ell(t, s, g) = \sum_{j=1}^{S} w_j(z) \left( \mathbf{c}(z)^\top \boldsymbol{\psi}_j(t) \right) + b_t.\]

By implementing this as a double tensor contraction-\(\mathbf{w}(z)^\top \boldsymbol{\Psi} \mathbf{c}(z)\)-we allow the model to dynamically "assemble" a custom temporal basis for every \((s, g)\) pair. This allows for better expressivity (capturing diverse "styles" of hazard functions) while maintaining the computational benefits of a single-head architecture, as the library \(\boldsymbol{\Psi}\) is a shared, learned parameters across the entire dataset.

7.5 Details on binned time modeling↩︎

7.5.0.1 Piecewise-constant per-step hazard (PCH)

Let’s assume the hazard function is constant within each bin: \[h^\pi_{\theta}(t\mid s,g)=\bar h^\pi_{\theta}(k\mid s,g) \quad t\in[b_k,b_{k+1}).\] For an observation \((s,g,\tau,c,\delta)\), we can write \(\tau=b_{k_\tau}+m_\tau\) and \(c=b_{k_c}+m_c\), where \(k_\tau = \max\{k: b_k\le \tau\}\), \(k_c = \max\{k: b_k\le c\}\), \(m_\tau\in\{0,\ldots,L_{k_\tau}-1\}\) and \(m_c\in\{0,\ldots,L_{k_c}\}\). The grouped-time negative log-likelihood is \[\label{eq:nll95pch} \mathcal{L}^{\text{PCH}}(\theta) =-\mathbb{E}_{(s,g,\tau,c,\delta)}\!\left[ \delta\,\ell_{\tau}(\theta)+(1-\delta)\,\ell_{c}(\theta) \right]\tag{14}\] With \[\begin{align} \ell^{\text{PCH}}_{\tau}(\theta) =\log \bar h^\pi_{\theta}(k_\tau\!\mid\! s,g)&+\sum_{j<k_\tau}L_j\log\!\bigl(1-\bar h^\pi_{\theta}(j\!\mid\! s,g)\bigr)\\ & +m_\tau\log\!\bigl(1-\bar h^\pi_{\theta}(k_\tau\!\mid\! s,g)\bigr)\\ \end{align}\] \[\begin{align} \ell^{\text{PCH}}_{c}(\theta) = &\sum_{j<k_c}L_j\log\!\bigl(1-\bar h^\pi_{\theta}(j\!\mid\! s,g)\bigr) \\ &+m_c\log\!\bigl(1-\bar h^\pi_{\theta}(k_c\!\mid\! s,g)\bigr). \end{align}\] Value evaluation admits a closed form for the prefix and a hazard-based tail. Let \(q_{\hat{\theta}}(s,g):=\Pr_{\hat{\theta}}(T^\pi(s,g)=0)\) denote the predicted probability of immediate success, so that \(S^\pi_{\hat{\theta}}(b_0\mid s,g)=1 - q_{\hat{\theta}}(s,g)\), and \(S^\pi_{\hat{\theta}}(b_{k+1}\!\mid\! s,g)\!=\!S^\pi_{\hat{\theta}}(b_k\!\mid \! s,g)(1-\!\bar h^\pi_{\hat{\theta}}(k\mid \!s,g))^{L_k}\). Then \[\label{eq:Q95pch} \begin{align} V_{\hat{\theta}}^{\pi,\text{PCH}}(s,g) = &-\sum_{k=0}^{K-1} \Delta_{\theta,k}(\gamma) S^\pi_{\hat{\theta}}(b_k\mid s,g)\;\\ &\;-\; \frac{\gamma^H}{1-\gamma(1-h_K)}\,S^\pi_{\hat{\theta}}(H\mid s,g) \end{align}\tag{15}\] where \(\Delta_{\theta,k}(\gamma)=\gamma^{b_k} \frac{1-\bigl(\gamma(1-\bar h^\pi_{\theta}(k\mid s,g)\bigr)^{L_k}}{1-\gamma\bigr(1-\bar h^\pi_{\theta}(k\mid s,g)\bigr)}\),
\(h_K\!=\!\bar h^\pi_{\hat{\theta}}(K\mid s,g)\) and the tail uses
\(\sum_{t=H}^\infty \gamma^t S(t)\approx \gamma^H S(H)/(1-\gamma(1-h_K))\).

7.5.0.2 Piecewise-constant survival per bin (PCS).

We also consider the standard grouped-time model [42] where the event time is observed at the bin level. Define the interval hazard \[\label{eq:pcs95interval95hazard} \tilde{h}^\pi_{\theta}(k\mid s,g) \!=\! \Pr\!\bigl(T^\pi(s,g)\in[b_k,b_{k+1}) \!\bigm| \!T^\pi(s,g)\ge b_k\bigr).\tag{16}\] Survival at bin boundaries satisfies \[\label{eq:pcs95surv} S^\pi_{\theta}(b_{k+1}\mid s,g) = S^\pi_{\theta}(b_k\mid s,g)\bigl(1-\tilde{h}^\pi_{\theta}(k\mid s,g)\bigr)\tag{17}\] If an episode produces an event time \(\tau\le H\), we observe a binned event \([b_{k_\tau}, b_{k_\tau + 1})\). If it is right-censored at \(c\le H\), we only observe survival up to the last completed boundary \(b_{k(c)}\). The grouped-time negative log-likelihood is \[\label{eq:nll95pcs} \mathcal{L}^{\text{PCS}}(\theta) =-\mathbb{E}_{(s,g,\tau,c,\delta)}\!\left[ \delta\,\ell_{\tau}(\theta)+(1-\delta)\,\ell_{c}(\theta) \right]\tag{18}\] With \[\begin{align} \ell^{\text{PCS}}_{\tau}(\theta) &=\log \tilde{h}^\pi_{\theta}(k(\tau)\!\mid\! s,g) +\sum_{j<k(\tau)}\log\!\bigl(1-\tilde{h}^\pi_{\theta}(j\!\mid\! s,g)\bigr)\\ \ell^{\text{PCS}}_{c}(\theta) &=\sum_{j<k(c)}\log\!\bigl(1-\tilde{h}^\pi_{\theta}(j\!\mid\! s,g)\bigr) \end{align}\]

As the survival function is constant within each bin, \(S_{\theta}(t\mid s,g)\approx S_{\theta}(b_k\mid s,g)\) for \(t\in[b_k,b_{k+1})\). Hence, \[\label{eq:Q95pcs} \begin{align} V_{\hat{\theta}}^{\pi,\text{PCS}}(s,g) = &-\sum_{k=0}^{K-1} S^\pi_{\hat{\theta}}(b_k\mid s,g)\, \frac{\gamma^{b_k}\bigl(1-\gamma^{L_k}\bigr)}{1-\gamma}\\ &\;-\; \frac{\gamma^H}{1-\gamma}\,S^\pi_{\hat{\theta}}(H\mid s,g), \end{align}\tag{19}\] where the tail uses \(\sum_{t=H}^{\infty}\gamma^t S(t)\approx \gamma^H S(H)/(1-\gamma)\).

6pt

6pt

4pt

4pt

4pt

References↩︎