1 Introduction↩︎

Reinforcement learning (RL) has recently achieved remarkable progress in a variety of complex decision-making domains, including Go [1], competitive multiplayer online games [2], continuous control and simulated robotics [3], as well as autonomous driving related tasks [4]. Despite these successes, an implicit assumption is often made that agent–environment interactions are instantaneous, such that actions take effect immediately and observations faithfully reflect the current state. In real-world systems, however, this assumption frequently breaks down due to communication latency, sensor pipelines, and the computational overhead of model inference. Without properly modeling and compensating for delays, RL agents may suffer severe degradation and even safety-critical failures [5]; for instance, substantial performance loss in trading [6], instability in dynamical systems [7], [8], and reduced training efficiency and reproducibility in real-world robotics [9]. Therefore, explicitly incorporating action and observation delays into RL algorithm design is a crucial step towards reliable real-world deployment. Existing studies on delayed RL can be broadly categorized into three main lines. Memoryless methods learn policies that depend only on the most recent observation, thereby ignoring the non-Markovian nature induced by observation delays [10]. While simple to implement, these methods typically incur significant performance drops under large delays. Augmentation-based methods restore the Markov property by constructing an information state that concatenates the last observed state with the sequence of actions executed thereafter, reducing delayed RL to an augmented-state delayed MDP [11]–[13], on which classical RL algorithms can be applied [14]. However, as the delay horizon increases, the augmented state space expands rapidly, resulting in substantially higher sample complexity and pronounced instability in TD-style learning under long delays, reflecting a classic manifestation of the curse of dimensionality. Model-based methods seek to avoid learning directly in the rapidly expanding augmented space by instead computing compact statistics about the unknown current state from history and using them as policy inputs [15]–[19]. In addition, some works accelerate learning under long delays by introducing auxiliary short-delay tasks; for example, boosting delayed RL via auxiliary short delays can guide learning in the long-delay task and improve sample efficiency [20].
Meanwhile, in real-world systems, delay durations are often stochastic rather than fixed. In random-delay environments, the discrepancy between delayed observations and the true current state generally grows with the delay length, causing the optimal policy to vary across different delay regimes. Ignoring such delay heterogeneity may lead the agent to behave overly conservatively under short delays or excessively aggressively under long delays, resulting in degraded performance and unstable training dynamics. In random MDPs, we theoretically establish the existence of delay-induced state discrepancy and show that it amplifies as the delay length increases. We further demonstrate that approaches based on single-point state prediction inevitably suffer performance degradation, as they fail to capture this constantly changing discrepancy. To explicitly account for delay-induced state discrepancy in delayed reinforcement learning, we propose Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization (DUPO). The key idea is to model the mismatch between delayed state message and the underlying true state during policy optimization. DUPO leverages a generative diffusion model to characterize the conditional relationship between delayed observations and the current true state, enabling a principled representation of potentially multi-modal discrepancy structures induced by varying delay lengths. Based on these multi-modal state representations, DUPO estimates the uncertainty of the action–value function conditioned on delayed observations and uses this uncertainty to adaptively weight delayed policy updates. This uncertainty-aware weighting yields delay-adaptive policies that adjust their behavior across delay regimes, leading to more stable and sample-efficient learning in random-delay environments.
We transform the originally deterministic continuous-control MuJoCo tasks into stochastic MDPs by injecting Gaussian noise into the action execution, and further introduce random observation delays to emulate realistic delayed-feedback settings. Across a wide range of delay configurations, DUPO consistently demonstrates superior performance and improved training stability compared to other strong baselines designed for stochastic delayed environments. These results indicate that DUPO is not only effective under random delays but also robust to the compounded challenges arising from environmental stochasticity and delay variability. Our contributions can be summarized as follows:

• We theoretically prove the existence of delay-induced state discrepancy and show that it amplifies with increasing delay length, and further demonstrate that single-point state prediction methods inevitably suffer from performance degradation.

• We model the conditional distribution between delayed state message and the true current state using a generative diffusion model instead of single-point state prediction, capturing multi-modal discrepancy structures induced by random delays.

• We estimate action-value uncertainty under delayed observations and leverage it to adaptively weight delayed policy updates, enabling delay-adaptive policy learning.

• DUPO consistently outperforms state-of-the-art baselines across multiple random delay settings.

2 Related Work↩︎

2.0.0.1 Reinforcement learning with delayed feedback.

Existing studies on delayed reinforcement learning can be broadly categorized into three main lines. Memoryless methods learn policies that depend only on the most recent observation, thereby ignoring the non-Markovian nature induced by observation delays [10]. While simple to implement, these methods typically suffer from severe performance degradation as the delay length increases.
Augmentation-based methods restore the Markov property by constructing an information state that concatenates the last observed state with the sequence of subsequently executed actions, reducing delayed reinforcement learning to an augmented-state MDP [11]–[13]. This formulation enables the direct application of standard RL algorithms [14]. However, the augmented state space grows rapidly with the delay horizon, leading to substantially increased sample complexity and pronounced instability in TD-style learning under long delays, reflecting a classic manifestation of the curse of dimensionality.
Model-based and belief-based methods seek to avoid learning directly in the rapidly expanding augmented space by instead estimating compact statistics or distributions of the unknown current state from historical information and using them as policy inputs [15]–[19]. Recent advances further incorporate principled probabilistic modeling and variational inference to address delayed decision making; for example, Variational Delayed Policy Optimization formulates delayed RL from a variational perspective and optimizes policies under delayed information constraints [21], while other works analyze and mitigate signal delays in deep RL through delay-aware architectures and training strategies [22]. In addition, several approaches accelerate learning under long delays by introducing auxiliary short-delay tasks; leveraging such auxiliary delays has been shown to effectively guide policy learning in long-delay settings and improve sample efficiency [20].

2.0.0.2 Diffusion models and uncertainty-aware reinforcement learning.

Generative modeling has recently become a powerful tool in reinforcement learning, enabling expressive and potentially multi-modal representations of policies, dynamics, and latent states [23]. Diffusion models, in particular, are well suited for learning complex conditional distributions and have been used to improve robustness against observation perturbations in offline and model-based RL [24]. In parallel, uncertainty-aware value estimation and policy optimization, such as ensemble-based critics and uncertainty-weighted updates, have been shown to improve robustness under partial observability and distribution shift [25]. Building on these advances, we leverage diffusion modeling to explicitly capture the multi-modal state discrepancy induced by random observation delays and incorporate the resulting uncertainty into policy optimization, yielding delay-regime-adaptive behavior in delayed RL environments.

3 Preliminaries↩︎

A Partially Observable Markov Decision Process (POMDP) [26] can be defined by a tuple \(\mathcal{M}=(\mathcal{S},\mathcal{A},T,r,\Omega,O,\gamma)\), where \(\mathcal{S}\), \(\mathcal{A}\), and \(\Omega\) denote the state, action, and observation spaces. The transition dynamics are defined by \(T(s'\mid s,a)\),the reward function by \(r(s,a)\), and the observation probabilities by \(O(o\mid s)\). At each timestep \(t\), the environment is in a latent state \(s_t\in\mathcal{S}\), the agent receives an observation \(o_t\sim O(\cdot\mid s_t)\), selects an action \(a_t\in\mathcal{A}\) according to a policy \(\pi\), obtains a reward \(r_t=r(s_t,a_t)\), and the environment transitions to the next state according to \(s_{t+1}\sim T(\cdot\mid s_t,a_t)\). The agent repeats this process to find an optimal policy \(\pi^*\) that maximizes the expected discounted cumulative reward (i.e., return) over a finite or infinite horizon \(H\), which is given by \[\pi^* = \arg\max_{\pi} \; \mathbb{E} \left[ \sum_{k=0}^{H-1} \gamma^{k} r_{k+1} \;\middle|\; \pi, d_0 \right]. \label{eq:optimal95policy}\tag{1}\] where \(d_0\) denotes the initial state distribution. In the fully observable case, \(o_t=s_t\) almost surely, and the process reduces to a standard Markov Decision Process (MDP).

3.1 Random Observation Delay POMDP↩︎

We model random observation delay by augmenting the above POMDP with a random delay process. Formally, a random observation delay environment is defined as a pair \(\langle \mathcal{M}, \mathcal{D} \rangle\), where \(\mathcal{M}\) is a POMDP and \(\mathcal{D}\) is a probability distribution over non-negative integers, representing random observation delays. At each timestep \(t\), instead of observing the current state \(s_t\), the agent receives a delayed state observation corresponding to the latent state \(s_{t-\Delta T}\), where the delay \(\Delta T \sim \mathcal{D}\). In addition to the delayed state, the agent has access to the sequence of actions \(a_{t-\Delta T:t-1}\) executed since the observation was generated. Together, the delayed state and the history actions form a delayed state message, denoted by \(M_t = \big(s_{t-\Delta T},\, a_{t-\Delta T:t-1}\big)\) which summarizes all information available to the agent at time \(t\) and serves as the input for decision making. Throughout this paper, we assume that the delay is finite and bounded by a maximum delay \(D\).

3.2 Conditional Diffusion Model↩︎

A conditional diffusion model is a generative model that learns a conditional data distribution \(p_\theta(x_0 \mid c)\), where \(x_0 \in \mathbb{R}^d\) denotes a target variable and \(c\) denotes conditional information. It introduces a sequence of latent variables \(\{x_k\}_{k=1}^{K}\) by gradually perturbing \(x_0\) with Gaussian noise according to a fixed variance schedule \(\{\beta_k\}_{k=1}^{K}\): \[q(x_k \mid x_{k-1}) = \mathcal{N}\!\left(\sqrt{1-\beta_k}\,x_{k-1},\,\beta_k I\right), \quad k=1,\dots,K.\] Let \(\bar{\alpha}_k=\prod_{i=1}^{k}(1-\beta_i)\), then the marginal at step \(k\) admits the closed form [27] \[x_k=\sqrt{\bar{\alpha}_k}\,x_0+\sqrt{1-\bar{\alpha}_k}\,\epsilon,\qquad \epsilon\sim\mathcal{N}(0,I).\] The reverse generative process is parameterized by a conditional noise predictor \(\epsilon_\theta(x_k,k,c)\) and is trained with the standard noise-prediction objective \[\min_\theta\;\mathbb{E}_{x_0,k,\epsilon}\Big[\big\|\epsilon-\epsilon_\theta(x_k,k,c)\big\|_2^2\Big],\] where \(k\) is sampled uniformly from \(\{1,\dots,K\}\). At inference time, given conditioning \(c\), a sample \(x_0 \sim p_\theta(\cdot \mid c)\) is obtained by starting from \(x_K \sim \mathcal{N}(0,I)\) and iteratively applying the learned reverse denoising transitions. This framework provides a flexible approximation to complex conditional distributions and naturally represents uncertainty through random sampling.

4 Our Approach↩︎

In this section, we introduce Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization (DUPO). DUPO first learns a conditional diffusion model to represent the posterior over the latent delay-free state from delayed state messages, and then we leverage the posterior distribution to characterize the state discrepancy induced by observation delays and quantify the resulting uncertainty in action value estimation. We then weight candidate actions according to their value uncertainty to obtain a more stable delayed policy. Finally, by integrating this uncertainty-weighted delayed policy into Soft Actor-Critic, we arrive at our DUPO algorithm.

4.1 Discrepancy under Random Delays↩︎

We study a stochastic delayed MDP with latent dynamics \(s_{t+1}=f(s_t,a_t)+\xi_t\) and delayed message \(M_t=(s_{t-\Delta T},a_{t-\Delta T:t-1})\). Random delays and system noise induce an inherent mismatch between \(M_t\) and the current state \(s_t\). The bounds below are stated conditional on a realized delay value \(\Delta T\); averaging over a random delay distribution yields the corresponding bounds with the last terms averaged over \(\Delta T\). The detailed proof is provided in the supplementary material.

A common workaround is to compensate delays by a point prediction \(\hat{s}_t=g(M_t)\) and act greedily on \(\hat{s}_t\). The next result shows that such point-estimate methods suffer a delay-amplified performance loss.

Together, these theorems motivate modeling the full posterior density \(p(s_t\mid M_t)\), rather than a single prediction, to properly account for delay-induced uncertainty.

4.2 Diffusion-based State Estimator↩︎

To model the relationship between delayed state message and the delay-free state, we employ a conditional diffusion model with strong expressive capacity. Specifically, the model captures the conditional distribution \(p_\theta(s_t \mid M_t)\) and provides a flexible representation of the potentially multi-modal posterior distribution induced by random observation delays [27].
The diffusion model introduces a sequence of latent variables \(\{s_t^n\}_{n=0}^N\) with \(s_t^0 = s_t\), obtained by progressively corrupting the clean state with Gaussian noise according to a predefined variance schedule. The forward noising process is defined by the Markov transition \[q(s_t^{n} \mid s_t^{n-1}) = \mathcal{N}\!\left( \sqrt{\alpha_n}\, s_t^{n-1}, (1-\alpha_n)I \right), \quad n = 1,\dots,N. \label{eq:dbde95forward}\tag{2}\] where \(\alpha_n\in (0,1), \forall n=1, \cdots, N\) controls the noise level at each diffusion step. By composing these transitions, the marginal distribution at step \(n\) admits the closed form \[q(s_t^{n} \mid s_t) = \mathcal{N}\!\left( \sqrt{\bar{\alpha}_n}\, s_t, (1-\bar{\alpha}_n)I \right), \qquad \bar{\alpha}_n = \prod_{i=1}^{n} \alpha_i. \label{eq:dbde95forward95closed}\tag{3}\] which enables efficient sampling of noisy states at arbitrary diffusion steps.
The reverse generative process is parameterized as a conditional Markov chain that inverts the noising process given delayed message \(M_t\) [28], \[p_\theta(s_t^{n-1} \mid s_t^{n}, M_t) = \mathcal{N}\!\left( \mu_\theta(s_t^{n}, M_t, n), \sigma_n^2 \right). \label{eq:dbde95reverse}\tag{4}\] where the mean \(\mu_\theta\) is computed via a neural noise predictor \(\epsilon_\theta(s_t^{n}, M_t, n)\) using the standard \(\epsilon\)-parameterization, \[\mu_\theta(s_t^{n}, M_t, n) = \frac{1}{\sqrt{\alpha_n}} \left( s_t^{n} - \frac{1-\alpha_n}{\sqrt{1-\bar{\alpha}_n}} \epsilon_\theta(s_t^{n}, M_t, n) \right). \label{eq:dbde95mean}\tag{5}\] Conditioning on \(M_t\) allows the denoising process to adapt to different delay realizations and action histories.
Model parameters are learned by minimizing the standard noise-prediction objective. Specifically, a diffusion step \(n\) is sampled uniformly from \(\{1,\dots,N\}\), Gaussian noise \(\epsilon \sim \mathcal{N}(0,I)\) is drawn, and a noisy state is constructed via \[s_t^{n} = \sqrt{\bar{\alpha}_n}\, s_t + \sqrt{1-\bar{\alpha}_n}\, \epsilon.\] The training loss is then given by \[\mathcal{L}_{\mathrm{diff}} = \mathbb{E}_{s_t, M_t, \epsilon, n} \left[ \left\| \epsilon - \epsilon_\theta(s_t^{n}, M_t, n) \right\|_2^2 \right]. \label{eq:dbde95loss}\tag{6}\]
After training, the learned reverse process defines an implicit conditional generative model for the delay-free state. Given delayed message \(M_t\), samples from \(p_\theta(s_t \mid M_t)\) are obtained by starting from \(s_t^N \sim \mathcal{N}(0,I)\) and iteratively applying the reverse transitions in Eq. 4 . This corresponds to the conditional density \[p_\theta(s_t \mid M_t) = \int p_\theta(s_t^{0:N-1} \mid s_t^{N}, M_t)\, p(s_t^{N})\, \mathrm{d}s_t^{1:N}. \label{eq:dbde95conditional}\tag{7}\] where \(p(s_t^{N}) = \mathcal{N}(0,I)\). In our framework, multiple samples from this distribution are used to characterize the discrepancy between delayed state message \(M_t\) and the delay-free state \(s_t\), which will be exploited by subsequent uncertainty-aware policy optimization.

4.3 Uncertainty-Weighted Delayed SAC↩︎

To cope with random observation delays, we explicitly treat the current delay-free state as a latent variable conditioned on the delayed message \(M_t=\big(s_{t-\Delta T},\,a_{t-\Delta T:t-1}\big).\) Using the trained conditional diffusion model, we draw \(N\) plausible delay-free states \(s_t^{(i)}\sim p_\theta(\cdot\mid M_t),\) which yields a Monte Carlo approximation of the discrepancy induced by delay. This discrepancy is then propagated to action value estimation: for any action \(a\), we quantify the delay-induced critic uncertainty by the sample variance \[\begin{align} \widehat{\mathrm{Var}}[Q(M_t,a)] = \frac{1}{N}\sum_{i=1}^N Q_\psi(s_t^{(i)},a)^2 - Q_\psi(s_t,a)^2. \end{align}\] We transform this uncertainty into a reliability weight via inverse variance scaling \[\omega(M_t,a)=\frac{\beta}{\widehat{\mathrm{Var}}\!\left[Q(M_t,a)\right]+\varepsilon}.\] and weight the base actor \(\pi_\phi(\cdot\mid M_t)\) to obtain an uncertainty-weighted delayed policy [25] \[\label{eq:pi95prime} \pi'_\phi(a\mid M_t)=\frac{\omega(M_t,a)\,\pi_\phi(a\mid M_t)}{\int \omega(M_t,\tilde{a})\,\pi_\phi(\tilde{a}\mid M_t)\,d\tilde{a}}.\tag{8}\] Intuitively, actions whose values are highly sensitive to the latent state (hence more risky under delay) receive smaller probability mass under \(\pi'_\phi\), leading to a more conservative policy when the delayed message is ambiguous.
We incorporate \(\pi'_\phi\) into SAC while training the critic on delay-free transition tuples, following the standard treatment in delayed RL [22]. Concretely, \[\begin{align} J_Q(\psi) &=\mathbb{E}_{\mathcal{D}}\!\left[\big(Q_\psi(s_t,a_t)-y_t\big)^2\right], \tag{9}\\ y_t &= r_t + \gamma\,\mathbb{E}_{a_{t+1}\sim \pi'(\cdot \mid s_{t+1})} \Big[\, Q_{\psi}(s_{t+1}, a_{t+1}) \notag\\ &\qquad - \alpha \log \pi'(a_{t+1}\mid s_{t+1}) \,\Big]. \tag{10} \end{align}\] Since \(s_{t+1}\) is delay-free, weighting is unnecessary and \(\pi'(\cdot\mid s_{t+1})=\pi(\cdot\mid s_{t+1})\).
In practice, we optimize the delayed policy by updating the base actor \(\pi_\phi\) under an uncertainty-weighted objective that preserves the effect of weighting. Specifically, the actor is trained using a weighted SAC surrogate loss, where the uncertainty weight \(\omega(M_t,a)\) modulates the policy update.

\[\begin{align} J_{\pi'}(\phi) &=\mathbb{E}_{(M_t,s_t)\sim\mathcal{D}} \Big[ \mathbb{E}_{a_t\sim\pi'_\phi(\cdot\mid M_t)} \Big[ \notag\\ &\qquad \alpha\log\pi'_\phi(a_t\mid M_t)-Q_\psi(s_t,a_t) \Big]\Big]\\ &=\mathbb{E}_{(M_t,s_t)\sim\mathcal{D}} \Big[ \mathbb{E}_{\mathbf{a}_t\sim\pi_\phi} \Big[ \omega(M_t,a)\Big( \notag\\ &\qquad \alpha\log\!\left( \frac{\omega(M_t,a)\pi_\phi(\mathbf{a}_t\mid M_t)}{\int_a \omega(M_t,a)\pi(a\mid M_t)\,da} \right) \notag\\ &\qquad -Q_\psi(s_t,\mathbf{a}_t) \Big)\Big]\Big]. \label{eq:actor95obj} \end{align}\tag{11}\] This objective preserves the effect of uncertainty-aware weighting while remaining amenable to standard SAC-style optimization.

5 Experimental Results↩︎

5.1 Benchmarks and Baseline Algorithms↩︎

We evaluate DUPO on deterministic continuous robotic locomotion control MuJoCo [29] with random observation delays, including Ant-v4, HalfCheetah-v4, Hopper-v4, Walker2d-v4, and Swimmer-v4. We focus on stochastic MDPs that better reflect real-world control systems, in which delays induce state transition discrepancies and amplify uncertainties. We therefore convert deterministic MuJoCo environments into stochastic MDPs by adding Gaussian noise \(\mathcal{N}(0,0.1)\) to the environment-executed actions. After constructing stochastic MuJoCo environments, we further introduce random observation delays. Let \(\Delta T\) denote the delay at time \(t\), drawn from a uniform distribution where \(\Delta T_{\max}\) specifies the maximum delay horizon. Accordingly, the agent does not directly observe the current state \(s_t\); instead, it receives a delayed message \(M_t\), which aggregates the delayed observation together with the associated action history.
The following algorithms were included as baselines in the experiments: DC/AC [30], State Augmentation [22], and State Prediction [22]. DC/AC is a SAC variant that integrates off-policy multi-step value estimation with partial trajectory resampling, yielding strong sample efficiency under random delays. State augmentation concatenates delayed observations with the aligned action history to build a time-calibrated input for standard actor–critic learning. State prediction trains an auxiliary model to infer the delay-free state from the delayed message and controls using the predicted state at decision time. Detailed experimental settings and hyperparameter configurations are provided in the supplementary material.

5.2 Results and Discussion↩︎

Table 1 reports the final return after \(1\)M global environment steps, summarized by the mean \(\mu\) and standard deviation \(\sigma\) over four random seeds. Since raw returns have different scales across MuJoCo tasks, we normalize results within each fixed \((\Delta T_{\max}, \text{env})\) setting using the DC/AC baseline mean return \(\mu_{\text{DC/AC}}\) as a scale factor: \(\tilde{\mu}=\mu/\mu_{\text{DC/AC}}, \qquad \tilde{\sigma}=\sigma/\mu_{\text{DC/AC}}.\) Consequently, DC/AC is always \(1.00\) in every \((\Delta T_{\max}, \text{env})\) entry, and other numbers can be read as a multiplicative ratio relative to DC/AC, enabling fair comparison across tasks and delay budgets.
Overall, DUPO achieves the best or near-best normalized mean in most environments, and its advantage becomes more pronounced as the delay budget increases. In particular, under severe random delays (\(\Delta T_{\max}=25\)), DUPO attains strong normalized performance on HalfCheetah-v4, Hopper-v4, and Walker2d-v4, consistently outperforming State-Pred and State-Aug, which tend to degrade substantially in the long-delay regime. We also observe that point-estimation baselines can be competitive in short-delay and simpler dynamics (e.g., Swimmer-v4 at \(\Delta T_{\max}=5\)), yet their gains diminish quickly as delays grow, whereas DUPO remains robust. Figure 3 complements the final scores by illustrating learning dynamics under \(\Delta T_{\max}=25\). Across five tasks, DUPO typically reaches high-return regimes faster and exhibits more stable training trajectories, with reduced variance and fewer late-stage plateaus compared to State-Pred and State-Aug. More learning curves are provided in the supplementary material. A representative case is Walker2d-v4, where long stochastic delays often induce inaccurate delayed-state estimates and value bias for the baselines, limiting further improvement; in contrast, DUPO continues to improve and sustains higher returns. These results align with DUPO’s design: modeling the distributional posterior from delayed messages and performing uncertainty-aware weighting yields more reliable policy updates under multi-modal and high-uncertainty delay conditions, leading to consistent gains especially in medium-to-long random delays.

5.3 Ablation Study↩︎

Figure 4 (left) studies the impact of the observation delay time distribution while keeping the same maximum delay budget \(\Delta T_{\max}=10\) on HalfCheetah-v4. We consider three commonly used delay models: a uniform distribution (Un), a truncated Gaussian distribution (Ga), and a discrete Poisson distribution (Po), all supported on \(\{1,\dots,\Delta T_{\max}\}\). Across these settings, DUPO exhibits consistently strong performance and stable learning dynamics, indicating that the proposed diffusion-based posterior modeling is not tied to a particular delay prior. Notably, the Gaussian-delay setting yields the fastest improvement and the highest final return, while the uniform and Poisson variants remain competitive but converge more slowly, suggesting that the shape of the delay distribution mainly affects learning speed rather than DUPO’s overall robustness.
Figure 4 (right) further ablates the posterior sampling mechanism used to represent the latent delay-free state. Our full DUPO uses a diffusion model to produce multimodal posterior samples, whereas MD-PO replaces diffusion sampling with Monte Carlo dropout sampling, and G-PO replaces it with a unimodal Gaussian sampler. The results show a clear performance gap in favor of DUPO: diffusion-based multimodal sampling reaches high-return regimes earlier and plateaus at substantially higher performance than both alternatives. This highlights the necessity of accurately capturing posterior multimodality under stochastic delays; simplified unimodal (Gaussian) or approximate uncertainty sampling (MC dropout) tends to under-represent ambiguous delayed observations, leading to less reliable value estimation and a weaker delayed policy.

Table 2 reports a Hyperparameter sensitivity study on Hopper-v4 with \(\Delta T_{\max}=10\), where we normalize returns by the default configuration \((\beta{=}0.8,\,N{=}10)\). The coefficient \(\beta\) controls the uncertainty weight \(\omega(M_t,a)=\beta/(\widehat{\mathrm{Var}}[Q(M_t,a)]+\varepsilon)\), and \(N\) denotes the number of posterior samples drawn from the diffusion model to approximate the multimodal latent state distribution. Varying \(\beta\) shows that performance is relatively stable around the default choice, while overly small or large values degrade normalized return, indicating that moderate uncertainty scaling yields the most reliable action weighting. Increasing \(N\) consistently improves performance and reduces variability, suggesting that richer posterior sampling provides a more accurate uncertainty estimate and hence a more effective delayed policy; in particular, larger \(N\) achieves the best normalized return under the same delay budget. More details of the ablation studies can be found in the supplementary material.

6 Conclusion↩︎

In this paper, we studied reinforcement learning under stochastic observation delays and theoretically analyzed how random delays in stochastic MDPs induce an intrinsic discrepancy between delayed observations and the true current state, which fundamentally limits approaches based on single-point state estimation. To address this challenge, we proposed Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization (DUPO), which models the multimodal posterior distribution of the latent delay-free state using a conditional diffusion model and incorporates uncertainty-aware action weighting to stabilize policy updates under delayed state information. By explicitly accounting for delay-induced uncertainty and posterior multimodality, DUPO provides a principled solution to delayed decision-making in stochastic environments. Extensive experiments on continuous-control benchmarks with multiple random delay settings demonstrate that DUPO consistently outperforms strong baselines, with particularly pronounced advantages under medium-to-long delays, validating both the theoretical motivation and practical effectiveness of the proposed approach. To facilitate reproducibility, the code will be publicly released upon acceptance.

7 Theoretic Analysis↩︎

7.0.0.1 Notation and regularity assumptions.

All statements below are written for a realized delay value \(d=\Delta T\). We treat the realized delay as part of the delayed message; equivalently, if an implementation stores the delay separately, read \(M_t\) below as the pair \((M_t,d)\). This is natural because the action-history length in \(M_t=(s_{t-d},a_{t-d:t-1})\) identifies \(d\). The sigma-field generated by the delayed message is denoted by \[\mathcal{F}_t := \sigma(M_t,d) = \sigma\!\left(s_{t-d},a_{t-d:t-1},d\right).\] Thus \(s_{t-d}\) and the intervening actions are known under \(\mathcal{F}_t\). For \(j=0,\ldots,d\), define \[S_j:=s_{t-d+j},\qquad A_j:=a_{t-d+j},\qquad \Sigma_j:=\mathrm{Cov}(S_j\mid\mathcal{F}_t),\qquad v_j:=\mathrm{tr}(\Sigma_j).\] We use the following standard local conditions for stochastic delayed dynamics:

1. The latent dynamics along the delayed window satisfy \(S_{j+1}=f(S_j,A_j)+\xi_j\).

2. The innovation noise is conditionally centered and conditionally orthogonal to the propagated state uncertainty: \[\mathbb{E}[\xi_j\mid\mathcal{F}_t]=0,\qquad \mathrm{tr}\!\left(\mathrm{Cov}(f(S_j,A_j),\xi_j\mid\mathcal{F}_t)\right)=0.\]

3. The transition noise has non-vanishing conditional variance and the deterministic part does not contract conditional uncertainty below a factor \(\kappa\ge1\): \[\mathrm{tr}\!\left(\mathrm{Cov}(\xi_j\mid\mathcal{F}_t)\right)\ge \nu,\qquad \mathrm{tr}\!\left(\mathrm{Cov}(f(S_j,A_j)\mid\mathcal{F}_t)\right)\ge \kappa\,\mathrm{tr}\!\left(\mathrm{Cov}(S_j\mid\mathcal{F}_t)\right),\] for some \(\nu>0\).

These assumptions formalize the setting in which stochastic transitions inject fresh uncertainty at every hidden step, while the delayed message cannot fully recover the intervening latent trajectory.

7.1 Proof of Theorem 1↩︎

7.2 Proof of Theorem 2↩︎

We fix any point estimator \(\hat{s}_t=g(M_t)\) and the induced greedy policy \(\pi_g\) defined by \(a_t=a^\star(g(M_t))\), where \(a^\star(s):=\arg\max_a Q^\star(s,a)\) (assumed unique) and \(V^\star(s):=\max_a Q^\star(s,a)\). Let the discounted occupancy over \((s,M)\) under a policy \(\pi\) be \[d^\pi(s,M):=(1-\gamma)\sum_{t=0}^{\infty}\gamma^t\Pr_\pi(s_t=s,M_t=M).\]

8 Implementation Details↩︎

8.1 Benchmark and Baseline Setting↩︎

We evaluate on MuJoCo-v4 continuous-control benchmarks (Ant, HalfCheetah, Hopper, Walker2d, and Swimmer) under random observation delays. To emulate stochastic control systems, we turn each originally deterministic environment into a stochastic MDP by adding Gaussian noise to the environment-executed actions: the simulator applies \(\tilde{a}_t = a_t + \epsilon_t\) with \(\epsilon_t \sim \mathcal{N}(0,0.1)\). The observation delay at time \(t\) is sampled as an integer-valued random variable \(\Delta T_t \sim \mathrm{Unif}\{1,\ldots,\Delta T_{\max}\}\), and the agent receives a delayed message \(M_t\) that aggregates the delayed observation together with the associated action history, e.g., \(M_t=(s_{t-\Delta T_t}, a_{t-\Delta T_t:t-1})\). For each baseline, we implement the method following its original paper and adopt the best-performing hyperparameters recommended by the authors whenever available; when multiple configurations are reported, we select the one that achieves the strongest final performance under the same delay budget. All baselines are trained with the identical environment construction, interaction budget, and evaluation protocol as DUPO. We further conduct ablations by varying only the targeted components while keeping the remaining training pipeline unchanged.

8.2 Ablation Study↩︎

Figure 4 investigates two factors while keeping the remaining training pipeline unchanged. In the left panel, we study the effect of the observation-delay time distribution on HalfCheetah-v4 with a fixed maximum delay budget \(\Delta T_{\max}=10\). At each time step, the delay length is an integer random variable \(\Delta T_t \in \{1,\ldots,\Delta T_{\max}\}\), and the agent observes a delayed message constructed from the delayed state and the corresponding action history. We consider three commonly used discrete delay models on this support: an integer-valued uniform distribution (Un), a truncated discrete Gaussian distribution (Ga), and a discrete Poisson distribution truncated to \(\{1,\ldots,\Delta T_{\max}\}\) (Po). Across all three priors, DUPO achieves consistently strong performance and stable learning dynamics, suggesting that diffusion-based posterior modeling is not tied to a particular delay prior. The Gaussian-delay setting improves fastest and attains the best final return, while the uniform and Poisson variants remain competitive but converge more slowly, indicating that the delay distribution primarily affects learning speed rather than overall robustness.

The right panel ablates how we obtain multiple samples of the latent delay-free state for uncertainty estimation. Our full DUPO draws multimodal posterior samples using the diffusion model. In contrast, MD-PO (Monte Carlo Dropout Posterior) replaces diffusion sampling with dropout-based sampling: we keep dropout active at inference time and perform multiple stochastic forward passes to generate a set of predicted states. G-PO (Gaussian Posterior) replaces diffusion sampling with a unimodal Gaussian sampler: we fit (or assume) a single Gaussian distribution over the predicted state and draw multiple state samples from this Gaussian. Both variants keep the downstream uncertainty-aware weighting and policy optimization unchanged, differing only in the state-sampling mechanism. The results show a clear advantage for DUPO: diffusion-based multimodal sampling reaches high-return regimes earlier and plateaus at substantially higher performance than MD-PO and G-PO. This highlights that accurately capturing posterior multimodality under stochastic delays is crucial; unimodal Gaussian sampling or approximate uncertainty sampling via MC dropout tends to under-represent ambiguous delayed observations, leading to less reliable value estimation and weaker delayed policies.

8.3 Hyperparameters setting↩︎

12pt

9 Learning Curves↩︎

References↩︎