1 Introduction↩︎

Policy gradient (PG) methods are among the most widely used algorithms in reinforcement learning (RL). Beginning with REINFORCE [1] and the policy gradient theorem [2], classical PG performs Euclidean gradient ascent over a parameterized policy class. While simple and scalable, Euclidean PG is sensitive to step sizes and parameterization, which has motivated geometry-aware alternatives. Natural policy gradient replaces the Euclidean metric with the information geometry induced by the Fisher information [3], [4], and is closely related to trust-region (TR) methods. In deep RL, TRPO constrains the average KL divergence between successive policies [5], while PPO provides a practical surrogate through clipping or adaptive KL penalties [6]. These developments underscore the central role of geometry in policy optimization.

Beyond information geometry, optimal transport (OT) provides a different geometry for policy space, one that is particularly natural for distributions over continuous action spaces. Existing Wasserstein policy optimization methods can be viewed as different implementations or approximations of Wasserstein gradient flow: through JKO schemes and particle approximations [7], kernelized Wasserstein natural-gradient approximations [8], [9], trust region formulations [10], [11], and Wasserstein gradient updates projected onto explicit [12] or implicit [13] policy classes. These methods share the same underlying mechanism: policy improvement is driven by transport in action space.

Entropy regularization is another powerful scheme in modern continuous-control RL. Maximum-entropy RL augments reward maximization with an entropy term on the policy distribution over actions, promoting exploration and robustness and leading to soft Bellman operators and Gibbs-type policy improvements [14]–[16]. From the OT perspective, this entropy term naturally induces diffusion. Consequently, Wasserstein policy optimization for the entropy-regularized objective takes the form of a drift-diffusion, or Fokker–Planck, equation, whose particle representation is a Langevin-type actor update. More concretely, the Wasserstein Policy Gradient (WPG) flow reads \[\label{eq:intro95wpgf} \partial_t \pi_t(a\mid s) = -\mathrm{div}_a\Big(\pi_t(a\mid s)\nabla_a\big(Q^{\pi_t}(s,a)-\tau\log\pi_t(a\mid s)\big)\Big),\tag{1}\] and its discrete-time Langevin counterpart is \[\label{eq:intro95wpgd} A_{k+1}^s = A_k^s + \eta \nabla_a Q^{\pi_k}(s,A_k^s) + \sqrt{2\tau\eta}\,\xi_{k+1}, \qquad \xi_{k+1}\sim \mathcal{N}(0,I_d),\tag{2}\] where \(\pi_k(\cdot\mid s)=\mathrm{Law}(A_k^s)\). Thus, each state-conditional policy is transported along the action gradient of the current soft Q-function, while entropy regularization contributes to the Langevin diffusion.

Despite the algorithmic appeal of WPG, its global convergence behavior remains largely open. Existing global analyses of PG or natural PG typically rely on Euclidean or KL/Bregman geometry in policy space, using mirror-descent identity to relate policy updates to global suboptimality [17]–[27]. These arguments do not transfer to Wasserstein geometry, which is non-Bregman and lacks the pointwise three-point structure underlying KL-based analyses. Existing OT-based convergence results also do not directly help for the Langevin WPG update in 2 . The convergence result in [7] is asymptotic and places the entropy regularization on the policy parameters rather than on state-conditional policy distributions. The trust-region analysis of [11] applies to finite tabular MDPs without entropy regularization, and [13] studies a JKO-type implicit proximal scheme. None of these analyses transfers to Langevin dynamics.

Our analysis is closely related to the broader literature on mean-field Langevin dynamics, which has been used to study training dynamics of large interacting particle systems and over-parameterized neural networks [28]–[32]. However, global convergence arguments in that literature typically rely on a convex objective and convexity-analysis based argument. RL objective has a different structure: the policy enters the objective through the Bellman recursion which destroys the convexity, and the soft Q-function entangles the reward and entropy term which make the regularity of the Langevin drift not taken for granted.

1.0.0.1 Contribution.

Our main contribution is a non-asymptotic global convergence analysis of WPG for entropy-regularized continuous-action RL. Rather than relying on the convex-analysis machinery commonly used in standard Mean-field Langevin dynamics, we exploit the Bellman structure of the RL objective. The analysis is built on the following ingredients.

• Value improvement-to-Fisher information via a Bellman resolvent identity. We show that policy-space dissipation propagates through the Bellman equation via a resolvent identity. This allows us to lower bound value improvement by Fisher-information dissipation, replacing the standard energy-dissipation identity used in convex settings.

• Optimality gap-to-KL via Bellman residuals and contraction. We represent the statewise soft Bellman residual as a KL divergence between the current policy and its associated Gibbs policy. Bellman contraction then relates this residual to the global optimality gap in sup-norm, replacing the entropy-sandwich arguments used in convex settings.

• Uniform control of the moving Gibbs family. To exploit the commonly used uniform logarithmic Sobolev inequalities to connect the KL divergence (controlling the optimality gap) with the Fisher information (controlling value improvement), we establish uniform bound estimates along the WPG iterates.

We first illustrate these ideas in the continuous-time WPG flow, where the Bellman residual, contraction, resolvent mechanisms appear most transparently. Then we provide careful analysis for the discrete-time Langevin update 2 , which requires additional regularity estimates for the Wasserstein gradient and a one-step interpolation argument [33] to control time-discretization error. To this end, we establish refined uniform a priori bounds on value functions, soft \(Q\)-functions, drift Lipschitz constants, action moments, drift moments, and the relevant KL quantities. Combining these ingredients yields geometric contraction of the global optimality gap up to a discretization bias.

1.0.0.2 Organization.

Section 2 introduces the entropy-regularized discounted RL model, the soft Bellman operators, and the Wasserstein/Langevin policy update. Section 3 presents the continuous-time analysis, isolating the core Bellman mechanisms underlying the proof. Section 4 proves the finite-time convergence theorem for the discrete-time update, with emphasis on the additional regularity estimates and discretization-error control. The appendices contain auxiliary results and detailed proofs.

2 Model Setup↩︎

We consider an infinite-horizon discounted Markov decision process \[\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma,\rho_0),\] where \(\mathcal{S}\) is the state space, \(\mathcal{A}\) is the action space, \(\gamma\in(0,1)\) is the discount factor, \(\rho_0\in\mathcal{P}(\mathcal{S})\) is the initial state distribution, \(P(\mathrm ds'\mid s,a)\) is a Markov transition kernel on \(\mathcal{S}\), and \(r:\mathcal{S}\times\mathcal{A}\to\mathbb{R}\) is a measurable reward function. A (stationary Markov) policy is a conditional distribution \(\pi(\mathrm da\mid s)\) that admits a density, still denoted \(\pi(a\mid s)\), with respect to Lebesgue measure on \(\mathcal{A}\). Given \(\pi\), a trajectory \((s_n,a_n)_{n\ge 0}\) is generated by \(s_0\sim\rho_0\), \(a_n\sim\pi(\cdot\mid s_n)\), and \(s_{n+1}\sim P(\cdot\mid s_n,a_n)\). We write \(\mathbb{E}_{\rho_0,\pi}[\cdot]\) for expectations under this rollout distribution. The normalized discounted state occupancy of \(\pi\) is the probability measure \(d^\pi\in\mathcal{P}(\mathcal{S})\) defined by \[\label{eq:occ95def} d^\pi(B)=(1-\gamma)\sum_{n\ge 0}\gamma^n\mathbb{P}_{\rho_0,\pi}(s_n\in B), \qquad B\in\mathcal{B}(\mathcal{S}).\tag{3}\] We assume throughout that the initial distribution \(\rho_0\) has full support on \(\mathcal{S}\). Hence, for every policy \(\pi\), \[d^\pi(B) = (1-\gamma)\sum_{t\ge0}\gamma^t\mathbb{P}_{\rho_0,\pi}(s_t\in B) \ge (1-\gamma)\rho_0(B), \qquad B\in\mathcal{B}(\mathcal{S}).\] Thus \(d^\pi\) has full support whenever \(\rho_0\) does. This ensures that the statewise policy-gradient updates below are specified on the same state space on which the sup-norm value bounds are proved. We use the following standing measurability convention. The model kernels and their action derivatives admit jointly measurable versions such that all displayed integrals are jointly measurable. Under this convention, Bellman fixed points are selected as bounded measurable fixed points of contraction maps, and Gibbs and WPG kernels are taken in their induced jointly measurable versions.

We work with an action space \(\mathcal{A}=\mathbb{R}^d\). We consider the regularized reward \[\tilde{r}(s,a)=r(s,a)-\frac{\beta}{2}\|a\|^2,\] where \(\beta>0\) is the quadratic action penalty. This penalty term stabilizes the training [34]–[36], and is also exploited in standard mean-field Langevin dynamics analysis [37], closely related to the dissipativity condition that helps to control the KL divergence along the trajectory. Fix a temperature \(\tau>0\), we study the discounted regularized objective \[\label{eq:J-def} J(\pi)=\mathbb{E}_{\rho_0,\pi}\left[\sum_{n\ge 0}\gamma^n\left(\tilde{r}(s_n,a_n)-\tau\log\pi(a_n\mid s_n)\right)\right],\tag{4}\] restricting attention to policies for which the above expectation is finite. The quadratic action penalty is essential on the unbounded action space. Without this term, the entropy bonus can be made arbitrarily large by spreading the policy mass over larger regions of \(\mathbb{R}^d\), so the entropy-regularized policy-improvement objective may be unbounded above. The quadratic penalty rules out this degeneracy. Equivalently, the quadratic term induces the Gaussian reference \[\label{eq:rho95beta95def} \rho_\beta(a)=Z_\beta^{-1}\exp\left(-\frac{\beta}{2\tau}\|a\|^2\right), \quad Z_\beta=\left(\frac{2\pi\tau}{\beta}\right)^{d/2}.\tag{5}\] For the discrete-time result, we work on the admissible class \[\label{eq:admissible95class} \begin{align} \Pi :=\Big\{\pi: &\;\pi(\cdot\mid s)\ll \mathrm da, \;(s,a)\mapsto \pi(a\mid s)\text{ jointly measurable}, \;\sup_{s\in\mathcal{S}}\mathrm{KL}(\pi(\cdot\mid s)\|\rho_\beta)<\infty\Big\}. \end{align}\tag{6}\] Finite relative entropy gives finite second moments; see Lemma [lem:gaussian-KL-second-moment] in Appendix 10.

2.1 Soft value functions and Bellman operators↩︎

Given a policy \(\pi\), define the soft value function \[\label{eq:Vpi95def} V^\pi(s)=\mathbb{E}\left[\sum_{n\ge 0}\gamma^n\big(\tilde{r}(s_n,a_n)-\tau\log\pi(a_n\mid s_n)\big)\mid s_0=s\right],\tag{7}\] so that \(J(\pi)=\int_{\mathcal{S}}V^\pi(s)\rho_0(\mathrm ds)\) whenever the integral is well-defined. Define the state-action value function \[\label{eq:Qpi95def} Q^\pi(s,a)=\tilde{r}(s,a)+\gamma\int_{\mathcal{S}}V^\pi(s')P(\mathrm ds'\mid s,a).\tag{8}\] Then \(V^\pi\) satisfies the soft Bellman identity \[\label{eq:bellman95Vpi} V^\pi(s)=\int_{\mathcal{A}}\left(Q^\pi(s,a)-\tau\log\pi(a\mid s)\right)\pi(a\mid s)\mathrm da.\tag{9}\]

For a fixed policy \(\pi\), define the policy evaluation operator acting on bounded measurable \(V:\mathcal{S}\to\mathbb{R}\) by \[\label{eq:Tpi} (\mathcal{T}^\pi V)(s)=\int_{\mathcal{A}}\left(\tilde{r}(s,a)+\gamma\int_{\mathcal{S}}V(s')P(\mathrm ds'\mid s,a)-\tau\log\pi(a\mid s)\right)\pi(a\mid s)\mathrm da.\tag{10}\] Under standard measurability and integrability conditions, \(\mathcal{T}^\pi\) is a \(\gamma\)-contraction in \(\|\cdot\|_\infty\) and its unique fixed point is \(V^\pi=\mathcal{T}^\pi V^\pi\). For a bounded candidate value \(V\), write \[Q_V(s,a):=\tilde{r}(s,a)+\gamma\int_{\mathcal{S}}V(s')P(\mathrm ds'\mid s,a).\] Define the soft optimality operator by \[\label{eq:Tstar} (\mathcal{T}^\star V)(s)=\sup_{\mu}\;\int_{\mathbb{R}^d}\bigl(Q_V(s,a)-\tau\log\mu(a)\bigr)\mu(a)\mathrm da,\tag{11}\] where the supremum ranges over action densities for which the expression is finite. The supremum is attained uniquely at a Gibbs density, and therefore \[\label{eq:Tstar95logsumexp} (\mathcal{T}^\star V)(s)=\tau\log\int_{\mathbb{R}^d}\exp\left(\frac{Q_V(s,a)}{\tau}\right)\mathrm da, \qquad \mathcal{G}[V](a\mid s) = \frac{\exp(Q_V(s,a)/\tau)}{\int_{\mathbb{R}^d}\exp(Q_V(s,\tilde{a})/\tau)\mathrm d\tilde{a}}.\tag{12}\] Moreover, \(\mathcal{T}^\star\) is a \(\gamma\)-contraction in \(\|\cdot\|_\infty\) and has a unique bounded fixed point \(V^\star=\mathcal{T}^\star V^\star\). We set \(Q^\star:=Q_{V^\star}\) and \(\pi^\star:=\mathcal{G}[V^\star]\).

For each \((\pi,s)\), define the Gibbs density by \[\label{eq:gibbs95ref95density} p_s^\pi(a) :=\mathcal{G}[V^\pi](a\mid s) = \frac{\exp(Q^\pi(s,a)/\tau)}{\int_{\mathbb{R}^d}\exp(Q^\pi(s,\tilde{a})/\tau)\mathrm d\tilde{a}}, \qquad a\in\mathbb{R}^d.\tag{13}\] The following identity relates the Bellman residual to the KL divergence between the current policy and the Gibbs density.

For every \(s \in \mathcal{S}\), \[\label{eq:residual95identity95ct} (\mathcal{T}^\star V^{\pi})(s) - V^{\pi}(s) = \tau \mathrm{KL}(\pi(\cdot \mid s)\|p_s^{\pi}) \ge 0,\tag{14}\] where \(p_s^{\pi}\) is the Gibbs density defined in 13 .

2.2 Wasserstein Policy Gradient↩︎

Fix a policy \(\pi\). The first-variation calculation in Appendix 6.2 shows that, for statewise transport perturbations of the action law, the ascent direction is \[\nabla_a\bigl(Q^\pi(s,a)-\tau\log\pi(a\mid s)\bigr).\] The discounted occupancy \(d^\pi\) enters this calculation through the usual policy-gradient weighting over states, as in the policy-gradient theorem and trust-region/natural-gradient policy methods [2], [4], [5]. Taking the Wasserstein gradient of this first variation state by state gives the Wasserstein policy-gradient flow \[}\label{eq:FP95actor} \partial_t\pi_t(a\mid s) = -\nabla_a\cdot\left[ \pi_t(a\mid s) \nabla_a\bigl(Q^{\pi_t}(s,a)-\tau\log\pi_t(a\mid s)\bigr) \right].\tag{15}\] Using the Gibbs density \(p_s^{\pi_t}\) in 13 , this can be rewritten as \[\partial_t\pi_t(a\mid s) = \tau\nabla_a\cdot\left( \pi_t(a\mid s) \nabla_a\log\frac{\pi_t(a\mid s)}{p_s^{\pi_t}(a)} \right).\]

From an equivalent particle viewpoint, for each fixed state \(s\), the above Fokker–Planck equation can be viewed formally as the forward equation of the nonlinear Langevin diffusion \[dA_t^s = \nabla_a Q^{\pi_t}(s,A_t^s)\,dt + \sqrt{2\tau}\,dB_t^s, \qquad \pi_t(\cdot\mid s):=\mathrm{Law}(A_t^s).\] Indeed, since \[\tau\nabla_a\log p_s^{\pi_t}(a)=\nabla_a Q^{\pi_t}(s,a),\]

Applying an explicit Euler–Maruyama step to this statewise Langevin diffusion gives the discrete-time update below.

The discrete-time counterpart of 15 is the Langevin policy update \[}\label{eq:ULA95update95dt} A_{k+1}^s = A_k^s+\eta\nabla_a Q^{\pi_k}(s,A_k^s) + \sqrt{2\tau\eta}\,\xi_{k+1}, \qquad \xi_{k+1}\sim N(0,I_d),\tag{16}\] where \(A_k^s\sim\pi_k(\cdot\mid s)\) and \[\pi_{k+1}(\cdot\mid s):=\mathrm{Law}(A_{k+1}^s).\] Thus WPGD is a statewise policy-space update specified for every \(s\in\mathcal{S}\). Under the full-support convention above, the discounted occupancy weighting does not remove any state from the gradient calculation, and the convergence theorem below controls the resulting value gap in sup-norm over the same state space.

3 Continuous-Time Analysis: Bellman Geometry↩︎

This section presents a formal continuous-time analysis of the Wasserstein policy gradient flow 15 . Its purpose is to make transparent the Bellman mechanisms that replace the standard mean-field Langevin dynamics proof chain. We do not claim here a complete well-posedness theorem for the nonlinear Fokker–Planck equation. Instead, the identities below are stated under a smooth-flow regularity convention that supplies the differentiability, integrability, and boundary-decay properties needed for the calculation.

3.0.0.1 Why the standard mean-field Langevin dynamics proof chain does not directly apply.

For a mean-field Langevin dynamics problem, exponential convergence is often proved through the schematic chain [30], [31]: \[\label{eq:standard95proof95chain} \frac{\mathrm d}{\mathrm dt}\mathcal{F}(\mu_t) \overset{\mathrm{(a)}}{=} -\,\mathcal{I}(\mu_t\|\nu_t) \overset{\mathrm{(b)}}{\lesssim} -\,\mathrm{KL}(\mu_t\|\nu_t) \overset{\mathrm{(c)}}{\lesssim} -\,\bigl(\mathcal{F}(\mu_t)-\mathcal{F}^\star\bigr),\tag{17}\] where (a) is the gradient-flow dissipation identity, (b) follows from a uniform LSI, and (c) is through a convex-analysis based comparison from local KL to global suboptimality. In entropy-regularized RL, these relationships do not hold in these forms. First, (a) is altered by the Bellman recursion: differentiating \(V^{\pi_t}\) also differentiates \(Q^{\pi_t}\) through the value function, so the value derivative is not merely a statewise Fisher-information dissipation. Lemma [prop:V95derivative95identity95ct] replaces this step by the value-resolvent identity \((I-\gamma P^{\pi_t})\dot{V}^{\pi_t}=g_t\), with \(g_t\) proportional to a statewise Fisher-information term. Second, (b) is not directly available from a convex functional, because the RL objective is nonconvex due to the Bellman recursion of the state-dependent policy distribution. Lemma [lem:uniform95bounds95ct] and [lem:uniform95LSI95ct] will derive a uniform LSI from uniform value/\(Q\) bounds. Third, (c) cannot be obtained from convexity due to the nonconvexity and aggregation over states in the RL objective. Instead, Lemma [prop:residual95identity95ct] identifies \(\tau\mathrm{KL}(\pi_t(\cdot\mid s)\|p_s^{\pi_t})\) with the statewise Bellman residual, and Bellman contraction in the proof of Theorem [thm:exp95rate] will compare this residual at the worst state directly with the \(\|\cdot\|_\infty\)-value gap.

Smooth-flow regularity convention. Throughout this section only, the flow 15 is considered for smooth positive solutions \(\pi_t(\cdot\mid s)\) satisfying the following properties on every finite time interval. For each state \(s\), \(t\mapsto V^{\pi_t}(s)\) is locally absolutely continuous, the density \(\pi_t(\cdot\mid s)\) has the moments, entropy, Fisher information, and decay at infinity needed for all displayed integrations by parts, and \(a\mapsto Q^{\pi_t}(s,a)\) is sufficiently smooth. Differentiation under the action/state integrals is assumed valid. These assumptions are used only for the formal PDE calculation in this section for illustration and will not be used in the discrete-time analysis in Section 4.

We assume the following throughout:

1. \(|r(s,a)|\le R_{\max}\) for all \((s,a)\in\mathcal{S}\times\mathcal{A}\).

2. \(\sup_{s\in\mathcal{S}}\mathrm{KL}(\pi_0(\cdot\mid s)\|\rho_\beta)\le K_0\), where \(\rho_\beta(a) \propto \exp\left(-\frac{\beta}{2\tau}\|a\|^2\right)\).

Our proof proceeds in three steps. First, we establish a resolvent identity for the value derivative along the flow (Lemma [prop:V95derivative95identity95ct]), which reveals a dissipation given by Fisher information relative to the moving Gibbs family \(p_s^{\pi_t}\) and yields monotone value improvement. Second, we use this monotonicity together with bounded rewards and the initialization condition to obtain uniform-in-time bounds on \(V^{\pi_t}\) and \(Q^{\pi_t}\) (Lemma [lem:uniform95bounds95ct]); these bounds imply that each \(p_s^{\pi_t}\) is a bounded perturbation of a Gaussian, enabling a uniform log-Sobolev inequality along the optimization trajectory via Holley-Stroock Perturbation Lemma (Lemma [lem:uniform95LSI95ct]). Third, we combine the uniform LSI with the Bellman-residual identity, Lemma [prop:residual95identity95ct], and Bellman contraction to convert KL dissipation into an exponential-decay estimate for the optimality gap (Theorem [thm:exp95rate]).

Note that RL policy differs from the standard mean-field Langevin dynamics in that it is concerned with a family of state-dependent policy distributions \(\{\pi(\cdot\mid s)\}_{s\in\mathcal{S}}\) rather than a single distribution. Thus, adapting the usual proof chain   17 to RL settings requires choosing how to aggregate statewise quantities. Another possible route is to aggregate the KL terms in (c) by a discounted state-visitation distribution via the performance difference lemma [lem:performance95difference], as in occupancy-weighted policy-gradient analyses. However, this creates a mismatch between the state distribution used to measure the optimality gap and the state aggregation naturally induced by the value derivative or resolvent identity in (a). Closing this gap typically requires density-ratio or coverage assumptions between state distributions [17], [24], [38], [39]. Our argument above circumvents this route by keeping the Bellman residual identity statewise and using Bellman contraction to lower bound the residual in terms of the sup-norm value gap, and thus does not require additional coverage assumptions.

For a policy \(\pi\), define the induced state kernel \[P^\pi(\mathrm ds'\mid s) := \int_{\mathbb{R}^d} P(\mathrm ds'\mid s,a)\pi(a\mid s)\mathrm da.\]

Let \(\mu\) and \(\nu\) be probability measures on \(\mathbb{R}^d\) with \(\mu\ll \nu\). The Kullback–Leibler divergence of \(\mu\) from \(\nu\) is defined by \[\mathrm{KL}(\mu\|\nu) := \int_{\mathbb{R}^d} \log\frac{d\mu}{d\nu}\,d\mu .\] The relative Fisher information of \(\mu\) with respect to \(\nu\) is defined by \[\mathcal{I}(\mu\|\nu) := \int_{\mathbb{R}^d} \left\| \nabla\log\frac{d\mu}{d\nu} \right\|^2 d\mu .\]

Let \(\nu\) be a probability measure on \(\mathbb{R}^d\) with positive density. We say that \(\nu\) satisfies a log-Sobolev inequality with constant \(\alpha>0\), or \(\alpha\)-LSI, if \[\mathcal{I}(\mu\|\nu) \ge 2\alpha\,\mathrm{KL}(\mu\|\nu), \qquad \forall \mu\ll \nu\] for every probability measure \(\mu\) for which the two sides are well-defined.

We first establish an identity for the time derivative of \(V^{\pi_t}\) along the flow 15 .

Under Assumption [ass:ct-standing] and the smooth-flow regularity convention, for every \(t\ge 0\), \[\label{eq:V95derivative95resolvent} (I-\gamma P^{\pi_t})\,\frac{\mathrm d}{\mathrm dt}V^{\pi_t} = g_t, \qquad g_t(s) := \tau^2 \mathcal{I}\big(\pi_t(\cdot\mid s)\|p_s^{\pi_t}\big)\ge0,\tag{18}\] where \(\mathcal{I}\big(\pi_t(\cdot\mid s)\|p_s^{\pi_t}\big)\) is the relative Fisher information. Moreover, \[\label{eq:dotV95ge95g} \frac{\mathrm d}{\mathrm dt}V^{\pi_t}(s)\ge g_t(s)\ge0 \qquad \forall s.\tag{19}\]

Lemma [prop:V95derivative95identity95ct] shows that the value derivative satisfies \[\frac{\mathrm d}{\mathrm dt}V^{\pi_t} = (I-\gamma P^{\pi_t})^{-1}g_t\] and is therefore pointwise nonnegative along the flow 15 . Note that this identity can be viewed as the statewise resolvent form of the entropy-regularized performance-difference lemma: integrating it against an initial distribution recovers the usual occupancy-weighted performance-difference formula, whereas keeping it pointwise is what enables the subsequent sup-norm contraction argument in 24 below without introducing state-distribution mismatch coefficients. Based on Lemma [prop:V95derivative95identity95ct], we derive uniform-in-time bounds on the value function, which further imply uniform bounds on the action-value Q-function.

Under Assumption [ass:ct-standing] and the smooth-flow regularity convention above, for all \(t\ge0\), \[\label{eq:V95upper95bound} \sup_{s\in\mathcal{S}}V^{\pi_t}(s) \le \frac{R_{\max}+\tau\log Z_\beta}{1-\gamma}.\tag{20}\] Moreover, \[\label{eq:V095lower95bound} \inf_{s\in\mathcal{S}}V^{\pi_t}(s) \ge \inf_{s\in\mathcal{S}}V^{\pi_0}(s) \ge \frac{-R_{\max}+\tau\log Z_\beta-\tau K_0}{1-\gamma}.\tag{21}\] Consequently, for all \(t\ge0\), \(s\in\mathcal{S}\), and \(a\in\mathcal{A}\), \[\label{eq:Q95pointwise95bounds} -R_{\max}-\frac{\beta}{2}\|a\|^2+\gamma\inf_{x\in\mathcal{S}}V^{\pi_0}(x) \le Q^{\pi_t}(s,a) \le R_{\max}-\frac{\beta}{2}\|a\|^2+\gamma\sup_{x\in\mathcal{S}}V^{\pi_t}(x).\tag{22}\]

Thanks to the bound 22 on the Q-function, the Gibbs densities \(p_s^{\pi_t}\) are bounded perturbations of Gaussian densities, and thus satisfy log-Sobolev inequalities with uniform constants along the optimization trajectory due to the Holley-Stroock perturbation lemma.

Define \[V_{\max} = \max\left\{ \frac{R_{\max}+\tau\log Z_\beta}{1-\gamma}, \frac{R_{\max}-\tau\log Z_\beta+\tau K_0}{1-\gamma} \right\}\] and \[U_{\max}=R_{\max}+\gamma V_{\max}.\] Then for every \(t\ge0\) and \(s\in\mathcal{S}\), the Gibbs density \(p_s^{\pi_t}\) satisfies the log-Sobolev inequality \[\mathcal{I}(\nu\|p_s^{\pi_t}) \ge 2\alpha\,\mathrm{KL}(\nu\|p_s^{\pi_t}) \qquad \forall \nu\ll p_s^{\pi_t},\] with the uniform constant \[\alpha = \frac{\beta}{\tau} \exp\left(-\frac{2U_{\max}}{\tau}\right).\]

We now state the exponential-decay consequence of the preceding identities. Let \(V^\star\) be the unique fixed point of \(\mathcal{T}^\star\), and let \(\pi^\star\) denote the optimal Gibbs policy attaining \(V^\star\). Since \(V^\star\) is the optimal value function and the flow remains in the admissible policy class, \(V^\star(s)\ge V^{\pi_t}(s)\) for all \(s,t\). Define the residual \[R_t(s) := (\mathcal{T}^\star V^{\pi_t})(s)-V^{\pi_t}(s)\ge0.\] By Lemma [prop:residual95identity95ct], \[\label{eq:residual95as95KL} R_t(s) = \tau \mathrm{KL}\big(\pi_t(\cdot\mid s)\|p_s^{\pi_t}\big).\tag{23}\] Define \[W_t(s) := V^\star(s)-V^{\pi_t}(s)\ge0, \qquad E(t):=\|W_t\|_\infty.\] Since \(V^\star=\mathcal{T}^\star V^\star\), \(\mathcal{T}^\star\) is monotone, and \(\mathcal{T}^\star\) is a \(\gamma\)-contraction in \(\|\cdot\|_\infty\), \[\label{eq:delta95contraction} 0 \le V^\star(s)-(\mathcal{T}^\star V^{\pi_t})(s) \le \gamma E(t) \qquad \forall s\in\mathcal{S},\;t\ge0.\tag{24}\]

Under Assumption [ass:ct-standing] and the smooth-flow regularity convention above, for all \(t\ge0\), \[J(\pi^\star)-J(\pi_t) \le \|V^\star-V^{\pi_t}\|_\infty \le e^{-2\alpha\tau(1-\gamma)t} \|V^\star-V^{\pi_0}\|_\infty.\]

Theorem [thm:exp95rate] is a formal continuous-time convergence for smooth solutions of 15 . It identifies the rate obtained from the value-resolvent identity, uniform value/\(Q\) bounds, the uniform LSI, and the statewise Bellman-residual identity. Its role is explanatory: it isolates the Bellman-residual mechanism and motivates the discrete-time argument.

After the first version of this paper was submitted for review, we became aware of a closely related note by [40] for finite-state continuous-action problems. Their analysis and resulting guarantee are structurally different from ours. In their cost-minimization notation, the argument uses an entropy-sandwich inequality to control the scalar distributional gap \(V^{\pi_t}(\rho)-V^{\pi^\star}(\rho)\) and states a bound of the form \[V^{\pi_t}(\rho)-V^{\pi^\star}(\rho) \le \bar\kappa \left( \int_{\mathcal{S}} \bigl(V^{\pi_0}(s)-V^{\pi^\star}(s)\bigr) \,d_\rho^{\pi^\star}(ds) \right) e^{-2\kappa\alpha\tau(1-\gamma) t},\] where \(\bar\kappa := \sup_{s\in\mathcal{S}} \frac{d\rho}{d d_\rho^{\pi^\star}}(s)\), \(\kappa := \inf_{s\in\mathcal{S}} \frac{d\rho}{d d_\rho^{\pi^\star}}(s)\le 1.\) Thus their estimate is distribution-dependent and requires coverage-type density-ratio control, in particular \(\kappa>0\), to obtain a positive exponential rate. By contrast, Theorem [thm:exp95rate] proves a state-uniform bound and therefore controls \(J(\pi^\star)-J(\pi_t)\) for any initial state distribution. The proof mechanisms are also different: rather than passing through an occupancy-weighted entropy sandwich, we use the exact statewise Bellman-residual identity and Bellman contraction to compare the residual directly with the \(\|\cdot\|_\infty\)-optimality gap. This removes the density-ratio constants \(\kappa\) from the exponential convergence rate, although with a different initial gap. Finally, both their and our analyses involve formal calculations; the main rigorous guarantee in the present paper is the discrete-time Langevin analysis in Section 4.

4 Discrete-Time Global Convergence Analysis↩︎

This section proves the non-asymptotic convergence guarantee for the explicit Langevin update 16 . The continuous-time section isolates the Bellman residual/resolvent mechanism. Apart from their counterparts in discrete time, an additional task is to control the finite-step Langevin discretization and the policy-dependent drift uniformly along the iterates.

4.1 Assumptions↩︎

The assumptions below are the quantitative inputs for the finite-step analysis. To apply the one step drift interpolation argument, we need uniform control of the drift Lipschitz constants, action moments, drift second moments, finite KL quantities, and the LSI constants of the Gibbs family. Since \[b_k(s,a):=\nabla_a Q^{\pi_k}(s,a)=\nabla_a r(s,a)-\beta a+ \gamma\int_{\mathcal{S}}V^{\pi_k}(s')\nabla_a p(s'\mid s,a)\lambda(\mathrm ds'),\] these constants cannot be read off from the Wasserstein gradient of a static functional; they must be derived from the Bellman recursion and propagated along the WPG iterates. Assumptions [ass:standing]-[ass:action-regularity], together with the a priori bounds that will be proved in Proposition [prop:WPG95uniform95stability], provide this uniform control.

We assume:

1. \(|r(s,a)|\le R_{\max}\) for all \((s,a)\in\mathcal{S}\times\mathbb{R}^d\).

2. \(\sup_{s\in\mathcal{S}}\mathrm{KL}(\pi_0(\cdot\mid s)\|\rho_\beta)\le K_0\).

For constants used later, set \[M_0:=\sup_{s\in\mathcal{S}}\mathbb{E}\|A_0^s\|^2, \qquad A_0^s\sim\pi_0(\cdot\mid s).\] This quantity is finite by Lemma [lem:gaussian-KL-second-moment] applied uniformly in \(s\). In particular, applying that lemma with \(c=\beta/(4\tau)\) and using Assumption [ass:standing](ii) gives \[\label{eq:M095explicit95bound} M_0 \le \frac{4\tau}{\beta} \left(K_0+\log\int_{\mathbb{R}^d} \exp\left(\frac{\beta}{4\tau}\|a\|^2\right)\rho_\beta(a)\,da\right) = \frac{4\tau}{\beta}\left(K_0+\frac{d}{2}\log 2\right).\tag{30}\]

There is a \(\sigma\)-finite reference measure \(\lambda\) on \(\mathcal{S}\) such that \(P(\mathrm ds'\mid s,a)=p(s'\mid s,a)\lambda(\mathrm ds')\). There are finite constants \(L_r,L_p,G_r,G_p\) such that:

1. For every \(s\in\mathcal{S}\), the map \(a\mapsto r(s,a)\) is continuously differentiable and \[\|\nabla_a r(s,a)\|\le G_r, \qquad \|\nabla_a r(s,a)-\nabla_a r(s,\bar a)\|\le L_r\|a-\bar a\|.\]

2. For every \(s\in\mathcal{S}\) and \(\lambda\)-a.e. \(s'\in\mathcal{S}\), the map \(a\mapsto p(s'\mid s,a)\) is continuously differentiable, and \[\begin{align} \int_{\mathcal{S}}\|\nabla_a p(s'\mid s,a)\|\lambda(\mathrm ds')&\le G_p,\\ \int_{\mathcal{S}}\|\nabla_a p(s'\mid s,a)-\nabla_a p(s'\mid s,\bar a)\|\lambda(\mathrm ds') &\le L_p\|a-\bar a\|. \end{align}\]

A concrete example satisfying the transition part of Assumption [ass:action-regularity] is the following additive-noise model. Let \(\mathcal{S}=\mathbb{R}^m\), let \(\lambda\) be Lebesgue measure, and suppose \[S'=F(s,a)+\sigma\zeta,\] where \(\sigma\in\mathbb{R}^{m\times m}\) is invertible and \(\zeta\) has a smooth density \(q\). Write \[q_\sigma(x):=|\det\sigma|^{-1}q(\sigma^{-1}x)\] for the density of \(\sigma\zeta\), and assume \[\|\nabla q_\sigma\|_{L^1}<\infty, \qquad \|D^2q_\sigma\|_{L^1}:= \int_{\mathbb{R}^m}\|D^2q_\sigma(x)\|_{\mathrm{op}}\,\mathrm dx<\infty .\] Assume further that, uniformly in \(s\), \[\|D_aF(s,a)\|_{\mathrm{op}}\le M_1, \qquad \|D_aF(s,a)-D_aF(s,\bar a)\|_{\mathrm{op}}\le M_2\|a-\bar a\|.\] Then \[p(y\mid s,a)=q_\sigma(y-F(s,a))\] satisfies Assumption [ass:action-regularity](ii). Indeed, \[\nabla_a p(y\mid s,a) = -D_aF(s,a)^\top \nabla q_\sigma(y-F(s,a)),\] so one may take \[G_p=M_1\|\nabla q_\sigma\|_{L^1}, \qquad L_p=M_2\|\nabla q_\sigma\|_{L^1}+M_1^2\|D^2q_\sigma\|_{L^1}.\] Nondegenerate Gaussian noise is a special case, since its first derivative and Hessian are integrable. Together with the reward condition in Assumption [ass:action-regularity](i), this gives a concrete class of controlled transition models covered by the theorem. Deterministic or degenerate-noise dynamics are not covered by this example.

With these assumptions in place, write \[W_k:=V^\star-V^{\pi_k},\qquad E_k:=\|W_k\|_\infty,\] and define the current-policy Langevin drift \[\label{eq:bk95pk95def95dt} b_k(s,a):=\nabla_a Q^{\pi_k}(s,a).\tag{31}\] The current-policy Gibbs stationarity condition is \[\label{eq:stationarity95pk95dt} \tau\nabla_a\log p_s^{\pi_k}(a)=\nabla_a Q^{\pi_k}(s,a)=b_k(s,a).\tag{32}\]

4.2 Proof sketch and main theorem↩︎

4.2.1 Bellman residual-to-optimality gap mechanism and uniform LSI↩︎

We first record the discrete counterparts of the residual and resolvent mechanisms from Section 3. They show that the KL to the current-policy Gibbs law is exactly the Bellman residual, that this residual controls the worst-state value gap, and that decreasing the same KL produces value improvement after applying the Bellman resolvent.

Fix \(k\ge 0\). Let \(R_k:=(\mathcal{T}^\star V^{\pi_k})-V^{\pi_k}\). Then, for every \(s\in\mathcal{S}\), \[R_k(s)\ge W_k(s)-\gamma E_k.\]

Fix \(k\ge 0\). Suppose \(\pi_k\) and the WPG update \(\pi_{k+1}\) are admissible, with measurability covered by the convention in Section 2, and the KL terms below are finite. Then \[\label{eq:V95increment95resolvent95dt} (I-\gamma P^{\pi_{k+1}})\big(V^{\pi_{k+1}}-V^{\pi_k}\big)=g_k, \qquad g_k(s):=(\mathcal{T}^{\pi_{k+1}}V^{\pi_k})(s)-V^{\pi_k}(s),\tag{33}\] and, for every \(s\in\mathcal{S}\), \[\label{eq:gk95as95KL95dt} g_k(s)= \tau\Big( \mathrm{KL}(\pi_k(\cdot\mid s)\|p_s^{\pi_k})- \mathrm{KL}(\pi_{k+1}(\cdot\mid s)\|p_s^{\pi_k}) \Big).\tag{34}\]

As a consequence of these two lemmas, once \(p_s^{\pi_k}\) satisfies an LSI with constant \(\bar\alpha\), the Bellman residual yields the following PŁ-type residual-to-gap inequality at an \(\varepsilon\)-maximizer of \(W_k\): \[\label{eq:bellman95PL95main} \mathcal{I}(\pi_k(\cdot\mid s)\|p_s^{\pi_k}) \ge 2\bar\alpha\mathrm{KL}(\pi_k(\cdot\mid s)\|p_s^{\pi_k}) =\frac{2\bar\alpha}{\tau}R_k(s) \ge \frac{2\bar\alpha}{\tau}\big((1-\gamma)E_k-\varepsilon\big).\tag{35}\] This is a PŁ inequality for the Bellman gap, not a consequence of flat convexity of \(J\).

The LSI used in 35 follows from the same Gaussian perturbation mechanism as in continuous time, but it must be made uniform along the discrete trajectory. The a priori bounds prove a uniform value bound; once \(\|V^{\pi_k}\|_\infty\le V_{\max}\), \[\label{eq:pk95bounded95tilt95main} p_s^{\pi_k}(a)=\frac{e^{\psi_{k,s}(a)}\rho_\beta(a)}{\int e^{\psi_{k,s}(\tilde{a})}\rho_\beta(\tilde{a})\mathrm d\tilde{a}}, \qquad \psi_{k,s}(a):=\frac{r(s,a)+\gamma\int V^{\pi_k}(s')P(\mathrm ds'\mid s,a)}{\tau}.\tag{36}\] The perturbation \(\psi_{k,s}\) may be nonconvex, but bounded rewards and the value bound give \(\|\psi_{k,s}\|_\infty\le (R_{\max}+\gamma V_{\max})/\tau\) and hence \(\mathrm{osc}(\psi_{k,s})\le 2(R_{\max}+\gamma V_{\max})/\tau\). The Holley-Stroock bounded-perturbation principle [41] then transfers the Gaussian LSI of \(\rho_\beta\) to every \(p_s^{\pi_k}\) uniformly.

Fix \(k\ge 0\) and suppose \(\|V^{\pi_k}\|_\infty\le V_{\max}\). Then, for every \(s\in\mathcal{S}\), the current-policy Gibbs density \(p_s^{\pi_k}\) satisfies \[\mathcal{I}(\mu\|p_s^{\pi_k})\ge 2\alpha(V_{\max})\mathrm{KL}(\mu\|p_s^{\pi_k}) \qquad\forall\mu\ll p_s^{\pi_k},\] where \[\label{eq:alpha95of95Vmax95dt} \alpha(V_{\max})=\frac{\beta}{\tau}\exp\left(-\frac{2(R_{\max}+\gamma V_{\max})}{\tau}\right).\tag{37}\]

4.2.2 Drift regularity and the discretization error↩︎

We now control the finite-step error introduced by the Langevin discretization. As explained in Section 4.1, the drift Langevin interpolation requires uniform action-Lipschitz and dissipativity estimates for the policy-dependent drift. The following lemma converts the action-regularity assumptions and a value bound into those estimates.

Assume Assumption [ass:action-regularity]. Fix \(k\ge0\) and suppose \(\|V^{\pi_k}\|_\infty\le V_{\max}<\infty\). Then, for every \(s\in\mathcal{S}\), \[\label{eq:dissipative95decomp95dt} b_k(s,a)=-\beta a+h_k(s,a),\tag{38}\] with \[\label{eq:hk95bound95dt} \sup_{s,a}\|h_k(s,a)\|\le G(V_{\max}):=G_r+\gamma G_pV_{\max}.\tag{39}\] Moreover, for all \(a,\bar a\in\mathbb{R}^d\), \[\|b_k(s,a)-b_k(s,\bar a)\| \le L_b(V_{\max})\|a-\bar a\|, \qquad L_b(V_{\max}):=\beta+L_r+\gamma L_pV_{\max}.\]

The Lipschitz bound controls the Euler interpolation error; the dissipative decomposition controls the second moment and hence the squared drift moment. Together with the LSI from Lemma [lem:uniform95LSI95dt], they give the only place where discrete time loses exact monotonicity. The analytic part of the next lemma is based on a one-step fixed-drift interpolation. The full calculation is given in Appendix 9.2.

Fix \(k\ge 0\) and \(s\in\mathcal{S}\). Suppose \(p_s^{\pi_k}>0\), \(\log p_s^{\pi_k}\in C^1(\mathbb{R}^d)\), \(p_s^{\pi_k}\) satisfies an LSI with constant \(\alpha>0\), and the Gibbs-score identity \[\tau\nabla_a\log p_s^{\pi_k}(a)=b_k(s,a) \qquad\text{for all }a\in\mathbb{R}^d\] holds. Suppose also that \(b_k(s,\cdot)\) is \(L_b\)-Lipschitz and \[\mathbb{E}_{a\sim\pi_k(\cdot\mid s)}\|a\|^2<\infty, \qquad \mathbb{E}_{a\sim\pi_k(\cdot\mid s)}\|b_k(s,a)\|^2\le B^2, \qquad \mathrm{KL}(\pi_k(\cdot\mid s)\|p_s^{\pi_k})<\infty.\] Set \[\delta_\eta(L_b,B^2):=\frac{L_b^2 d}{2}\eta^2+\frac{L_b^2B^2}{6\tau}\eta^3.\] Then \[\label{eq:KL95contraction95step} \mathrm{KL}\big(\pi_{k+1}(\cdot\mid s)\|p_s^{\pi_k}\big) \le e^{-\alpha\tau\eta} \mathrm{KL}\big(\pi_k(\cdot\mid s)\|p_s^{\pi_k}\big)+\delta_\eta(L_b,B^2).\tag{40}\] Fix \(k\ge0\) and \(s\in\mathcal{S}\). Suppose the admissibility and finite-KL hypotheses of Lemma [lem:V95increment95resolvent95dt] hold for \((\pi_k,\pi_{k+1})\). If, for some \(c\in[0,1]\) and \(\delta\ge0\), \[\label{eq:abstract95KL95contraction95to95g95assumption} \mathrm{KL}(\pi_{k+1}(\cdot\mid s)\|p_s^{\pi_k}) \le (1-c)\mathrm{KL}(\pi_k(\cdot\mid s)\|p_s^{\pi_k})+\delta,\tag{41}\] then, with \(R_k(s)=\tau\mathrm{KL}(\pi_k(\cdot\mid s)\|p_s^{\pi_k})\) and \(g_k\) as in Lemma [lem:V95increment95resolvent95dt], \[\label{eq:gk95lower95from95KL95contraction} g_k(s)\ge cR_k(s)-\tau\delta.\tag{42}\] In particular, Lemma [lem:KL95contraction95frozen] supplies 41 with \(c=1-e^{-\alpha\tau\eta}\) and \(\delta=\delta_\eta(L_b,B^2)\) whenever its analytic hypotheses also hold.

With the uniform constants supplied by Proposition [prop:WPG95uniform95stability], the preceding lemmas yield a closed one-step recursion for the sup-norm gap. Throughout this argument, \(\bar\alpha\) denotes the trajectory-uniform LSI constant, \(\bar L_b\) the uniform Lipschitz constant of the drift, \(\bar B^2\) the uniform second-moment bound for the drift, and \(\bar\delta_\eta\) the corresponding one-step Langevin discretization error. Set \[c_\eta := 1-e^{-\bar\alpha\tau\eta}.\] Then, for every \(s\in\mathcal{S}\), the estimates compose as follows: \[\label{eq:four-line-proof-chain-main} \begin{align} \mathrm{KL}(\pi_{k+1}(\cdot\mid s)\|p_s^{\pi_k}) &\le (1-c_\eta)\mathrm{KL}(\pi_k(\cdot\mid s)\|p_s^{\pi_k}) +\bar\delta_\eta \\[-1mm] & \text{by Lemma~\ref{lem:KL95contraction95frozen},} \\[1mm] &\Longrightarrow\quad g_k(s)\ge c_\eta R_k(s)-\tau\bar\delta_\eta \\[-1mm] & \text{by Lemma~\ref{lem:KL95contraction95bellman95consequence},} \\[1mm] &\Longrightarrow\quad V^{\pi_{k+1}}(s)-V^{\pi_k}(s) \ge c_\eta R_k(s)-\frac{\tau}{1-\gamma}\bar\delta_\eta \\[-1mm] & \text{by Lemma~\ref{lem:V95increment95resolvent95dt},} \\[1mm] &\Longrightarrow\quad E_{k+1} \le \bigl(1-(1-\gamma)c_\eta\bigr)E_k +\frac{\tau}{1-\gamma}\bar\delta_\eta \\[-1mm] & \text{by Lemma~\ref{lem:residual95lower95epsmax95dt}.} \end{align}\tag{43}\]

Here the KL terms are statewise, \(R_k=\mathcal{T}^\star V^{\pi_k}-V^{\pi_k}\), and \(E_k=\|V^\star-V^{\pi_k}\|_\infty\). The substantive RL step is the middle of 43 : a local KL decrease becomes a global value-gap contraction through the Bellman residual and the value resolvent, not through flat convexity of a probability functional. The implications in 43 reduce the convergence proof to estimates that hold uniformly along the WPGD trajectory. The constants entering this recursion are policy dependent, since both \(p_s^{\pi_k}\) and \(b_k\) are defined through \(Q^{\pi_k}\) and hence through the Bellman fixed point \(V^{\pi_k}\). The next proposition supplies the required a priori estimates. Approximate value monotonicity controls the lower value envelope, the quadratic action penalty gives uniform action-moment bounds, and these estimates feed back into uniform drift regularity, finite KL controls, and a trajectory-uniform LSI for the moving Gibbs family.

Under Assumptions [ass:standing] and [ass:action-regularity], there exist explicit finite constants \[\bar V,\quad \bar L_b,\quad \bar G,\quad \bar M,\quad \bar B,\quad \bar\alpha>0, \quad \eta_0>0,\] depending only on the problem data and the initialization such that, for every \(0<\eta\le\eta_0\), the WPG iterates are admissible and, for all \(k\ge0\), \[\|V^{\pi_k}\|_\infty\le\bar V, \qquad \sup_{s\in\mathcal{S}}\mathbb{E}\|A_k^s\|^2\le \bar M, \qquad \sup_{s\in\mathcal{S}}\mathbb{E}_{\pi_k}\|\nabla_a Q^{\pi_k}(s,A)\|^2\le \bar B^2.\] Moreover, \(a\mapsto \nabla_a Q^{\pi_k}(s,a)\) is \(\bar L_b\)-Lipschitz uniformly in \((k,s)\), every current-policy Gibbs density \(p_s^{\pi_k}\) satisfies LSI with constant at least \(\bar\alpha\), and the generated policies satisfy \[\begin{align} \sup_{k\ge0}\sup_{s\in\mathcal{S}}\mathrm{KL}(\pi_k(\cdot\mid s)\|\rho_\beta)&\le \bar K_\eta,\tag{44}\\ \sup_{k\ge0}\sup_{s\in\mathcal{S}}\mathrm{KL}(\pi_k(\cdot\mid s)\|p_s^{\pi_k})&\le \bar H_\eta.\tag{45} \end{align}\] All constants and the feasible step-size range are given in Appendix 8.

Proposition [prop:WPG95uniform95stability] provides the uniform constants needed to run the one-step recursion in 43 at every iteration. The value bound yields a common Holley–Stroock perturbation bound and hence a uniform LSI constant \(\bar\alpha\); the drift Lipschitz and second-moment estimates give a uniform discretization error \(\bar\delta_\eta=O(\eta^2)\); and the admissibility and finite-KL bounds ensure that the resolvent identity and the one step drift interpolation estimate apply throughout the trajectory. Substituting these uniform estimates into 43 gives a closed scalar recursion for the sup-norm optimality gap, which yields the geometric contraction up to the accumulated Langevin discretization bias stated in the main theorem.

Under Assumptions [ass:standing] and [ass:action-regularity], there exist explicit finite constants \(\bar\alpha,C_\delta,\eta_0>0\), such that for every \(0<\eta\le\eta_0\) and every \(k\ge0\), \[\label{eq:main95discrete95bound} J(\pi^\star)-J(\pi_k) \le \|V^\star-V^{\pi_k}\|_\infty \le \exp\Big(-\tfrac12\bar\alpha\tau(1-\gamma)\eta k\Big) \|V^\star-V^{\pi_0}\|_\infty + \frac{2C_\delta}{\bar\alpha(1-\gamma)^2}\,\eta.\tag{46}\]

Theorem [thm:main95discrete] gives a finite-step geometric contraction for the WPGD update 16 , up to a fixed-step discretization bias. The drift KL estimate has an additive error \(\bar\delta_\eta=O(\eta^2)\), and the scalar gap recursion divides this error by \(1-e^{-\bar\alpha\tau\eta}\asymp \bar\alpha\tau\eta\). The proof in Appendix 9.4 keeps the sharper quantity \(\bar\delta_\eta\) throughout and substitutes \(\bar\delta_\eta\le C_\delta\eta^2\) only at the end.

The theorem is closest in form to discrete-time mean-field Langevin guarantees, but the source of the contraction is different. Mean-field convex analyses convert KL decrease to objective decrease through flat convexity of a distributional objective [30], [31], [42]. Here the Gibbs law \(p_s^{\pi_k}\) moves with the policy through the soft Bellman equation that depends on both the policy gradient and the KL regularization. The required PŁ-type relation is therefore reconstructed from the RL-specific ingredients.

The result is also complementary to entropy-regularized natural policy gradient and policy mirror descent, where KL/Bregman geometry gives a closed-form policy-improvement step and Bellman monotonicity [16], [19], [20], [25], [26]. WPGD instead uses Wasserstein geometry: local progress is Fisher-information dissipation in action space, and the main technical burden is to show that this transport-based dissipation remains uniformly controlled under the policy-dependent Bellman recursion and relate it to the global optimality gap.

5 Conclusion↩︎

We developed a convergence analysis framework for Wasserstein policy gradient in entropy-regularized discounted RL with continuous actions. The proof combines Bellman-residual identities, value-resolvent relation, one-step interpolation, uniform value/Q-function and moment bounds, and uniform LSI for the iterated Gibbs family. These ingredients yield exponential convergence for the continuous-time WPG flow and geometric convergence for the discrete-time WPG Langevin dynamics up to a discretization bias. An important direction for future work is to integrate these policy-space guarantees with practical approximations, such as projection onto parametric policy classes and finite-sample critic estimation.

6 Auxiliary Results↩︎

6.1 Entropy-Regularized Performance Difference Lemma↩︎

[25] Let \(\rho_0\in\mathcal{P}(S)\) be the initial state distribution, and write \(J(\pi):=J_{\rho_0}(\pi)=\int_S V^\pi(s)\rho_0(ds)\). For any two feasible policies \(\pi\) and \(\pi'\), we have \[J(\pi')-J(\pi) = \frac{1}{1-\gamma}\int_S \left[ \int_{\mathbb{R}^d} Q^\pi(s,a)\bigl(\pi'(a\mid s)-\pi(a\mid s)\bigr)\,da -\tau\mathsf{H}^{\pi'}(s)+\tau\mathsf{H}^\pi(s) \right]d_{\rho_0}^{\pi'}(ds),\] where \[\mathsf{H}^\pi(s):=\int_{\mathbb{R}^d}\pi(a\mid s)\log\pi(a\mid s)\,da\] is the statewise negative entropy, and \[d_{\rho_0}^{\pi'}(B):=(1-\gamma)\sum_{t=0}^\infty \gamma^t\mathbb{P}_{\rho,\pi'}(s_t\in B),\qquad B\in\mathcal{B}(S),\] is the normalized discounted state occupancy measure of \(\pi'\) initialized from \(\rho\).

For completeness, we provide the proof here and adapt it to our notation.

6.2 Derivation of the first variation and Wasserstein policy-gradient direction↩︎

We justify the statewise direction used in Section 2.2. Throughout this subsection, we write \(J(\pi)=J_{\rho_0}(\pi)\) for the entropy-regularized objective with initial distribution \(\rho_0\). Fix a policy \(\pi\), and let \(d^\pi=d_{\rho_0}^\pi\) be its normalized discounted state occupancy.

For each state \(s\), consider a smooth transport perturbation \(t\mapsto \pi_t(\cdot\mid s)\) with \(\pi_0=\pi\) and velocity field \(u(s,\cdot)\), so that \[\label{eq:app95transport95perturbation} \partial_t\pi_t(a\mid s)\big|_{t=0} =-\nabla_a\cdot\bigl(\pi(a\mid s)u(s,a)\bigr).\tag{47}\] The following proposition identifies the first variation of \(J\) along such statewise perturbations.

Let \(t\mapsto \pi_t\) be a smooth curve of feasible policies with \(\pi_0=\pi\), and write \[\dot{\pi}(a\mid s):=\partial_t\pi_t(a\mid s)\big|_{t=0}.\] Then \[\label{eq:app95first95variation95density} \frac{\mathrm d}{\mathrm dt}J(\pi_t)\bigg|_{t=0} = \frac{1}{1-\gamma}\int_{\mathcal{S}}\int_{\mathbb{R}^d} \bigl(Q^\pi(s,a)-\tau\log\pi(a\mid s)\bigr)\dot{\pi}(a\mid s) \,da\,d^\pi(ds).\tag{48}\] If, in addition, \(\dot{\pi}\) is generated by the transport perturbation 47 , then \[\label{eq:app95first95variation95velocity} \frac{\mathrm d}{\mathrm dt}J(\pi_t)\bigg|_{t=0} = \frac{1}{1-\gamma}\int_{\mathcal{S}}\int_{\mathbb{R}^d} \left\langle \nabla_a\bigl(Q^\pi(s,a)-\tau\log\pi(a\mid s)\bigr),u(s,a) \right\rangle \pi(a\mid s)\,da\,d^\pi(ds).\tag{49}\]

Thus, the action-space gradient direction associated with the first variation is \[\label{eq:app95wpg95direction} \nabla_a\bigl(Q^\pi(s,a)-\tau\log\pi(a\mid s)\bigr).\tag{50}\] Equivalently, one may view 49 as taking the Wasserstein gradient with respect to the statewise \(2\)-Wasserstein geometry on the conditional action laws, averaged over states using the policy-gradient weight \(d^\pi\). This is the standard Otto-calculus interpretation of gradients on probability measures [43], [44]; the appearance of \(d^\pi\) is the standard discounted state weighting from the policy-gradient theorem [2], and analogous visitation-weighted averages appear in natural-gradient and trust-region policy optimization [4], [5].

Under the full-support convention on \(\rho\), the induced occupancy \(d^\pi\) has full support, so this statewise direction is determined on all states relevant to the sup-norm analysis. Substituting the direction 50 into the continuity equation gives the Wasserstein policy-gradient flow: \[\partial_t\pi_t(a\mid s) = -\nabla_a\cdot\left( \pi_t(a\mid s)\nabla_a \bigl(Q^{\pi_t}(s,a)-\tau\log\pi_t(a\mid s)\bigr) \right),\] which is 15 .

7 Proofs for Section 3↩︎

8 Uniform regularity bounds↩︎

This appendix verifies the quantitative bounds needed to use the core Bellman proof uniformly along WPGD. Pure measurability of kernels and conditional versions remains covered by the measurability convention in Section 2; Normalizability, Bellman differentiability, value bounds, drift bounds, moments, KL bounds, and LSI are proved below.

8.1 Explicit constants and feasible step sizes↩︎

For \(V_{\max}\ge0\), define \[L_b(V_{\max}) := \beta+L_r+\gamma L_p V_{\max}, \qquad G(V_{\max}) := G_r+\gamma G_pV_{\max}.\] Let \[U:=\frac{R_{\max}+\tau\log Z_\beta}{1-\gamma}, \quad L_\star:=\frac{-R_{\max}+\tau\log Z_\beta}{1-\gamma}, \quad \bar E_{0}:=\frac{2R_{\max}+\tau K_0}{1-\gamma},\] and set \[\label{eq:self95consistent95Vbar} \bar V:=\max\{1,\;U,\;\bar E_{0}-L_\star+1\}.\tag{55}\] The induced constants are \[\bar L_b:=L_b(\bar V), \qquad \bar G:=G(\bar V), \qquad \bar\alpha:=\frac{\beta}{\tau}\exp\left(-\frac{2(R_{\max}+\gamma\bar V)}{\tau}\right), \qquad c_\eta:=1-e^{-\bar\alpha\tau\eta}.\] With \(M_0=\sup_s\mathbb{E}\|A_0^s\|^2\), define \[\bar M_\infty(\eta):=\frac{2}{\beta}\left(\frac{\bar G^2}{\beta}+2\tau d\right)+\frac{4\bar G^2}{\beta}\eta, \qquad B^2:=2\beta^2\max\{M_0\bar M_\infty(\eta)\}+2\bar G^2\]

The statewise one-step KL discretization error is \[\label{eq:delta95bar95eta} \bar\delta_\eta:=\frac{\bar L_b^2d}{2}\eta^2+\frac{\bar L_b^2B_\infty^2}{6\tau}\eta^3.\tag{56}\] The auxiliary KL bounds are \[\label{eq:Kbar95eta95def} \bar K_\eta :=\max\left\{K_0,\;0,\;\frac{\beta M_\infty}{2\tau}+\log Z_\beta-\frac{d}{2}\log(4\pi e\tau\eta)\right\},\tag{57}\] \[\label{eq:Hbar95eta95def} \bar H_\eta:=\bar K_\eta+\frac{2(R_{\max}+\gamma\bar V)}{\tau}.\tag{58}\] The exact stability condition used in Proposition [prop:WPG95uniform95stability] is \[\label{eq:uniform95stepsize95varyingK} 0<\eta \le \frac{1}{4\beta}, \qquad \bar\alpha\tau\eta \le 1, \qquad \frac{\tau}{(1-\gamma)^2}\frac{\bar\delta_\eta}{c_\eta} \le 1.\tag{59}\] For the simplified theorem statement, let \[\bar M:= \max\left\{M_0,\frac{3\bar G^2}{\beta^2}+\frac{4\tau d}{\beta}\right\}, \qquad \bar B^2:=2\beta^2\bar M+2\bar G^2,\] \[\label{eq:Cdelta95def} C_\delta:=\frac{\bar L_b^2d}{2}+\frac{\bar L_b^2\bar B^2}{6\tau},\tag{60}\] and \[\label{eq:eta095def} \eta_0:=\min\left\{1,\frac{1}{4\beta},\frac{1}{\bar\alpha\tau},\frac{\bar\alpha(1-\gamma)^2}{2C_\delta}\right\}.\tag{61}\] For every \(0<\eta\le\eta_0\), Lemma [lem:stepsize-feasibility] implies 59 , and 56 satisfies \(\bar\delta_\eta\le C_\delta\eta^2\).

8.2 Bellman well-posedness and optimal Gibbs policies↩︎

Under Assumptions [ass:standing] and [ass:action-regularity], with the measurability convention of Section 2, the soft optimality operator \(\mathcal{T}^\star\) maps bounded measurable functions to bounded measurable functions, is monotone, and is a \(\gamma\)-contraction in \(\|\cdot\|_\infty\): \[\|\mathcal{T}^\star V-\mathcal{T}^\star W\|_\infty\le \gamma\|V-W\|_\infty.\] Consequently, \(\mathcal{T}^\star\) has a unique bounded measurable fixed point \(V^\star\).

Assume the bounded-reward condition in Assumption 4.1(i), the measurability convention in Section 2, and the action-regularity bounds in Assumption 4.2. Let \(\pi\in\Pi\). Then the following statements hold.

(i) The policy-evaluation Bellman operator \(\mathcal{T}^\pi\) is well-defined on bounded measurable functions, maps them to bounded functions, is a \(\gamma\)-contraction in \(\|\cdot\|_\infty\), and has a unique bounded fixed point \(V^\pi\).

(ii) For every state \(s\), the action-value function \[Q^\pi(s,a) = r(s,a)-\frac{\beta}{2}\|a\|^2 + \gamma\int_S V^\pi(s')P(ds'\mid s,a)\] is continuously differentiable in \(a\), and \[\nabla_a Q^\pi(s,a) = \nabla_a r(s,a)-\beta a + \gamma\int_S V^\pi(s')\nabla_a p(s'\mid s,a)\lambda(ds').\] Moreover, if \(\|V^\pi\|_\infty\le V_{\max}\), then \[\|\nabla_a Q^\pi(s,a)-\nabla_a Q^\pi(s,\bar a)\| \le (\beta+L_r+\gamma L_pV_{\max})\|a-\bar a\|,\] and \[\nabla_a Q^\pi(s,a)=-\beta a+h^\pi(s,a), \qquad \sup_{s,a}\|h^\pi(s,a)\| \le G_r+\gamma G_pV_{\max}.\]

(iii) For every bounded measurable \(V:S\to\mathbb{R}\), the Gibbs kernel \[\mathcal{G}[V](a\mid s) = \frac{\exp(Q_V(s,a)/\tau)}{\int_{\mathbb{R}^d}\exp(Q_V(s,\tilde{a})/\tau)d\tilde{a}}\] is well-defined, belongs to \(\Pi\), and attains the variational supremum defining \(\mathcal{T}^\star V\).

(iv) In particular, with \(V^\star\) from[lem:Tstar95contraction], the Gibbs kernel \[\pi^\star:=\mathcal{G}[V^\star]\] belongs to \(\Pi\), satisfies \(V^{\pi^\star}=V^\star\), and is an optimal admissible policy.

(v) If \(\pi_k\in\Pi\) and \(b_k(s,a):=\nabla_aQ^{\pi_k}(s,a)\), then the WPG update has the statewise density \[\pi_{k+1}(y\mid s) = \int_{\mathbb{R}^d} \phi_{2\tau\eta}(y-a-\eta b_k(s,a))\pi_k(a\mid s)\,da,\] where \[\phi_{2\tau\eta}(z) = (4\pi\tau\eta)^{-d/2} \exp\left(-\frac{\|z\|^2}{4\tau\eta}\right).\] Equivalently, \[A^s_{k+1} = A^s_k+\eta b_k(s,A^s_k)+\sqrt{2\tau\eta}\,\xi_{k+1}, \qquad A^s_k\sim\pi_k(\cdot\mid s), \quad \xi_{k+1}\sim N(0,I_d).\]

For \(t>0\), the corresponding one step drift interpolation has density \[q_t^s(y) = \int_{\mathbb{R}^d} \phi_{2\tau t}(y-a-tb_k(s,a))\pi_k(a\mid s)\,da,\] and its conditional drift is \[u_t^s(y) = \frac{ \int_{\mathbb{R}^d} b_k(s,a)\phi_{2\tau t}(y-a-tb_k(s,a))\pi_k(a\mid s)\,da }{ q_t^s(y) }\] for \(q_t^s(y)dy\)-a.e. \(y\).

Finally, along the WPG sequence, admissibility of the updated densities is obtained from the moment and Gaussian-relative-entropy estimates in Proposition 4.1: under its step-size condition, \[\sup_{k\ge0}\sup_{s\in S}\mathrm{KL}(\pi_k(\cdot\mid s)\|\rho_\beta)<\infty,\] and hence all iterates remain in \(\Pi\).

8.2.1 Optimal Gibbs policy↩︎

Let \(V^\star\) be the unique bounded fixed point of \(\mathcal{T}^\star\), and define \[Q^\star(s,a) := \tilde{r}(s,a) + \gamma\int_S V^\star(s')P(ds'\mid s,a),\] and \[\label{eq:optimal-gibbs-policy} \pi^\star(a\mid s) := \frac{\exp(Q^\star(s,a)/\tau)}{\int_{\mathbb{R}^d}\exp(Q^\star(s,\tilde{a})/\tau)\,d\tilde{a}}.\tag{63}\] Then \(\pi^\star\in\Pi\), \[\mathcal{T}^{\pi^\star}V^\star=\mathcal{T}^\star V^\star=V^\star,\] and hence \[V^{\pi^\star}=V^\star.\] Moreover, for every admissible policy \(\pi\), \[V^\pi\le V^\star \qquad\text{pointwise on }S.\] Consequently, \(\pi^\star\) is an optimal soft policy within the admissible class, and \[J(\pi^\star) = \int_S V^\star(s)\rho_0(ds).\]

8.3 Value bounds and Bellman drift estimates↩︎

For every \(\pi\in\Pi\), the one-step regularized reward obeys \[\label{eq:rbar95admissible95bound} \begin{align} \bar r^\pi(s) &:=\int_{\mathbb{R}^d}\bigl(\tilde{r}(s,a)-\tau\log\pi(a\mid s)\bigr)\pi(a\mid s)\,\mathrm da\\ &=\int r(s,a)\pi(\mathrm da\mid s)+\tau\log Z_\beta- \tau\mathrm{KL}(\pi(\cdot\mid s)\|\rho_\beta). \end{align}\tag{64}\] Consequently, under Assumption [ass:standing], \[\label{eq:rbar95admissible95range} -R_{\max}+\tau\log Z_\beta-\tau C_\pi \le \bar r^\pi(s)\le R_{\max}+\tau\log Z_\beta, \qquad C_\pi:=\sup_s\mathrm{KL}(\pi(\cdot\mid s)\|\rho_\beta).\tag{65}\] This coercive identity is the basic reason admissibility relative to \(\rho_\beta\) makes Bellman evaluation finite on the unbounded action space.

Let \[U:=\frac{R_{\max}+\tau\log Z_\beta}{1-\gamma}, \qquad L_\star:=\frac{-R_{\max}+\tau\log Z_\beta}{1-\gamma}, \qquad \bar E_{0}:=\frac{2R_{\max}+\tau K_0}{1-\gamma}.\] Under Assumption [ass:standing], every admissible policy \(\pi\in\Pi\) satisfies \[V^\pi(s)\le U\qquad\forall s\in\mathcal{S}.\] The optimal value satisfies \[L_\star\le V^\star(s)\le U\qquad\forall s\in\mathcal{S},\] and the initial policy satisfies \[V^{\pi_0}(s)\ge \frac{-R_{\max}+\tau\log Z_\beta-\tau K_0}{1-\gamma}, \qquad \|V^\star-V^{\pi_0}\|_\infty\le \bar E_0.\]

8.4 Moment bounds, step sizes, and uniform stability↩︎

Suppose that for all \(0\le k\le K-1\), Lemma [lem:bellman-drift-regularity] holds with \(G(V_{\max}) = G\). Assume \(M_0:=\sup_{s\in\mathcal{S}}\mathbb{E}\|A_0^s\|^2<\infty\) and \(0<\eta\le \frac{1}{4\beta}\). Then \[\label{eq:moment95bound95result} \sup_{0\le k\le K}\sup_{s\in\mathcal{S}}\mathbb{E}\|A_k^s\|^2 \;\le\; \max \{M_0,\;M_\infty(\eta)\},\tag{66}\] where one may take \[\label{eq:Minfty95def} M_\infty(\eta) := \frac{2}{\beta}\Big(\frac{G^2}{\beta}+2\tau d\Big) +\frac{4G^2}{\beta}\,\eta.\tag{67}\]

Let the constants in 55 61 be defined as above. Then every \(0<\eta\le\eta_0\) satisfies the exact stability condition 59 . Moreover, \[\bar\delta_\eta\le C_\delta\eta^2.\]

9 Core Bellman Contraction Proof↩︎

This appendix proves the Bellman part of the argument. These are the nonstandard identities that replace the flat-convex gap-conversion step in mean-field Langevin analysis.

9.1 Bellman residual, residual-to-gap, and resolvent identities↩︎

9.2 One-step KL contraction with Discretization Error↩︎

Let \(A_0\sim \pi_k(\cdot\mid s)\) and let \((B_t)_{t\ge 0}\) be standard Brownian motion in \(\mathbb{R}^d\), independent of \(A_0\). Let \(b:\mathbb{R}^d\to\mathbb{R}^d\) be Borel and assume \(\mathbb{E}\|b(A_0)\|<\infty\). For \(t\in[0,\eta]\) define \[A_t := A_0 + t\,b(A_0) + \sqrt{2\tau}\,B_t,\qquad \mu_t := \mathrm{Law}(A_t).\] Define the conditional drift \[u_t(a):=\mathbb{E}[b(A_0)\mid A_t=a].\] Then, for \(t>0\), \(\mu_t\) satisfies the Fokker–Planck equation \[\partial_t \mu_t = -\nabla\cdot(u_t\,\mu_t) + \tau \Delta \mu_t\] in the sense of distributions.

9.3 From statewise KL contraction to Bellman improvement↩︎

9.4 Proof of the main theorem↩︎

10 Standard analytic tools↩︎

This appendix collects standard analytic inputs used in the fixed-drift Langevin estimate and the uniform bounds. These tools are stated separately so that the paper’s main proof remains focused on the Bellman residual/resolvent mechanism.

10.1 Gaussian entropy, smoothing, and bounded perturbations↩︎

Let \(\mu\) be a probability measure on \(\mathbb{R}^d\). If \(\mathrm{KL}(\mu\|\rho_\beta)<\infty\), then \(\int \|a\|^2\,\mu(da)<\infty\). More quantitatively, for every \(c\in(0,\beta/(2\tau))\), \[c\int_{\mathbb{R}^d}\|a\|^2\,\mu(da) \le \mathrm{KL}(\mu\|\rho_\beta) + \log\int_{\mathbb{R}^d} e^{c\|a\|^2}\rho_\beta(a)\,da <\infty.\]

Let \(\mu\) be a probability measure on \(\mathbb{R}^n\) satisfying \(\mathrm{LSI}(\alpha)\), namely, \[\mathcal{I}(\nu\|\mu)\ge 2\alpha\,\mathrm{KL}(\nu\|\mu), \qquad \forall\,\nu\ll\mu .\] Let \(\psi\in L^\infty(\mu)\) and define \[d\tilde{\mu} := \frac{e^\psi}{\int e^\psi\,d\mu}\,d\mu, \qquad \operatorname{Osc}(\psi) := \operatorname*{ess\,sup}\psi-\operatorname*{ess\,inf}\psi.\] Then \(\tilde{\mu}\) satisfies \(\mathrm{LSI}(\tilde{\alpha})\) with \[\tilde{\alpha} \ge e^{-\operatorname{Osc}(\psi)}\alpha.\] Equivalently, \[\mathcal{I}(\nu\|\tilde{\mu}) \ge 2\alpha e^{-\operatorname{Osc}(\psi)} \mathrm{KL}(\nu\|\tilde{\mu}), \qquad \forall\,\nu\ll\tilde{\mu}.\]

Let \(\rho_\beta(a)=Z_\beta^{-1}\exp(-\beta\|a\|^2/(2\tau))\) on \(\mathbb{R}^d\). Let \(Y=X+\sigma\xi\), where \(\sigma>0\), \(\xi\sim\mathcal{N}(0,I_d)\) is independent of \(X\), and \(\mathbb{E}\|Y\|^2<\infty\). Then \(\nu:=\mathrm{Law}(Y)\) has a density and \[\label{eq:quant95gaussian95KL} \mathrm{KL}(\nu\|\rho_\beta) \le \frac{\beta}{2\tau}\mathbb{E}\|Y\|^2+\log Z_\beta-\frac{d}{2}\log(2\pi e\sigma^2).\tag{96}\] In particular, if \(\sigma^2=2\tau\eta\) and \(\mathbb{E}\|Y\|^2\le M\), then \[\label{eq:quant95gaussian95KL95eta} \mathrm{KL}(\nu\|\rho_\beta) \le \frac{\beta M}{2\tau}+\log Z_\beta-\frac{d}{2}\log(4\pi e\tau\eta).\tag{97}\] Moreover, if \(p(a)=Z_p^{-1}e^{\psi(a)}\rho_\beta(a)\) with \(\|\psi\|_\infty\le C\), then every probability measure \(\mu\) satisfying \(\mathrm{KL}(\mu\|\rho_\beta)<\infty\) obeys \[\label{eq:bounded95tilt95KL} \mathrm{KL}(\mu\|p)\le \mathrm{KL}(\mu\|\rho_\beta)+2C.\tag{98}\]

References↩︎