Proof. We discuss the case of \(t =0\), while the following argument works for any \(t\geq 0\). By Itô’s formula, \(v^\tau\) satisfies (for \(h>0\)) \[\begin{align} &e^{-\beta t}v^{\tau}(t,X^{\tau+h}_t) - v^{\tau}(0,x)\nonumber\\ =& \int_0^{t} e^{-\beta s} \left(v^{\tau}_t -\beta v^{\tau} + \tilde{b}^\top(X^{\tau+h}_s,\pi_{\tau+h-s}(X^{\tau+h}_s))v^{\tau}_x(s,X^{\tau+h}_s)\right.\nonumber\\ &\left.+ \frac{1}{2}\mathrm{Tr}[\tilde{\sigma}\tilde{\sigma}^\top(X^{\tau+h}_s,\pi_{\tau+h-s}(X^{\tau+h}_s)):v^{\tau}_{xx}(s,X^{\tau+h}_s)]\right)ds\nonumber\\ &+ \int_0^{t} e^{-\beta s} \tilde{\sigma}^\top(X^{\tau+h}_s,\pi_{\tau+h-s}(X^{\tau+h}_s))v^{\tau}_x(s,X^{\tau+h}_s)dW_s.\nonumber \end{align}\] Since \(v^{\tau}_x(s,X^{\tau+h}_s)\) is square integrable for sufficiently small \(h\) by Assumption 2 and \(\tilde{\sigma}\) is bounded, the integral with respect to \(W\) is a martingale. Letting \(t=\tau+h\) and taking expectation on both sides of the above equation (in the following discussion, we summarize the arguments of all the functions inside an integral at the end of the integrand for the ease of notation, unless ambiguity arises), \[\begin{align} &v^{\tau}(0,x) \nonumber \\ =& \mathbb{E}\left[\int_0^{\tau+h} e^{-\beta s} \left(-v^{\tau}_t +\beta v^{\tau} - \tilde{b}^\top(\cdot,\pi_{\tau+h-s})v^{\tau}_x - \frac{1}{2}\mathrm{Tr}[\tilde{\sigma}\tilde{\sigma}^\top(\cdot,\pi_{\tau+h-s}):v^{\tau}_{xx}]\right)(s,X^{\tau+h}_s)ds\right.\nonumber\\ &\left.+e^{-\beta(\tau+h)}v^{\tau}(\tau+h,X^{\tau+h}_{\tau+h})\right].\label{eq:v-tau-zero-1} \end{align}\tag{6}\] On the other hand, from the definition of \(v^{\tau}\) in Definition ?? , \[\begin{align} v^{\tau}(t,X^\tau_t) =& \mathbb{E}\left[\int_t^{t+h} e^{-\beta(s-t)}\left(\tilde{f}(\cdot,\pi_{\tau-s}) - \lambda\int_{\mathbb{R}^n}\pi_{\tau-s}\ln\pi_{\tau-s}(\cdot,u)du\right)(X^\tau_s)ds\right.\nonumber\\ +& \left. e^{-\beta h}v^\tau(t+h,X^\tau_{t+h})\,\Big|\,\mathcal{F}^W_t\right], \end{align}\] where \(\{\mathcal{F}^W_t\}_{t\geq 0}\) is the filtration generated by \(W\). Thus \[\begin{align} &e^{-\beta t}v^{\tau}(t,X^\tau_t) + \int_0^t e^{-\beta s}\left(\tilde{f}(\cdot,\pi_{\tau-s}) - \lambda\int_{\mathbb{R}^n}\pi_{\tau-s}\ln\pi_{\tau-s}(\cdot,u)du\right)(X^\tau_s)ds\nonumber \\ =& \mathbb{E}\left[\int_0^{t+h} e^{-\beta s}\left(\tilde{f}(\cdot,\pi_{\tau-s}) - \lambda\int_{\mathbb{R}^n}\pi_{\tau-s}\ln\pi_{\tau-s}(\cdot,u)du\right)(X^\tau_s)ds\right.\nonumber\\ &\left.+ e^{-\beta (t+h)}v^\tau(t+h,X^\tau_{t+h})\,\Big|\,\mathcal{F}^W_t\right]. \end{align}\] In other words, \[M_t = e^{-\beta t}v^{\tau}(t,X^\tau_t) + \int_0^t e^{-\beta s}\left(\tilde{f}(\cdot,\pi_{\tau-s}) - \lambda\int_{\mathbb{R}^n}\pi_{\tau-s}\ln\pi_{\tau-s}(\cdot,u)du\right)(X^\tau_s)ds,\] is a martingale. Based on the smoothness assumption on \(v^\tau\) in Assumption 2, by standard arguments, \(v^{\tau}(t,x)\) satisfies, \[\begin{align} &v^{\tau}_t(t,x) -\beta v^{\tau}(t,x) +\tilde{b}^\top(\cdot,\pi_{\tau-t})v^{\tau}_x(t,x) + \frac{1}{2}\mathrm{Tr}[\tilde{\sigma}\tilde{\sigma}^\top(\cdot,\pi_{\tau-t}):v^{\tau}_{xx}(t,x)] \nonumber\\ &+ \int_{\mathbb{R}^n}\left(f- \lambda \ln\pi_{\tau-t}\right)\pi_{\tau-t}(x,u)du = 0.\label{v-pde} \end{align}\tag{7}\] Substituting this into 6 , \[\begin{align} &v^{\tau}(0,x) \nonumber\\ =& \mathbb{E}\left[\int_0^{\tau +h} e^{-\beta s} \left(\tilde{b}(\cdot,\pi_{\tau-s}) - \tilde{b}(\cdot,\pi_{\tau+h-s})\right)^\top v^{\tau}_x(s,X^{\tau+h}_s)ds\right]\nonumber\\ &+\frac{1}{2}\mathbb{E}\left[\int_0^{\tau +h} e^{-\beta s} \mathrm{Tr}[\left(\tilde{\sigma}\tilde{\sigma}^\top(\cdot,\pi_{\tau-s}) - \tilde{\sigma}\tilde{\sigma}^\top(\cdot,\pi_{\tau+h-s})\right):v^{\tau}_{xx}(s,X^{\tau+h}_s)]ds\right]\nonumber\\ &+ \mathbb{E}\left[\int_0^{\tau +h}e^{-\beta s}\left( \int_{\mathbb{R}^n}\left(f(\cdot,u) - \lambda \ln\pi_{\tau-s}\right)\pi_{\tau-s}(X^{\tau+h}_s,u)du\right) ds\right]\nonumber\\ & + \mathbb{E}\left[e^{-\beta(\tau +h)}v^{\tau}(\tau +h,X^{\tau+h}_{\tau+h})\right].\label{eq:v-tau-zero-2} \end{align}\tag{8}\] Notice that \(\rho^\tau_t = \pi_0\) for \(t\geq \tau+h\), thus starting from \(t=\tau+h\) and initial state \(x\), adopting strategy \(\rho^\tau\), the agent obtains the value \[\begin{align} v^\tau(\tau+h,x) =& \mathbb{E}\left[\int_{\tau+h}^\infty e^{-\beta (t-\tau-h)}\int_{\mathbb{R}^n}\left(f(X^{\tau}_t,u)- \lambda \ln\rho^\tau_t(u)\right)\rho^\tau_t(u)du\,dt\,\Big|\,X^\tau_{\tau+h} = x\right]\\ =&\mathbb{E}\left[\int_{\tau+h}^\infty e^{-\beta (t-\tau-h)} \int_{\mathbb{R}^n}\left(f(X^\tau_t,u)- \lambda\ln\pi_0(u)\right)\pi_0(u)du\,dt\,\Big|\,X^\tau_{\tau+h}=x\right]\\ =& \mathbb{E}\left[\int_{0}^\infty e^{-\beta t}\int_{\mathbb{R}^n}\left(f(X^0_t,u)- \lambda \ln\pi_0(u)\right)\pi_0(u)du\,dt\,\Big|\,X^0_0 = x\right]\\ =&v^0(0,x). \end{align}\] A similar argument shows that \(v^0(0,x) = v^{\tau+h}(\tau+h,x)\). Thus \(v^{\tau}(\tau +h,X^{\tau+h}_{\tau+h}) = v^{\tau+h}(\tau +h,X^{\tau+h}_{\tau+h})\). Plugging this into 8 , also using the fact that \[\begin{align} &\mathbb{E}\left[e^{-\beta(\tau +h)}v^{\tau+h}(\tau +h,X^{\tau+h}_{\tau+h})\right] \nonumber\\ =& v^{\tau+h}(0,x) -\mathbb{E}\left[\int_0^{\tau+h} e^{-\beta s}\left(\int_{\mathbb{R}^n} \left(f - \lambda\ln\pi_{\tau+h-s}\right)\pi_{\tau+h-s}(X^{\tau+h}_s,u)du\right)ds \right], \end{align}\] the following holds \[\begin{align} v^\tau(0,x)=&v^{\tau+h}(0,x)+\mathbb{E}\left[\int_0^{\tau +h}e^{-\beta s}\left(\tilde{f}\left(\cdot,\pi_{\tau-s}\right)-\tilde{f}\left(\cdot,\pi_{\tau+h-s}\right)\right)(X^{\tau+h}_s)ds\right]\nonumber\\ &+\mathbb{E}\left[\int_0^{\tau +h} e^{-\beta s} \left(\tilde{b}(\cdot,\pi_{\tau-s}) - \tilde{b}(\cdot,\pi_{\tau+h-s})\right)^\top v^{\tau-s}_x(X^{\tau+h}_s) ds\right]\nonumber\\ &+\frac{1}{2}\mathbb{E}\left[\int_0^{\tau +h} e^{-\beta s} \mathrm{Tr}[\left(\tilde{\sigma}\tilde{\sigma}^\top(\cdot,\pi_{\tau-s}) - \tilde{\sigma}\tilde{\sigma}^\top(\cdot,\pi_{\tau+h-s})\right):v^{\tau-s}_{xx}(X^{\tau+h}_s)]ds\right]\nonumber\\ &+ \mathbb{E}\left[\int_0^{\tau +h}e^{-\beta s}\left( \int_{\mathbb{R}^n}\lambda\left(\pi_{\tau+h-s}\ln\pi_{\tau+h-s} - \pi_{\tau-s} \ln\pi_{\tau-s}\right)(X^{\tau+h}_s,u)du \right)ds\right], \end{align}\] where we also used fact that \(v^{\tau}(s,x) = v^{\tau-s}(0,x) = v^{\tau-s}(x)\) by our convention.

Since \(\pi_{\tau}\) is the distribution density function of \(u_\tau\), and with deterministic coefficients in 3 , \(u\) is independent of the Brownian motion \(W\), the above can be written as (with a slight abuse of notation where the expectation is taken in a larger probability space which supports the two independent Brownian motions \(B\) and \(W\)) \[\begin{align} \label{eq:v-difference} &v^{\tau}(0,x) - v^{\tau+h}(0,x)\nonumber\\ =&\mathbb{E}\left[\int_0^{\tau +h}e^{-\beta s}\mathbb{E}\left[f(X^{\tau+h}_s,u_{\tau-s})-f(X^{\tau+h}_s,u_{\tau+h-s})\,\big|\,\mathcal{F}^W_s\right]ds\right]\nonumber\\ &+\mathbb{E}\left[\int_0^{\tau +h} e^{-\beta s} \mathbb{E}\left[\left(b(\cdot,u_{\tau-s}) - b(\cdot,u_{\tau+h-s})\right)^\top v^{\tau-s}_x(X^{\tau+h}_s)\,\big|\,\mathcal{F}^W_s\right] ds\right]\nonumber\\ &+\frac{1}{2}\mathbb{E}\left[\int_0^{\tau +h} e^{-\beta s} \mathbb{E}\left[\mathrm{Tr}[\left(\sigma\sigma^\top(\cdot,u_{\tau-s}) - \sigma\sigma^\top(\cdot,u_{\tau+h-s})\right):v^{\tau-s}_{xx}(X^{\tau+h}_s)]\,\big|\,\mathcal{F}^W_s\right]ds\right]\nonumber\\ &+\lambda \mathbb{E}\left[\int_0^{\tau +h}e^{-\beta s}\mathbb{E}\left[\ln\pi_{\tau+h-s}(X^{\tau+h}_s,u_{\tau+h-s}) - \ln\pi_{\tau-s}(X^{\tau+h}_s,u_{\tau-s})\,\big|\,\mathcal{F}^W_s\right] ds\right]. \end{align}\tag{9}\] With the notation \(\mathbf{H}^\tau\) defined in 5 and denoting by \(\nabla_u\) the Jacobi matrix, applying Itô’s formula to \(f\) and \(b\) as a function of \(u_\tau\), which follows 4 , for each \(0\leq s\leq \tau+h\), \[\begin{align} &\mathbb{E}\left[f(X^{\tau+h}_s,u_{\tau-s})-f(X^{\tau+h}_s,u_{\tau+h-s})\,\big|\,\mathcal{F}^W_s\right] \nonumber\\ =&\mathbb{E}\left[- \int_0^{h}\left((\nabla_u f)^\top\nabla_u \mathbf{H}^{\tau-s+l} + \lambda\mathrm{Tr}[\nabla^2_{uu}f]\right)(X^{\tau+h}_s,u_{\tau-s+l})dl\,\Big|\,\mathcal{F}^W_s\right]\tag{10}\\ &\mathbb{E}\left[\left(b(\cdot,u_{\tau-s}) - b(\cdot,u_{\tau+h-s})\right)^\top v^{\tau-s}_{x}(X^{\tau+h}_s)\,\big|\,\mathcal{F}^W_s\right]\nonumber\\ =& \mathbb{E}\left[-\int_0^{h}\left((\nabla_u b\,\nabla_u \mathbf{H}^{\tau-s+l})^\top v^{\tau-s}_{x} + \lambda\sum\limits_{i=1}^d\mathrm{Tr}[\nabla^2_{uu}b_i]v^{\tau-s}_{x_i}\right)(X^{\tau+h}_s,u_{\tau-s+l})dl\,\Big|\,\mathcal{F}^W_s\right]\tag{11}\\ &\mathbb{E}\left[\mathrm{Tr}[\left(\sigma\sigma^\top(\cdot,u_{\tau-s}) - \sigma\sigma^\top(\cdot,u_{\tau+h-s})\right):v^{\tau-s}_{xx}(X^{\tau+h}_s)]\,\big|\,\mathcal{F}^W_s\right]\nonumber\\ =& \mathbb{E}\left[-\int_0^{h} \left(\nabla_u \mathrm{Tr}[\sigma\sigma^\top \,v^{\tau-s}_{xx}]^\top\nabla_u \mathbf{H}^{\tau-s+l} + \lambda\nabla^2_{uu}\mathrm{Tr}[\sigma\sigma^\top\, v^{\tau-s}_{xx}]\right)(X^{\tau+h}_s,u_{\tau-s+l})dl\,\Big|\,\mathcal{F}^W_s\right]\tag{12}, \end{align}\] which follow from the fact that the integral with respect to Brownian motion \(B\) is a martingale by Assumption 1. Furthermore, \[\begin{align} &\lambda\mathbb{E}\left[\ln\pi_{\tau+h-s}(X^{\tau+h}_s,u_{\tau+h-s}) - \ln\pi_{\tau-s}(X^{\tau+h}_s,u_{\tau-s})\,\big|\,\mathcal{F}^W_s\right]\nonumber\\ =&\lambda \mathbb{E}\left[\int_0^{h}\left(\frac{\nabla_\tau \pi_{\tau-s+l}}{\pi_{\tau-s+l}} + \frac{(\nabla_u \pi_{\tau-s+l})^\top\nabla_u \mathbf{H}^{\tau-s+l}}{\pi_{\tau-s+l}}\right)(X^{\tau+h}_s,u_{\tau-s+l})dl\,\Big|\,\mathcal{F}^{W}_s\right]\nonumber\\ +&\lambda^2 \mathbb{E}\left[\int_0^{h}\left(\frac{\mathrm{Tr}[\nabla^2_{uu} \pi_{\tau-s+l}]}{\pi_{\tau-s+l}}- \frac{(\nabla_u \pi_{\tau-s+l})^\top\nabla_u \pi_{\tau-s+l}}{\pi^2_{\tau-s+l}}\right)(X^{\tau+h}_s,u_{\tau-s+l})dl\,\Big|\,\mathcal{F}^{W}_s\right]\\ =& \lambda\mathbb{E}\left[\int_0^{h} \left(-\mathrm{Tr}[\nabla^2_{uu} \mathbf{H}^{\tau-s+l}] + 2\lambda\frac{\mathrm{Tr}[\nabla^2_{uu} \pi_{\tau-s+l}]}{\pi_{\tau-s+l}} - \lambda\|\nabla_u\ln \pi_{\tau-s+l}\|^2\right)(X^{\tau+h}_s,u_{\tau-s+l})dl\,\Big|\,\mathcal{F}^{W}_s\right]\\ =&\lambda\mathbb{E}\left[\int_0^{h} \left(-\mathrm{Tr}[\nabla^2_{uu} \mathbf{H}^{\tau-s+l}]- \lambda\|\nabla_u\ln \pi_{\tau-s+l}\|^2\right)(X^{\tau+h}_s,u_{\tau-s+l})dl\,\Big|\,\mathcal{F}^{W}_s\right], \end{align}\] where the second equation follows from the Fokker-Planck equation for \(\pi_{\tau-s+l}\), and the last equation holds because \[\mathbb{E}\left[\frac{\mathrm{Tr}[\nabla^2_{uu} \pi_{\tau-s+l}]}{\pi_{\tau-s+l}}(X^{\tau+h}_s,u_{\tau-s+l})\,\Big|\,\mathcal{F}^{W}_s\right] =\int_{\mathbb{R}^n}\mathrm{Tr}[\nabla^2_{uu} \pi_{\tau-s+l}](X^{\tau+h}_s,u)du = 0.\nonumber\] Plugging these calculations back into 9 , \[\begin{align} &v^{\tau}(0,x) - v^{\tau+h}(0,x)\nonumber\\ =&-\mathbb{E}\left[\int_0^{\tau +h}e^{-\beta s}\int_0^{h} \left((\nabla_u f)^\top\nabla_u \mathbf{H}^{\tau-s+l} + \lambda\mathrm{Tr}[\nabla^2_{uu}f]\right)(X^{\tau+h}_s,u_{\tau-s+l})dl\,ds\right]\nonumber\\ &-\mathbb{E}\left[\int_0^{\tau +h} e^{-\beta s} \int_0^{h}\left((\nabla_u b^\top v^{\tau-s}_x)^\top\nabla_u \mathbf{H}^{\tau-s+l}+ \lambda\sum\limits_{i=1}^d\mathrm{Tr}[\nabla^2_{uu}b_i]v^{\tau-s}_{x_i}\right)(X^{\tau+h}_s,u_{\tau-s+l})dl\, ds\right]\nonumber\\ &-\frac{1}{2}\mathbb{E}\left[\int_0^{\tau +h} e^{-\beta s} \int_0^{h}\left(\nabla_u \mathrm{Tr}[\sigma\sigma^\top\, :v^{\tau-s}_{xx}]^\top\nabla_u \mathbf{H}^{\tau-s+l} + \lambda\nabla^2_{uu}\mathrm{Tr}[\sigma\sigma^\top\, : v^{\tau-s}_{xx}]\right)(X^{\tau+h}_s,u_{\tau-s+l})dl\, ds\right]\nonumber\\ &+\lambda \mathbb{E}\left[\int_0^{\tau +h}e^{-\beta s}\int_0^{h}\left(-\mathrm{Tr}[\nabla^2_{uu} \mathbf{H}^{\tau-s+l}]- \lambda\|\nabla_u\ln \pi_{\tau-s+l}\|^2 \right)(X^{\tau+h}_s,u_{\tau-s+l})dl\, ds\right]\\ =& -\mathbb{E}\left[\int_0^{\tau+h} e^{-\beta s}\int_0^{h}(\nabla_u\mathbf{H}^{\tau-s})^\top\nabla_u \mathbf{H}^{\tau-s+l}(X^{\tau+h}_s,u_{\tau-s+l})dl\,ds \right]\nonumber\\ &-\lambda\mathbb{E}\left[\int_0^{\tau+h} e^{-\beta s}\int_0^{h}\left(\mathrm{Tr}[\nabla^2_{uu} \mathbf{H}^{\tau-s+l}]+\mathrm{Tr}[\nabla^2_{uu} \mathbf{H}^{\tau-s}]\right)(X^{\tau+h}_s,u_{\tau-s+l})dl\,ds \right]\nonumber\\ &-\mathbb{E}\left[\int_0^{\tau+h} e^{-\beta s}\int_0^{h} \lambda^2\|\nabla_u\ln \pi_{\tau-s+l}\|^2 (X^{\tau+h}_s,u_{\tau-s+l})dl\,ds \right]. \end{align}\] Notice that (and the same applies to \(\mathbb{E}\left[\mathrm{Tr}[\nabla^2_{uu} \mathbf{H}^{\tau-s}](X^{\tau+h}_s,u_{\tau-s+l})\,\big|\,X^{\tau+h}_s\right]\)) \[\begin{align} &\mathbb{E}\left[\mathrm{Tr}[\nabla^2_{uu} \mathbf{H}^{\tau-s+l}](X^{\tau+h}_s,u_{\tau-s+l})\,\big|\,X^{\tau+h}_s\right] = \int_{\mathbb{R}^n}\mathrm{Tr}[\nabla^2_{uu} \mathbf{H}^{\tau-s+l}]\pi_{\tau-s+l}(X^{\tau+h}_s,u)du\nonumber\\ =&-\int_{\mathbb{R}^n}(\nabla_u \mathbf{H}^{\tau-s+l})^\top\nabla_u\pi_{\tau-s+l}(X^{\tau+h}_s,u)du\nonumber\\ =& -\mathbb{E}\left[(\nabla_u \mathbf{H}^{\tau-s+l})^\top\nabla_u\ln\pi_{\tau-s+l}(X^{\tau+h}_s,u_{\tau-s+l})\,\big|\,X^{\tau+h}_s\right],\label{eq:H-2nd} \end{align}\tag{13}\] where the last equation follows from the boundedness of the model parameters and that \(\pi_{\tau-s+l}\), as a probability density function, vanishes at \(\partial \mathbb{R}^n\). Thus by the dominated convergence theorem, the above implies that \[\begin{align} &\frac{1}{h}(v^{\tau}(0,x) - v^{\tau+h}(0,x)) \nonumber\\ =& -\frac{1}{h}\mathbb{E}\left[\int_0^{\tau+h} e^{-\beta s}\int_0^{h}(\nabla_u \mathbf{H}^{\tau-s})^\top\nabla_u \mathbf{H}^{\tau-s+l}(X^{\tau+h}_s,u_{\tau-s+l})dl\,ds \right]\nonumber\\ &+\frac{\lambda}{h}\mathbb{E}\left[\int_0^{\tau+h} e^{-\beta s}\int_0^{h}\left(\nabla_u \mathbf{H}^{\tau-s+l}+\nabla_u \mathbf{H}^{\tau-s}\right)^\top\nabla_u\ln\pi_{\tau-s+l}(X^{\tau+h}_s,u_{\tau-s+l})dl\,ds \right]\nonumber\\ &-\frac{1}{h}\mathbb{E}\left[\int_0^{\tau+h} e^{-\beta s}\int_0^{h} \lambda^2\|\nabla_u\ln \pi_{\tau-s+l}\|^2 (X^{\tau+h}_s,u_{\tau-s+l})dl\,ds \right]\nonumber\\ &\xrightarrow{h\rightarrow 0} -\mathbb{E}\left[\int_0^{\tau} e^{-\beta s}\|\nabla_u \mathbf{H}^{\tau-s})\|^2(X^{\tau}_s,u_{\tau-s})ds \right]\nonumber\\ &+2\lambda\mathbb{E}\left[\int_0^{\tau} e^{-\beta s}(\nabla_u \mathbf{H}^{\tau-s})^\top\nabla_u\ln\pi_{\tau-s}(X^{\tau}_s,u_{\tau-s})ds \right]\nonumber\\ &-\mathbb{E}\left[\int_0^{\tau} e^{-\beta s}\lambda^2\|\nabla_u\ln \pi_{\tau-s}\|^2 (X^{\tau}_s,u_{\tau-s})ds \right]\nonumber\\ =& -\mathbb{E}\left[\int_0^{\tau} e^{-\beta s}\|\nabla_u \mathbf{H}^{\tau-s} - \lambda \nabla_u\ln \pi_{\tau-s}\|^2(X^{\tau}_s,u_{\tau-s})ds \right]\leq 0. \end{align}\] Therefore \(v^{\tau}(0,x)\) is non-decreasing in \(\tau\). ◻

Proof. For any \(h>0\), since \(v^{\tau+h}(h,x) = v^\tau(0,x)\), \[\begin{align} &v^{\tau+h}(0,x) - v^\tau(0,x)\nonumber\\ =&\mathbb{E}\left[\int_0^{h} e^{-\beta s} \left(\int_{\mathbb{R}^n}\left(f - \lambda\ln\pi_{\tau+h-s}\right)\pi_{\tau+h-s}(X^{\tau+h}_s,u)du\right) ds + e^{-\beta h}v^{\tau+h}\left(h,X^{\tau+h}_{h}\right) -v^\tau(0,x)\right]\\ =&\mathbb{E}\left[\int_0^{h} e^{-\beta s} \left(\int_{\mathbb{R}^n}\left(f - \lambda\ln\pi_{\tau+h-s}\right)\pi_{\tau+h-s}(X^{\tau+h}_s,u)du\right) ds + e^{-\beta h}v^{\tau}\left(0,X^{\tau+h}_{h}\right) -v^\tau(0,x)\right]\\ =&\mathbb{E}\left[\int_0^{h} e^{-\beta s} \left(\int_{\mathbb{R}^n}\left(f - \lambda\ln\pi_{\tau+h-s}\right)\pi_{\tau+h-s}(X^{\tau+h}_s,u)du\right) ds\right]\nonumber\\ & + \mathbb{E}\left[\int_0^h e^{-\beta s} \left(-\beta v^\tau + \tilde{b}^\top v^{\tau}_x + \frac{1}{2}\mathrm{Tr}[\tilde{\sigma}\tilde{\sigma}^\top:v^{\tau}_{xx}]\right)(X^{\tau+h}_s,\pi_{\tau+h-s})ds \right], \end{align}\] where the last equation follows from Itô’s formula, and the stochastic integral with respect to the Brownian motion \(W\) in expectation is \(0\) due to Assumption 2. Dividing by \(h\) on both sides of the above equations and letting \(h\rightarrow 0\), the dominated convergence theorem implies that \[\begin{align} dv^\tau(x) =&\left(\int_{\mathbb{R}^n} \left(f - \lambda\ln\pi_{\tau} + b^\top v^\tau_x + \frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top \,v_{xx}^\tau]\right)\pi_{\tau}(x,u)du -\beta v^{\tau} \right)d\tau\nonumber\\ =&\left(\int_{\mathbb{R}^n} \left(\mathbf{H}^\tau - \lambda\ln \pi_\tau\right)\pi_{\tau}(x,u)du -\beta v^{\tau} \right)d\tau. \label{eq:v-dynamics} \end{align}\tag{14}\] Notice that the integral on the right hand side is with respect to \(u\). As a function of \(x\), \(\mathbf{H}(x,u,v^\tau_x,v^\tau_{xx}) - \lambda\ln\pi_\tau\) is smooth and its derivatives are locally bounded. Thus by thedominated convergence theorem, switching the order of derivative with respect to \(x\) and the integral with respect to \(u\) does not change its value. Therefore taking derivative on both sides with respect to \(x\) yields the dynamics for \(v^\tau_x\) and \(v^\tau_{xx}\). ◻

Proof. Since for given \(x\in\mathbb{R}^d\), \(v^\tau(t,x)\) is non-decreasing in \(\tau\) and is bounded above by the optimal \(V(x)\), there exists a limit \(v^*(t,x)\). Notice that for each \(\tau\), \(v^\tau(t,x)\) solves \[\begin{align} &v^{\tau}_t(t,x) -\beta v^{\tau}(t,x) +\tilde{b}^\top (x,\pi_{\tau-t})v^{\tau}_x(t,x) + \frac{1}{2}\mathrm{Tr}[\tilde{\sigma}\tilde{\sigma}^\top \,v^{\tau}_{xx}(t,x)] \nonumber\\ &+ \int_{\mathbb{R}^n}\left(f(x,u) - \lambda \ln\pi_{\tau-t}\right)\pi_{\tau-t}(u)du = 0,\nonumber \end{align}\] and by Proposition 1, \(v^0(t,x) \leq v^{\tau}(t,x) \leq v^*(t,x)\). Also notice that since \(\pi_0\) is continuous in \(x\), and \(\int_{\mathbb{R}^n}\pi_\tau\ln\pi_\tau(x,u)du\) is bounded uniformly in \(\tau\), the latter is locally bounded in \(x\), independent of \(\tau\). Then by Schauder’s interior estimates (c.f. krylov1996lectures?[Chapter 8], which gives an estimated of the norm of the derivatives of the solution to a PDE in the interior of a bounded domain) imply that \(v^\tau_t(t,x)\), \(v_x^\tau(t,x)\) and \(v_{xx}^\tau(t,x)\) are locally uniformly equicontinuous, and thus by the Arzelà-Ascoli theorem, \(v_x^\tau(t,x)\), \(v_t^\tau(t,x)\) and \(v_{xx}^\tau(t,x)\), each as a family of locally uniformaly bounded and equicontinuos functions, converge (up to a subsequence) locally uniformly to \(v_t^*(t,x)\), \(v_x^*(t,x)\) and \(v^*_{xx}(t,x)\). Furthermore, by definition \(v^\tau(t,x) = v^{\tau-t}(0,x)\) for \(\tau \geq t\). Thus for any \(t \geq 0\), \[v^*(t,x) = \lim_{\tau\rightarrow\infty}v^\tau(t,x) = \lim_{\tau\rightarrow\infty}v^{\tau -t}(0,x) = v^*(0,x).\] Therefore \(v^*\) is independent of \(t\), \(v^*_t(t,x)=0\) and we can drop the argument \(t\) in \(v^*\), \(v^*_x\) and \(v^*_{xx}\). ◻

Proof. With \(x\in\mathbb{R}^d\) given, since \(v^\tau (x)\) converges (cf. 14 ), \[\begin{align} &\int_{\mathbb{R}^n} \left(f - \lambda\ln\pi_{\tau} + b^\top v_x^\tau + \frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top:v_{xx}^\tau]\right)\pi_{\tau}(x,u)du -\beta v^{\tau}(x)\nonumber\\ =&\int_{\mathbb{R}^n} \left(\mathbf{H}^\tau- \lambda\ln\pi_{\tau}\right)\pi_{\tau}(x,u)du -\beta v^{\tau}, \end{align}\] converges to \(0\) as \(\tau\rightarrow\infty\). In other words, \[g^\tau(x) = \int_{\mathbb{R}^n} \left(\mathbf{H}^\tau- \lambda\ln\pi_{\tau}\right)\pi_{\tau}(x,u)du,\] converges to a constant \(\beta v^*(x)\). Notice that \(\pi_{\tau}\) is the distribution density function of \(u_\tau\), thus \[\begin{align} &g^{\tau}(x)- g^{\tau+h}(x)\nonumber\\ =&\mathbb{E}\left[f(\cdot,u_{\tau})-f(\cdot,u_{\tau+h})+ b^\top(\cdot,u_\tau)v^{\tau}_x -b^\top(\cdot,u_{\tau+h})v^{\tau+h}_x\right](x)\nonumber\\ &+ \mathbb{E}\left[\frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top(\cdot,u_\tau):v^\tau_{xx} - \sigma\sigma^\top(\cdot,u_{\tau+h}):v^{\tau+h}_{xx}](x) + \lambda\left(\ln\pi_{\tau+h}(x,u_{\tau+h}) - \ln\pi_{\tau}(x,u_{\tau})\right)\right]\nonumber\\ =&\mathbb{E}\left[f(\cdot,u_{\tau})-f(\cdot,u_{\tau+h})+ (b(\cdot,u_\tau) - b(\cdot,u_{\tau+h}))^\top v^{\tau}_x + b^\top(\cdot,u_{\tau+h})(v^{\tau}_x-v^{\tau+h}_x) \right](x)\nonumber\\ &+ \mathbb{E}\left[\frac{1}{2}\mathrm{Tr}[(\sigma\sigma^\top(\cdot,u_\tau)-\sigma\sigma^\top(\cdot,u_{\tau+h})):v^\tau_{xx}] + \frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top(\cdot,u_{\tau+h}):(v^\tau_{xx}-v^{\tau+h}_{xx})]\right](x)\nonumber\\ &+ \mathbb{E}\left[\lambda(\ln\pi_{\tau+h}(x,u_{\tau+h}) - \ln\pi_{\tau}(x,u_{\tau}))\right]. \end{align}\] Then by similar argument to the proof of Proposition 1 (cf. 9 ), \[dg^\tau = \mathbb{E}\left[\|\nabla_u \mathbf{H}^\tau - \lambda \nabla_u \ln\pi_{\tau}\|^2(x,u_\tau)\right]d\tau -\tilde{b}^\top(x,\pi_\tau)dv^\tau_x-\frac{1}{2}\mathrm{Tr}[\tilde{\sigma}\tilde{\sigma}^\top(x,\pi_\tau):dv^{\tau}_{xx}].\label{eq:g-tau}\tag{15}\] The fact that \(v^\tau_x\) and \(v^{\tau}_{xx}\) converge combined with the boundedness of \(\tilde{b}\) and \(\tilde{\sigma}\) implies that the last two terms converge to \(0\) as \(\tau\rightarrow \infty\). Since the diffusion coefficient for \(u\) is constant \(\sqrt{2\lambda} >0\), \(u_\tau\) is supported on \(\mathbb{R}^n\) for every \(\tau >0\). Then \(g^\tau\) being convergent implies that \(\nabla_u \ln\pi_{\tau}(x,u)\) converges to \(\nabla_u \mathbf{H}(x,u,v^*_x,v^*_{xx})/\lambda\). In other words, with \(h_\tau(x,u) = \nabla_u (\ln\pi_{\tau}-\mathbf{H}^\tau/\lambda)(x,u)\), \(\lim\limits_{\tau\rightarrow\infty}h_\tau(x,u) =0\). Furthermore, since \(v^\tau_x\) and \(v^\tau_{xx}\) converge and \(b\), \(\sigma\) and \(f\) are bounded by assumption, \(\nabla_u \mathbf{H}^\tau\) is locally bounded in \(u\), and the bounds are uniform in \(\tau\). By assumption on \(\nabla_u\ln\pi_\tau\), the same holds for \(h_\tau\).

Furthermore, since \(\pi_\tau\) is a probability density function, \[\pi_\tau(x,u) = \frac{e^{\mathbf{H}(x,u,v^\tau_x,v^\tau_{xx})/\lambda + \int_0^{u_1}\cdots\int_0^{u_n}h_\tau(x,r)dr_1\dots dr_n}}{\int_{\mathbb{R}^n}e^{\mathbf{H}^\tau(x,u')/\lambda + \int_0^{u'_1}\cdots\int_0^{u'_n}h_\tau(x,r)dr_1\dots dr_n}du'}.\] Notice that by Fatou’s lemma, \[\begin{align} &\int_{\mathbb{R}^n}e^{\mathbf{H}^\tau(x,u')/\lambda + \int_0^{u'_1}\cdots\int_0^{u'_n}h_\tau(x,r)dr_1\dots dr_n}du'\geq \int_{\mathbb{R}^n}e^{\mathbf{H}^*(x,u)/\lambda + \liminf\limits_{\tau\rightarrow\infty}\int_0^{u'_1}\cdots\int_0^{u'_n}h_\tau(x,r)dr_1\dots dr_n}du\\ =&\int_{\mathbb{R}^n}e^{\mathbf{H}^*(x,u)/\lambda}du, \end{align}\] where the last equation follows from the dominated convergence theorem. The denominator of \(\pi_\tau\) above can be approximated similarly, and \[\limsup\limits_{\tau\rightarrow\infty}\pi_\tau(x,u) \leq \frac{\limsup\limits_{\tau\rightarrow\infty}e^{\mathbf{H}^\tau(x,u,v^\tau_x,v^\tau_{xx})/\lambda + \int_0^{u_1}\cdots\int_0^{u_n}h_\tau(x,r)dr_1\dots dr_n}}{\liminf\limits_{\tau\rightarrow\infty}\int_{\mathbb{R}^n}e^{\mathbf{H}^\tau(x,u)/\lambda + \int_0^{u_1}\cdots\int_0^{u_n}h_\tau(x,r)dr_1\dots dr_n}du} \leq \frac{e^{\mathbf{H}^*(x,u)/\lambda}}{\int_{\mathbb{R}^n}e^{\mathbf{H}^*(x,u')/\lambda} du'}.\] On the other hand, following a similar argument, \(\liminf\limits_{\tau\rightarrow\infty}\pi_\tau(x,u) \geq \frac{e^{\mathbf{H}^*(x,u)/\lambda}}{\int_{\mathbb{R}^n}e^{\mathbf{H}^*(x,u')/\lambda} du'}\), which concludes the proof. ◻

Proof. In this proof we focus on showing that \(v^*\) is indeed the solution to the HJB equation of the relaxed control problem and that \(\pi^*\) is the associated maximizer of the Hamiltonian. The verification that \(\pi^*\) is the optimal strategy and \(v^*\) is the corresponding value follows standard arguments which we omit here. With \(\pi^*(x,u) = \frac{e^{\mathbf{H}^*(x,u)/\lambda}}{\int_{\mathbb{R}^n}e^{\mathbf{H}^*(x,u')/\lambda} du'}\) and \(c = \int_{\mathbb{R}^n}e^{\mathbf{H}^*(x,u)/\lambda} du\), \[\begin{align} &\int_{\mathbb{R}^n} \left(f - \lambda\ln\pi^* + b^\top v_x^* +\frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top \,v_{xx}^*]\right)\pi^*(x,u)du -\beta v^* = \lambda\ln c - \beta v^*.\nonumber \end{align}\] On the other hand, since \(v^\tau\) converges, from the dynamics of \(v^\tau(x)\) in 14 , \[\lim_{\tau\rightarrow\infty}\left(\int_{\mathbb{R}^n} \left(f - \lambda\ln\pi_{\tau} + b^\top v_x^\tau+\frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top \,v^\tau_{xx}]\right)\pi_{\tau}(x,u)du -\beta v^{\tau}\right) = 0. \nonumber\] With \(h_\tau\) defined in the proof of Proposition 2, \(\pi_\tau(x,u) = \frac{e^{\mathbf{H}^\tau(x,u)/\lambda + \int_0^{u_1}\cdots\int_0^{u_n}h_\tau(x,r)dr_1\dots dr_n}}{\int_{\mathbb{R}^n}e^{\mathbf{H}^\tau(x,u')/\lambda + \int_0^{u'_1}\cdots\int_0^{u'_n}h_\tau(x,r)dr_1\dots dr_n}du'}\), and \(\lim\limits_{\tau\rightarrow \infty}h_{\tau} = 0\). Thus the above equation implies that \(\lim\limits_{\tau\rightarrow\infty}\lambda \ln c_\tau -\beta v^* = 0\), where \[c_\tau = \int_{\mathbb{R}^n}e^{\mathbf{H}^\tau(x,u)/\lambda + \int_0^{u_1}\cdots\int_0^{u_n}h_\tau(x,r)dr_1\dots dr_n}du = \frac{e^{\mathbf{H}^\tau(x,u)/\lambda + \int_0^{u_1}\cdots\int_0^{u_n}h_\tau(x,r)dr_1\dots dr_n}}{\pi_\tau(x,u)}.\] Notice that \[\begin{align} \lim_{\tau\rightarrow\infty} c_\tau = \lim_{\tau\rightarrow\infty}\frac{e^{\mathbf{H}^\tau(x,u)/\lambda + \int_0^{u_1}\cdots\int_0^{u_n}h_\tau(x,r)dr_1\dots dr_n}}{\pi_\tau(x,u)} = \frac{e^{\mathbf{H}^*(x,u)/\lambda}}{\pi^*(x,u)} = c.\nonumber \end{align}\] Thus \(\beta v^* = \lim\limits_{\tau\rightarrow\infty}\lambda \ln c_\tau = \lambda\ln c\), and \[\int_{\mathbb{R}^n} \left(f- \lambda\ln\pi^* + b^\top v_x^*+\frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top:v_{xx}^*]\right)\pi^*(x,u)du -\beta v^* = 0.\nonumber\] Furthermore, since the maximizer of (which is pointwise concave in \(\pi(u)\)) \[-\beta v(x) +\sup_{\pi\in\mathcal{P}(\mathbb{R}^n)} \int_{\mathbb{R}^n} \left(\mathbf{H}(x,u,v_x,v_{xx}) -\lambda \ln\pi(u)\right)\pi(u)du, \nonumber\] is \(\hat{\pi} = \frac{e^{\mathbf{H}(x,u,v_x,v_{xx})/\lambda}}{\int_{\mathbb{R}^n}e^{\mathbf{H}(x,u',v_x,v_{xx})/\lambda} du'}\), it implies that \[\sup_{\pi\in\mathcal{P}(\mathbb{R}^n)}\int_{\mathbb{R}^n} \left(\mathbf{H}(x,u,v^*_x,v^*_{xx}) -\lambda\ln \pi(u)\right)\pi(u)du -\beta v^*= 0. \nonumber\] Therefore, \(v^*\) solves the HJB equation, and \(\pi^*\) is the corresponding maximizer. ◻

Proof. Following an argument similar to that for the relaxed control problem in Lemma 1, for any \(h>0\), since \(v^{\tau+h}(h,x) = v^\tau(0,x)\), \[\begin{align} &v^{\tau+h}(0,x) - v^\tau(0,x)\nonumber\\ =&\mathbb{E}\left[\int_0^{h} e^{-\beta s} f\, ds + e^{-\beta h}v^{\tau+h}\left(h,X^{\tau+h}_{h}\right) -v^\tau(0,x)\,\Big|\,X_0=x,\mathcal{F}^B_\tau\right] \\ =& \mathbb{E}\Big[\int_0^h e^{-\beta s} \Big(-\beta v^\tau + f+b^\top v^{\tau}_x + \frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top :v^{\tau}_{xx}]\Big)(X^{\tau+h}_s,u_{\tau+h-s}) ds \,\Big|\,X_0=x,\mathcal{F}^B_\tau\Big]. \end{align}\] The rest follows similarly to Lemma 1. ◻

Proof. Similarly to the case of relaxed control, we only show the proof for \(t=0\), and other cases follow by the same argument. By Itô’s formula, \[\begin{align} &e^{-\beta t}v^{\tau}(t,X^{\tau+h}_t) - v^{\tau}(0,x)\nonumber\\ =& \int_0^{t} e^{-\beta s} \left(v^{\tau}_t(s,X^{\tau+h}_s) -\beta v^{\tau}(s,X^{\tau+h}_s) + b^\top(X^{\tau+h}_s,u_{\tau+h-s}) v^{\tau}_x(s,X^{\tau+h}_s)\right.\nonumber\\ &\left. + \frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top(X^{\tau+h}_s,u_{\tau+h-s}):v^{\tau}_{xx}(s,X^{\tau+h}_s)]\right)ds+ \int_0^{t} e^{-\beta s} v^{\tau}_x(s,X^{\tau+h}_s)^\top \sigma(X^{\tau+h}_s,u_{\tau+h-s})dW_s\\ =& \int_0^{t} e^{-\beta s} \left(v^{\tau-s}_t -\beta v^{\tau-s} + b^\top v^{\tau-s}_x + \frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top:v^{\tau-s}_{xx}]\right)(X^{\tau+h}_s,u_{\tau+h-s})ds\nonumber\\ &+ \int_0^{t} e^{-\beta s} v^{\tau-s}_x{}^\top \sigma(X^{\tau+h}_s,u_{\tau+h-s})dW_s. \end{align}\] Under Assumption 5, the integral with respect to \(W\) is a martingale, and \[\begin{align} &\mathbb{E}\left[v^{\tau}(0,x)\right] \nonumber\\ =& \mathbb{E}\left[\int_0^{\tau+h} e^{-\beta s} \left(-v^{\tau-s}_t +\beta v^{\tau-s} - b^\top v^{\tau-s}_x -\frac{1}{2} \mathrm{Tr}[\sigma\sigma^\top:v^{\tau-s}_{xx}]\right)(X^{\tau+h}_s,u_{\tau+h-s})ds\right]\nonumber\\ &+ \mathbb{E}\left[e^{-\beta(\tau+h)}v^{\tau}(\tau+h,X^{\tau+h}_{\tau+h})\right]. \end{align}\] On the other hand, by the definition of \(v^{\tau}\) and standard argument similar to that for 7 , \(v^{\tau}(t,x)\) satisfies, \[\begin{align} &v^{\tau}_t(t,x) -\beta v^{\tau}(t,x) +b^\top(x,u_{\tau-t})v^{\tau}_x(t,x) + \frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top(x,u_{\tau-t}):v^{\tau}_{xx}(t,x)] + f(x,u_{\tau-t}) = 0. \end{align}\] Substituting this back into the last equation for \(v^\tau(0,x)\), \[\begin{align} &\mathbb{E}[v^{\tau}(0,x)] \nonumber\\ =& \mathbb{E}\left[\int_0^{\tau +h} e^{-\beta s} \left(\left(b(X^{\tau+h}_s,u_{\tau-s}) - b(X^{\tau+h}_s,u_{\tau+h-s})\right)^\top v^{\tau-s}_x(X^{\tau+h}_s)\right) ds\right]\nonumber\\ &+ \frac{1}{2}\mathbb{E}\left[\int_0^{\tau +h} e^{-\beta s} \mathrm{Tr}[\left(\sigma\sigma^\top(X^{\tau+h}_s,u_{\tau-s}) - \sigma\sigma^\top(X^{\tau+h}_s,u_{\tau+h-s})\right):v^{\tau-s}_{xx}(X^{\tau+h}_s)] ds\right]\nonumber\\ &+ \mathbb{E}\left[\int_0^{\tau +h}e^{-\beta s} f(X^{\tau+h}_s,u_{\tau-s}) ds\right] + \mathbb{E}\left[e^{-\beta(\tau +h)}v^{\tau}(\tau +h,X^{\tau+h}_{\tau+h})\right]\\ =&\mathbb{E}[v^{\tau+h}(0,x)]+\mathbb{E}\left[\int_0^{\tau +h}e^{-\beta s}\left( f(X^{\tau+h}_s,u_{\tau-s})-f(X^{\tau+h}_s,u_{\tau+h-s})\right)ds\right]\nonumber\\ &+\mathbb{E}\left[\int_0^{\tau +h} e^{-\beta s} \left((b(X^{\tau+h}_s,u_{\tau-s}) - b(X^{\tau+h}_s,u_{\tau+h-s}))^\top v^{\tau-s}_x(X^{\tau+h}_s)\right) ds\right]\nonumber\\ &+ \frac{1}{2}\mathbb{E}\left[\int_0^{\tau +h} e^{-\beta s} \mathrm{Tr}\Big[\left(\sigma\sigma^\top(X^{\tau+h}_s,u_{\tau-s}) - \sigma\sigma^\top(X^{\tau+h}_s,u_{\tau+h-s})\right):v^{\tau-s}_{xx}(s,X^{\tau+h}_s)\Big] ds\right],\label{eq:v-difference-c} \end{align}\tag{17}\] where the last equation follows from a similar argument as that for 9 and the fact that by definition \(v^{\tau}(\tau +h,X^{\tau+h}_{\tau+h}) = v^{0}(0,X^{\tau+h}_{\tau+h}) = v^{\tau+h}(\tau +h,X^{\tau+h}_{\tau+h})\). Notice that by Itô’s formula, and the fact that the integral with respect to Brownian motion \(B\) is a martingale under Assumption 5, following similar calculations as in previous sections, for \(0\leq s\leq \tau+h\), 10 , 11 and 12 hold, and imply that \[\begin{align} &\mathbb{E}[v^{\tau}(0,x) - v^{\tau+h}(0,x)]\nonumber\\ =&-\mathbb{E}\left[\int_0^{\tau +h}e^{-\beta s}\left(\int_0^{h}(\nabla_u\mathbf{H}^{\tau-s})^\top\nabla_u \mathbf{H}^{\tau-s+l}(X^{\tau+h}_s,u_{\tau-s+l})dl\right)ds\right]\nonumber\\ &-\lambda\mathbb{E}\left[\int_0^{\tau+h} e^{-\beta s}\int_0^{h}tr[\nabla^2_{uu} \mathbf{H}^{\tau-s}](X^{\tau+h}_s,u_{\tau-s+l})dl\,ds \right]. \end{align}\] Furthermore, 13 implies that, \[\mathbb{E}\left[\mathrm{Tr}[\nabla^2_{uu} \mathbf{H}^{\tau-s}](x,u_{\tau-s+l})\right] = -\mathbb{E}\left[(\nabla_u \mathbf{H}^{\tau-s})^\top\nabla_u\ln\pi_{\tau-s+l}(x,u_{\tau-s+l})\right],\] where \(\pi_{\tau-s+l}\) is the distribution density function of \(u_{\tau-s+l}\). Plugging this into \(\mathbb{E}[v^{\tau}(0,x) - v^{\tau+h}(0,x)]\) above, and dividing by \(h\) on both sides, \[\begin{align} &\frac{\mathbb{E}[v^{\tau+h}(0,x) - v^{\tau}(0,x)]}{h}\nonumber\\ =& \mathbb{E}\left[\int_0^{\tau +h}e^{-\beta s}\left(\int_0^{h}\frac{1}{h}(\nabla_u\mathbf{H}^{\tau-s})^\top\nabla_u \mathbf{H}^{\tau-s+l}(X^{\tau+h}_s,u_{\tau-s+l})dl\right)ds\right]\nonumber\\ &-\lambda\mathbb{E}\left[\int_0^{\tau +h}e^{-\beta s}\int_0^{h} \frac{1}{h}(\nabla_u \mathbf{H}^{\tau-s})^\top\nabla_u\ln\pi_{\tau-s+l}(X^{\tau+h}_s,u_{\tau-s+l})dl\, ds\right]. \end{align}\] Condition (iii) of Assumption 5 guarantees that the dominated convergence theorem applies, and letting \(h\rightarrow 0\), \[\begin{align} \frac{d\mathbb{E}[v^\tau(0,x)]}{d\tau} =& \mathbb{E}\left[\int_0^{\tau}e^{-\beta s}\left(\|\nabla_u\mathbf{H}^{\tau-s})\|^2-\lambda(\nabla_u \mathbf{H}^{\tau-s})^\top\nabla_u\ln\pi_{\tau-s}\right)(X^{\tau}_s,u_{\tau-s})ds\right]\nonumber\\ =&\mathbb{E}\left[\int_0^{\tau} e^{-\beta s}\left(\|\nabla_u \mathbf{H}^{\tau-s} -\frac{\lambda}{2}\nabla_u\ln\pi_{\tau-s}\|^2 -\frac{\lambda^2}{4}\|\nabla_u \ln\pi_{\tau-s}\|^2\right)(X^{\tau}_s,u_{\tau-s})ds\right]\nonumber\\ &\geq 0, \end{align}\] which holds because of assumption (ii) of the lemma, and covers \(\lambda =0\) as a special case. Thus \(\mathbb{E}[v^{\tau}(0,x)]\) is non-decreasing in \(\tau\). ◻

Proof. If \(\lambda = 0\), then \(u_\tau\) and thus \(v^\tau\) are deterministic. Lemma 4 implies that \(v^\tau\) is non-decreasing in \(\tau\), and because the value function is finite, it converges to \(v^*\). Since for any \(\tau\), \(v^\tau(x) \in [v^0(x),v^*(x)]\), it is locally bounded. Then by the boundedness of \(f,b\) and \(\sigma\), following similar arguments as for Lemma 2, \((v^\tau_t,v^\tau,v^\tau_x,v^\tau_{xx})\) converge (up to a subsequence) locally uniformly to \((v^*_t=0,v^*,v^*_x,v^*_{xx})\).

Furthermore, since the monotone function \(v^\tau(x)\) converges, 16 implies that \[-\beta v^\tau(x) + f(x,u_\tau) + b^\top(x,u_\tau)v^\tau_x(x) + \frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top(x,u_\tau):v^\tau_{xx}(x)],\] converges to \(0\). In other words, for every \(x\) \[g_\tau = f(x,u_\tau) + b^\top(x,u_\tau)v^\tau_x(x)+\frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top(x,u_\tau):v^\tau_{xx}(x)],\] converges to constant \(\beta v^*(x)\). By calculations similar to 15 , \[dg_\tau =\|\nabla_u \mathbf{H}^\tau(x,u_\tau)\|^2d\tau + b^\top(x,u_\tau)dv^\tau_x(x) +\frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top(x,u_\tau):dv^\tau_{xx}(x)].\] Since \(g_\tau, v^\tau_x(x)\) and \(v^\tau_{xx}(x)\) converge in \(\tau\), their derivative with respect to \(\tau\) should converge to \(0\), which implies that \(\nabla_u \mathbf{H}^\tau(x,u_\tau)\) converges to \(0\). Since \(\nabla_u(\mathbf{H}(x,u^*,v^*_x(x),v^*_{xx}(x)))|_{|u|=\infty} \neq 0\), \(\{u_\tau\}_{\tau\geq 0}\) is bounded, and thus up to a subsequence converges to \(u^*\), and by the continuity of \(\mathbf{H}\), \(\nabla_u (\mathbf{H}(x,u^*,v^*_x(x),v^*_{xx}(x))) =0\).

Finally, since \((v^\tau,v^\tau_x,v^\tau_{xx},u_\tau)\) converge, from ?? , \[\begin{align} 0 =& \lim_{\tau\rightarrow\infty}\left(-\beta v^\tau(x) + f(x,u_\tau) + b^\top(x,u_\tau) v^\tau_x(x)+ \frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top(x,u_\tau):v^\tau_{xx}(x)]\right)\nonumber\\ =& -\beta v^*(x) + f(x,u^*) + b^\top(x,u^*)v^*_x(x) + \frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top(x,u^*):v^*_{xx}(x)]. \end{align}\] If \(u^*(x) = \mathop{\mathrm{arg\,max}}\limits_u \mathbf{H}(x,u(x),v^*_x(x),v^*_{xx}(x))\), then \(\sup_{u}\left(-\beta v^*(x) + \mathbf{H}(x,u,v^*_x,v^*_{xx})\right) = 0\). Thus \(v^*\) satisfies the HJB equation corresponding to the control problem \(V(x) = \sup\limits_{u\in\mathcal{A}(x)} J(x,u)\), with \(u^*\) being the maximizer of the associated Hamiltonian. The verification of optimality follows standard arguments and is omitted here. ◻

Proof. Since \(v^{(i+1)0} = v^{(i)*}\), thus \[\begin{align} &dv^{(i+1)\tau}(x)|_{\tau = 0} \nonumber\\ =& \left(-\beta v^{(i)*}(x) + f(x,u^{(i+1)}_0) + b^\top(x,u^{(i+1)}_0) v^{(i)*}_x(x) + \frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top(x,u^{(i+1)}_0):v^{(i)*}_{xx}(x)]\right)d\tau. \end{align}\] By Definition 5, \[\mathbf{H}(x,u^{(i+1)}_0(x),v^{(i)*}_x(x),v^{(i)*}_{xx}(x)) > \mathbf{H}(x,u^{(i)*}(x),v^{(i)*}_x(x),v^{(i)*}_{xx}(x)).\] Thus, \[dv^{(i+1)\tau}(x)|_{\tau = 0} > \left(-\beta v^{(i)*}(x) + f(x,u^{(i)*}) + b^\top(x,u^{(i)*})v^{(i)*}_x(x) + \frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top(x,u^{(i)*}):v^{(i)*}_{xx}]\right)d\tau.\] On the other hand, since \((u^{(i)*}, v^{(i)*})\) is the limit of the \(i\)-th round of iteration, \[-\beta v^{(i)*}(x) + f(x,u^{(i)*}) + b^\top(x,u^{(i)*})v^{(i)*}_x(x) + \frac{1}{2}\mathrm{Tr}[\sigma\sigma^\top(x,u^{(i)*}):v^{(i)*}_{xx}] = 0,\] which implies that \(dv^{(i+1)\tau}(x)|_{\tau = 0} >0\). If the policy iteration follows 16 with \(\lambda=0\), Lemma 4 implies that \(v^{(i)\tau_1} \leq v^{(i)\tau_2}\) for any \(i\geq 1\) and \(\tau_1 < \tau_2 \leq \infty\). Thus in order to prove the claim, it suffices to check \(dv^{(i+1)\tau}>0\) at \(\tau = 0\) as above, because that is the only instance where we manually choose the drift, in particular the control \(u\) in the Hamiltonian \(\mathbf{H}\), of the value iteration, which otherwise should be the current iterates for \(u\), according to 16 . ◻

Proof. The convexity is immediate: (i), (iii) and (iv) from Definition 7 are stable under convex combinations, and for (ii) note that a mixture of \(\rho\) and \(\rho'\) is bounded above by \(e^{M\langle u\rangle^{2}}\), while the lower bound follows from concavity of the logarithm. As for the compactness. Let \((\rho_{k})\subset\mathcal{D}_{M}\). By (i), \(\|\rho_{k}(x,\cdot)-\rho_{j}(x,\cdot)\|_{L^{1}_{2}}\le 2M\) for every \(x\), so the region \(\{|x|>R\}\) contributes at most \(2M\langle R\rangle^{-\kappa}\) to \(\mathsf d_{\kappa}\). It therefore suffices to extract a subsequence converging uniformly on each ball \(B_{R}\) and conclude by a diagonal argument. Fix \(R\), for \(x\in B_{R}\) the densities \(\rho_{k}(x,\cdot)\) lie in a fixed relatively compact subset of \(L^{1}_{2}\): the \(H^{1}\) bound (iv) gives equicontinuity under translations in \(u\), the moment bound (i) prevents mass from escaping to \(|u|=\infty\) in the weighted norm, and the Fréchet-Riesz-Kolmogorov theorem applies, see hancheolsen2010kolmogorov? for a standard reference. Since (iii) provides equicontinuity in \(x\), Arzelà–Ascoli yields a subsequence converging uniformly on \(B_{R}\), as required. It remains to check that the limit \(\rho\) belongs to \(\mathcal{D}_{M}\). Passing to a further subsequence, \(\rho_{k}(x,\cdot)\to\rho(x,\cdot)\) a.e., so (i) and (ii) follow from Fatou’s lemma, (iii) from convergence of the \(L^{1}_{2}\)-norms, and (iv) from weak lower semicontinuity of the \(H^{1}\)-norm. ◻

Proof. We first prove the pointwise (unweighted) inequalities \[\label{eq:metric-equiv-ptw} \mathcal{W}_{2}(\mu,\nu)\le\sqrt{2}\,\|\mu-\nu\|_{L^{1}_{2}}^{1/2}, \qquad \|\mu-\nu\|_{L^{1}_{2}}\le C_{M}\,\mathcal{W}_{2}(\mu,\nu)^{\gamma},\tag{20}\] for \(\mu=\rho(x,\cdot)\), \(\nu=\rho'(x,\cdot)\) at a fixed \(x\); all constants are uniform in \(x\) and may depend on \(M,n,\delta\).

Forward bound. Let \(\eta=\mu\wedge\nu\) be the overlap, and \(\mu_{1}=(\mu-\nu)_{+}\), \(\nu_{1}=(\nu-\mu)_{+}\) the positive and negative parts of \(\mu-\nu\), respectively. Since \(\mu,\nu\) are both probability measures, \(\mu_{1}\) and \(\nu_{1}\) have equal mass \(m=\tfrac12\int_{\mathbb{R}^{n}}|\mu-\nu|\,du\), and \(\int_{\mathbb{R}^{n}}\eta\,du=1-m\). If \(m=0\) then \(\mu=\nu\). Otherwise \(\gamma_{0}=(\mathrm{id},\mathrm{id})_{\#}\eta+\tfrac1m\mu_{1}\otimes\nu_{1}\) has marginals \(\mu,\nu\), and by the definition of \(\mathcal{W}_{2}\), \[\begin{align} \mathcal{W}_{2}^{2}(\mu,\nu) &\le\int_{\mathbb{R}^{n}}\int_{\mathbb{R}^{n}}|u-v|^{2}\,d\gamma_{0} =\frac{1}{m}\int_{\mathbb{R}^{n}}\int_{\mathbb{R}^{n}}|u-v|^{2}\,\mu_{1}(du)\nu_{1}(dv)\\ &\le\frac{2}{m}\int_{\mathbb{R}^{n}}\int_{\mathbb{R}^{n}} \big(|u|^{2}+|v|^{2}\big)\mu_{1}(du)\nu_{1}(dv) =2\!\int_{\mathbb{R}^{n}}|u|^{2}\,(\mu_{1}+\nu_{1})\,du . \end{align}\] Since \(\mu_{1}+\nu_{1}=|\mu-\nu|\), \[\mathcal{W}_{2}^{2}(\mu,\nu)\le 2\!\int_{\mathbb{R}^{n}}|u|^{2}|\mu-\nu|\,du \le 2\!\int_{\mathbb{R}^{n}}\langle u\rangle^{2}|\mu-\nu|\,du=2\,\|\mu-\nu\|_{L^{1}_{2}},\] which is the first inequality in 20 .

Reverse bound. Set \(W=\mathcal{W}_{2}(\mu,\nu)\). If \(W=0\) then \(\mu=\nu\), and since \(\|\mu-\nu\|_{L^{1}_{2}}\le 2M\) by (i), the case \(W>1\) holds with \(C_{M}=2M\). Thus we focus on \(0<W\le1\). The difficulty is that \(\|\mu-\nu\|_{L^{1}_{2}}\) is a strong distance while \(W\) is a weak one: two densities can be \(\mathcal{W}_{2}\)-close yet far apart in \(L^{1}\), either by spreading mass to large \(|u|\), where the weight \(\langle u\rangle^{2}\) amplifies it, or by oscillating rapidly in \(u\). We control the first effect by the moment bound (i) and the second by the Sobolev bound (iv), through a truncation radius \(R\ge1\) and a mollification scale \(t>0\) fixed at the end. We now separate the bound as \(\|\mu-\nu\|_{L^{1}_{2}}=I_{R}+II_{R}\) with \(I_{R}=\int_{|u|\le R}\langle u\rangle^{2}|\mu-\nu|\) and \(II_{R}=\int_{|u|>R}\langle u\rangle^{2}|\mu-\nu|\). On \(\{|u|>R\}\), one has \(\langle u\rangle^{2}\le R^{-\delta}\langle u\rangle^{2+\delta}\), so (i) gives \[II_{R}\le R^{-\delta}\!\int_{\mathbb{R}^{n}}\langle u\rangle^{2+\delta}(\mu+\nu)\,du \le 2M\,R^{-\delta}.\] On the ball, \(\langle u\rangle^{2}\le 2R^{2}\), so \(I_{R}\le 2R^{2}\|\mu-\nu\|_{L^{1}(B_{R})}\), and it remains to bound the unweighted mass \(\|\mu-\nu\|_{L^{1}(B_{R})}\) by \(W\). Mollifying with the heat semigroup \(P_{t}=e^{t\Delta_{u}}\) in the control variable — a purely auxiliary smoothing over length scale \(\sqrt t\), with \(t\) unrelated to the relaxation time — we write \[\|\mu-\nu\|_{L^{1}(B_{R})} \le\|P_{t}(\mu-\nu)\|_{L^{1}}+\|(I-P_{t})(\mu-\nu)\|_{L^{1}(B_{R})},\] where the smoothed part is controlled by transport and the remainder by the \(H^{1}\) bound. We use two standard heat-semigroup facts for \(P_{t}=e^{t\Delta}\) with kernel \(p_{t}\): \(\|\nabla p_{t}\|_{L^{1}}\le C_{n}t^{-1/2}\), and \(1-e^{-t|\xi|^{2}}\le\sqrt{t}\,|\xi|\), whence \(\|(I-P_{t})g\|_{L^{2}}\le\sqrt{t}\,\|\nabla g\|_{L^{2}}\). Indeed, by \(L^{1}\)–\(L^{\infty}\) duality, self-adjointness of \(P_{t}\), the bound \(\mathrm{Lip}(P_{t}\phi)\le\|\nabla p_{t}\|_{L^{1}}\le C_{n}t^{-1/2}\) for \(\|\phi\|_{\infty}\le1\), the Kantorovich–Rubinstein duality, and \(\mathcal{W}_{1}\le\mathcal{W}_{2}\), \[\|P_{t}(\mu-\nu)\|_{L^{1}} =\sup_{\|\phi\|_{\infty}\le1}\int_{\mathbb{R}^{n}}(P_{t}\phi)\,d(\mu-\nu) \le C_{n}t^{-1/2}\,\mathcal{W}_{1}(\mu,\nu)\le C_{n}t^{-1/2}W;\] and by Cauchy–Schwarz, \(\|(I-P_{t})g\|_{L^{2}}\le\sqrt t\,\|\nabla g\|_{L^{2}}\), and assumption (iv) for \(\mu\) and \(\nu\), \[\|(I-P_{t})(\mu-\nu)\|_{L^{1}(B_{R})} \le|B_{R}|^{1/2}\|(I-P_{t})(\mu-\nu)\|_{L^{2}} \le C_{n,M}R^{n/2}\sqrt t .\] The two terms balance at \(t=WR^{-n/2}\), giving \[\|\mu-\nu\|_{L^{1}(B_{R})} \le C_{n,M}\big(t^{-1/2}W+R^{n/2}\sqrt t\big)\Big|_{t=WR^{-n/2}} =C_{n,M}\,R^{n/4}W^{1/2}.\] Combining bulk and tail, \(\|\mu-\nu\|_{L^{1}_{2}}\le C_{n,M}R^{2+n/4}W^{1/2}+2MR^{-\delta}\), which for \(0<W\le1\) is optimized at \(R=W^{-1/(2(2+n/4+\delta))}\ge1\) and yields the second inequality in 20 with \[\gamma=\frac{\delta}{2\big(2+\tfrac n4+\delta\big)} =\frac{2\delta}{\,n+8+4\delta\,}\in(0,1).\]

In the end, we show the inequalities for the weighted norm. Since \(\langle x\rangle\ge1\) and the exponents \(1/2\), \(\gamma\) lie in \((0,1]\), we have \(\langle x\rangle^{-\kappa}\le \langle x\rangle^{-\kappa\beta'}\) for any \(\beta'\in(0,1]\). Multiplying the pointwise inequalities 20 by \(\langle x\rangle^{-\kappa}\) therefore gives \(\langle x\rangle^{-\kappa} W\le\sqrt2\, (\langle x\rangle^{-\kappa}\|\mu-\nu\|_{L^{1}_{2}})^{1/2}\) and \(\langle x\rangle^{-\kappa}\|\mu-\nu\|_{L^{1}_{2}}\le C_{M}(\langle x\rangle^{-\kappa} W)^{\gamma}\) (for the case \(W>1\) use \(\|\mu-\nu\|_{L^{1}_{2}}\le 2M\) and \(\langle x\rangle^{-\kappa}\le \langle x\rangle^{-\kappa*\gamma}\) for \(\langle x\rangle^{-\kappa}\in[0,1]\)). Taking suprema over \(x\) yields ?? . ◻

Proof. (a). We first bound the entropy term. Since \(\langle u\rangle^{2}\le\langle u\rangle^{2+\delta}\), condition (i) of Definition 7 gives the second-moment bound \(\int\langle u\rangle^{2}\rho(x,u)\,du\le M\), and combined with the two-sided bound (ii), \(|\ln\rho|\le M\langle u\rangle^{2}\), \[|\mathcal{H}_{\rho}(x)|=\Big|\int\rho\ln\rho\,du\Big| \le\int\rho\,|\ln\rho|\,du \le M\!\int\langle u\rangle^{2}\rho\,du\le M^{2}.\] For the \(x\)-modulus, apply the mean-value identity \(t\ln t-t'\ln t'=(\ln\xi+1)(t-t')\) with \(\xi\) between \(t=\rho(x,u)\) and \(t'=\rho(x',u)\); by (ii), \(|\ln\xi+1|\le M\langle u\rangle^{2}+1\), so (iii) yields \[|\mathcal{H}_{\rho}(x)-\mathcal{H}_{\rho}(x')| \le\int(M\langle u\rangle^{2}+1)\,|\rho(x,\cdot)-\rho(x',\cdot)|\,du \le(M+1)\,\|\rho(x,\cdot)-\rho(x',\cdot)\|_{L^{1}_{2}} \le C_{M}|x-x'|^{\alpha'}.\] Hence \(\mathcal{H}_{\rho}\in C^{0,\alpha'}_{b}\) with norm \(\le C_{M}\). The same computation, with the \(C^{0,\alpha'}_{b}\)-in-\(x\), bounded-in-\(u\) maps \(\sigma\sigma^{\top},b,f\) (Assumption 6) in place of \(\ln\rho+1\), gives \(\bar a_{\rho},\bar b_{\rho},\tilde{f}(\cdot,\rho)\in C^{0,\alpha'}_{b}\) with norm \(\le C_{M}\), and therefore \(\bar f_{\rho}=\tilde{f}(\cdot,\rho)-\lambda\mathcal{H}_{\rho}\in C^{0,\alpha'}_{b}\). Finally, averaging \(\sigma\sigma^{\top}\succeq\theta_{0}I_{d}\) against \(\rho\) preserves the inequality, so \(\bar a_{\rho}\succeq\theta_{0}I_{d}\).

(b). Fix \(\rho,\rho'\in\mathcal{D}_{M}\) and \(x\in\mathbb{R}^{d}\). The same bracketing as in (a), now comparing \(\rho\) and \(\rho'\) at the common point \(x\), gives \(|\mathcal{H}_{\rho}(x)-\mathcal{H}_{\rho'}(x)| \le(M+1)\|\rho(x,\cdot)-\rho'(x,\cdot)\|_{L^{1}_{2}}\), and likewise for \(\bar a,\bar b,\tilde{f}(\cdot,\rho)\), which implies ?? . Multiplying by \(\langle x\rangle^{-\kappa}\) and taking the supremum over \(x\) gives the weighted Lipschitz bound. For the Hölder upgrade, set \(g=\bar f_{\rho}-\bar f_{\rho'}\), which by (a) is bounded in \(C^{0,\alpha'}_{b}\) by \(2C_{M}\) while \(\|g\|_{C^{0}_{b}}\le C_{M}\mathsf d_{\infty}(\rho,\rho')\) by ?? . The interpolation inequality \(\|g\|_{C^{0,\alpha}_{b}}\le C\,\|g\|_{C^{0}_{b}}^{1-\alpha/\alpha'} \|g\|_{C^{0,\alpha'}_{b}}^{\alpha/\alpha'}\) krylov1996lectures? then gives \(\|g\|_{C^{0,\alpha}_{b}}\le C_{M}\mathsf d_{\infty}(\rho,\rho')^{\theta}\).

(c). The marginal \(\pi_{\tau}(x,\cdot)=\mathrm{Law}(u^{x}_{\tau})\) solves the Fokker–Planck equation \[\partial_{\tau}\pi^{\tau} =\lambda\Delta_{u}\pi^{\tau}-\mathrm{div}_{u}\big(G\,\pi^{\tau}\big),\] which is uniformly parabolic in \(u\) (since \(\lambda>0\)), with a drift \(G\) of linear growth in \(u\) and bounded in \(C^{0,\alpha'}_{b}\) in \(x\). We check that the four defining conditions of \(\mathcal{D}_{M}\) are preserved on \([0,T]\), starting from \(\pi_0 = \mathrm{Law}(u_{0})\in\mathcal{D}_{M_{0}}\). The two-sided bound (ii) propagates by Aronson’s Gaussian estimates for non-degenerate parabolic equations Aronson1967?, Sheu1991?. The \((2+\delta)\)-moment (i) follows from Itô’s formula applied to \(\langle u^{x}_{\tau}\rangle^{2+\delta}\) and Grönwall, using the linear growth of \(G\). The spatial modulus (iii) follows from the weighted-\(L^{1}\) estimate for the equation satisfied by the difference \(\rho^{\tau}(x,\cdot)-\rho^{\tau}(x',\cdot)\), whose source is controlled by the \(C^{0,\alpha'}_{b}\) regularity of \(G\). The Sobolev bound (iv) follows from the standard parabolic energy estimate, which preserves — and for \(\tau>0\) improves — \(\|\pi_{\tau}(x,\cdot)\|_{H^{1}}\), with constants depending only on \(\|G\|_{C^{0,\alpha'}_{b}}\) and \(T\). Hence \(\pi_{\tau}\in\mathcal{D}_{M}\) for all \(\tau\le T\), with \(M=M(T,M_{0})\). In particular, under Assumption 6 (ii), the dissipative condition of \(\nabla_u\mathbf{H}\) caps its moments uniformly, independent of \(M_0\), and thus \(M=M(T)\). ◻

Proof. Proof of ?? . \(v^\tau\) defined in ?? is the Feynman–Kac representation of ??; boundedness of the running cost (Lemma 7(a)) and \(\beta>0\) make it the unique bounded classical solution, by the maximum principle for parabolic equations friedman1964parabolic?. By Assumption 1 and Lemma 7(a), ?? is uniformly parabolic with coefficients bounded in \(C^{0,\alpha'}_{b}\) uniformly in \(t\). The global parabolic Schauder estimate krylov1996lectures?, combined with the exponential damping furnished by \(-\beta\), gives \(\sup_{t}\|v(t,\cdot)\|_{C^{2,\alpha'}_{b}}\le C_{M}\) (see also lunardi1995analytic?).

Proof of ?? . The difference \(w:=v[\boldsymbol{\rho}]-v[\boldsymbol{\rho}']\) solves ?? (with policy \(\boldsymbol{\rho}\)) with the additional source \[g_{t}=(\bar f_{\rho_{t}}-\bar f_{\rho'_{t}}) +(\bar{\mathcal{L}}_{\rho_{t}}-\bar{\mathcal{L}}_{\rho'_{t}}) v[\boldsymbol{\rho}'] .\] By the pointwise Lipschitz bound ?? of Lemma 7(b) and the uniform bound ?? applied to \(v[\boldsymbol{\rho}']\), \[\label{eq:source-ptw} |g_{t}(x)| \le C_{M}\big(1+\|v[\boldsymbol{\rho}'](t,\cdot)\|_{C^{2}_{b}}\big) \|\rho_{t}(x,\cdot)-\rho'_{t}(x,\cdot)\|_{L^{1}_{2}} \le C_{M}\,\langle x\rangle^{\kappa}\,\bar{\mathsf d}_{\kappa} \qquad\text{for all }(t,x).\tag{21}\] Let \(X^{t,x}\) denote the diffusion generated by \(\bar{\mathcal{L}}_{\rho_{\cdot}}\) started at \(x\) at time \(t\). Since \(\bar b_{\rho},\bar a_{\rho}\) are bounded uniformly over \(\mathcal{D}_{M}\) (Lemma 7(a)), the Burkholder–Davis–Gundy inequality gives the polynomial moment estimate \[\label{eq:X-moment} \mathbb{E}\big[\langle X^{t,x}_{t+s}\rangle^{\kappa}\big] \le C_{\kappa}\big(\langle x\rangle^{\kappa}+1+s^{\kappa}\big), \qquad s\ge0 .\tag{22}\] The Feynman–Kac representation of \(w\), \(w(t,x)=\mathbb{E}\int_{0}^{\infty}e^{-\beta s}g_{t+s}(X^{t,x}_{t+s})\,ds\), together with 21 –22 , yields \[|w(t,x)| \le C_{M}\,\bar{\mathsf d}_{\kappa} \int_{0}^{\infty}e^{-\beta s}\, \mathbb{E}\big[\langle X^{t,x}_{t+s}\rangle^{\kappa}\big]\,ds \le C_{M,\beta,\kappa}\,\langle x\rangle^{\kappa}\,\bar{\mathsf d}_{\kappa},\] since the polynomial growth in \(s\) in 22 is integrable against \(e^{-\beta s}\) for every \(\beta>0\). This implies ?? .

Proof of ?? . Fix \(x\). By the interpolation inequality on balls krylov1996lectures?, \[\|w(t,\cdot)\|_{C^{2,\alpha}(B_{1}(x))} \le C\,\|w(t,\cdot)\|_{C^{0}(B_{2}(x))}^{1-\lambda_{*}} \|w(t,\cdot)\|_{C^{2,\alpha'}(B_{2}(x))}^{\lambda_{*}}, \qquad \lambda_{*}=\tfrac{2+\alpha}{2+\alpha'} .\] On \(B_{2}(x)\) one has \(\langle y\rangle^{\kappa}\le C\langle x\rangle^{\kappa}\), the pointwise form of ?? , \(|w(t,y)|\le C_{M,\beta,\kappa}\langle y\rangle^{\kappa}\bar{\mathsf d}_{\kappa}\), yields \(\|w(t,\cdot)\|_{C^{0}(B_{2}(x))} \le C_{M,\beta,\kappa}\,\langle x\rangle^{\kappa}\,\bar{\mathsf d}_{\kappa}\), and ?? bounds the second factor by \((2C_{M})^{\lambda_{*}}\). Hence \(\|w(t,\cdot)\|_{C^{2,\alpha}(B_{1}(x))} \le C\,\langle x\rangle^{\kappa(1-\lambda_{*})} \bar{\mathsf d}_{\kappa}^{\,1-\lambda_{*}}\), and since \(\langle x\rangle^{-\kappa}\le\langle x\rangle^{-\kappa(1-\lambda_{*})}\), ?? follows. ◻

Proof. The proof has five steps: we (1) define a convex set of policy flows \(\mathcal{K}_{T}\) and a map \(\Phi\) on it whose fixed points are exactly the solutions of the coupled system on \([0,T]\); (2) show \(\Phi(\mathcal{K}_{T})\subseteq\mathcal{K}_{T}\); (3) show \(\mathcal{K}_{T}\) is compact; (4) show \(\Phi\) is continuous; (5) apply Schauder’s theorem and extend the solution to \([0,\infty)\) by concatenation.

Step 1 (The fixed-point set and the map \(\Phi\)). Fix \(T>0\) and let \(M=M(T,M_{0})\) be the constant from Lemma 7(c). With a constant \(C_{T}\) determined in Step 2, define \[\mathcal{K}_{T}:=\Big\{\boldsymbol{\rho}=(\rho_{\tau})_{\tau\in[0,T]} \in C([0,T];\mathcal{D}_{M}):\; \mathsf d_{\kappa}(\rho_{\tau},\rho_{\tau'}) \le C_{T}|\tau-\tau'|^{1/2}\;\;\forall\,\tau,\tau'\Big\},\] which is nonempty (it contains constant-in-\(\tau\) flows) and convex, since \(\mathcal{D}_{M}\) is convex (Lemma 5) and the time modulus is stable under convex combinations. Given \(\boldsymbol{\rho}\in\mathcal{K}_{T}\), define \(\Phi(\boldsymbol{\rho})\) in two stages:

Value stage. Define \(\bar{\boldsymbol{\rho}} = \{\bar\rho_t\}_{t\geq 0}\) by \[\bar\rho_t = \begin{cases} \rho_{T-t}, &0\leq t \leq T;\\ \rho_0, & t > T. \end{cases}\] Let \(v^{\tau}:=v[\bar{\boldsymbol{\rho}}](0,\cdot)\). By Lemma 8, \(\sup_{\tau\le T}\|v^{\tau}\|_{C^{2,\alpha'}_{b}}\le C_{M}\), and by ?? the map \(\boldsymbol{\rho}\mapsto(\nabla v^{\tau},\nabla^{2}v^{\tau})\) is continuous from \((\mathcal{K}_{T},\sup_{\tau}\mathsf d_{\kappa})\) into \(C([0,T];C^{1,\alpha}(B_{R}))\) for every \(R>0\).

Policy stage. With this \(v^{\tau}\), the drift \(G(\tau,x,u)=\nabla_{u}\mathbf{H}\big(x,u,\nabla v^{\tau}(x), \nabla^{2}v^{\tau}(x)\big)\) in ?? (which is same as 4 ) is Lipschitz with linear growth in \(u\) and bounded in \(C^{0,\alpha'}_{b}\) in \(x\), uniformly in \(\tau\le T\), by the bound in (a). Solve 4 and denote \(\Phi(\boldsymbol{\rho})_{\tau}(x,\cdot):=\mathrm{Law}(u^{x}_{\tau})\).

By construction, \(\boldsymbol{\rho}^{\star}=\Phi(\boldsymbol{\rho}^{\star})\) if and only if \((v^{\tau},(u^{x}_{\tau})_{x})_{\tau\le T}\) solves the coupled system 4 –?? on \([0,T]\).

Step 2 (Invariance: \(\Phi(\mathcal{K}_{T})\subseteq\mathcal{K}_{T}\)). \(\Phi(\boldsymbol{\rho})_{\tau}\in\mathcal{D}_{M}\) for all \(\tau\le T\) is exactly Lemma 7(c), since the drift \(G\) satisfies its hypotheses by Step 1(b). For the time modulus, write \(\tilde{\rho}_{\tau}:=\Phi(\boldsymbol{\rho})_{\tau}\) and recall from Lemma 7(c) that, for each \(x\), \(\tilde{\rho}_{\tau}(x,\cdot)\) solves the Fokker–Planck equation \(\partial_{\tau}\tilde{\rho}=\lambda\Delta_{u}\tilde{\rho} -\mathrm{div}_{u}(G\tilde{\rho})\). Integrating in time and taking the \(L^{1}_{2}\)-norm, for each fixed \(x\), \[\|\tilde{\rho}_{\tau}(x,\cdot)-\tilde{\rho}_{\tau'}(x,\cdot)\|_{L^{1}_{2}} \le\int_{\tau'}^{\tau} \big\|\lambda\Delta_{u}\tilde{\rho}_{s} -\mathrm{div}_{u}(G\tilde{\rho}_{s})\big\|_{L^{1}_{2}}\,ds \le C_{T}'\,|\tau-\tau'|^{1/2},\] since \(\mathrm{div}_{u}G\) is bounded and \(|G|\le C\langle u\rangle\), the moment bound (i) gives \(\|(\mathrm{div}_{u}G)\tilde{\rho}_{s}\|_{L^{1}_{2}}\le C_{T}\). On the other hand, since the diffusion coefficient is a constant, the parabolic estimate for \(\|\Delta_{u}\tilde{\rho}^{s}\|_{L^{1}_{2}}\) degenerates like \(s^{-1/2}\) near \(s=0\). Integrating it directly leads to an upper bound which is multiple of \(|\tau-\tau'|^{1/2}\). Since the weight satisfies \(\langle x\rangle^{-\kappa}\le1\), and \(|\tau-\tau'|\le T^{1/2}|\tau-\tau'|^{1/2}\) on \([0,T]\), taking \(\sup_{x}\) gives \(\mathsf d_{\kappa}(\tilde{\rho}^{\tau},\tilde{\rho}^{\tau'}) \le C_{T}|\tau-\tau'|^{1/2}\) with \(C_{T}:=T^{1/2}C_{T}'\). Hence \(\Phi(\boldsymbol{\rho})\in\mathcal{K}_{T}\).

Step 3 (Compactness of \(\mathcal{K}_{T}\)). By Lemma 5, \((\mathcal{D}_{M},\mathsf d_{\kappa})\) is compact, and \(\mathcal{K}_{T}\subset C([0,T];\mathcal{D}_{M})\) is equicontinuous in \(\tau\) by the uniform \(\tfrac12\)-Hölder time modulus. The Arzelà–Ascoli theorem therefore shows that \(\mathcal{K}_{T}\) is compact in the metric \(\sup_{\tau\le T}\mathsf d_{\kappa}\), and closedness of the modulus condition under this convergence is immediate.

Step 4 (Continuity of \(\Phi\)). Let \(\boldsymbol{\rho}^{k}\to\boldsymbol{\rho} \in \mathcal{K}_T\) in \(\sup_{\tau}\mathsf d_{\kappa}\). By ?? , \(\nabla v[\boldsymbol{\rho}^{k}],\nabla^{2}v[\boldsymbol{\rho}^{k}] \to\nabla v[\boldsymbol{\rho}],\nabla^{2}v[\boldsymbol{\rho}]\) uniformly on \([0,T]\times B_{R}\) for every \(R>0\). Hence, for each fixed \(x\), the frozen-\(x\) drifts \(G^{k}(\cdot,x,\cdot)\to G(\cdot,x,\cdot)\) locally uniformly, with Lipschitz and growth constants uniform in \((k,x)\). Standard stability of SDEs with respect to convergence of coefficients then gives \[\sup_{\tau\le T}\;\mathcal{W}_{2}\big(\Phi(\boldsymbol{\rho}^{k})^{\tau}(x,\cdot), \Phi(\boldsymbol{\rho})^{\tau}(x,\cdot)\big)\longrightarrow0, \qquad\text{uniformly for }x\in B_{R},\;\forall R>0 .\] Moreover \(\mathcal{W}_{2}\) is bounded on \(\mathcal{D}_{M}\): by the moment bound (i), \(\mathcal{W}_{2}(\mu,\nu)^{2}\le2\int|u|^{2}(\mu+\nu)\le4M\). Splitting \(\sup_{x}=\sup_{|x|\le R}+\sup_{|x|>R}\) and using \(\langle x\rangle^{-\kappa}\le\langle R\rangle^{-\kappa}\) on the tail, we conclude \(\sup_{\tau}\mathcal{W}_{2,\kappa}\big(\Phi(\boldsymbol{\rho}^{k})^{\tau}, \Phi(\boldsymbol{\rho})^{\tau}\big)\to0\). Since all laws involved lie in \(\mathcal{D}_{M}\) by Step 2, the Hölder equivalence of \(\mathcal{W}_{2,\kappa}\) and \(\mathsf d_{\kappa}\) on \(\mathcal{D}_{M}\) (Lemma 6) yields \(\Phi(\boldsymbol{\rho}^{k})\to\Phi(\boldsymbol{\rho})\) in \(\sup_{\tau}\mathsf d_{\kappa}\), i.e.\(\Phi\) is continuous on \(\mathcal{K}_{T}\).

Step 5 (Fixed point and global extension). By Steps 2–4, \(\Phi\) is a continuous self-map of the nonempty convex and compact set \(\mathcal{K}_{T}\), so Schauder’s fixed point theorem yields \(\boldsymbol{\rho}^{\star}=\Phi(\boldsymbol{\rho}^{\star})\), i.e.a solution of the coupled system on \([0,T]\), with \(\rho^*_{\tau}\in\mathcal{D}_{M}\) and \(\sup_{\tau\le T}\|v^{\tau}\|_{C^{2,\alpha'}_{b}}\le L:=C_{M}\) by Lemma 8. Since the constant \(L\) does not depend on \(T\), we may iterate: for each \(k\ge 1\), the terminal law \(\mathrm{Law}(u^{x}_{kT})\in\mathcal{D}_{M}\) serves as initial datum for the next window \([kT,(k+1)T]\), and the solutions concatenate into a solution on \([0,\infty)\). ◻

Proof. The difference between the coupled processes \(v^{\tau}\) (and \(v^{\tau}_x\), \(v^\tau_{xx}\) respectively\()\) and \(\tilde{v}^{\tau}\) (and \(\tilde{v}^{\tau}_x\), \(\tilde{v}^\tau_{xx}\) respectively\()\) follows the dynamics below (omitting the argument \(x\) for simplicity), \[\begin{align} d(v^{\tau}-\tilde{v}^{\tau})=&-\beta(v^\tau-\tilde{v}^\tau)d\tau \\ &+\Big[ \int_{\mathbb{R}^n} (\mathbf{H}^\tau- \lambda \ln\pi_\tau)\pi_\tau(x,u)du -\int_{\mathbb{R}^n}(\tilde{\mathbf{H}}^\tau -\lambda\ln\tilde{\pi}_\tau)\tilde{\pi}_\tau(x,\tilde{u})d\tilde{u} \Big]d\tau,\\ d(v^{\tau}_x-\tilde{v}^{\tau}_x)=&-\beta(v^\tau_x-\tilde{v}_x^\tau) d\tau\\ &+ \nabla_x \Big[\int_{\mathbb{R}^n}(\mathbf{H}^\tau- \lambda \ln \pi_\tau)\pi_\tau(x,u)\,du -\int_{\mathbb{R}^n}(\tilde{\mathbf{H}}^\tau -\lambda\ln\tilde{\pi}_\tau)\tilde{\pi}_\tau(x,\tilde{u})\,d\tilde{u} \Big]d\tau ,\\ d(v^{\tau}_{xx}-\tilde{v}^{\tau}_{xx})=& -\beta(v^\tau_{xx}-\tilde{v}_{xx}^\tau)d\tau\\ &+\nabla^2_{xx}\Big[ \int_{\mathbb{R}^n} (\mathbf{H}^\tau- \lambda \ln \pi_\tau)\pi_\tau(x,u)\,du -\int_{\mathbb{R}^n}(\tilde{\mathbf{H}}^\tau -\lambda\ln\tilde{\pi}_\tau)\tilde{\pi}_\tau(x,\tilde{u})\,d\tilde{u}\Big] d\tau . \end{align}\] We show the convergence for \(v^{\tau}_x-\tilde{v}^{\tau}_x\), and the estimates for \(\|v^{\tau}(x)-\tilde{v}^{\tau}(x)\|^2\) and \(\|v_{xx}^{\tau}(x)-\tilde{v}_{xx}^{\tau}(x)\|^2\) follow similarly. From its dynamics above, \[\begin{align} &\frac{d}{d\tau} \|v^{\tau}_x-\tilde{v}^{\tau}_x\|^2\\ =&2\Big(\nabla_x\Big[ \int_{\mathbb{R}^n} (\mathbf{H}^\tau- \lambda \ln\pi_\tau)\pi_\tau(x,u)\,du -\int_{\mathbb{R}^n}(\tilde{\mathbf{H}}^\tau -\lambda\ln\tilde{\pi}_\tau)\tilde{\pi}_\tau(x,\tilde{u})\,d\tilde{u} \Big]\Big)^\top(v^{\tau}_x-\tilde{v}^{\tau}_x)\\ &-2\beta \|v^{\tau}_x-\tilde{v}^{\tau}_x\|^2 \\ \le & -2\beta\|v^{\tau}_x-\tilde{v}^{\tau}_x\|^2, \end{align}\] where the last inequality follows from the monotonicity condition in Assumption 7. This then implies \[\label{convergence:32v39} \|v^{\tau}_x(x)-\tilde{v}^{\tau}_x(x)\|^2\le e^{-2\beta \tau} \|v^0_x(x)-\tilde{v}^0_x(x)\|^2.\tag{25}\]  ◻

Proof. Motivated by synchronous coupling techniques for Langevin dynamics EGZ19?, Dalalyan2020?, we adapt this idea to the inhomogeneous Langevin dynamics governing the control process \(u_\tau\) and derive a convergence rate for the distribution of the randomized control in our policy iteration scheme under the monotonicity condition. Following the proof of Lemma 9, denote as \((v^{\tau}_x,u_{\tau})\) and \((\tilde{v}^{\tau}_x,\tilde{u}_{\tau})\) the two coupled processes driven by the same Brownian motion \((B_{\tau})_{\tau\ge 0}\). The difference \(u_{\tau}-\tilde{u}_{\tau}\) follows \[d(u_\tau-\tilde{u}_\tau)=[\nabla_u \mathbf{H}^\tau(u_\tau) -\nabla_{u} \tilde{\mathbf{H}}^\tau(\tilde{u}_\tau)]d\tau.\] Next, we show the convergence of the difference process \(u_\tau-\tilde{u}_\tau\). Observe that \[\begin{align} &\nabla_u \mathbf{H}^\tau(u_\tau) -\nabla_{u} \tilde{\mathbf{H}}^\tau(\tilde{u}_\tau)\\ =& \nabla_u f (x,u_\tau) -\nabla_{u} f(x,\tilde{u}_\tau)+ \sum_{i=1}^n\nabla_u b_i(x,u_\tau)v^{\tau}_{x,i}(x)-\sum_{i=1}^n\nabla_{u} b_i(x,\tilde{u}_\tau) \tilde{v}^{\tau}_{x,i}(x)\\ &+\sum_{i,j,k=1}^n\sigma_{ik}(x,u_\tau)\nabla_u\sigma_{jk}(x,u_\tau)v^{\tau}_{xx,ij}(x)-\sum_{i,j,k=1}^n\sigma_{ik}(x,\tilde{u}_\tau)\nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau)\tilde{v}^{\tau}_{xx, ij}(x)\\ =&\int_0^1\frac{d}{dr} \nabla_u f(x,r u_\tau+(1-r)\tilde{u}_\tau ) dr\\ &+\sum_{i=1}^n\Big(v^{\tau}_{x,i}(x)\int_0^1 \frac{d}{dr}\nabla_u b_i(x,r u_\tau+(1-r)\tilde{u}_\tau ) dr +\nabla_{u}b_i(x,\tilde{u}_\tau)[v^{\tau}_{x,i}(x)-\tilde{v}^{\tau}_{x,i}(x)]\Big)\\ &+\sum_{i,j,k=1}^n \Big(\sigma_{ik}(x,u_\tau)v^{\tau}_{xx,ij}\int_0^1 \frac{d}{dr}\nabla_u \sigma_{jk}(x,r u_\tau+(1-r)\tilde{u}_\tau ) dr\Big.\\ &\Big.+\sigma_{ik}(x,u_\tau)\nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau)[v^{\tau}_{xx,ij}-\tilde{v}^{\tau}_{xx,ij}]\Big) \\ &+\sum_{i,j,k=1}^n\tilde{v}^{\tau}_{xx,ij}\nabla_u\sigma_{jk}(x,\tilde{u}_\tau)\int_0^1 \frac{d}{dr}\sigma_{ik}(x,r u_\tau+(1-r)\tilde{u}_\tau ) dr \\ =&\int_0^1\langle u_\tau-\tilde{u}_\tau, \nabla^2_{uu} f(x,r u_\tau+(1-r)\tilde{u}_\tau ) \rangle dr\\ &+ \sum_{i=1}^nv^{\tau}_{x,i}(x) \int_0^1 \langle u_\tau-\tilde{u}_\tau, \nabla^2_{uu} b_i(x,r u_\tau+(1-r)\tilde{u}_\tau ) \rangle dr +\sum_{i=1}^n\nabla_{u}b_i(x,\tilde{u}_\tau)[v^{\tau}_{x,i}(x)-\tilde{v}^{\tau}_{x,i}(x)]\\ &+\sum_{i,j,k=1}^n\sigma_{ik}(x,u_\tau)v^{\tau}_{xx,ij}\int_0^1 \langle u_\tau-\tilde{u}_\tau, \nabla^2_{uu} \sigma_{jk}(x,r u_\tau+(1-r)\tilde{u}_\tau ) \rangle dr\\ &+\sum_{i,j,k=1}^n\sigma_{ik}(x,u_\tau)\nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau)[v^{\tau}_{xx,ij}-\tilde{v}^{\tau}_{xx,ij}] \\ &+\sum_{i,j,k=1}^n \tilde{v}^{\tau}_{xx,ij}\nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau)\int_0^1 \langle u_\tau-\tilde{u}_\tau, \nabla_{u} \sigma_{ik}(x,r u_\tau+(1-r)\tilde{u}_\tau ) \rangle dr\\ =& \mathbf{U}^f_\tau (u_\tau-\tilde{u}_\tau) +\sum_{i=1}^nv^{\tau}_{x,i}(x) \mathbf{U}^{b_i}_\tau (u_\tau-\tilde{u}_\tau)+\sum_{i=1}^n\nabla_{u}b_i(x,\tilde{u}_\tau)[v^{\tau}_{x,i}(x)-\tilde{v}^\tau_{x,i}(x)]\\ &+\sum_{i,j,k=1}^n\mathbf{U}^{\sigma_{jk}}_\tau (u_\tau-\tilde{u}_\tau)\sigma_{ik}(x,u_\tau)v^{\tau}_{xx,ij}+\sum_{i,j,k=1}^n\sigma_{ik}(x,u_\tau)\nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau)[v^{\tau}_{xx,ij}-\tilde{v}^{\tau}_{xx,ij}]\\ &+\sum_{i,j,k=1}^n(\hat{\mathbf{U}}^{\sigma_{ik}}_\tau)^\top(u_\tau-\tilde{u}_\tau)\tilde{v}^{\tau}_{xx,ij}\nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau), \end{align}\] where we denote \[\begin{align} \mathbf{U}^f_\tau :=& \int_0^1 \nabla^2_{uu} f(x, r u_\tau+(1-r)\tilde{u}_\tau ) dr,\nonumber\\ \mathbf{U}^{b_i}_\tau :=& \int_0^1 \nabla^2_{uu} b_i(x,r u_\tau+(1-r)\tilde{u}_\tau ) dr,\quad \text{for}\quad i=1,\cdots,n,\nonumber\\ \mathbf{U}^{\sigma_{ij}}_\tau :=& \int_0^1 \nabla^2_{uu} \sigma_{ij}(x,r u_\tau+(1-r)\tilde{u}_\tau ) dr, \quad \text{for}\quad i,j=1,\cdots,n,\nonumber\\ \hat{\mathbf{U}}^{\sigma_{ij}}_\tau :=& \int_0^1 \nabla_{u} \sigma_{ij}(x,r u_\tau+(1-r)\tilde{u}_\tau ) dr, \quad \text{for}\quad i,j=1,\cdots,n. \end{align}\] Plugging the above terms into the difference equation for the control process, we have \[\begin{align} &\frac{d}{d\tau} (u_\tau-\tilde{u}_\tau)\\ =&\mathbf{U}^f_\tau (u_\tau-\tilde{u}_\tau) +\sum_{i=1}^nv^{\tau}_{x,i}(x) \mathbf{U}^{b_i}_\tau (u_\tau-\tilde{u}_\tau)+\sum_{i=1}^n\nabla_{ u}b_i(x,\tilde{u}_\tau)[v^{\tau}_{x,i}(x)-\tilde{v}^{\tau}_{x,i}(x)]\\ &+\sum_{i,j,k=1}^n\mathbf{U}^{\sigma_{jk}}_\tau (u_\tau-\tilde{u}_\tau)\sigma_{ik}(x,u_\tau)v^{\tau}_{xx,ij}+\sum_{i,j,k=1}^n\sigma_{ik}(x,u_\tau)\nabla_{ u}\sigma_{jk}(x,\tilde{u}_\tau)[v^{\tau}_{xx,ij}-\tilde{v}^{\tau}_{xx,ij}]\\ &+\sum_{i,j,k=1}^n(\hat{\mathbf{U}}^{\sigma_{ik}}_\tau)^\top(u_\tau-\tilde{u}_\tau)\tilde{v}^{\tau}_{xx,ij}\nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau)\\ =&\Big[\mathbf{U}^f_\tau +\sum_{i=1}^nv^{\tau}_{x,i}(x) \mathbf{U}^{b_i}_\tau+ \sum_{i,j,k=1}^n\mathbf{U}^{\sigma_{jk}}_\tau \sigma_{ik}(x,u_\tau)v^{\tau}_{xx,ij}\\ &+\sum_{i,j,k=1}^nv^{\tau}_{xx,ij}\hat{\mathbf{U}}^{\sigma_{ik}}_\tau(\nabla_{ u}\sigma_{jk}(x,\tilde{u}_\tau))^\top\Big] (u_\tau-\tilde{u}_\tau)\\ &+\sum_{i=1}^n\nabla_{u}b_i(x,\tilde{u}_\tau)[v^{\tau}_{x,i}(x)-\tilde{v}^{\tau}_{x,i}(x)]+\sum_{i,j,k=1}^n\sigma_{ik}(x,u_\tau)\nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau)[v^{\tau}_{xx,ij}-\tilde{v}^{\tau}_{xx,ij}]. \end{align}\] Applying the above equation and Young’s inequality (\(2ab\le \varepsilon a^2+\frac{1}{\varepsilon}b^2\), for \(\varepsilon>0\)), \[\begin{align} \label{difference32u} &\frac{d}{d\tau} \|u_\tau-\tilde{u}_\tau\|^2\nonumber\\ =&2(u_\tau-\tilde{u}_\tau)^\top\Big[\mathbf{U}^f_\tau +\sum_{i=1}^nv^{\tau}_{x,i}(x) \mathbf{U}^{b_i}_\tau+ \sum_{i,j,k=1}^n\mathbf{U}^{\sigma_{jk}}_\tau \sigma_{ik}(x,u_\tau)v^{\tau}_{xx,ij}\nonumber \\ &\quad \quad \quad\quad \quad\quad +\sum_{i,j,k=1}^n\hat{\mathbf{U}}^{\sigma_{ik}}_\tau( \nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau))^\top v^{\tau}_{xx,ij}\Big] (u_\tau-\tilde{u}_\tau) \nonumber\\ &+2\sum_{i=1}^n(\nabla_{u}b_i(x,\tilde{u}_\tau))^\top[u_\tau-\tilde{u}_\tau][v^{\tau}_{x,i}(x)-\tilde{v}^{\tau}_{x,i}(x)]\nonumber\\ &+2\sum_{i,j,k=1}^n\sigma_{ik}(x,u_\tau)[v^{\tau}_{xx,ij}-\tilde{v}^{\tau}_{xx,ij}](\nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau))^\top[u_\tau-\tilde{u}_\tau]\nonumber\\ \le &2(u_\tau-\tilde{u}_\tau)^\top\Big[\mathbf{U}^f_\tau +\sum_{i=1}^nv^{\tau}_{x,i}(x) \mathbf{U}^{b_i}_\tau+ \sum_{i,j,k=1}^n\mathbf{U}^{\sigma_{jk}}_\tau \sigma_{ik}(x,u_\tau)v^{\tau}_{xx,ij}\nonumber \\ &\quad \quad \quad\quad \quad\quad +\sum_{i,j,k=1}^n\hat{\mathbf{U}}^{\sigma_{ik}}_\tau( \nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau))^\top v^{\tau}_{xx,ij}\Big] (u_\tau-\tilde{u}_\tau) \nonumber\\ &+(\sum_{i=1}^n\varepsilon_{1i}+\sum_{i,j=1}^n\varepsilon_{2ij})\|u_\tau-\tilde{u}_\tau\|^2+ \sum_{i=1}^n\frac{1}{\varepsilon_{1i}}\| \nabla_{u}b_i(x,\tilde{u}_\tau)\|^2|v^{\tau}_{x,i}(x)-\tilde{v}^{\tau}_{x,i}(x)|^2\nonumber\\ &+\sum_{i,j=1}^n\frac{1}{\varepsilon_{2ij}}|\sigma_{ik}(x,u_\tau)|^2|v^{\tau}_{xx,ij}-\tilde{v}^{\tau}_{xx,ij}|^2\|\nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau)\|^2, \end{align}\tag{26}\] where \(\varepsilon_{1i}\)’s and \(\varepsilon_{2ij}\)’s are any positive constants, satisfying \(\max\left\{\varepsilon_{1i},\varepsilon_{2ij},1\leq i,j\leq n\right\} = \overline{\varepsilon}(x)\). Notice that by Assumption 1, \(\nabla_u b(x,u_\tau)\), \(\sigma\) and \(\nabla_u \sigma\) are bounded. Furthermore, Lemma 9 implies that \(v^\tau_x\) and \(v^\tau_{xx}\) converge, and thus are also bounded. Therefore there exists \(L>0\), such that the bound in ?? holds. Applying Lemma 9 and using the upper bound \(L\), the above inequality implies that \[\begin{align} & \frac{d}{d\tau} \|u_\tau-\tilde{u}_\tau\|^2\\ \le & 2(u_\tau-\tilde{u}_\tau)^\top\Big[\mathbf{U}^f_\tau +\sum_{i=1}^nv^{\tau}_{x,i}(x) \mathbf{U}^{b_i}_\tau+ \sum_{i,j,k=1}^n\mathbf{U}^{\sigma_{jk}}_\tau \sigma_{ik}(x,u_\tau)v^{\tau}_{xx,ij}\nonumber \\ &\quad \quad \quad\quad \quad\quad +\sum_{i,j,k=1}^n\hat{\mathbf{U}}^{\sigma_{ik}}_\tau( \nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau))^\top v^{\tau}_{xx,ij}+\overline{\varepsilon}(x)(n+n^2)\mathbf{I}_n\Big] (u_\tau-\tilde{u}_\tau) \nonumber\\ &+\frac{1}{\underline{\varepsilon}(x)}L\left(\|v^{\tau}_{x}(x)-\tilde{v}^{\tau}_{x}(x)\|^2+L\|v^{\tau}_{xx}(x)-\tilde{v}^{\tau}_{xx}(x)\|^2\right)\\ \le &2(u_\tau-\tilde{u}_\tau)^\top\Big[\mathbf{U}^f_\tau +\sum_{i=1}^nv^{\tau}_{x,i}(x) \mathbf{U}^{b_i}_\tau+ \sum_{i,j,k=1}^n\mathbf{U}^{\sigma_{jk}}_\tau \sigma_{ik}(x,u_\tau)v^{\tau}_{xx,ij}\nonumber \\ &\quad \quad \quad\quad \quad\quad +\sum_{i,j,k=1}^n\hat{\mathbf{U}}^{\sigma_{ik}}_\tau( \nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau))^\top v^{\tau}_{xx,ij}+\overline{\varepsilon}(x)(n+n^2)\mathbf{I}_n\Big] (u_\tau-\tilde{u}_\tau) \nonumber\\ &+\frac{1}{\underline{\varepsilon}(x)}Le^{-2\beta\tau}\left(\|v^{0}_{x}(x)-\tilde{v}^{0}_{x}(x)\|^2+L\|v^{0}_{xx}(x)-\tilde{v}^0_{xx}(x)\|^2\right), \end{align}\] where \(\underline{\varepsilon}(x) = \min\left\{\varepsilon_{1i},\varepsilon_{2ij},1\leq i,j\leq n\right\}\). Then condition ?? implies that \[\begin{align} &\frac{d}{d\tau} \|u_\tau-\tilde{u}_\tau\|^2\\ \le& -2\kappa(x) \|u_\tau-\tilde{u}_\tau\|^2+\frac{1}{\underline{\varepsilon}(x)}Le^{-2\beta\tau}\left(\|v^{0}_{x}(x)-\tilde{v}^{0}_{x}(x)\|^2+L\|v^{0}_{xx}(x)-\tilde{v}^0_{xx}(x)\|^2\right). \end{align}\] Applying Gronwall’s inequality, the following estimate holds \[\begin{align} \label{grownall32relax} \|u_\tau-\tilde{u}_\tau\|^2 \le& e^{-2\kappa(x)\tau} \|u_0-\tilde{u}_0\|^2\\ &+e^{-2\kappa(x)\tau}\Big[ \frac{1}{\underline{\varepsilon}(x)}L\Big(\|v^{0}_{x}(x)-\tilde{v}^{0}_{x}(x)\|^2+L\|v^{0}_{xx}(x)-\tilde{v}^0_{xx}(x)\|^2\Big) \int_0^\tau e^{2(\kappa(x)-\beta)r}dr\Big]\\ \le& e^{-2\kappa(x)\tau} \|u_0-\tilde{u}_0\|^2\\ &+\frac{e^{-2\beta\tau}-e^{-2\kappa(x)\tau}}{2(\kappa(x)-\beta)}\frac{1}{\underline{\varepsilon}(x)}L\Big( \|v^{0}_{x}(x)-\tilde{v}^{0}_{x}(x)\|^2+L\|v^{0}_{xx}(x)-\tilde{v}^0_{xx}(x)\|^2\Big) \\ =& e^{-2\kappa(x)\tau} \|u_0-\tilde{u}_0\|^2+\frac{e^{-2\beta\tau }-e^{-2\kappa(x)\tau}}{2(\kappa(x)-\beta)} C_0. \end{align}\tag{27}\] Taking expectation on both sides, we conclude that \[\mathcal{W}_2^2(\pi_\tau,\tilde{\pi}_\tau)\le e^{-2\kappa(x)\tau} \mathbb{E}[\|u_0-\tilde{u}_0\|^2]+\frac{e^{-2\beta\tau }-e^{-2\kappa(x)\tau}}{2(\kappa(x)-\beta)} C_0.\] ◻

Proof. If \(\pi_0 = \pi^*\), then \(v^0\) is independent of \(t\), and \[v^0 = \mathbb{E}\left[\int_0^\infty e^{-\beta t}\left(\tilde{f}(X^*_t,\pi^*(X^*_t))- \lambda \int_{\mathbb{R}^n}\pi^*\ln\pi^*(X^*_t,u)du\right)dt\right],\] where \(X^*\) is the state process under the strategy \(\pi_t = \pi^*\) for \(t \geq 0\). Since Proposition 3 shows that \(\pi^*\) is the optimal strategy and \(v^*\) is the corresponding value, \(v^0 = v^*\). Furthermore, by the policy improvement result in Proposition 1, \(v^\tau = v^*\) for \(\tau \geq 0\), thus the diffusion 3 that \(u_\tau\) follows becomes \[du^x_\tau = \nabla_u \mathbf{H} (x,u^x_\tau,v^*_x,v^*_{xx})d\tau+ \sqrt{2\lambda}dB_\tau,\] which has an invariant distribution \(\pi^*= \frac{e^{\mathbf{H}(x,u,v^*_x,v^*_{xx})/\lambda}}{\int_{\mathbb{R}^n}e^{\mathbf{H}(x,u',v^*_x,v^*_{xx})/\lambda} du'}\). Since \(\pi_0 = \pi^*\), it implies that \(\pi_{\tau} = \pi_0\). ◻

Proof. Similar to the proof of Lemma 9, the difference \(v_x^{\tau} - \tilde{v}^\tau_x\) follows \[\label{eq:v-dynamics-c-convergence} d[v_x^\tau(x)-\tilde{v}_x^\tau(x)] = \left(-\beta [v_x^\tau(x)-\tilde{v}_x^\tau(x)] + \nabla_x \mathbf{H}^\tau(u_\tau) -\nabla_x\tilde{\mathbf{H}}^\tau (\tilde{u}_\tau)\right)d\tau.\tag{29}\] Thus \[\begin{align} \frac{d}{d\tau} \|v^{\tau}_x-\tilde{v}^{\tau}_x\|^2 =&2\Big[ \nabla_x \mathbf{H}^\tau(u_\tau) -\nabla_x \tilde{\mathbf{H}}^\tau(\tilde{u}_\tau) \Big]^\top(v^{\tau}_x-\tilde{v}^{\tau}_x) -2\beta\|v^{\tau}_x-\tilde{v}^{\tau}_x\|^2 \\ =& 2\Big[\nabla_x(\mathbf{H}^\tau(u_\tau)-\tilde{\mathbf{H}}^\tau(\tilde{u}_\tau))^\top(v^{\tau}_x(x)-\tilde{v}^{\tau}_x(x)) \Big] -2\beta\|v^{\tau}_x-\tilde{v}^{\tau}_x\|^2. \end{align}\] Taking expectation on both sides, \[\begin{align} \frac{d}{d\tau} \mathbb{E}[\|v^{\tau}_x-\tilde{v}^{\tau}_x\|^2] =& 2\mathbb{E}\Big[\nabla_x(\mathbf{H}^\tau(u_\tau)-\tilde{\mathbf{H}}^\tau(\tilde{u}_\tau))^\top(v^{\tau}_x(x)-\tilde{v}^{\tau}_x(x)) \Big] -2\beta\mathbb{E} [\|v^{\tau}_x-\tilde{v}^{\tau}_x\|^2]\\ \le & -2\beta \mathbb{E}[\|v^{\tau}_x-\tilde{v}^{\tau}_x\|^2], \end{align}\] where the last inequality follows from the monotonicity condition in Assumption 8. Then Gronwall’s inequality implies that \[\label{convergence:32v3932classic} \mathbb{E}[\|v^{\tau}_x(x)-\tilde{v}^{\tau}_x(x)\|^2]\le e^{-2\beta \tau}\mathbb{E}[ \|v^0_x(x)-\tilde{v}^0_x(x)\|^2].\tag{30}\] The estimates for \(\mathbb{E}[\|v^{\tau}(x)-\tilde{v}^{\tau}(x)\|^2]\) and \(\mathbb{E}[\|v^{\tau}_{xx}(x)-\tilde{v}^{\tau}_{xx}(x)\|^2]\) follow similar arguments. ◻

Proof. Following the proof of Proposition 7, and equation 26 , we have \[\begin{align} &\frac{d}{d\tau} \|u_\tau-\tilde{u}_\tau\|^2 \\ \le &2(u_\tau-\tilde{u}_\tau)^\top\Big[\mathbf{U}^f_\tau +\sum_{i=1}^nv^{\tau}_{x,i}(x) \mathbf{U}^{b_i}_\tau+ \sum_{i,j,k=1}^n\mathbf{U}^{\sigma_{jk}}_\tau \sigma_{ik}(x,u_\tau)v^{\tau}_{xx,ij}\\ &\quad\quad\quad\quad\quad\quad +\sum_{i,j,k=1}^n\hat{\mathbf{U}}^{\sigma_{ik}}_\tau\nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau) v^{\tau}_{xx,ij}\Big] (u_\tau-\tilde{u}_\tau)\\ &+(\sum_{i=1}^n\varepsilon_{1i}+\sum_{i,j=1}^n\varepsilon_{2ij})\|u_\tau-\tilde{u}_\tau\|^2+ \sum_{i=1}^n\frac{1}{\varepsilon_{1i}}\| \nabla_{u}b_i(x,\tilde{u}_\tau)\|^2|v^{\tau}_{x,i}(x)-\tilde{v}^{\tau}_{x,i}(x)|^2\\ &+\sum_{i,j=1}^n\frac{1}{\varepsilon_{2ij}}|\sigma_{ik}(x,u_\tau)|^2|v^{\tau}_{xx,ij}-\tilde{v}^{\tau}_{xx,ij}|^2\|\nabla_{u}\sigma_{jk}(x,\tilde{u}_\tau)\|^2. \end{align}\] Then the claim of the proposition holds by taking expectation on both sides, following the proof for Proposition 7, and the estimates in Assumption 8. ◻

Proof. Since \(v^\tau\) is quadratic in \(x\) for all \(0\leq s\leq \tau\), \(dv^\tau_{xx}\) reduces to \[\begin{align} dv^\tau_{xx} =&\left((C^2+2A)v^\tau_{xx}-M +2Bv^\tau_{xx}\partial_x \mathbb{E}[u_\tau]+(Bv^\tau_x -Q)\partial_{xx} \mathbb{E}[u_\tau] - \frac{N}{2}\partial_{xx}\mathbb{E}[u^2_\tau]\right.\nonumber\\ &\left. - \lambda\partial_{xx}\int_{\mathbb{R}}\pi_{\tau}\ln \pi_{\tau}(u)du -\beta v^{\tau}_{xx} \right)d\tau.\nonumber \end{align}\] Furthermore, since \(v^s_{xx}\) is constant for \(0\leq s\leq \tau\) and \(\mathsf{Var}(u_0)\) is independent of \(x\), 31 , 32 and 33 imply that \(\mathbb{E}[u_\tau]\) is linear in \(x\), and \(\mathsf{Var}(u_\tau)\) and \(\int_{\mathbb{R}}\pi_{\tau}\ln \pi_{\tau}(u)du\) are independent of \(x\). Therefore, \(\mathbb{E}[u_\tau^2]\) is quadratic in \(x\) and \[\begin{align} \frac{dv^\tau_{xx}}{d\tau} =&(C^2+2A)v^\tau_{xx}-M +2Bv^\tau_{xx}\partial_x \mathbb{E}[u_\tau] - \frac{N}{2}\partial_{xx}\mathbb{E}[u^2_\tau] -\beta v^{\tau}_{xx},\nonumber \end{align}\] is independent of \(x\). ◻

Proof. With the strategy \(\rho^0_t = \pi_0\) for every \(t\geq 0\) and initial value \(X_0 = x\), \[\begin{align} dX_t = \left(AX_t +B\mu\right)dt +CX_tdW_t.\nonumber \end{align}\] Then \(X_t = xe^{(A-\frac{1}{2}C^2)t +CW_t} + B\mu\int_0^te^{(A-\frac{1}{2}C^2)(t-s) +C(W_t-W_s)}ds\), and \[\begin{align} v^0 =& \mathbb{E}\left[\int_0^\infty e^{-\beta t}\left(-\frac{M}{2}X_t^2 - \frac{N}{2}\left(\mu^2+\sigma^2\right) -PX_t-Q\mu\right)dt\right]\nonumber\\ =&-\frac{M}{\beta-2A-C^2}x^2-\left(\frac{BMC^2\mu}{(A+C^2)(\beta-2A-C^2)(\beta - A)}+\frac{P}{\beta-A}\right)x + K,\nonumber \end{align}\] for some constant \(K\). ◻

Proof. By 35 , \(I_2^\tau\) and \(a^\tau_2\) satisfy the following coupled equations \[\begin{align} \frac{da^\tau_2}{d\tau}=&\left(C^2+2A + 2B^2I^\tau_2 -\beta\right)a^\tau_2-M- NB^2(I^\tau_2)^2,\tag{36}\\ \frac{dI^\tau_2}{d\tau} =& -NI^\tau_2 + a^\tau_2\tag{37}. \end{align}\] Since \(a^\tau_2=v^\tau_{xx}\) converges to \(a^*_2\) and thus \(\lim\limits_{\tau\rightarrow\infty} I^\tau_2=I^*_2 = a^*_2/N\). The limit can be found by checking the equilibrium point \((a^*_2,I^*_2)\) of the above equations, i.e., \(\frac{da^\tau_2}{d\tau}=\frac{dI^\tau_2}{d\tau}=0\) at \(\tau = \infty\), \[\begin{align} \left(C^2+2A + 2B^2I^*_2 -\beta\right)a^*_2-M - NB^2(I^*_2)^2 =& 0,\\ -NI^*_2 + a^*_2 =& 0. \end{align}\] Since \(\mathbb{E}[u_\tau]\) is linear in \(x\), and thus \(\tilde{f}(x,\pi_\tau)\) is concave in \(x\), and so is \(v^\tau\). Thus we choose the negative root of the quadratic function of \(a^*_2\) above, and \(a^*_2 = -\frac{2M}{\beta-C^2-2A + \sqrt{(\beta-C^2-2A)^2+\frac{4MB^2}{N}}}\). Next, since \(v^\tau_x\) converges, \(a^\tau_1\) converges to a constant \(a^*_1\), and \(I^\tau_1 = \int_0^\tau e^{N(s-\tau)}a^s_1ds\) converges to \(I^*_1 = a^*_1/N\). Its dynamics is \[\begin{align} \frac{da^\tau_1}{d\tau}=\frac{dv^\tau_x}{d\tau} -\frac{da^\tau_2}{d\tau}x =&\left(A-\beta\right)a^\tau_1-P+(Ba^\tau_1- Q)BI^\tau_2\nonumber\\ &+\left(Ba^\tau_2- NBI^\tau_2\right)\left(e^{-N\tau}\mu + BI^\tau_1 - \frac{1-e^{-N\tau}}{N}Q\right),\nonumber \end{align}\] and the right hand side should converge to \(0\) at \(\tau = \infty\). It implies that \[\begin{align} (A-\beta)a^*_1 - P + Ba^*_2(Ba^*_1-Q)/N = 0.\nonumber \end{align}\] Thus, we have \(a^*_1 = \frac{BQa^*_2+NP}{N(A-\beta) + B^2a^*_2}\). ◻

Proof. Suppose the initial values \(u\sim \mathcal{N}(\mu,\sigma^2)\) and \(\tilde{u}\sim \mathcal{N}(\tilde{\mu},\tilde{\sigma}^2)\). Lemma 12 implies that \(a^0_2 = \tilde{a}^0_2\), and since both \(a^\tau_2\) and \(\tilde{a}^\tau_2\) satisfy the differential equations 36 and 37 , \(a^\tau_2 = \tilde{a}^\tau_2\) and \(I^\tau_2 = \tilde{I}^\tau_2\). Thus monotonicity condition (MC III) is satisfied. Next, since \(\mathbb{E}[\mathbf{H}^\tau(u_\tau)]\) is quadratic in \(x\), \(\partial_x\mathbb{E}[\mathbf{H}^\tau(u_\tau)] = \partial_{xx}\mathbb{E}[\mathbf{H}^\tau(u_\tau)]x + C_\tau\), where \[\begin{align} C_\tau = &-\left(Ne^{-N\tau}\mu + BNI^\tau_1 +e^{-N\tau}Q\right)BI^\tau_2\nonumber\\ &+\left(A+B^2I^\tau_2\right)a^\tau_1 + B\left(e^{-N\tau}\mu + BI^\tau_1 - \frac{1-e^{-N\tau}}{N}Q\right)a^\tau_2-P. \nonumber \end{align}\] and \(a^\tau_1\) follows \[\begin{align} \frac{da^\tau_1}{d\tau}=&\frac{dv^\tau_x}{d\tau} -\frac{da^\tau_2}{d\tau}x\nonumber\\ =&\left(A-\beta\right)a^\tau_1-P+(Ba^\tau_1- Q)BI^\tau_2+B\left(a^\tau_2- NI^\tau_2\right)\left(e^{-N\tau}\mu - \frac{1-e^{-N\tau}}{N}Q + BI^\tau_1\right). \nonumber \end{align}\] Thus (notice that \(\partial_{xx}\mathbb{E}[\mathbf{H}^\tau(u_\tau)] - \partial_{xx}\mathbb{E}[\tilde{\mathbf{H}}^\tau(\tilde{u}_\tau)]=0\)) \[\begin{align} &\left(\partial_{x}\mathbb{E}[\mathbf{H}^\tau(u_\tau)] - \partial_{x}\mathbb{E}[\tilde{\mathbf{H}}^\tau(\tilde{u}_\tau)]\right)(v^\tau_{x}-\tilde{v}^\tau_{x}) \nonumber\\ =&\left((\partial_{xx}\mathbb{E}[\mathbf{H}^\tau(u_\tau)] - \partial_{xx}\mathbb{E}[\tilde{\mathbf{H}}^\tau(\tilde{u}_\tau)])x + C_\tau-\tilde{C}_\tau\right)((a^\tau_{2}-\tilde{a}^\tau_{2})x + a^\tau_1-\tilde{a}^\tau_1)\nonumber \\ =&(C_\tau-\tilde{C}_\tau)(a^\tau_{1}-\tilde{a}^\tau_{1})\\ =&e^{-N\tau}B(a^\tau_2-NI^\tau_2)(\mu-\tilde{\mu})(a^\tau_1-\tilde{a}^\tau_1)\nonumber\\ &+ B^2\left(a^\tau_2-NI^\tau_2\right)\left(I^\tau_1-\tilde{I}^\tau_1\right)(a^\tau_1-\tilde{a}^\tau_1)+\left(A+ B^2 I^\tau_2\right)\left(a^\tau_1-\tilde{a}^\tau_1\right)^2.\label{eq:Ca} \end{align}\tag{38}\] Let \(Y_{\tau} = \frac{I^\tau_1 - \tilde{I}^\tau_1}{a^\tau_1 - \tilde{a}^\tau_1}\) and \(Z_{\tau} = e^{-N\tau}\frac{\mu - \tilde{\mu}}{a^\tau_1 - \tilde{a}^\tau_1}\). With \(Y_{0} = 0\) and \(Z_0 = -\frac{(A+C^2)(\beta-2A-C^2)(\beta - A)}{BMC^2}\), \[\begin{align} \frac{dY_{\tau}}{d\tau} =& -\left(N+A-\beta + B^2I^\tau_2\right)Y_\tau+1-B(a^\tau_2-NI^\tau_2)Z_\tau Y_{\tau} - B^2(a^\tau_2-NI^\tau_2)Y^2_{\tau},\tag{39}\\ \frac{dZ_\tau}{d\tau} =& -(N+A-\beta+B^2I^\tau_2)Z_\tau - B^2(a^\tau_2 - NI^\tau_2)Z_\tau Y_\tau+B(a^\tau_2 - NI^\tau_2)Z^2_\tau.\tag{40} \end{align}\] Since \(a^\tau_2\) converges to \(a^*_2\) and \(I^\tau_2\) converges to \(I^*_2 = a^*_2/N\), \(I^\tau_2\) is bounded, and the bound does not depend on initial distribution \(\pi_0\). For \(A<0\) and \(|A|\) sufficiently large, \(A +B^2 I^\tau_2<0\), and with sufficiently large \(N>|A-\beta|\), \(N+A-\beta+B^2I^\tau_2>0\). In 39 and 40 , the coefficients of \(Z_\tau Y_\tau\), \(Y^2_\tau\) and \(Z_\tau^2\) are all bounded, and the bounds are independent of the initial distribution \(\pi_0\) and \(N\). Then with sufficiently large \(M\) and \(N\), such that \(|Z_0|\) is sufficiently small, since \(Y_0 = 0\), the mean reverting term in both 39 and 40 dominates other terms. Thus \(Y_\tau\) and \(Z_\tau\) are bounded. Furthermore, since \(a^\tau_2 - NI^\tau_2\) converges to \(0\), \(\lim\limits_{\tau\rightarrow \infty}Y_\tau = \frac{1}{N+A-\beta + B^2a^*_2/N}\) and \(\lim\limits_{\tau\rightarrow \infty}Z_\tau = 0\). Therefore, 38 implies that \[\begin{align} &(C_\tau - \tilde{C}_\tau)(a^\tau_1-\tilde{a}^\tau_1) \nonumber\\ =&\left(e^{-N\tau}B(a^\tau_2-NI^\tau_2)Z_\tau+ B^2\left(a^\tau_2-NI^\tau_2\right)Y_\tau+ A+ B^2 I^\tau_2\right)\left(a^\tau_1-\tilde{a}^\tau_1\right)^2. \label{a32tau32difference} \end{align}\tag{41}\] Notice that Lemma 12 implies that \(|a^0_2|\) is decreasing in \(|A|\), and \(Z_0\) can be made small by choosing large \(M\) (e.g. \(M = |A|^4\)). For sufficiently large \(|A|\), the coefficient of \((a^\tau_1 - \tilde{a}^\tau_1)^2\) in 41 is negative. ◻