Keywords: Calculus of variations, stochastic optimal control, functional derivative

1 Introduction↩︎

The standard setup for a finite time stochastic control problem is as follows. Fix a time horizon \(T < \infty\) and a probability space \((\Omega,\mathscr{F},\mathbb{P})\). Let \(W = (W_t)_{0 \leq t \leq T}\) be a Brownian motion and let \(\mathbb{F}= (\mathscr{F}_t)_{0 \leq t \leq T}\) be a filtration for \(W\). Consider a controlled diffusion process \(X^c = (X_t^c)_{0 \leq t \leq T}\) whose dynamics are given by \[\begin{align} \mathrm{d}X_t^c &= b(t,X_t^c,c_t) \mathrm{d}t + a(t,X_t^c,c_t) \mathrm{d}W_t , \end{align}\] where the process \(c = (c_t)_{0 \leq t \leq T}\) is an \(\mathbb{F}\)-adapted control. Suppose we wish to maximize the following functional of the control \(c\) \[\begin{align} J(c) &:= \mathbb{E}\int_0^T \mathrm{d}t \, f(t,X_t^c,c_t) + \mathbb{E}g(X_T^c) . \label{eq:J} \end{align}\tag{1}\] If there exists an optimal control \(c^*\) such that \(J(c^*) \geq J(c)\) for all \(c\), and if \(c^*\) is Markov \(c_t^* = c^*(t,X_t^{c^*})\), then one way to find \(c^*\) is to solve the Hamilton-Jacobi-Bellman (HJB) equation \[\begin{align} 0 &= \partial_t v(t,x) + \max_c \Big( f(t,x,c) + b(t,x,c) \partial_x v(t,x) + \tfrac{1}{2} \partial_x a^2(t,x,c) v(t,x) \Big) , & v(T,x) &= g(x) . \end{align}\] The optimal control is then obtained by setting \[\begin{align} c^*(t,x) &= \arg\max_c \Big( f(t,x,c) + b(t,x,c) \partial_x v(t,x) + \tfrac{1}{2} a^2(t,x,c) \partial_x^2 v(t,x) \Big) . \end{align}\] To derive the HJB equation one typically relies on the Dynamic Programming Principle (DPP) with Bellman’s Principle of Optimality. Specifically, one considers a problem over an infinitesimally small time step and uses a limiting argument to obtain the HJB equation; see, e.g., [1]. The purpose of this short note is to show how to find the optimal control \(c^*\) using calculus of variations, a topic that may be more intuitive than DPP for anyone that has familiarity with classical mechanics; see, e.g., [2].

2 A calculus of variations approach↩︎

Throughout this section, we will make the same assumptions that one makes when formally deriving the HJB equation. Specifically, we assume that an optimal control exists and that it is Markov. Under this assumption, for a fixed control \(c\), the controlled diffusion \(X^c\) is the solution of the following stochastic differential equation (SDE) \[\begin{align} \mathrm{d}X_t^c &= b_c(t,X_t^c) \mathrm{d}t + a_c(t,X_t^c) \mathrm{d}W_t , \label{eq:dXc} \end{align}\tag{2}\] where we have defined \[\begin{align} b_c(t,x) &:= b(t,x,c(t,x)) , & a_c(t,x) &:= a(t,x,c(t,x)) . \label{eq:bc-ac} \end{align}\tag{3}\] As solutions of SDEs are Markov processes, we can associate with \(X^c\) an infinitesimal generator of \(\mathscr{A}_c\), which is defined as follows \[\begin{align} \mathscr{A}_c \phi (x) &:= \lim_{s \searrow t} \frac{ \mathbb{E}( \phi (X_s^c) | X_t^c = x ) - \phi (x) }{s-t} , \end{align}\] and is given explicitly by \[\begin{align} \mathscr{A}_c &= b_c(t,x) \partial_x + \tfrac{1}{2} a_c^2(t,x) \partial_x^2 . \end{align}\] We can also associate with \(X^c\) a transition density \(p_c\), which is defined as follows \[\begin{align} p_c(t,x;s,y) \mathrm{d}y &:= \mathbb{P}(X_s^c \in \mathrm{d}y | X_t^c = x ) . \end{align}\] It is well-known that \(p_c\) satisfies the Kolmogorov forward equation (KFE), also called the Fokker-Planck equation (FPE), in the forward variables \((s,y)\) with the backward variables \((t,x)\) fixed \[\begin{align} 0 &= ( - \partial_s + \mathscr{A}_c^\dagger ) p_c(t,x;s,y) , & p_c(t,x;t,y) &= \delta_x(y) , \label{eq:KFE} \end{align}\tag{4}\] where \(\mathscr{A}_c^\dagger\) is the formal \(L^2\) adjoint of \(\mathscr{A}_c\), which is given explicitly by \[\begin{align} \mathscr{A}_c^\dagger &:= - \partial_y b_c(t,y) + \tfrac{1}{2} \partial_y^2 a_c^2(t,y) . \end{align}\] Now, recall that our goal is to maximize the functional \(J\), given by 1 , over all Markov controls. Let us express \(J\) using the transition density \(p_c\). Omitting the dependence on the initial condition \(p_c(s,y) \equiv p_c(0,x;s,y)\) to ease notation, we have \[\begin{align} J(c) &= \int_0^T \mathrm{d}s \int \mathrm{d}y \, f_c(s,y) p_c(s,y) + \int \mathrm{d}y \, g(y) p_c(T,y) , & f_c(s,y) &:= f(s,y,c(s,y)) , \label{eq:J2} \end{align}\tag{5}\] Thus, we have a PDE-constrained optimization problem \[\begin{align} &\max_c J(c)& &\text{with J given by \eqref{eq:J2}}& &\text{subject to}& &\text{p_c satisfying \eqref{eq:KFE}}. \label{eq:constrained} \end{align}\tag{6}\] Much like one can solve the constrained optimization problem \(\max_{x,y} f(x,y)\) subject to \(g(x,y) = 0\) by introducing a Lagrange multiplier \(\lambda\) and finding the critical points \((x^*,y^*,\lambda^*)\) of the unconstrained function \(\ell(x,y,\lambda) := f(x,y) + \lambda g(x,y)\), one can solve the PDE-constrained optimization problem 6 by searching for critical points \((p^*,c^*,\lambda^*,\mu^*)\) of the following Lagrangian \[\begin{align} L(p,c,\lambda,\mu) &:= J(c) + \int_0^T \mathrm{d}s \int \mathrm{d}y \, \lambda(s,y) ( - \partial_s + \mathscr{A}_c^\dagger ) p(s,y) + \int \mathrm{d}y \, \mu(y) ( p(0,y) - \delta_x(y) ) \\ &= \int_0^T \mathrm{d}s \int \mathrm{d}y \Big( f_c(s,y) + \lambda(s,y) ( - \partial_s + \mathscr{A}_c^\dagger ) \Big) p(s,y) \\ &\quad + \int \mathrm{d}y \Big( g(y) p(T,y) + \mu(y) ( p(0,y) - \delta_x(y) ) \Big) . \label{eq:L} \end{align}\tag{7}\] Here, the functions \(\lambda\) and \(\mu\) play the role of Lagrange multipliers. Note that the function \(p\) in 7 is not constrained to satisfy the KFE 4 , which is why we have removed the subscript \(c\) from \(p_c\). For future reference, it will be helpful to provide an alternative expression for \(L\). Using integration by parts, we have \[\begin{align} L(p,c,\lambda,\mu) &= \int_0^T \mathrm{d}s \int \mathrm{d}y \, p(s,y) \Big( f_c(s,y) + ( \partial_s + \mathscr{A}_c ) \lambda(s,y) \Big) \\ &\quad - \int \mathrm{d}y \, \lambda(s,y) p(s,y) \Big|_{s=0}^{s=T} + \int \mathrm{d}y \Big( g(y) p(T,y) + \mu(y) ( p(0,y) - \delta_x(y) ) \Big) \\ &= \int_0^T \mathrm{d}s \int \mathrm{d}y \, p(s,y) \Big( f_c(s,y) + ( \partial_s + \mathscr{A}_c ) \lambda(s,y) \Big) \\ &\quad + \int \mathrm{d}y \Big( \lambda(0,y) \delta_x(y) - \lambda(T,y) p(T,y) + g(y) p(T,y) + \mu(y) ( p(0,y) - \delta_x(y) ) \Big) . \label{eq:L2} \end{align}\tag{8}\] Now, we wish to find functions \((p^*,c^*,\lambda^*,\mu^*)\) that together are a critical point of \(L\). A necessary condition for \((p^*,c^*,\lambda^*,\mu^*)\) to be a critical point of \(L\) is that the following functional derivatives are simultaneously equal to zero \[\begin{align} 0 &= \frac{\mathrm{d}}{\mathrm{d}\epsilon } L(p^* + \epsilon q, c^*, \lambda^*, \mu^*) \Big|_{\epsilon = 0} , & 0 &= \frac{\mathrm{d}}{\mathrm{d}\epsilon } L(p^*, c^* + \epsilon \gamma, \lambda^*, \mu^*) \Big|_{\epsilon = 0} , \\ 0 &= \frac{\mathrm{d}}{\mathrm{d}\epsilon } L(p^*, c^*, \lambda^* + \epsilon \nu, \mu^*) \Big|_{\epsilon = 0} , & 0 &= \frac{\mathrm{d}}{\mathrm{d}\epsilon } L(p^*, c^*, \lambda^*, \mu^* + \epsilon \pi) \Big|_{\epsilon = 0} , \end{align}\] for any choice of functions \((q,\gamma,\nu,\pi)\) such that \(q(0,y) = 0\).
Let us take the functional derivative of \(L\) with respect to \(p\). From 8 we have \[\begin{align} 0 &= \frac{\mathrm{d}}{\mathrm{d}\epsilon } L(p^* + \epsilon q, c^*, \lambda^*, \mu^*) \Big|_{\epsilon = 0} \\ &= \int_0^T \mathrm{d}s \int \mathrm{d}y \, q(s,y) \Big( f_{c^*}(s,y) + ( \partial_s + \mathscr{A}_{c^*} ) \lambda^*(s,y) \Big) + \int \mathrm{d}y \Big( - q(T,y) \lambda^*(T,y) + g(y) q(T,y) \Big) . \label{eq:p-variation} \end{align}\tag{9}\] In order for 9 to hold for any choice of \(q\) we must have \[\begin{align} 0 &= f_{c^*}(t,y) + ( \partial_t + \mathscr{A}_{c^*} ) \lambda^*(t,y) , & \lambda^*(T,y) &= g(y) . \label{eq:lam} \end{align}\tag{10}\] Next, we compute functional derivative of \(L\) with respect to \(c\). From 8 we have \[\begin{align} 0 &= \frac{\mathrm{d}}{\mathrm{d}\epsilon } L(p^* , c^* + \epsilon \gamma, \lambda^*, \mu^*) \Big|_{\epsilon = 0} \\ &= \int_0^T \mathrm{d}s \int \mathrm{d}y \, p^*(s,y) \frac{\mathrm{d}}{\mathrm{d}\epsilon } \Big( f_{c^* + \epsilon \gamma}(s,y) + ( \partial_s + \mathscr{A}_{c^* + \epsilon \gamma} ) \lambda^*(s,y) \Big) \Big|_{\epsilon = 0} \\ &= \int_0^T \mathrm{d}t \int \mathrm{d}y \, p^*(t,y) \gamma(t,y) \partial_{c^*} \Big( f_{c^*}(t,y) + \mathscr{A}_{c^*} \lambda^*(t,y) \Big) , \label{eq:c-variation} \end{align}\tag{11}\] In order for 11 to hold for any choice of \(\gamma\) we must have \[\begin{align} 0 &= \partial_{c^*} \Big( f_{c^*}(t,y) + \mathscr{A}_{c^*} \lambda^*(t,y) \Big) . \label{eq:c} \end{align}\tag{12}\] Next, we compute functional derivative of \(L\) with respect to \(\lambda\). From 7 we find \[\begin{align} 0 &= \frac{\mathrm{d}}{\mathrm{d}\epsilon } L(p^* , c^* , \lambda^* + \epsilon \nu, \mu^*) \Big|_{\epsilon = 0} \\ &= \int_0^T \mathrm{d}t \int \mathrm{d}y \, \nu(t,y) ( - \partial_t + \mathscr{A}_{c^*}^\dagger ) p^*(t,y) . \label{eq:lam-variation} \end{align}\tag{13}\] In order for 13 to hold for any \(\nu\) we must have \[\begin{align} 0 &= ( - \partial_t + \mathscr{A}_{c^*}^\dagger ) p^*(t,y) . \label{eq:p} \end{align}\tag{14}\] Lastly, we compute the functional derivative of \(L\) with respect to \(\mu\). From 7 we have \[\begin{align} 0 &= \frac{\mathrm{d}}{\mathrm{d}\epsilon } L(p^* , c^* , \lambda^* , \mu^* + \epsilon \pi) \Big|_{\epsilon = 0} \\ &= \int \mathrm{d}y \, \pi(y) ( p^*(0,y) - \delta_x(y) ) . \label{eq:mu-variation} \end{align}\tag{15}\] In order for 15 to hold for any choice of \(\pi\) we must have \[\begin{align} p^*(0,y) &= \delta_x(y) . \label{eq:p0} \end{align}\tag{16}\] Note that PDE 14 along with 16 say that \(p^*\) satisfies the KFE 4 with \(c = c^*\).
To summarize, in order for \((p^*,c^*,\lambda^*,\mu^*)\) to be a critical point of \(L\), equations 10 , 12 , 14 and 16 must be satisfied.

3 Relation to the stochastic maximum principle↩︎

The necessary conditions 10 , 12 , 14 and 16 can be interpreted as a “density-level” formulation of the stochastic maximum principle for the controlled diffusion \(X^c\) in 2 . To explain this link, we first recall the classical stochastic maximum principle and then show how it emerges from our calculus of variations approach.
The classical stochastic maximum principle characterizes the optimal control through a forward-backward SDE (FBSDE) system. The forward equation governs the optimal state process \(X^{c^*}\), while the backward equation governs an adjoint process \(Y^{c^*} = (Y_t^{c^*})_{0 \leq t \leq T}\) (and an auxiliary process \(Z^{c^*} = (Z_t^{c^*})_{0 \leq t \leq T}\)) that captures the sensitivity of the objective to changes in the state. The FBSDE system reads \[\begin{align} \mathrm{d}X_t^{c^*} &= b\big(t, X_t^{c^*}, c_t^*\big) \mathrm{d}t + a\big(t, X_t^{c^*}, c_t^*\big) \mathrm{d}W_t , & X_0^{c^*} &= x , \tag{17} \\ - \mathrm{d}Y_t^{c^*} &= \partial_x H\big(t, X_t^{c^*}, c_t^*, Y_t^{c^*}, Z_t^{c^*}\big) \mathrm{d}t - Z_t^{c^*} \mathrm{d}W_t , & Y_T^{c^*} &= g'(X_T^{c^*}) , \tag{18} \end{align}\] where the Hamiltonian \(H\) is defined by \[\begin{align} H(t,x,c,y,z) &:= f(t,x,c) + b(t,x,c)\, y + a(t,x,c)\, z . \end{align}\] The forward equation 17 is an ordinary SDE solved forward in time from the initial condition \(X_0^{c^*} = x\). The backward equation 18 is a backward SDE (BSDE): it is solved backward in time from the terminal condition \(Y_T^{c^*} = g'(X_T^{c^*})\), which is prescribed at \(t = T\). The process \(Z^{c^*}\) arises naturally in the BSDE as the diffusion coefficient needed to keep \(Y^{c^*}\) adapted to the filtration \(\mathbb{F}\). The optimal control satisfies the pointwise first-order condition \[\begin{align} \partial_c H\big(t, X_t^{c^*}, c_t^*, Y_t^{c^*}, Z_t^{c^*}\big) &= 0 ; \label{eq:smp-foc} \end{align}\tag{19}\] see, e.g., [1] for a precise statement in a Markov diffusion setting.
In the present approach, the role of the forward state is played by the transition density \(p^*\), which solves the Kolmogorov forward equation 14 –16 under the Markov control \(c^*\). This is the density-level counterpart of the forward SDE 17 .
Likewise, the adjoint equation 10 can be viewed as the backward Kolmogorov equation associated with the pair \((X^{c^*}, c^*)\). For each \((t,y)\), the solution \(\lambda^*(t,y)\) coincides with the conditional value function \[\begin{align} \lambda^*(t,y) &= \mathbb{E}\!\left( \int_t^T f\big(s,X_s^{c^*},c_s^*\big) \mathrm{d}s + g(X_T^{c^*}) \,\bigg|\, X_t^{c^*} = y \right) . \label{eq:lam-interp} \end{align}\tag{20}\] The pathwise adjoint processes of the classical stochastic maximum principle can be recovered from \(\lambda^*\) by setting \[\begin{align} Y_t^{c^*} &:= \partial_x \lambda^*\big(t, X_t^{c^*}\big) , & Z_t^{c^*} &:= \partial_x \lambda^*\big(t, X_t^{c^*}\big)\, a\big(t, X_t^{c^*}, c_t^*\big) . \label{eq:YZ} \end{align}\tag{21}\] To see why, apply Itô’s formula to \(\lambda^*(t, X_t^{c^*})\). Using 10 (with \(f_{c^*} = 0\) absorbed into the \(g\) terminal condition for simplicity of illustration) one finds that \(Y_t^{c^*} = \partial_x \lambda^*(t, X_t^{c^*})\) satisfies the backward SDE 18 , with \(Z_t^{c^*}\) as defined in 21 playing the role of the diffusion coefficient.
Finally, the first-order condition 12 can be written as \[\begin{align} 0 &= \partial_{c^*} \Big( f_{c^*}(t,y) + \mathscr{A}_{c^*} \lambda^*(t,y) \Big) = \partial_c \mathscr{H}\big(t,y,c,\lambda^*\big)\big|_{c = c^*(t,y)} , \end{align}\] where \[\begin{align} \mathscr{H}(t,y,c,\lambda) &:= f(t,y,c) + \mathscr{A}_c \lambda(t,y) = f(t,y,c) + b(t,y,c)\, \partial_y \lambda(t,y) + \tfrac{1}{2} a^2(t,y,c)\, \partial_y^2 \lambda(t,y) \end{align}\] is a Hamiltonian density. Evaluating along the optimal trajectory using 21 , we have \[\begin{align} \partial_c \mathscr{H}\big(t, X_t^{c^*}, c_t^*, \lambda^*\big) &= \partial_c H\big(t, X_t^{c^*}, c_t^*, Y_t^{c^*}, Z_t^{c^*}\big) = 0 , \end{align}\] which is precisely the Pontryagin-type first-order optimality condition 19 of the stochastic maximum principle, expressed here in terms of the fields \((p^*, \lambda^*)\) rather than the pathwise pair \((X^{c^*}, Y^{c^*})\).

4 Example: Merton portfolio problem↩︎

Let us see how we can use the results of Section 2 to find the optimal control in the well-known Merton portfolio optimization problem [3]. To this end, consider a financial market consisting of a single stock \(S = (S_t)_{0 \leq t \leq T}\) and a money market account \(M = (M_t)_{0 \leq t \leq T}\), whose real-world dynamics are of the form \[\begin{align} \mathrm{d}S_t &= \mu S_t \mathrm{d}t + \sigma S_t \mathrm{d}W_t , & \mathrm{d}M_t &= r M_t \mathrm{d}t . \end{align}\] We will take \(r=0\) for simplicity. Denote by \(c = (c_t)_{0 \leq t \leq T}\) the percentage of wealth that an agent invests in the stock \(S\). Then, denoting by \(X^c = (X_t^c)_{0 \leq t \leq T}\) the wealth process of the agent, we have \[\begin{align} \mathrm{d}X_t^c &= \frac{ c_t X_t^c }{S_t} \mathrm{d}S_t + \frac{(1-c_t)X_t^c}{M_t}\mathrm{d}M_t = \mu c_t X_t^c \mathrm{d}t + \sigma c_t X_t^c \mathrm{d}W_t . \label{eq:dXc-merton} \end{align}\tag{22}\] Comparing 2 3 with 22 , we identify \[\begin{align} b(t,x,c) &= \mu c \, x , & a(t,x,c) &= \sigma c \, x . \end{align}\] Suppose the agent wishes to maximize his expected utility of wealth at time \(T\). Denoting by \(U\) the agent’s utility function, the agent’s goal is to maximize \[\begin{align} J(c) &:= \mathbb{E}U(X_T^c) . \label{eq:J-merton} \end{align}\tag{23}\] Comparing 1 with 23 , we identify \[\begin{align} f(t,x,c) &= 0 , & g(x) &= U(x) . \end{align}\] From 12 , we have \[\begin{align} 0 &= \partial_{c^*} \mathscr{A}_{c^*} \lambda^*(t,y) \\ &= \mu y \partial_y \lambda^*(t,y) + \sigma^2 c^*(t,y) y^2 \partial_y^2 \lambda^*(t,y) , & &\Rightarrow& c^*(t,y) &= \frac{ - \mu y \partial_y \lambda^*(t,y) }{ \sigma^2 y^2 \partial_y^2 \lambda^*(t,y)} . \label{eq:c-merton} \end{align}\tag{24}\] And, from 10 , we have \[\begin{align} 0 &= ( \partial_t + \mathscr{A}_{c^*} ) \lambda^*(t,y) \\ &= \Big( \partial_t + \mu c^*(t,y) y \partial_y + \tfrac{1}{2} \sigma^2 (c^*(t,y))^2 y^2 \partial_y^2 \Big) \lambda^*(t,y) \\ &= \partial_t \lambda^*(t,y) - \frac{\mu^2 (\partial_y \lambda^*(t,y))^2}{ 2 \sigma^2 \partial_y^2 \lambda^*(t,y)} , & \lambda^*(T,y) &= g(y) = U(y) . \end{align}\] Suppose the agent’s utility function is of the power form: \(U(y) = y^q / q\). Using the Ansatz \(\lambda^*(t,y) = h(t) U(y)\) we obtain \[\begin{align} h'(t) &= \frac{q \mu^2}{2 (q-1) \sigma^2} h(t) , & h(T) &= 1 , & &\Rightarrow& h(t) &= \exp \Big( \frac{q \mu^2}{2 (1-q) \sigma^2} (T-t) \Big) . \end{align}\] Hence, from 24 , the optimal control is given by \[\begin{align} c^*(t,y) &= \frac{\mu}{(1-q) \sigma^2} . \end{align}\] Although we did not need it in order to find the optimal control \(c^*\), it may be of to compute the transition density \(p^*\) of the optimal wealth process \(X^{c^*}\), which can be used to compute \(J(c^*)\). From 14 we have \[\begin{align} 0 &= ( - \partial_t + \mathscr{A}_{c^*}^\dagger ) p^*(t,y) \\ &= \Big( - \partial_t - \partial_y \frac{\mu^2 y}{(1-q)\sigma^2} + \tfrac{1}{2} \partial_y^2 \frac{\mu^2 y^2}{(1-q)^2\sigma^2} \Big) p^*(t,y) , & p^*(0,y) &= \delta_x(y) . \end{align}\] One can check by direct substitution that \[\begin{align} p^*(t,y) \equiv p^*(0,x,t,y) &= \frac{1}{y \sqrt{2 \pi \Sigma^2 t}} \exp \Big( -\frac{1}{2 \Sigma^2 t} (\log y - \log x - (m - \tfrac{1}{2} \Sigma^2) t)^2 \Big) , \\ \Sigma &= \frac{\mu}{(1-q)\sigma} , \quad \quad m = \frac{\mu^2}{(1-q) \sigma^2} . \end{align}\] Lastly, we can compute the optimal expected utility of wealth. We have \[\begin{align} J(c^*) &= \mathbb{E}U(X_T^{c^*}) = \int \mathrm{d}y \, p^*(T,y) \frac{x^q}{q} = \frac{x^q}{q} \exp \Big( \frac{q \mu^2 T}{2(1-q)\sigma^2} \Big) . \end{align}\]

References↩︎