Value function

The value function of an optimization problem gives the value attained by the objective function at a solution, while only depending on the parameters of the problem.[1][2] In a controlled dynamical system, the value function represents the optimal payoff of the system over the interval [t, T] when started at the time-t state variable x(t)=x.[3] If the objective function represents some cost that is to be minimized, the value function can be interpreted as the cost to finish the optimal program, and is thus referred to as "cost-to-go function."[4] In an economic context, where the objective function usually represents utility, the value function is conceptually equivalent to the indirect utility function.[5][6]

In a problem of optimal control, the value function is defined as the supremum of the objective function taken over the set of admissible controls. Given ${\displaystyle (t_{0},x_{0})\in [0,t_{1}]\times \mathbb {R} ^{d}}$, a typical optimal control problem is

${\displaystyle {\text{maximize}}\quad J(t_{0},x_{0};u)=\int _{t_{0}}^{t_{1}}I(t,x(t),u(t))\,\mathrm {d} t+\phi (x(t_{1}))}$

subject to

${\displaystyle {\frac {\mathrm {d} x(t)}{\mathrm {d} t}}=f(t,x(t),u(t))}$

with initial state variable ${\displaystyle x(t_{0})=x_{0}}$.[7] The objective function ${\displaystyle J(t_{0},x_{0};u)}$ is to be maximized over all admissible controls ${\displaystyle u\in U[t_{0},t_{1}]}$, where ${\displaystyle u}$ is a Lebesgue measurable function from ${\displaystyle [t_{0},t_{1}]}$ to some prescribed arbitrary set in ${\displaystyle \mathbb {R} ^{m}}$. The value function is then defined as

If the optimal pair of control and state trajectories is ${\displaystyle (x^{\ast },u^{\ast })}$, then ${\displaystyle V(t_{0},x_{0})=J(t_{0},x_{0};u^{\ast })}$. In economics, the function ${\displaystyle u^{\ast }=h(x)}$ is called a policy function.[8]

Bellman's principle of optimality roughly states that any optimal policy at time ${\displaystyle t}$, ${\displaystyle t_{0}\leq t\leq t_{1}}$ taking the current state ${\displaystyle x(t)}$ as "new" initial condition must be optimal for the remaining problem. If the value function happens to be continuously differentiable,[9] this gives rise to an important functional recurrence equation known as Hamilton–Jacobi–Bellman equation,

${\displaystyle -{\frac {\partial V(t,x)}{\partial t}}=\max _{u}H\left(t,x,u,{\frac {\partial V(t,x)}{\partial x}}\right)}$

where the maximand on the right-hand side is the Hamiltonian, with ${\displaystyle \partial V(t,x)/\partial x}$ playing the role of the costate variables.[10] The value function is a viscosity solution to the Hamilton–Jacobi–Bellman equation.[11]

In an online closed-loop approximate optimal control, the value function is also a Lyapunov function that establishes global asymptotic stability of the closed-loop system.[12]