- Hamilton-Jacobi-Bellman equation
The Hamilton-Jacobi-Bellman (HJB) equation is a
partial differential equation which is central tooptimal control theory.The solution of the HJB equation is the 'value function', which gives the optimal cost-to-go for a given
dynamical system with an associated cost function. Classical variational problems, for example, thebrachistochrone problem can be solved using this method as well.The equation is a result of the theory of
dynamic programming which was pioneered in the 1950s byRichard Bellman and coworkers. [R. E. Bellman. Dynamic Programming. Princeton, NJ, 1957.] The corresponding discrete-time equation is usually referred to as theBellman equation . In continuous time, the result can be seen as an extension of earlier work inclassical physics on theHamilton-Jacobi equation byWilliam Rowan Hamilton andCarl Gustav Jacob Jacobi .Consider the following problem in deterministic optimal control
:min int_0^T C [x(t),u(t)] ,dt + D [x(T)]
subject to
:dot{x}(t)=F [x(t),u(t)]
where x(t) is the system state, x(0) is assumed given, and u(t) for 0leq tleq T is the control that we are trying to find.For this simple system, the Hamilton Jacobi Bellman partial differential equation is
:frac{partial}{partial t} V(x,t) + min_u left{ leftlangle frac{partial}{partial x}V(x,t), F(x, u) ight angle + C(x,u) ight} = 0
subject to the terminal condition
:V(x,T) = D(x).,
The unknown V(t, x) in the above PDE is the Bellman '
value function ', which represents the cost incurred from starting in state x at time t and controlling the system optimally from then until time T.The HJB equation needs to be solved backwards in time, starting from t = T and ending at t = 0. (The notation langle a,b angle means the inner product of the vectors a and b).The HJB equation is a
sufficient condition for an optimum.Fact|date=May 2008 If we can solve for V then we can find from it a control u that achieves the minimum cost.The HJB method can be generalized to
stochastic systems as well.In general case, the HJB equation does not have a classical (smooth) solution. Several notions of generalized solutions have been developed to cover such situations, including
viscosity solution (Pierre-Louis Lions andMichael Crandall ),minimax solution (Andrei Izmailovich Subbotin ), and others.References
Wikimedia Foundation. 2010.