- Hamilton-Jacobi-Bellman equation
The Hamilton-Jacobi-Bellman (HJB) equation is a
partial differential equation which is central tooptimal control theory.The solution of the HJB equation is the 'value function', which gives the optimal cost-to-go for a given
dynamical system with an associated cost function. Classical variational problems, for example, thebrachistochrone problem can be solved using this method as well.The equation is a result of the theory of
dynamic programming which was pioneered in the 1950s byRichard Bellman and coworkers. [R. E. Bellman. Dynamic Programming. Princeton, NJ, 1957.] The corresponding discrete-time equation is usually referred to as theBellman equation . In continuous time, the result can be seen as an extension of earlier work inclassical physics on theHamilton-Jacobi equation byWilliam Rowan Hamilton andCarl Gustav Jacob Jacobi .Consider the following problem in deterministic optimal control
:
subject to
:
where is the system state, is assumed given, and for is the control that we are trying to find.For this simple system, the Hamilton Jacobi Bellman partial differential equation is
:
subject to the terminal condition
:
The unknown in the above PDE is the Bellman '
value function ', which represents the cost incurred from starting in state at time and controlling the system optimally from then until time .The HJB equation needs to be solved backwards in time, starting from and ending at . (The notation means the inner product of the vectors a and b).The HJB equation is a
sufficient condition for an optimum.Fact|date=May 2008 If we can solve for then we can find from it a control that achieves the minimum cost.The HJB method can be generalized to
stochastic systems as well.In general case, the HJB equation does not have a classical (smooth) solution. Several notions of generalized solutions have been developed to cover such situations, including
viscosity solution (Pierre-Louis Lions andMichael Crandall ),minimax solution (Andrei Izmailovich Subbotin ), and others.References
Wikimedia Foundation. 2010.