BFGS method

In mathematics, the Broyden-Fletcher-Goldfarb-Shanno (BFGS) method is a method to solve an unconstrained nonlinear optimization problem.

The BFGS method is derived from the Newton's method in optimization, a class of hill-climbing optimization techniques that seeks the stationary point of a function, where the gradient is 0. Newton's method assumes that the function can be locally approximated as a quadratic in the region around the optimum,and use the first and second derivatives to find the stationary point. In Quasi-Newton methods the Hessian matrix of second derivatives of the function to be minimized does not need to be computed at any stage. The Hessian is updated by analyzing successive gradient vectors instead. Quasi-Newton methods are a generalization of the secant method to find the root of the first derivative for multidimensional problems. In multi-dimensions the secant equation is under-determined, and quasi-Newton methods differ in how they constrain the solution. The BFGS method is one of the most successful members of this class Fact|date=December 2007.

Rationale

The search direction $mathbf\{p\}\_k$ at stage "k" is given by the solution of the analogue of the Newton equation

:$B\_k\; mathbf\{p\}\_k\; =\; -\; abla\; f(mathbf\{x\}\_k).$

A line search in the direction $mathbf\{p\}\_k$ is then used to find the next point $mathbf\{x\}\_\{k+1\}$.

Instead of requiring the full Hessian matrix at the point $mathbf\{x\}\_\{k+1\}$ to be computed as $B\_\{k+1\}$, the approximate Hessian at stage "k" is updated by the addition of two matrices. :$B\_\{k+1\}=B\_k+U\_k+V\_k$Both $U\_k$ and $V\_k$ are rank-one matrices but have different bases. The rank one assumption here means that we may write:$C=mathbf\{a\}mathbf\{b\}^T$So equivalently, $U\_k$ and $V\_k$ construct a rank-two update matrix which is robust against the scale problem often suffered in the gradient descent searching.

(as in Broyden's method, the multidimensional analogue of the secant method). The quasi-Newton condition imposed on this update is

:$B\_\{k+1\}\; (mathbf\{x\}\_\{k+1\}-mathbf\{x\}\_k\; )\; =\; abla\; f(mathbf\{x\}\_\{k+1\})\; -\; abla\; f(mathbf\{x\}\_k\; ).$

Algorithm

From an initial guess $mathbf\{x\}\_0$ and an approximate Hessian matrix $B\_0$ the following steps are repeated until $x$ converges to the solution.

# Obtain $mathbf\{s\}\_k$ by solving: $B\_k\; mathbf\{s\}\_k\; =\; -\; abla\; f(mathbf\{x\}\_k).$
# Perform a line search to find the optimal $alpha\_k$ in the direction found in the first step, then update $mathbf\{x\}\_\{k+1\}\; =\; mathbf\{x\}\_k\; +\; alpha\_kmathbf\{s\}\_k.$
# $mathbf\{y\}\_k\; =\; \{\; abla\; f(mathbf\{x\}\_\{k+1\})\; -\; abla\; f(mathbf\{x\}\_k)\}.$
# $B\_\{k+1\}\; =\; B\_k\; +\; frac\{mathbf\{y\}\_k\; mathbf\{y\}\_k^\{\; op\{mathbf\{y\}\_k^\{\; op\}\; mathbf\{s\}\_k\}\; -\; frac\{B\_k\; mathbf\{s\}\_k\; (B\_k\; mathbf\{s\}\_k)^\{\; op\{mathbf\{s\}\_k^\{\; op\}\; B\_k\; mathbf\{s\}\_k\}.$

$f(mathbf\{x\})$ denotes the objective function to be minimized. Convergence can be checked by observing the norm of the gradient, $left|\; abla\; f(mathbf\{x\}\_k)\; ight|$. Practically, $B\_0$ can be initialized with $B\_0\; =\; I$, so that the first step will be equivalent to a gradient descent, but further steps are more and more refined by $B\_\{k\}$, the approximation to the Hessian.

The first step of the algorithm is carried out using an approximate inverse of the matrix $B\_k$, which is usually obtained efficiently by applying the Sherman–Morrison formula to the fourth line of the algorithm, giving

: $B\_\{k+1\}^\{-1\}\; =\; B\_k^\{-1\}\; +\; frac\{(mathbf\{s\}\_k\; mathbf\{s\}\_k^\{\; op\})\; (mathbf\{s\}\_k^\{\; op\}mathbf\{y\}\_k+mathbf\{y\}\_k^\{\; op\}\; B\_k^\{-1\}\; mathbf\{y\}\_k)\}\{(mathbf\{s\}\_k^\{\; op\}\; mathbf\{y\}\_k)^2\}\; -\; frac\{B\_k^\{-1\}\; mathbf\{y\}\_k\; mathbf\{s\}\_k^\{\; op\}\; +\; mathbf\{s\}\_k\; mathbf\{y\}\_k^\{\; op\}B\_k^\{-1\{mathbf\{s\}\_k^\{\; op\}\; mathbf\{y\}\_k\}.$

Credible intervals or confidence intervals for the solution can be obtained from the inverse of the final Hessian matrix.

ee also

* L-BFGS method
* Newton's method in optimization

Bibliography

* Broyden, C. G., "The Convergence of a Class of Double-rank Minimization Algorithms" ,"Journal of the Institute of Mathematics and Its Applications" 1970, "6", 76-90
* Fletcher, R., " A New Approach to Variable Metric Algorithms", "Computer Journal" 1970, "13", 317-322
* Goldfarb, D., "A Family of Variable Metric Updates Derived by Variational Means", "Mathematics of Computation" 1970, "24", 23-26
* Shanno, D. F.,"Conditioning of Quasi-Newton Methods for Function Minimization" , "Mathematics of Computation" 1970, "24", 647-656
* Avriel, Mordecai 2003. "Nonlinear Programming: Analysis and Methods." Dover Publishing. ISBN 0-486-43227-0.