BFGS In this previous post, we described how Quasi-Newton methods can be used to minimize a twice-differentiable function whose domain is all of . BFGS is a popular quasi-Newton method. At each iteration, we take the following steps: Solve for in . Update with . Update according to the equation where and . Limited memory […]
Quasi-Newton methods: L-BFGS
Tag: iteration
Quasi-Newton methods in optimization
Consider the unconstrained minimization problem where is twice differentiable and . Newton’s method is a second-order descent method for finding the minimum. Starting at some initial point, at the th iteration we update the candidate solution with the formula where and are the gradient and Hessian of respectively, and is a step size chosen appropriately […]
Quasi-Newton methods in optimization
Exact line search and backtracking line search
Assume we are trying to minimize some convex function which is differentiable. One way to do this is to use a descent method. A general descent method can be described as follows: Given a starting point in the domain of , iterate over the following 3 steps until some stopping criterion is satisfied: Choose a […]
Exact line search and backtracking line search