BFGS In this previous post, we described how Quasi-Newton methods can be used to minimize a twice-differentiable function whose domain is all of . BFGS is a popular quasi-Newton method. At each iteration, we take the following steps: Solve for in . Update with . Update according to the equation where and . Limited memory […]
Quasi-Newton methods: L-BFGS
Category: Metamatemática, Matemática Pura y Matemática Aplicada
Higher Sense of Purpose in Life May Be Linked to Lower Mortality Risk
Source: Boston University Growing research indicates that one’s purpose—i.e., the extent to which someone perceives a sense of direction and goals in their life—may be linked to health-protective benefits such as better physical functioning and lower risks of cardiovascular disease or cognitive decline. Now, a new study led by a Boston University School of Public […]
Higher Sense of Purpose in Life May Be Linked to Lower Mortality Risk
Quasi-Newton methods in optimization
Consider the unconstrained minimization problem where is twice differentiable and . Newton’s method is a second-order descent method for finding the minimum. Starting at some initial point, at the th iteration we update the candidate solution with the formula where and are the gradient and Hessian of respectively, and is a step size chosen appropriately […]
Quasi-Newton methods in optimization
Affine hull vs. convex hull
The affine hull and convex hull are closely related concepts. Let be a set in . The affine hull of is the set of all affine combinations of elements of : The convex hull of is the set of all convex combinations of the elements of : Putting the definitions side by side, we see […]
Affine hull vs. convex hull
Exact line search and backtracking line search
Assume we are trying to minimize some convex function which is differentiable. One way to do this is to use a descent method. A general descent method can be described as follows: Given a starting point in the domain of , iterate over the following 3 steps until some stopping criterion is satisfied: Choose a […]
Exact line search and backtracking line search
What are the KKT conditions?
Consider an optimization problem in standard form: with the variable . Assume that the ‘s and ‘s are differentiable. (At this point, we are not assuming anything about their convexity.) As before, define the Lagrangian as the function Let and be the primal and dual optimal points respectively (i.e. points where the primal and dual […]
What are the KKT conditions?
Lagrange dual, weak duality and strong duality
Consider an optimization problem in standard form: with the variable . Let be the domain for , i.e. the intersection of the domains of the ‘s and the ‘s. Let denote the optimal value of the problem. The Lagrange dual function is the function defined as the minimum value of the Lagrangian over : The […]
Lagrange dual, weak duality and strong duality
The Lagrange dual function is always concave
Consider an optimization problem in standard form: with the variable . Let be the domain for , i.e. the intersection of the domains of the ‘s and the ‘s. The Lagrangian associated with this problem is the function defined as with domain . The Lagrange dual function is the function defined as the minimum value […]
The Lagrange dual function is always concave
A Bayesian probability worksheet
This is a spinoff from the previous post. In that post, we remarked that whenever one receives a new piece of information , the prior odds ratio between an alternative hypothesis and a null hypothesis is updated to a posterior odds ratio , which can be computed via Bayes’ theorem by the formula where is […]
A Bayesian probability worksheet
What are the odds?
An unusual lottery result made the news recently: on October 1, 2022, the PCSO Grand Lotto in the Philippines, which draws six numbers from to at random, managed to draw the numbers (though the balls were actually drawn in the order ). In other words, they drew exactly six multiples of nine from to . […]
What are the odds?