Quasi-Newton methods: L-BFGS

BFGS In this previous post, we described how Quasi-Newton methods can be used to minimize a twice-differentiable function whose domain is all of . BFGS is a popular quasi-Newton method. At each iteration, we take the following steps: Solve for in . Update with . Update according to the equation where and . Limited memory […]

Quasi-Newton methods: L-BFGS

Quasi-Newton methods in optimization

Consider the unconstrained minimization problem where is twice differentiable and . Newton’s method is a second-order descent method for finding the minimum. Starting at some initial point, at the th iteration we update the candidate solution with the formula where and are the gradient and Hessian of respectively, and is a step size chosen appropriately […]

Quasi-Newton methods in optimization

Exact line search and backtracking line search

Assume we are trying to minimize some convex function which is differentiable. One way to do this is to use a descent method. A general descent method can be described as follows: Given a starting point in the domain of , iterate over the following 3 steps until some stopping criterion is satisfied: Choose a […]

Exact line search and backtracking line search




Aldrich, J. H., & Nelson, F. D. (1984). Linear Probability, Logit, and Probit Models. Beverly Hills: Sage University Papers Series. Quantitative Applications in the Social Sciences.

Allen, M. (2017). The SAGE Encyclopedia of COMMUNICATION RESEARCH METHODS. London: SAGE Publications, Inc.

AMERICAN PSYCHOLOGICAL ASSOCIATION. (2021, Julio 15). level. Retrieved from APA Dictionary of Pyschology: https://dictionary.apa.org/level

AMERICAN PYSCHOLOGICAL ASSOCIATION. (2021, Julio 15). factor. Retrieved from APA Dictionary of Pyschology: https://dictionary.apa.org/factor

AMERICAN PYSCHOLOGY ASSOCIATION. (2021, Julio 15). logistic regression (LR). Retrieved from APA Dictionary of Pyschology: https://dictionary.apa.org/logistic-regression

Barrios, J. (2019, Julio 19). La matriz de confusión y sus métricas . Retrieved from Health BIG DATA: https://www.juanbarrios.com/la-matriz-de-confusion-y-sus-metricas/

Bhuptani, R. (2020, Julio 13). Quora. Retrieved from What is the difference between linear regression and least squares?: https://www.quora.com/What-is-the-difference-between-linear-regression-and-least-squares

Birnbaum, Z. W., & Sirken, M. G. (1950, Marzo). Bias Due to Non-Availability in Sampling Surveys. Journal of the American Statistical Association, 45(249), 98-111.

Burrus, C. S. (2021, Julio 7). Iterative Reweighted Least Squares. Retrieved from https://cnx.org/exports/92b90377-2b34-49e4-b26f-7fe572db78a1@12.pdf/iterative-reweighted-least-squares-12.pdf

Centro Centroamericano de Población. (2021, Abril 28). Variables y escalas de medición. Retrieved from Universidad de Costa Rica: https://ccp.ucr.ac.cr/cursos/epidistancia/contenido/2_escmed.html

Cochran, W. G. (1991). Técnicas de Muestreo. México, D.F.: Compañía Editorial Continental.

Departamento Administrativo Nacional de Estadística. (2003). Metodología de Diseño Muestral. Bogotá: Dirección Sistema Nacional de Información Estadística. Retrieved from https://www.dane.gov.co/files/EDI/anexos_generales/Metodologia_diseno_muestral_anexo1.pdf?phpMyAdmin=a9ticq8rv198vhk5e8cck52r11

Díaz-Narváez, V. P. (2017). Regresión logística y decisiones clínicas. Nutrición Hospitalaria, 34(6), 1505-1505. Retrieved from https://scielo.isciii.es/pdf/nh/v34n6/36_diaz.pdf

Google Developers. (2021, Julio 19). Clasificación: Exactitud. Retrieved from https://developers.google.com/machine-learning/crash-course/classification/accuracy

Greene, W. (2012). Econometric Analysis (Séptima ed.). Harlow, Essex, England: Pearson Education Limited.

Gujarati, D., & Porter, D. (2010, Julio 8). Econometría (Quinta ed.). México, D.F.: McGrawHill Educación. Retrieved from Homocedasticidad.

Haskett, D. R. (2014, Octubre 10). “Mitochondrial DNA and Human Evolution” (1987), by “Mitochondrial DNA and Human Evolution” (1987), by Rebecca Louise Cann, Mark Stoneking, and Allan Charles Wilson. Retrieved from The Embryo Project Encyclopedia: https://embryo.asu.edu/pages/mitochondrial-dna-and-human-evolution-1987-rebecca-louise-cann-mark-stoneking-and-allan

Hastie, T., Tibshirani, R., & Friedman, J. (2017). The Elements of Statistical Learning. Data Mining, Inference, and Prediction (Segunda ed.). New York: Springer.

Instituto dei Sistemi Complessi. (2021, Febrero 27). Topolical vs Metric Distance. Retrieved from Biological Systems: https://www.isc.cnr.it/research/topics/physical-biology/biological-systems/topological-vs-metric-distance/

Instituto Nacional de Estadística y Censos de Costa Rica. (2016, Julio). Manual de Clasificación Geográfica con Fines Estadísticos de Costa Rica. Retrieved from Biblioteca Virtual: https://www.inec.cr/sites/default/files/documetos-biblioteca-virtual/meinstitucionalmcgfecr.pdf

Instituto Nacional de Estadística y Censos de Costa Rica. (2019). ENIGH. 2018. Cuadros sobre ingresos de los hogares. San José: INEC. Retrieved from https://www.inec.cr/sites/default/files/documetos-biblioteca-virtual/reenigh2018-ingreso.xlsx

Instituto Nacional de Estadística y Censos de Costa Rica. (2021, 7 14). Factor de Expansión. Retrieved from INEC: https://www.inec.cr/sites/default/files/_book/F.html

Instituto Nacional de Estadística y Censos de la República Argentina. (2019). Encuesta de Actividades de Niños, Niñas y Adolescentes 2016-2017. Factores de expansión, estimación y cálculo de los errores por muestra para el dominio rural. Buenos Aires: Ministerio de Hacienda. Retrieved from https://www.indec.gob.ar/ftp/cuadros/menusuperior/eanna/anexo_bases_eanna_rural.pdf

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An Introduction to Statistical Learning with Applications in R. New York: Springer.

Jose, K. (2020, Junio 27). Graph Theory | Isomorphic Trees. Retrieved from Towards Data Science: https://towardsdatascience.com/graph-theory-isomorphic-trees-7d48aa577e46

Köhler, T. (2016). Income and Wealth Poverty in Germany. SOEP papers on Multidisciplinary Panel Data Research, 1-48. Retrieved from https://www.diw.de/documents/publikationen/73/diw_01.c.540534.de/diw_sp0857.pdf

Kolmogórov, A. N., & Fomin, S. V. (1978). Elementos de la Teoría de Funciones y del Análisis Funcional (Tercera ed.). (q. e.-m. Traducido del ruso por Carlos Vega, Trans.) Moscú: MIR.

Liao, T. F. (1994). INTERPRETING PROBABILITY MODELS. Logit, Probit, and Other Generalized Linear Models. Iowa: Sage University Papers Series. Quantitative Applications in the Social Sciences.

Lipschutz, S. (1992). Álgebra Lineal. Madrid: McGraw-Hill.

Lohr, S. L. (2019). Sampling: Design and Analysis (Segunda ed.). Boca Raton: CRC Press.

Lohr, S. L. (2019). Sampling: Design and Analysis (Segunda ed.). Boca Raton: CRC Press.

McCullagah, P., & Nelder, J. A. (1989). Generalized Linear Models (Segunda ed.). London: Chapman and Hall.

McCullagh, P., & Nelder, J. A. (1989). Generalized Linear Models (Segunda ed.). London: Chapman and Hall.

Nelder, J. A., & Wedderburn, R. W. (1972). Generalized Linear Models. Journal of the Royal Statistical Society, 135(3), 370-384.

Online Stat Book. (2021, Julio 15). Levels of an Independent Variable. Retrieved from Independent and dependent variables: https://onlinestatbook.com/2/introduction/variables.html

Patil, G. P., & Shorrock, R. (1965). On Certain Properties of the Exponential-type Families. Journal of the Royal Statistical, 27(1), 94-99.

Perry, J. (2014, Abril 2). NORM TO/FROM METRIC. Retrieved from The University of Southern Mississippi: https://www.math.usm.edu/perry/old_classes/mat681sp14/norm_and_metric.pdf

Ritchey, F. (2002). ESTADÍSTICA PARA LAS CIENCIAS SOCIALES. El potencial de la imaginación estadística. México, D.F.: McGRAW-HILL/INTERAMERICANA EDITORES, S.A. DE C.V.

Samuels, S. (2014, 11 19). Can I get to an approximation of the population with knowledge of the expansion factor? Retrieved from Cross Validated. StackExchange: https://stats.stackexchange.com/questions/124750/can-i-get-to-an-approximation-of-the-population-with-knowledge-of-the-expansion

StackExchange Cross Validated. (2017, Febrero 2). “Least Squares” and “Linear Regression”, are they synonyms? Retrieved from What is the difference between least squares and linear regression? Is it the same thing?: https://stats.stackexchange.com/questions/259525/least-squares-and-linear-regression-are-they-synonyms

StackExchange Data Science. (2016, Junio 19). Is GLM a statistical or machine learning model? Retrieved from https://datascience.stackexchange.com/questions/488/is-glm-a-statistical-or-machine-learning-model

StackOverFlow. (2014, Marzo 15). Supervised Learning, Unsupervised Learning, Regression. Retrieved from https://stackoverflow.com/questions/22419136/supervised-learning-unsupervised-learning-regression

TalkStats. (2011, Noviembre 29). SPSS. Retrieved from Forums: http://www.talkstats.com/threads/what-is-the-difference-between-a-factor-and-a-covariate-for-multinomial-logistic-reg.21864/

UNITED NATIONS ECONOMIC COMMISSION FOR EUROPE. (2017). Guide on Poverty Measure. New York and Geneva: UNITED NATIONS. Retrieved from https://ec.europa.eu/eurostat/ramon/statmanuals/files/UNECE_Guide_on_Poverty_Measurement.pdf

van den Berg, R. G. (2021, Julio 15). Measurement Levels – What and Why? Retrieved from SPSS Tutorials: https://www.spss-tutorials.com/measurement-levels/

Weisstein, E. W. (2021, Julio 15). Sigmoid Function. Retrieved from MathWorld – A Wolfram Web Resource: https://mathworld.wolfram.com/SigmoidFunction.html

Weisstein, E. W. (2021, Mayo 21). Sigmoid Function. Retrieved from MathWorld – A Wolfram Web Resource: https://mathworld.wolfram.com/SigmoidFunction.html

Weisstein, E. W. (2021, Mayo 18). Smooth Function. Retrieved from Wolfram MathWorld – A Wolfram Web Resource: https://mathworld.wolfram.com/SmoothFunction.html

Wikimedia. (2021, Abril 6). Commons. Retrieved from Wikipedia: https://upload.wikimedia.org/wikipedia/commons/b/bf/Undirected.svg

Wikipedia. (2021, Julio 6). Graph isomorphism. Retrieved from Morphism: https://en.wikipedia.org/wiki/Graph_isomorphism

Wikipedia. (2021, Mayo 21). Iterative proportional fitting. Retrieved from Statistical algorithms: https://en.wikipedia.org/wiki/Iterative_proportional_fitting

Wikipedia. (2021, Febrero 25). Iteratively reweighted least squares. Retrieved from Least squares: https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares

Wikipedia. (2021, Julio 13). Logistic function. Retrieved from Growth curves: https://en.wikipedia.org/wiki/Logistic_function

Wikipedia. (2021, Mayo 22). Logistic regression. Retrieved from Regression models: https://en.wikipedia.org/wiki/Logistic_regression

Wikipedia. (2021, Junio 14). Logit. Retrieved from Special functions: https://en.wikipedia.org/wiki/Logistic_function

Wikipedia. (2021, Julio 8). Lp space. Retrieved from Measure theory: https://www.wikiwand.com/en/Lp_space

Wikipedia. (2021, Abril 15). Odds. Retrieved from Wagering: https://en.wikipedia.org/wiki/Odds

Wikipedia. (2021, Julio 10). Precision and recall. Retrieved from Bioinformatics: https://en.wikipedia.org/wiki/Precision_and_recall

Wooldridge, J. (2010). Econometric Analysis of Cross Section and Panel Data (Segunda ed.). Cambridge, Massachusetts: MIT Press.