Newton’s Method and Scoring
The MM and EM algorithms are hardly the only methods of optimization. Newton’s method is better known and more widely applied. We encountered Newton’s method in Section 5.4 of Chapter 5. Here we focus on the multidimensional version. Despite its defects, Newton’s method is the gold standard for speed of convergence and forms the basis of many modern optimization algorithms. Its variants seek to retain its fast convergence while taming its defects. The variants all revolve around the core idea of locally approximating the objective function by a strictly convex quadratic function. At each iteration the quadratic approximation is optimized. Safeguards are introduced to keep the iterates from veering toward irrelevant stationary points.
KeywordsExponential Family Secant Condition Dirichlet Distribution Multinomial Family Ascent Algorithm
Unable to display preview. Download preview PDF.
- 7.Davidon WC (1959) Variable metric methods for minimization. AEC Research and Development Report ANL-5990, Argonne National Laboratory, USAGoogle Scholar
- 8.De Leeuw J, Heiser WJ (1980) Multidimensional scaling with restrictions on the configuration. In Multivariate Analysis, Volume V, Krishnaiah PR, North-Holland, Amsterdam, pp 501-522Google Scholar
- 15.Jennrich RI, Moore RH (1975) Maximum likelihood estimation by means of nonlinear least squares. Proceedings of the Statistical Computing Section: Amer Stat Assoc 57-65Google Scholar
- 22.Magnus JR, Neudecker H (1988) Matrix Differential Calculus withApplications in Statistics and Econometrics. Wiley, New YorkGoogle Scholar
- 27.Whyte BM, Gold J, Dobson AJ, Cooper DA (1987) Epidemiology of acquired immunodeficiency syndrome in Australia. Med J Aust 147:65-69Google Scholar