Summary
We study the asymptotic behaviour of Forsythe's s-optimum gradient algorithm for the minimization of a quadratic function in \({\mathbb R}^d\) using a renormalization that converts the algorithm into iterations applied to a probability measure. Bounds on the performance of the algorithm (rate of convergence) are obtained through optimum design theory and the limiting behaviour of the algorithm for s = 2 is investigated into details. Algorithms that switch periodically between s = 1 and s = 2 are shown to converge much faster than when s is fixed at 2.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Akaike, H. (1959). On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method. Annals of the Institute Statistical Mathematics Tokyo, 11, 1–16.
Elaydi, S. (2005). An Introduction to Difference Equations. Springer, Berlin. Third Edition.
Fedorov, V. (1972). Theory of Optimal Experiments. Academic Press, New York.
Forsythe, G. (1968). On the asymptotic directions of the s-dimensional optimum gradient method. Numerische Mathematik, 11, 57–76.
Hoel, P. and Levine, A. (1964). Optimal spacing and weighting in polynomial prediction. Annals of Mathematical Statistics, 35(4), 1553–1560.
Kantorovich, L. and Akilov, G. (1982). Functional Analysis. Pergamon Press, London. Second edition.
Kiefer, J. and Wolfowitz, J. (1959). Optimum designs in regression problems. Annals of Mathematical Statistics, 30, 271–294.
Luenberger, D. (1973). Introduction to Linear and Nonlinear Programming. Addison-Wesley, Reading, Massachusetts.
Meinardus, G. (1963). Über eine Verallgemeinerung einer Ungleichung von L.V. Kantorowitsch. Numerische Mathematik, 5, 14–23.
Molchanov, I. and Zuyev, S. (2001). Variational calculus in the space of measures and optimal design. In A. Atkinson, B. Bogacka, and A. Zhigljavsky, editors, Optimum Design 2000, chapter 8, pages 79–90. Kluwer, Dordrecht.
Nocedal, J., Sartenaer, A., and Zhu, C. (1998). On the accuracy of nonlinear optimization algorithms. Technical Report Nov. 1998, ECE Department, Northwestern Univ., Evanston, Il 60208.
Nocedal, J., Sartenaer, A., and Zhu, C. (2002). On the behavior of the gradient norm in the steepest descent method. Computational Optimization and Applications, 22, 5–35.
Pronzato, L., Wynn, H., and Zhigljavsky, A. (2000). Dynamical Search. Chapman & Hall/CRC, Boca Raton.
Pronzato, L., Wynn, H., and Zhigljavsky, A. (2001). Renormalised steepest descent in Hilbert space converges to a two-point attractor. Acta Applicandae Mathematicae, 67, 1–18.
Pronzato, L., Wynn, H., and Zhigljavsky, A. (2005). Kantorovich-type inequalities for operators via D-optimal design theory. Linear Algebra and its Applications (Special Issue on Linear Algebra and Statistics), 410, 160–169.
Pronzato, L., Wynn, H., and Zhigljavsky, A. (2006). Asymptotic behaviour of a family of gradient algorithms in \({\mathbb R}^d\) and Hilbert spaces. Mathematical Programming, A107, 409–438.
Sahm, M. (1998). Optimal designs for estimating individual coefficients in polynomial regression. Ph.D. Thesis, Fakultät für Mathematik, Ruhr Universität Bochum.
Silvey, S. (1980). Optimal Design. Chapman & Hall, London.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer Science+Business Media LLC
About this chapter
Cite this chapter
Pronzato, L., Wynn, H., Zhigljavsky, A. (2009). A Dynamical-System Analysis of the Optimum s-Gradient Algorithm. In: Pronzato, L., Zhigljavsky, A. (eds) Optimal Design and Related Areas in Optimization and Statistics. Springer Optimization and Its Applications, vol 28. Springer, New York, NY. https://doi.org/10.1007/978-0-387-79936-0_3
Download citation
DOI: https://doi.org/10.1007/978-0-387-79936-0_3
Publisher Name: Springer, New York, NY
Print ISBN: 978-0-387-79935-3
Online ISBN: 978-0-387-79936-0
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)