On the asymptotic behaviour of some new gradient methods
The Barzilai-Borwein (BB) gradient method, and some other new gradient methods have shown themselves to be competitive with conjugate gradient methods for solving large dimension nonlinear unconstrained optimization problems. Little is known about the asymptotic behaviour, even when applied to n−dimensional quadratic functions, except in the case that n=2. We show in the quadratic case how it is possible to compute this asymptotic behaviour, and observe that as n increases there is a transition from superlinear to linear convergence at some value of n≥4, depending on the method. By neglecting certain terms in the recurrence relations we define simplified versions of the methods, which are able to predict this transition. The simplified methods also predict that for larger values of n, the eigencomponents of the gradient vectors converge in modulus to a common value, which is a similar to a property observed to hold in the real methods. Some unusual and interesting recurrence relations are analysed in the course of the study.
KeywordsAsymptotic Behaviour Mathematical Method Conjugate Gradient Gradient Method Quadratic Function
Unable to display preview. Download preview PDF.
- 1.Akaike, H.: On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method. Ann. Inst. Statist. Math. Tokyo 11, 1–17 (1959)Google Scholar
- 5.Barzilai, J., Borwein, J.M.: Two-point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)Google Scholar
- 6.Cauchy, A.: Méthode générale pour la résolution des systèms d’équations simultanées. Comp. Rend. Sci. Paris 25, 536–538 (1847)Google Scholar
- 7.Dai, Y.H.: Alternate Step Gradient Method. Research report, State Key Laboratory of Scientific and Engineering Computing, Institute of Computational Mathematics and Scientific/Engineering computing, Academy of Mathematics and System Sciences, Chinese Academy of Sciences. 2001Google Scholar
- 9.Fletcher, R.: Low storage methods for unconstrained optimization. Lect. Appl. Math. (AMS) 26, 165–179 (1999)Google Scholar
- 10.Fletcher, R.: On the Barzilai-Borwein Method. Research report, Department of Mathematics, University of Dundee, 2001Google Scholar
- 16.Raydan, M.: On the Barzilai and Borwein choice of steplength for the gradient method. IMA J. Numer. Anal. 13, 321–326 (1993)Google Scholar
- 18.Raydan, M.: Nonmonotone spectral methods for large-scale nonlinear systems. Report in the International Workshop on “Optimization and Control with Applications”, Erice, Italy, July 9–17, 2001Google Scholar