Abstract
The asymptotic behaviour of a family of gradient algorithms (including the methods of steepest descent and minimum residues) for the optimisation of bounded quadratic operators in ℝd and Hilbert spaces is analyzed. The results obtained generalize those of Akaike (1959) in several directions. First, all algorithms in the family are shown to have the same asymptotic behaviour (convergence to a two-point attractor), which implies in particular that they have similar asymptotic convergence rates. Second, the analysis also covers the Hilbert space case. A detailed analysis of the stability property of the attractor is provided.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akaike, H.: On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method. Ann. Inst. Statist. Math. Tokyo 11, 1–16 (1959)
Barzilai, J., Borwein, J.M.: Two-point step size gradient methods. IMA J. Numerical Anal. 8, 141–148 (1988)
Birgin, E.G., Martínez, J.M.: A spectral conjugate gradient method for unconstrained optimization. Appl. Math. Optim. 43, 117–128 (2001)
Fedorov, V.V.: Theory of Optimal Experiments. Academic Press, New York, 1972
Forsythe, G.E.: On the asymptotic directions of the s-dimensional optimum gradient method. Numerische Mathematik 11, 57–76 (1968)
Hale, J., Koçak, H.: Dynamics and Bifurcations. Springer-Verlag, Heidelberg, 1991
Kantorovich, L.V., Akilov, G.P.: Functional Analysis. Pergamon Press, London, 1982. Second edition
Kiefer, J., Wolfowitz, J.: The equivalence of two extremum problems. Canadian J. Math. 12, 363–366 (1960)
Kozjakin, V.S., Krasnosel'skii, M.A.: Some remarks on the method of minimal residues. Numer. Funct. Anal. Optim. 4 (3), 211–239 (1982)
Krasnosel'skii, M.A., Krein, S.G.: An iteration process with minimal residues. Mat. Sb. (in Russian) 31 (4), 315–334 (1952)
LaSalle, J.P.: The Stability of Dynamical Systems. SIAM, Philadelphia, 1976
Luenberger, D.G.: Introduction to Linear and Nonlinear Programming. Addison-Wesley, Reading, Massachusetts, 1973
Nocedal, J., Sartenaer, A., Zhu, C.: On the accuracy of nonlinear optimization algorithms. Technical Report Nov. 1998, ECE Department, Northwestern Univ., Evanston, Il 60208, 1998
Nocedal, J., Sartenaer, A., Zhu, C.: On the behavior of the gradient norm in the steepest descent method. Computational Optimization and Applications 22, 5–35 (2002)
Pronzato, L., Wynn, H.P., Zhigljavsky, A.A.: Dynamical Search. Chapman & Hall/CRC, Boca Raton, 2000
Pronzato, L., Wynn, H.P., Zhigljavsky, A.A.: Renormalised steepest descent in Hilbert space converges to a two-point attractor. Acta Applicandae Mathematicae 67, 1–18 (2001)
Pronzato, L., Wynn, H.P., Zhigljavsky, A.A.: Kantorovich-type inequalities for operators via D-optimal design theory. Linear Algebra and Its Applications (Special Issue on Linear Algebra and Statistics), 410, 160–169 (2005)
Raydan, M.: The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem. SIAM J. Optimi. 7 (1), 26–33 (1997)
Raydan, M., Svaiter, B.F.: Relaxed steepest descent and Cauchy-Barzilai-Borwein method. Comput. Optim. Appl. 21, 155–167 (2002)
Shiryaev, A.N.: Probability. Springer, Berlin, 1996
Silvey, S.D.: Optimal Design. Chapman & Hall, London, 1980
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Pronzato, L., Wynn, H. & Zhigljavsky, A. Asymptotic behaviour of a family of gradient algorithms in ℝd and Hilbert spaces. Math. Program. 107, 409–438 (2006). https://doi.org/10.1007/s10107-005-0602-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10107-005-0602-7