Mathematical Programming

, Volume 107, Issue 3, pp 409–438 | Cite as

Asymptotic behaviour of a family of gradient algorithms in ℝ d and Hilbert spaces

  • Luc Pronzato
  • Henry P. Wynn
  • Anatoly A. Zhigljavsky


The asymptotic behaviour of a family of gradient algorithms (including the methods of steepest descent and minimum residues) for the optimisation of bounded quadratic operators in ℝ d and Hilbert spaces is analyzed. The results obtained generalize those of Akaike (1959) in several directions. First, all algorithms in the family are shown to have the same asymptotic behaviour (convergence to a two-point attractor), which implies in particular that they have similar asymptotic convergence rates. Second, the analysis also covers the Hilbert space case. A detailed analysis of the stability property of the attractor is provided.

Mathematics Subject Classification (2000)

90C25 68Q25 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Akaike, H.: On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method. Ann. Inst. Statist. Math. Tokyo 11, 1–16 (1959)CrossRefzbMATHMathSciNetGoogle Scholar
  2. 2.
    Barzilai, J., Borwein, J.M.: Two-point step size gradient methods. IMA J. Numerical Anal. 8, 141–148 (1988)zbMATHMathSciNetGoogle Scholar
  3. 3.
    Birgin, E.G., Martínez, J.M.: A spectral conjugate gradient method for unconstrained optimization. Appl. Math. Optim. 43, 117–128 (2001)CrossRefzbMATHMathSciNetGoogle Scholar
  4. 4.
    Fedorov, V.V.: Theory of Optimal Experiments. Academic Press, New York, 1972Google Scholar
  5. 5.
    Forsythe, G.E.: On the asymptotic directions of the s-dimensional optimum gradient method. Numerische Mathematik 11, 57–76 (1968)CrossRefzbMATHMathSciNetGoogle Scholar
  6. 6.
    Hale, J., Koçak, H.: Dynamics and Bifurcations. Springer-Verlag, Heidelberg, 1991Google Scholar
  7. 7.
    Kantorovich, L.V., Akilov, G.P.: Functional Analysis. Pergamon Press, London, 1982. Second editionGoogle Scholar
  8. 8.
    Kiefer, J., Wolfowitz, J.: The equivalence of two extremum problems. Canadian J. Math. 12, 363–366 (1960)zbMATHMathSciNetGoogle Scholar
  9. 9.
    Kozjakin, V.S., Krasnosel'skii, M.A.: Some remarks on the method of minimal residues. Numer. Funct. Anal. Optim. 4 (3), 211–239 (1982)Google Scholar
  10. 10.
    Krasnosel'skii, M.A., Krein, S.G.: An iteration process with minimal residues. Mat. Sb. (in Russian) 31 (4), 315–334 (1952)MathSciNetGoogle Scholar
  11. 11.
    LaSalle, J.P.: The Stability of Dynamical Systems. SIAM, Philadelphia, 1976Google Scholar
  12. 12.
    Luenberger, D.G.: Introduction to Linear and Nonlinear Programming. Addison-Wesley, Reading, Massachusetts, 1973Google Scholar
  13. 13.
    Nocedal, J., Sartenaer, A., Zhu, C.: On the accuracy of nonlinear optimization algorithms. Technical Report Nov. 1998, ECE Department, Northwestern Univ., Evanston, Il 60208, 1998Google Scholar
  14. 14.
    Nocedal, J., Sartenaer, A., Zhu, C.: On the behavior of the gradient norm in the steepest descent method. Computational Optimization and Applications 22, 5–35 (2002)CrossRefzbMATHMathSciNetGoogle Scholar
  15. 15.
    Pronzato, L., Wynn, H.P., Zhigljavsky, A.A.: Dynamical Search. Chapman & Hall/CRC, Boca Raton, 2000Google Scholar
  16. 16.
    Pronzato, L., Wynn, H.P., Zhigljavsky, A.A.: Renormalised steepest descent in Hilbert space converges to a two-point attractor. Acta Applicandae Mathematicae 67, 1–18 (2001)CrossRefzbMATHMathSciNetGoogle Scholar
  17. 17.
    Pronzato, L., Wynn, H.P., Zhigljavsky, A.A.: Kantorovich-type inequalities for operators via D-optimal design theory. Linear Algebra and Its Applications (Special Issue on Linear Algebra and Statistics), 410, 160–169 (2005)zbMATHMathSciNetGoogle Scholar
  18. 18.
    Raydan, M.: The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem. SIAM J. Optimi. 7 (1), 26–33 (1997)CrossRefMathSciNetGoogle Scholar
  19. 19.
    Raydan, M., Svaiter, B.F.: Relaxed steepest descent and Cauchy-Barzilai-Borwein method. Comput. Optim. Appl. 21, 155–167 (2002)CrossRefzbMATHMathSciNetGoogle Scholar
  20. 20.
    Shiryaev, A.N.: Probability. Springer, Berlin, 1996Google Scholar
  21. 21.
    Silvey, S.D.: Optimal Design. Chapman & Hall, London, 1980Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Luc Pronzato
    • 1
  • Henry P. Wynn
    • 2
  • Anatoly A. Zhigljavsky
    • 3
  1. 1.Laboratoire I3S, CNRS - UNSALes Algorithmes - Bât. Euclide BSophia Antipolis CedexFrance
  2. 2.Department of StatisticsLondon School of EconomicsLondonUK
  3. 3.School of MathematicsCardiff UniversityCardiffUK

Personalised recommendations