On the Behavior of the Gradient Norm in the Steepest Descent Method

Computational Optimization and Applications volume 22, pages5–35(2002)Cite this article

Abstract

It is well known that the norm of the gradient may be unreliable as a stopping test in unconstrained optimization, and that it often exhibits oscillations in the course of the optimization. In this paper we present results descibing the properties of the gradient norm for the steepest descent method applied to quadratic objective functions. We also make some general observations that apply to nonlinear problems, relating the gradient norm, the objective function value, and the path generated by the iterates.

This is a preview of subscription content, log in to check access.

References

  1. 1.

    H. Akaike, “On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method,” Ann. Inst. Stat. Math. Tokyo, vol. 11, pp. 1–16, 1959.

    Google Scholar 

  2. 2.

    J. Barzilai and J.M. Borwein, “Two-point step size gradient methods,” IMA Journal of Numerical Analysis, vol. 8, pp. 141–148, 1988.

    Google Scholar 

  3. 3.

    M.S. Bazaraa, H.D. Sherali, and C.M. Shetty, Nonlinear Programming, 2nd edn., John Wiley & Sons: New York, 1993.

    Google Scholar 

  4. 4.

    I. Bongartz, A.R. Conn, N.I.M. Gould, and Ph.L. Toint, “CUTE: Constrained and unconstrained testing environment,” ACM Transactions on Mathematical Software, vol. 21, no. 1, pp. 123–160, 1995.

    Google Scholar 

  5. 5.

    R.H. Byrd, P. Lu, J. Nocedal, and C. Zhu, “A limited memory algorithm for bound constrained optimization,” SIAM Journal on Scientific Computing, vol. 16, no. 5, pp. 1190–1208, 1995.

    Google Scholar 

  6. 6.

    R. Byrd, J. Nocedal, and C. Zhu, “Towards a discrete Newton method with memory for large-scale optimization,” in Nonlinear Optimization and Applications. G. Di Pillo and F. Giannessi (Eds.), Plenum: New York, 1996.

    Google Scholar 

  7. 7.

    A.R. Conn, N.I.M. Gould, and Ph.L. Toint, “LANCELOT: A FORTRAN package for large-scale nonlinear optimization (Release A),” Number 17 in Springer Series in Computational Mathematics, Springer-Verlag: New York, 1992.

    Google Scholar 

  8. 8.

    G.E. Forsythe, “On the asymptotic directions of the s-dimensional optimum gradient method,” Numerische Mathematik, vol. 11, pp. 57–76, 1968.

    Google Scholar 

  9. 9.

    J.Ch. Gilbert, Private communication, 1994.

  10. 10.

    P.E. Gill, W. Murray, and M.H. Wright, Practical Optimization, Academic Press: London, 1981.

    Google Scholar 

  11. 11.

    D.G. Luenberger, Linear and Nonlinear Programming, 2nd edn., Addison-Wesley: Reading, MA, 1984.

    Google Scholar 

Download references

Author information

Affiliations

  1. ECE Department, Northwestern University, Evanston, Il, 60208, USA

    Jorge Nocedal

  2. Department of Mathematics, Facultés Universitaires Notre-Dame de la Paix, Namur, Belgium

    Annick Sartenaer

  3. Department of Electrical and Computer Engineering, Northwestern University, Evanston, Il, 60208-3118, USA

    Ciyou Zhu

Authors
  1. Jorge Nocedal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Annick Sartenaer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ciyou Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Nocedal, J., Sartenaer, A. & Zhu, C. On the Behavior of the Gradient Norm in the Steepest Descent Method. Computational Optimization and Applications 22, 5–35 (2002). https://doi.org/10.1023/A:1014897230089

Download citation

  • nonlinear optimization
  • unconstrained optimization
  • steepest descent method
  • behavior of gradient norm