Advertisement

A Positive Barzilai–Borwein-Like Stepsize and an Extension for Symmetric Linear Systems

  • Yu-Hong DaiEmail author
  • Mehiddin Al-Baali
  • Xiaoqi Yang
Part of the Springer Proceedings in Mathematics & Statistics book series (PROMS, volume 134)

Abstract

The Barzilai and Borwein (BB) gradient method has achieved a lot of attention since it performs much more better than the classical steepest descent method. In this paper, we analyze a positive BB-like gradient stepsize and discuss its possible uses. Specifically, we present an analysis of the positive stepsize for two-dimensional strictly convex quadratic functions and prove the R-superlinear convergence under some assumption. Meanwhile, we extend BB-like methods for solving symmetric linear systems and find that a variant of the positive stepsize is very useful in the context. Some useful discussions on the positive stepsize are also given.

Keywords

Unconstrained optimization Barzilai and Borwein gradient method Quadratic function R-superlinear convergence Condition number 

Notes

Acknowledgements

The authors are grateful to Dr. Bo Jiang for checking an early version of this manuscript and to Ms. Liaoyuan Zeng for her editing of this paper. They also thank an anonymous referee for his/her useful suggestions and comments.

References

  1. 1.
    Al-Baali, M.: On alternate steps for gradient methods. Talk at 22-nd Biennial Conference on Numerical Analysis, University of Dundee, Scotland, 26–29 June 2007Google Scholar
  2. 2.
    Barzilai, J., Borwein, J.M.: Two-point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)CrossRefMathSciNetzbMATHGoogle Scholar
  3. 3.
    Birgin, E.G., Martinez, J.M., Raydan, M.: Nonmonotone spectral projected gradient methods on convex sets. SIAM J. Optim. 10, 1196–1211 (2000)CrossRefMathSciNetzbMATHGoogle Scholar
  4. 4.
    Cauchy, A.: Méthode générale pour la résolution des systèms d’équations simultanées. Comput. Rend. Sci. Paris 25, 536–538 (1847)Google Scholar
  5. 5.
    Cheng, M., Dai, Y.H.: Adaptive nonmonotone spectral residual method for large-scale nonlinear systems. Pac. J. Optim. 8, 15–25 (2012)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Cruz, W.L., Martínez, J.M., Raydan, M.: Spectral residual method without gradient information for solving large-scale nonlinear systems of equations. Math. Comput. 75, 1429–1448 (2006)CrossRefzbMATHGoogle Scholar
  7. 7.
    Dai, Y.H.: Alternate step gradient method. Optimization 52, 395–415 (2003)CrossRefMathSciNetzbMATHGoogle Scholar
  8. 8.
    Dai, Y.H.: A new analysis on the Barzilai-Borwein gradient method. J. Oper. Res. Soc. China 1, 187–198 (2013)CrossRefzbMATHGoogle Scholar
  9. 9.
    Dai, Y.H., Fletcher, R.: On the asymptotic behaviour of some new gradient methods. Math. Program. Ser. A 103, 541–559 (2005)CrossRefMathSciNetzbMATHGoogle Scholar
  10. 10.
    Dai, Y.H., Fletcher, R.: New algorithms for singly linearly constrained quadratic programs subject to lower and upper bounds. Math. Program. Ser. A 106, 403–421 (2006)CrossRefMathSciNetzbMATHGoogle Scholar
  11. 11.
    Dai, Y.H., Liao, L.Z.: R-linear convergence of the Barzilai and Borwein gradient method. IMA J. Numer. Anal. 26, 1–10 (2002)CrossRefMathSciNetGoogle Scholar
  12. 12.
    Dai, Y.H., Liao, L.Z.: A new first-order neural network for unconstrained nonconvex optimization. Research Report, Academy of Mathematics and Systems Science, Chinese Academy of Sciences (1999)Google Scholar
  13. 13.
    Dai, Y.H., Yang, X.Q.: A new gradient method with an optimal stepsize property. Comput. Optim. Appl. 33, 73–88 (2006)CrossRefMathSciNetzbMATHGoogle Scholar
  14. 14.
    Elman, H.C., Golub, G.H.: Inexact and preconditioning Uzawa algorithms for saddle point problems. SIAM J. Numer. Anal. 36, 1645–1661 (1994)CrossRefMathSciNetGoogle Scholar
  15. 15.
    Grippo, L., Lampariello, F., Lucidi, S.: A nonmonotone line search technique for Newton’s method. SIAM J. Numer. Anal. 23, 707–716 (1986)CrossRefMathSciNetzbMATHGoogle Scholar
  16. 16.
    Raydan, M.: On the Barzilai and Borwein choice of steplength for the gradient method. IMA J. Numer. Anal. 13, 321–326 (1993)CrossRefMathSciNetzbMATHGoogle Scholar
  17. 17.
    Raydan, J.: The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem. SIAM J. Optim. 7, 26–33 (1997)CrossRefMathSciNetzbMATHGoogle Scholar
  18. 18.
    Serafini, T., Zanghirati, G., Zanni, L.: Gradient projection methods for quadratic programs and applications in training support vector machines. Optim. Methods Softw. 20, 347–372 (2005)MathSciNetGoogle Scholar
  19. 19.
    Vrahatis, M.N., Androulakis, G.S., Lambrinos, J.N., Magoulas, G.D.: A class of gradient unconstrained minimization algorithms with adaptive stepsize. J. Comput. Appl. Math. 114, 367–386 (2000)CrossRefMathSciNetzbMATHGoogle Scholar
  20. 20.
    Wright, S.J., Nowak, R.D., Figueiredo, M.A.T.: Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57, 2479–2493 (2009)CrossRefMathSciNetGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.State Key Laboratory of Scientific and Engineering ComputingChinese Academy of SciencesBeijingP.R. China
  2. 2.Department of Mathematics and StatisticsSultan Qaboos UniversityMuscatOman
  3. 3.Department of Applied MathematicsThe Hong Kong Polytechnic UniversityKowloonHong Kong

Personalised recommendations