Advertisement

Mathematical Programming

, Volume 127, Issue 2, pp 245–295 | Cite as

Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results

  • Coralia Cartis
  • Nicholas I. M. Gould
  • Philippe L. Toint
Full Length Paper Series A

Abstract

An Adaptive Regularisation algorithm using Cubics (ARC) is proposed for unconstrained optimization, generalizing at the same time an unpublished method due to Griewank (Technical Report NA/12, 1981, DAMTP, University of Cambridge), an algorithm by Nesterov and Polyak (Math Program 108(1):177–205, 2006) and a proposal by Weiser et al. (Optim Methods Softw 22(3):413–431, 2007). At each iteration of our approach, an approximate global minimizer of a local cubic regularisation of the objective function is determined, and this ensures a significant improvement in the objective so long as the Hessian of the objective is locally Lipschitz continuous. The new method uses an adaptive estimation of the local Lipschitz constant and approximations to the global model-minimizer which remain computationally-viable even for large-scale problems. We show that the excellent global and local convergence properties obtained by Nesterov and Polyak are retained, and sometimes extended to a wider class of problems, by our ARC approach. Preliminary numerical experiments with small-scale test problems from the CUTEr set show encouraging performance of the ARC algorithm when compared to a basic trust-region implementation.

Keywords

Nonlinear optimization Unconstrained optimization Cubic regularization Newton’s method Trust-region methods Global convergence Local convergence 

Mathematics Subject Classification (2000)

90C30 65K05 49M37 49M15 58C15 65F10 65H05 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Byrd R.H., Khalfan H.F., Schnabel R.B.: Analysis of a symmetric rank-one trust region method. SIAM J. Optim. 6(4), 1025–1039 (1996)MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Cartis, C., Gould, N.I.M., Toint, Ph.L.: Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity, 2007Google Scholar
  3. 3.
    Conn A.R., Gould N.I.M., Toint Ph.L.: Convergence of quasi-Newton matrices generated by the symmetric rank one update. Math. Program. 50(2), 177–196 (1991)MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Conn A.R., Gould N.I.M., Toint Ph.L.: Trust-Region Methods. SIAM, Philadelphia (2000)zbMATHCrossRefGoogle Scholar
  5. 5.
    Dembo R.S., Eisenstat S.C., Steihaug T.: Inexact-Newton methods. SIAM J. Numer. Anal. 19(2), 400–408 (1982)MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Dembo R.S., Steihaug T.: Truncated-Newton algorithms for large-scale unconstrained optimization. Math. Program. 26(2), 190–212 (1983)MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Dennis J.E., Moré J.J.: A characterization of superlinear convergence and its application to quasi-Newton methods. Math. Comput. 28(126), 549–560 (1974)zbMATHCrossRefGoogle Scholar
  8. 8.
    Dennis, J.E., Schnabel, R.B.: Numerical methods for unconstrained optimization and nonlinear equations. Prentice-Hall, Englewood Cliffs, New Jersey, USA, 1983. Reprinted as Classics in Applied Mathematics, vol. 16. SIAM, Philadelphia, USA (1996)Google Scholar
  9. 9.
    Deuflhard, P.: Newton Methods for Nonlinear Problems. Affine Invariance and Adaptive Algorithms. Springer Series in Computational Mathematics, vol. 35. Springer, Berlin (2004)Google Scholar
  10. 10.
    Dolan E.D., Moré J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91(2), 201–213 (2002)MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    Gerasoulis A.: A fast algorithm for the multiplication of generalized Hilbert matrices with vectors. Math. Comput. 50(181), 179–188 (1988)MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Gerasoulis A., Grigoriadis M.D., Sun L.: A fast algorithm for Trummer’s problem. SIAM J. Sci. Stat. Comput. 8(1), 135–138 (1987)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Golub G.H., Van Loan C.F.: Matrix Computations. The John Hopkins University Press, Baltimore (1996)zbMATHGoogle Scholar
  14. 14.
    Gould N.I.M., Lucidi S., Roma M., Toint Ph.L.: Solving the trust-region subproblem using the Lanczos method. SIAM J. Optim. 9(2), 504–525 (1999)MathSciNetzbMATHCrossRefGoogle Scholar
  15. 15.
    Gould N.I.M., Orban D., Sartenaer A., Toint Ph.L.: Sensitivity of trust-region algorithms on their parameters. 4OR Q. J. Ital. French Belg. Soc. 3(3), 227–241 (2005)MathSciNetzbMATHGoogle Scholar
  16. 16.
    Gould N.I.M., Orban D., Toint Ph.L.: CUTEr (and SifDec), a constrained and unconstrained testing environment, revisited. ACM Trans. Math. Softw. 29(4), 373–394 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  17. 17.
    Griewank, A.: The modification of Newton’s method for unconstrained optimization by bounding cubic terms. Technical Report NA/12 (1981). Department of Applied Mathematics and Theoretical Physics, University of Cambridge (1981)Google Scholar
  18. 18.
    Griewank A.: The “global” convergence of Broyden-like methods with a suitable line search. J. Aust. Math. Soc. Ser. B 28, 75–92 (1986)MathSciNetzbMATHCrossRefGoogle Scholar
  19. 19.
    Griewank, A., Toint, Ph.L.: Numerical experiments with partially separable optimization problems. Numerical Analysis: Proceedings Dundee 1983. Lecture Notes in Mathematics vol. 1066, pp. 203–220. Springer, Heidelberg (1984)Google Scholar
  20. 20.
    Moré, J.J.: Recent developments in algorithms and software for trust region methods. In: Bachem, A., Grötschel, M., Korte, B. Mathematical Programming: The State of the Art, pp. 258–287. Springer, Heidelberg (1983)Google Scholar
  21. 21.
    Nesterov Yu.: Introductory Lectures on Convex Optimization. Kluwer Academic Publishers, Dordrecht (2004)zbMATHGoogle Scholar
  22. 22.
    Nesterov Yu.: Accelerating the cubic regularization of Newton’s method on convex problems. Math. Program. 112(1), 159–181 (2008)MathSciNetzbMATHCrossRefGoogle Scholar
  23. 23.
    Nesterov Yu., Polyak B.T.: Cubic regularization of Newton’s method and its global performance. Math. Program. 108(1), 177–205 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  24. 24.
    Nocedal J., Wright S.J.: Numerical Optimization. Springer, New York (1999)zbMATHCrossRefGoogle Scholar
  25. 25.
    Thomas, S.W.: Sequential estimation techniques for quasi-Newton algorithms. Ph.D. Thesis, Cornell University, Ithaca (1975)Google Scholar
  26. 26.
    Weiser M., Deuflhard P., Erdmann B.: Affine conjugate adaptive Newton methods for nonlinear elastomechanics. Optim. Methods Softw. 22(3), 413–431 (2007)MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer and Mathematical Programming Society 2009

Authors and Affiliations

  • Coralia Cartis
    • 1
    • 2
  • Nicholas I. M. Gould
    • 2
    • 3
  • Philippe L. Toint
    • 4
  1. 1.School of MathematicsUniversity of EdinburghEdinburghScotland, UK
  2. 2.Computational Science and Engineering DepartmentRutherford Appleton LaboratoryChiltonUK
  3. 3.Numerical Analysis GroupOxford University Computing LaboratoryOxfordUK
  4. 4.Department of MathematicsFUNDP, University of NamurNamurBelgium

Personalised recommendations