Advertisement

Cubic regularization in symmetric rank-1 quasi-Newton methods

Full Length Paper
  • 53 Downloads

Abstract

Quasi-Newton methods based on the symmetric rank-one (SR1) update have been known to be fast and provide better approximations of the true Hessian than popular rank-two approaches, but these properties are guaranteed under certain conditions which frequently do not hold. Additionally, SR1 is plagued by the lack of guarantee of positive definiteness for the Hessian estimate. In this paper, we propose cubic regularization as a remedy to relax the conditions on the proofs of convergence for both speed and accuracy and to provide a positive definite approximation at each step. We show that the n-step convergence property for strictly convex quadratic programs is retained by the proposed approach. Extensive numerical results on unconstrained problems from the CUTEr test set are provided to demonstrate the computational efficiency and robustness of the approach.

Keywords

Interior-point methods Nonlinear programming Cubic regularization Newton’s method 

Mathematics Subject Classification

90C30 90C53 90–08 

Notes

Acknowledgements

We would like to thank Daniel Bienstock and Andreas Waechter for their handling of the paper as Editor and Associate Editor, respectively, for MPC. We would also like to thank the two anonymous referees whose feedback and suggestions have greatly improved the paper.

References

  1. 1.
    Anandkumar, A., Ge, R.: Efficient approaches for escaping higher order saddle points in non-convex optimization. (2016). arXiv preprint arXiv:1602.05908
  2. 2.
    Bellavia, S., Morini, B.: Strong local convergence properties of adaptive regularized methods for nonlinear least squares. IMA J. Numer. Anal. 35, dru021 (2014)MathSciNetGoogle Scholar
  3. 3.
    Benson, H.Y., Shanno, D.F.: Interior-point methods for nonconvex nonlinear programming: cubic regularization. Comput. Optim. Appl. 58, 323 (2014)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Bianconcini, T., Liuzzi, G., Morini, B., Sciandrone, M.: On the use of iterative methods in cubic regularization for unconstrained optimization. Comput. Optim. Appl. 60(1), 35–57 (2015)MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Bianconcini, T., Sciandrone, M.: A cubic regularization algorithm for unconstrained optimization using line search and nonmonotone techniques. Optim. Methods Softw. 1–28 (2016)Google Scholar
  6. 6.
    Birgin, E.G., Gardenghi, J.L., Martınez, J.M., Santos, S.A., Toint, Ph. L.: Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models. Report naXys-05-2015, University of Namur, Belgium (2015)Google Scholar
  7. 7.
    Broyden, C.G.: Quasi-newton methods and their application to function minimisation. Math. Comput. 21(99), 368–381 (1967)MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Broyden, G.C.: The convergence of a class of double-rank minimization algorithms 2. the new algorithm. IMA J. Appl. Math. 6(3), 222–231 (1970)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Byrd, R.H., Khalfan, H.F., Schnabel, R.B.: Analysis of a symmetric rank-one trust region method. SIAM J. Optim. 6, 1025–1039 (1996)MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Cartis, C., Gould, N.I.M., Toint, P.L.: Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results. Math. Program. Ser. A 127, 245–295 (2011)MathSciNetCrossRefMATHGoogle Scholar
  11. 11.
    Cartis, C., Gould, N.I.M., Toint, P.L.: Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity. Math. Program. Ser. A 130, 295–319 (2011)MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    Conn, A.R., Gould, N.I.M., Toint, P.L.: Convergence of quasi-newton matrices generated by the symmetric rank one update. Math. Program. 50(1–3), 177–195 (1991)MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    Conn, A.R., Gould, N., Toint, Ph.L.: Constrained and unconstrained testing environment. http://www.cuter.rl.ac.uk/Problems/mastsif.shtml. Accessed 01 Feb 2018
  14. 14.
    Conn, A.R., Gould, N.I.M., Toint, P.L.: Convergence of quasi-newton matrices generated by the symmetric rank one update. Math. Program. 50, 177–195 (1991)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Davidon, W.C.: Variance algorithm for minimization. Comput. J. 10(4), 406–410 (1968)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Davidon, W.C.: Variable metric method for minimization. SIAM J. Optim. 1(1), 1–17 (1991)MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Technical report, Argonne National Laboratory (2001)Google Scholar
  18. 18.
    Dussault, Jean-Pierre: Simple unified convergence proofs for the trust-region and a new arc variant. Technical report, Technical report, University of Sherbrooke, Sherbrooke, Canada (2015)Google Scholar
  19. 19.
    Feldman, S.I.: A fortran to c converter. In: ACM SIGPLAN Fortran Forum, vol. 9, pp. 21–22. ACM (1990)Google Scholar
  20. 20.
    Fiacco, A.V., McCormick, G.P.: Nonlinear programming: sequential unconstrained minimization techniques. Research Analysis Corporation, McLean Virginia. Republished in 1990 by SIAM, Philadelphia (1968)Google Scholar
  21. 21.
    Fletcher, R.: Practical Methods of Optimization. Wiley, Chichester (1987)MATHGoogle Scholar
  22. 22.
    Fletcher, R.: A new approach to variable metric algorithms. Comput. J. 13(3), 317–322 (1970)CrossRefMATHGoogle Scholar
  23. 23.
    Fletcher, R., Powell, M.J.D.: A rapidly convergent descent method for minimization. Comput. J. 6(2), 163–168 (1963)MathSciNetCrossRefMATHGoogle Scholar
  24. 24.
    Fourer, R., Gay, D.M., Kernighan, B.W.: AMPL: A Modeling Language for Mathematical Programming. Scientific Press, London (1993)MATHGoogle Scholar
  25. 25.
    Goldfarb, D.: Sufficient conditions for the convergence of a variable metric algorithm. In: Fletcher, R. (ed.) Optimization, pp. 273–281. Academic Press, New York (1969)Google Scholar
  26. 26.
    Goldfarb, D.: A family of variable-metric methods derived by variational means. Mathematics of computation 24(109), 23–26 (1970)MathSciNetCrossRefMATHGoogle Scholar
  27. 27.
    Göllner, T., Hess, W., Ulbrich, S.: Geometry optimization of branched sheet metal products. PAMM 12(1), 619–620 (2012)CrossRefGoogle Scholar
  28. 28.
    Gould, N.I.M., Porcelli, M., Toint, P.L.: Updating the regularization parameter in the adaptive cubic regularization algorithm. Comput. Optim. Appl. 53(1), 1–22 (2012)MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Griewank, A.: The modification of Newton’s method for unconstrained optimization by bounding cubic terms. Technical Report NA/12, Department of Applied Mathematics and Theoretical Physics, University of Cambridge (1981)Google Scholar
  30. 30.
    Griewank, A., Fischer, J., Bosse, T.: Cubic overestimation and secant updating for unconstrained optimization of c 2, 1 functions. Optim. Methods Softw. 29(5), 1075–1089 (2014)MathSciNetCrossRefMATHGoogle Scholar
  31. 31.
    Hsia, Y., Sheu, R.-L., Yuan, Y.-X.: On the p-regularized trust region subproblem (2014). arXiv preprint arXiv:1409.4665
  32. 32.
    Huang, H.Y.: Unified approach to quadratically convergent algorithms for function minimization. J. Optim. Theory Appl. 5(6), 405–423 (1970)MathSciNetCrossRefMATHGoogle Scholar
  33. 33.
    Khalfan, H.F., Byrd, R.H., Schnabel, R.B.: A theoretical and experimental study of the symmetric rank-one update. SIAM J. Optim. 3(1), 1–24 (1993)MathSciNetCrossRefMATHGoogle Scholar
  34. 34.
    Liu, X., Sun, J.: Global convergence analysis of line search interior-point methods for nonlinear programming without regularity assumptions. J. Optim. Theory Appl. 125(3), 609–628 (2005)MathSciNetCrossRefMATHGoogle Scholar
  35. 35.
    Sha, L., Wei, Z., Li, L.: A trust region algorithm with adaptive cubic regularization methods for nonsmooth convex minimization. Comput. Optim. Appl. 51(2), 551–573 (2012)MathSciNetCrossRefMATHGoogle Scholar
  36. 36.
    Martınez, J.M., Raydan, M.: Cubic-regularization counterpart of a variable-norm trust-region method for unconstrained minimization. Technical report (2015)Google Scholar
  37. 37.
    Murtagh, B.A., Sargent, R.W.H.: A constrained minimization method with quadratic convergence. Optimization, pp. 215–246 (1969)Google Scholar
  38. 38.
    Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Math. Program. Ser. A 108, 177–205 (2006)MathSciNetCrossRefMATHGoogle Scholar
  39. 39.
    Nocedal, J., Wright, S.J.: Numerical Optimization. Springer Series in Operations Research. Springer, Berlin (1999)Google Scholar
  40. 40.
    Oren, S.S., Luenberger, D.G.: Self-scaling variable metric (ssvm) algorithms. Manag. Sci. 20(5), 863–874 (1974)CrossRefMATHGoogle Scholar
  41. 41.
    Oren, S.S., Luenberger, D.G.: Self-scaling variable metric (ssvm) algorithms: Part i: Criteria and sufficient conditions for scaling a class of algorithms. Manag. Sci. 20(5), 845–862 (1974)CrossRefMATHGoogle Scholar
  42. 42.
    Oren, S.S., Spedicato, E.: Optimal conditioning of self-scaling variable metric algorithms. Math. Program. 10(1), 70–90 (1976)MathSciNetCrossRefMATHGoogle Scholar
  43. 43.
    Powell, M.J.D.: Recent advances in unconstrained optimization. Math. Program. 1, 26–57 (1971)MathSciNetCrossRefMATHGoogle Scholar
  44. 44.
    Schiela, A.: A flexible framework for cubic regularization algorithms for non-convex optimization in function space. Technical report (2014)Google Scholar
  45. 45.
    Shanno, D.D., Phua, K.H.: Remark on algorithm 500. minimization of unconstrained multivariate functions. Trans. Math. Softw. 6(4), 618–622 (1980)CrossRefGoogle Scholar
  46. 46.
    Shanno, D.F.: Conditioning of quasi-newton methods for function minimization. Math. Comput. 24(111), 647–656 (1970)MathSciNetCrossRefMATHGoogle Scholar
  47. 47.
    Shanno, D.F.: Conjugate gradient methods with inexact searches. Math. Oper. Res. 3(3), 244–256 (1978)MathSciNetCrossRefMATHGoogle Scholar
  48. 48.
    Shanno, D.F., Phua, K.H.: Matrix conditioning and nonlinear optimization. Math. Program. 14(1), 149–160 (1978)MathSciNetCrossRefMATHGoogle Scholar
  49. 49.
    Sherman, J., Morrison, W.J.: Adjustment of an inverse matrix corresponding to changes in the elements of a given column or a given row of the original matrix. In: Annals of Mathematical Statistics, volume 20(4), pp. 621–621. INST MATHEMATICAL STATISTICS IMS BUSINESS OFFICE-SUITE 7, 3401 INVESTMENT BLVD, HAYWARD, CA 94545 (1949)Google Scholar
  50. 50.
    Vanderbei, R.J.: AMPL models. http://orfe.princeton.edu/~rvdb/ampl/nlmodels. Accessed 01 August 2016
  51. 51.
    Vanderbei, R.J., Shanno, D.F.: An interior-point algorithm for nonconvex nonlinear programming. Comput. Optim. Appl. 13, 231–252 (1999)MathSciNetCrossRefMATHGoogle Scholar
  52. 52.
    Weiser, M., Deuflhard, P., Erdmann, B.: Affine conjugate adaptive Newton methods for nonlinear elastomechanics. Optim. Methods Softw. 22(3), 413–431 (2007)MathSciNetCrossRefMATHGoogle Scholar
  53. 53.
    Wolfe, P.: Another variable metric method. Technical report, working paper (1967)Google Scholar
  54. 54.
    Wolfe, P.: Convergence conditions for ascent methods. SIAM Rev. 11(2), 226–235 (1969)MathSciNetCrossRefMATHGoogle Scholar
  55. 55.
    Wolfe, P.: Convergence conditions for ascent methods. ii: Some corrections. SIAM Rev. 13(2), 185–188 (1971)MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature and The Mathematical Programming Society 2018

Authors and Affiliations

  1. 1.Drexel UniversityPhiladelphiaUSA
  2. 2.EmeritusRutgers UniversityNew BrunswickUSA

Personalised recommendations