Skip to main content
SpringerLink
Log in

An adaptive high order method for finding third-order critical points of nonconvex optimization

  • Published:
Journal of Global Optimization Aims and scope Submit manuscript

Abstract

Recently, the optimization methods for computing higher-order critical points of nonconvex problems attract growing research interest (Anandkumar Conference on Learning Theory 81-102, 2016), (Cartis Found Comput Math 18:1073-1107, 2018), (Cartis SIAM J Optim 30:513-541, 2020), (Chen Math Program 187:47-78, 2021) , as they are able to exclude the so-called degenerate saddle points and reach a solution with better quality. Despite theoretical developments in (Anandkumar Conference on Learning Theory 81-102, 2016), (Cartis Found Comput Math 18:1073-1107, 2018), (Cartis SIAM J Optim 30:513-541, 2020), (Chen Math Program 187:47-78, 2021) , the corresponding numerical experiments are missing. This paper proposes an implementable higher-order method, named adaptive high order method (AHOM), to find the third-order critical points. AHOM is achieved by solving an “easier” subproblem and incorporating the adaptive strategy of parameter-tuning in each iteration of the algorithm. The iteration complexity of the proposed method is established. Some preliminary numerical results are provided to show that AHOM can escape from the degenerate saddle points, where the second-order method could possibly get stuck.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. https://github.com/dalab/subsampled cubic regularization.

References

  1. Amaral, V., Andreani, R., Birgin, E., Marcondes, D., Martínez, J.: On complexity and convergence of high-order coordinate descent algorithms for smooth nonconvex box-constrained minimization. (2020) arXiv preprint arXiv:2009.01811

  2. Anandkumar, A., Ge, R.: Efficient approaches for escaping higher order saddle points in non-convex optimization. In: Conference on Learning Theory, pp 81–102 (2016)

  3. Baes, M.: Estimate sequence methods: extensions and approximations. Institute for Operations Research, ETH, Z\(^{..1}\)rich, Switzerland (2009)

  4. Bellavia, S., Gurioli, G., Morini, B., Toint, P.L.: Adaptive regularization algorithms with inexact evaluations for nonconvex optimization. SIAM J. Optim. 29(4), 2881–2915 (2019)

    Article  MathSciNet  Google Scholar 

  5. Birgin, E., Krejić, N., Martínez, J.: Economic inexact restoration for derivative-free expensive function minimization and applications. (2020) arXiv preprint arXiv:2009.09062

  6. Birgin, E.G., Gardenghi, J., Martínez, J.M., Santos, S.A.: On the use of third-order models with fourth-order regularization for unconstrained optimization. Optim. Lett. 14, 815–838 (2020)

    Article  MathSciNet  Google Scholar 

  7. Birgin, E.G., Gardenghi, J., Martinez, J.M., Santos, S.A., Toint, P.L.: Evaluation complexity for nonlinear constrained optimization using unscaled KKT conditions and high-order models. SIAM J. Optim. 26(2), 951–967 (2016)

    Article  MathSciNet  Google Scholar 

  8. Birgin, E.G., Gardenghi, J., Martínez, J.M., Santos, S.A., Toint, P.L.: Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models. Math. Prog. 163(1–2), 359–368 (2017)

    Article  MathSciNet  Google Scholar 

  9. Bubeck, S., Jiang, Q., Lee, Y.T., Li, Y., Sidford, A.: Near-optimal method for highly smooth convex optimization. In: Conference on Learning Theory, pp. 492–507. PMLR (2019)

  10. Carmon, Y., Duchi, J.C., Hinder, O., Sidford, A.: Lower bounds for finding stationary points I. Math. Prog. 184, 1–50 (2017)

    MathSciNet  MATH  Google Scholar 

  11. Carmon, Y., Duchi, J.C., Hinder, O., Sidford, A.: Accelerated methods for nonconvex optimization. SIAM J. Optim. 28(2), 1751–1772 (2018)

    Article  MathSciNet  Google Scholar 

  12. Cartis, C., Gould, N.I., Toint, P.L.: Adaptive cubic regularisation methods for unconstrained optimization. part I: motivation, convergence and numerical results. Math. Program. 127(2), 245–295 (2011)

    Article  MathSciNet  Google Scholar 

  13. Cartis, C., Gould, N.I., Toint, P.L.: Adaptive cubic regularisation methods for unconstrained optimization. part II: worst-case function-and derivative-evaluation complexity. Math. Program. 130(2), 295–319 (2011)

    Article  MathSciNet  Google Scholar 

  14. Cartis, C., Gould, N.I., Toint, P.L.: Second-order optimality and beyond: characterization and evaluation complexity in convexly constrained nonlinear optimization. Found. Comput. Math. 18(5), 1073–1107 (2018)

    Article  MathSciNet  Google Scholar 

  15. Cartis, C., Gould, N.I., Toint, P.L.: A concise second-order complexity analysis for unconstrained optimization using high-order regularized models. Opt. Methods Softw. 35(2), 243–256 (2020)

    Article  MathSciNet  Google Scholar 

  16. Cartis, C., Gould, N.I., Toint, P.L.: Sharp worst-case evaluation complexity bounds for arbitrary-order nonconvex optimization with inexpensive constraints. SIAM J. Optim. 30(1), 513–541 (2020)

    Article  MathSciNet  Google Scholar 

  17. Chen, X., Jiang, B., Lin, T., Zhang, S.: Accelerating adaptive cubic regularization of Newton’s method via random sampling. J. Mach. Learn. Res. (2022)

  18. Chen, X., Toint, P.L.: High-order evaluation complexity for convexly-constrained optimization with non-lipschitzian group sparsity terms. Math. Program. 187(1), 47–78 (2021)

    Article  MathSciNet  Google Scholar 

  19. Chen, X., Toint, P.L., Wang, H.: Complexity of partially separable convexly constrained optimization with Non-Lipschitzian singularities. SIAM J. Optim. 29(1), 874–903 (2019)

    Article  MathSciNet  Google Scholar 

  20. Curtis, F.E., Robinson, D.P., Samadi, M.: An inexact regularized Newton framework with a worst-case iteration complexity of for nonconvex optimization. IMA J. Numer. Anal. 39(3), 1296–1327 (2018)

    Article  MathSciNet  Google Scholar 

  21. Cushing, J.M.: Extremal tests for scalar functions of several real variables at degenerate critical points. Aequationes Math. 13(1–2), 89–96 (1975)

    Article  MathSciNet  Google Scholar 

  22. Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12, 2121–2159 (2011)

    MathSciNet  MATH  Google Scholar 

  23. Gasnikov, A., Kovalev, D., Mohhamed, A., Chernousova, E.: The global rate of convergence for optimal tensor methods in smooth convex optimization. (2018) arXiv preprint arXiv:1809.00382

  24. Ghadimi, S., Lan, G., Zhang, H.: Generalized uniformly optimal methods for nonlinear programming. J. Sci. Comput. 79(3), 1854–1881 (2019)

    Article  MathSciNet  Google Scholar 

  25. Gould, N.I., Lucidi, S., Roma, M., Toint, P.L.: Solving the trust-region subproblem using the lanczos method. SIAM J. Optim. 9(2), 504–525 (1999)

    Article  MathSciNet  Google Scholar 

  26. Gould, N.I., Rees, T., Scott, J.A.: Convergence and evaluation-complexity analysis of a regularized tensor-Newton method for solving nonlinear least-squares problems. Comput. Optim. Appl. 73(1), 1–35 (2019)

    Article  MathSciNet  Google Scholar 

  27. Grapiglia, G.N., Nesterov, Y.: Tensor methods for finding approximate stationary points of convex functions. Optim. Methods Softw. pp. 1–34 (2020)

  28. Grapiglia, G.N., Nesterov, Y.: Tensor methods for minimizing convex functions with hölder continuous higher-order derivatives. SIAM J. Optim. 30(4), 2750–2779 (2020)

    Article  MathSciNet  Google Scholar 

  29. Gratton, S., Simon, E., Toint, P.L.: An algorithm for the minimization of nonsmooth nonconvex functions using inexact evaluations and its worst-case complexity. Math. Program. 187(1), 1–24 (2021)

    Article  MathSciNet  Google Scholar 

  30. Jiang, B., Lin, T., Ma, S., Zhang, S.: Structured nonconvex and nonsmooth optimization: algorithms and iteration complexity analysis. Comput. Optim. Appl. 72, 115–157 (2019)

    Article  MathSciNet  Google Scholar 

  31. Jiang, B., Lin, T., Zhang, S.: A unified adaptive tensor approximation scheme to accelerate composite convex optimization. SIAM J. Optim. 30, 2897–2926 (2020)

    Article  MathSciNet  Google Scholar 

  32. Jiang, B., Wang, H., Zhang, S.: An optimal high-order tensor method for convex optimization. Math. Oper. Res. 46, 1390–1412 (2021)

    Article  MathSciNet  Google Scholar 

  33. Lucchi, A., Kohler, J.: A stochastic tensor method for non-convex optimization. (2019) arXiv preprint arXiv:1911.10367

  34. Martínez, J.M.: On high-order model regularization for constrained optimization. SIAM J. Optim. 27(4), 2447–2458 (2017)

    Article  MathSciNet  Google Scholar 

  35. Mason, L., Baxter, J., Bartlett, P., Frean, M.: Boosting algorithms as gradient descent. In: Solla S., Leen T., Müller K. (eds.) Advances in neural information processing systems, vol 12, pp. 512–518 (1999)

  36. Motzkin, T.S.: The arithmetic-geometric inequality. Inequalities (Proc. Sympos. Wright-Patterson Air Force Base, Ohio, 1965) pp. 205–224 (1967)

  37. Murty, K., Kabadi, S.: Some np-complete problems in quadratic and nonlinear programming. Math. Program. 39, 117–129 (1987)

    Article  MathSciNet  Google Scholar 

  38. Nesterov, Y.: Introductory lectures on convex optimization a basic course. Appl. Optim. 87(5), 236 (2004)

    MATH  Google Scholar 

  39. Nesterov, Y.: Accelerating the cubic regularization of Newton’s method on convex problems. Math. Program. 112(1), 159–181 (2008)

    Article  MathSciNet  Google Scholar 

  40. Nesterov, Y.: Implementable tensor methods in unconstrained convex optimization. Math. Program. 186(1), 157–183 (2021)

    Article  MathSciNet  Google Scholar 

  41. Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Math. Program. 108(1), 177–205 (2006)

    Article  MathSciNet  Google Scholar 

  42. Nie, J.: The hierarchy of local minimums in polynomial optimization. Math. Program. 151(2), 555–583 (2015)

    Article  MathSciNet  Google Scholar 

  43. Udell, M., Boyd, S.: Maximizing a sum of sigmoids. Optim. Eng.pp. 1–25 (2013)

  44. Zheng, Y., Zheng, B.: A modified adaptive cubic regularization method for large-scale unconstrained optimization problem. (2019) arXiv preprint arXiv:1904.07440

Download references

Acknowledgements

We would like to thank the two anonymous referees for their insightful comments, and we would like to also thank Professor Qi Deng at Shanghai University of Finance and Economics for the discussion on the numerical experiment of this paper.

Author information

Authors and Affiliations

  1. School of Information Management and Engineering, Shanghai University of Finance and Economics, Shanghai, 200433, People’s Republic of China

    Xihua Zhu

  2. UBC Sauder School of Business, the University of British Columbia, Vancouver, BC, V6T 1Z2, Canada

    Jiangze Han

  3. Research Institute for Interdisciplinary Sciences, School of Information Management and Engineering, Shanghai University of Finance and Economics, Shanghai, 200433, People’s Republic of China

    Bo Jiang

Authors
  1. Xihua Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiangze Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bo Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Research supported by NSFC Grants 72171141, NSFC Grants 71971132, NSFC Grants 72150001, NSFC Grants 11831002, GIFSUFE Grants CXJJ-2019-391, and Program for Innovative Research Team of Shanghai University of Finance and Economics.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, X., Han, J. & Jiang, B. An adaptive high order method for finding third-order critical points of nonconvex optimization. J Glob Optim 84, 369–392 (2022). https://doi.org/10.1007/s10898-022-01151-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10898-022-01151-1

Keywords

Mathematics Subject Classification

Search

Navigation