Skip to main content

A matrix-free trust-region newton algorithm for convex-constrained optimization

Abstract

We describe a matrix-free trust-region algorithm for solving convex-constrained optimization problems that uses the spectral projected gradient method to compute trial steps. To project onto the intersection of the feasible set and the trust region, we reformulate and solve the dual projection problem as a one-dimensional root finding problem. We demonstrate our algorithm’s performance on various problems from data science and PDE-constrained optimization. Our algorithm shows superior performance when compared with five existing trust-region and spectral projected gradient methods, and has the added benefit that it is simple to implement.

This is a preview of subscription content, access via your institution.

References

  1. 1.

    Andretta, M., Birgin, E.G., Martínez, J.M.: Practical active-set Euclidian trust-region method with spectral projected gradients for bound-constrained minimization. Optimization 54(3), 305–325 (2005)

    MathSciNet  Article  Google Scholar 

  2. 2.

    Antil, H., Kouri, D.P., Lacasse, M.D., Ridzal, D.: Frontiers in PDE-constrained optimization, vol. 163. Springer, Cham (2018)

    Book  Google Scholar 

  3. 3.

    Baker, C.G., Heroux, M.A.: Tpetra, and the use of generic programming in scientific computing. Sci. Program. 20(2), 115–128 (2012)

    Google Scholar 

  4. 4.

    Bendsoe, M.P., Sigmund, O.: Topology optimization: theory, methods, and applications. Springer, Cham (2013)

    MATH  Google Scholar 

  5. 5.

    Birgin, E.G., Martínez, J.M., Raydan, M.: Nonmonotone spectral projected gradient methods on convex sets. SIAM J. Opt. 10(4), 1196–1211 (2000)

    MathSciNet  Article  Google Scholar 

  6. 6.

    Birgin, E.G., Martínez, J.M., Raydan, M.: Inexact spectral projected gradient methods on convex sets. IMA J. Numer. Anal. 23(4), 539–559 (2003)

    MathSciNet  Article  Google Scholar 

  7. 7.

    Birgin, E.G., Martínez, J.M., Raydan, M.: Spectral projected gradient methods: review and perspectives. J. Stat. Softw 60(3), 1–21 (2014)

    Article  Google Scholar 

  8. 8.

    Bochev, P., Edwards, H.C., Kirby, R.C., Peterson, K., Ridzal, D.: Solving PDEs with intrepid. Sci. Program. 20(2), 151–180 (2012)

    Google Scholar 

  9. 9.

    Borrvall, T., Petersson, J.: Topology optimization of fluids in stokes flow. Int. J. Numer. Methods Fluids 41(1), 77–107 (2003)

    MathSciNet  Article  Google Scholar 

  10. 10.

    Brent, R.P.: Algorithms for minimization without derivatives. Courier Corporation (2013)

  11. 11.

    Burke, J.V., Moré, J.J., Toraldo, G.: Convergence properties of trust region methods for linear and convex constraints. Math. Program. 47(1–3), 305–336 (1990)

    MathSciNet  Article  Google Scholar 

  12. 12.

    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(3), 1–27 (2011)

    Article  Google Scholar 

  13. 13.

    Conn, A.R., Gould, N.I.M., Sartenaer, A., Toint, P.L.: Global convergence of a class of trust region algorithms for optimization using inexact projections on convex constraints. SIAM J. Opt. 3(1), 164–221 (1993). https://doi.org/10.1137/0803009

    MathSciNet  Article  MATH  Google Scholar 

  14. 14.

    Conn, A.R., Gould, N.I.M., Sartenaer, A., Toint, P.L.: Convergence properties of minimization algorithms for convex constraints using a structured trust region. SIAM J. Opt. 6(4), 1059–1086 (1996). https://doi.org/10.1137/S1052623492236481

    MathSciNet  Article  MATH  Google Scholar 

  15. 15.

    Conn, A.R., Gould, N.I.M., Toint, P.L.: A globally convergent augmented Lagrangian algorithm for optimization with general constraints and simple bounds. SIAM J. Numer. Anal. 28(2), 545–572 (1991). https://doi.org/10.1137/0728030

    MathSciNet  Article  MATH  Google Scholar 

  16. 16.

    Dai, Y.H., Fletcher, R.: New algorithms for singly linearly constrained quadratic programs subject to lower and upper bounds. Math. Program. 106(3), 403–421 (2006)

    MathSciNet  Article  Google Scholar 

  17. 17.

    Deng, Y., Liu, Z., Wu, J., Wu, Y.: Topology optimization of steady Navier-Stokes flow with body force. Comput. Methods Appl. Mech. Eng. 255, 306–321 (2013)

    MathSciNet  Article  Google Scholar 

  18. 18.

    Dua, D., Graff, C.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml

  19. 19.

    Hartung, J.: An extension of Sion’s minimax theorem with an application to a method for constrained games. Pacific J. Math. 103(2), 401–408 (1982)

    MathSciNet  Article  Google Scholar 

  20. 20.

    Heinkenschloss, M.: Numerical solution of implicitly constrained optimization problems. Rice University, Tech. rep. (2008)

  21. 21.

    Kouri, D.P., von Winckel, G., Ridzal, D.: ROL: Rapid Optimization Library. https://trilinos.org/packages/rol (2017)

  22. 22.

    Lazarov, B.S., Sigmund, O.: Filters in topology optimization based on Helmholtz-type differential equations. Int. J. Numer. Methods Eng. 86(6), 765–781 (2011)

    MathSciNet  Article  Google Scholar 

  23. 23.

    Lin, C.J., Moré, J.J.: Newton’s method for large bound-constrained optimization problems. SIAM J. Opt. 9(4), 1100–1127 (1999)

    MathSciNet  Article  Google Scholar 

  24. 24.

    Maciel, M.C., Mendonça, M.G., Verdiell, A.B.: Monotone and nonmonotone trust-region-based algorithms for large scale unconstrained optimization problems. Comput. Opt. Appl. 54(1), 27–43 (2013)

    MathSciNet  Article  Google Scholar 

  25. 25.

    Meyer, M., Vlachos, P.: StatLib—datasets archive. http://lib.stat.cmu.edu/datasets/. Accessed: 2021-05-10

  26. 26.

    Pace, R.K., Barry, R.: Sparse spatial autoregressions. Stat. Probab. Lett. 33(3), 291–297 (1997)

    Article  Google Scholar 

  27. 27.

    Sala, M., Stanley, K., Heroux, M.: Amesos: A set of general interfaces to sparse direct solver libraries. In: Proceedings of PARA’06 Conference, Umea, Sweden (2006)

  28. 28.

    Schenk, O., Gärtner, K.: Solving unsymmetric sparse systems of linear equations with PARDISO. Future Gener. Comput. Syst. 20(3), 475–487 (2004)

    Article  Google Scholar 

  29. 29.

    Schmidt, M., Berg, E., Friedlander, M., Murphy, K.: Optimizing costly functions with simple constraints: A limited-memory projected quasi-Newton algorithm. In: D. van Dyk, M. Welling (eds.) Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, vol. 5, pp. 456–463. PMLR, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA (2009)

  30. 30.

    Toint, P.L.: Global convergence of a clas of trust-region methods for nonconvex minimization in Hilbert space. IMA J. Numer. Anal. 8(2), 231–252 (1988). https://doi.org/10.1093/imanum/8.2.231

    MathSciNet  Article  MATH  Google Scholar 

  31. 31.

    Tröltzsch, F.: Optimal control of partial differential equations: theory, methods, and applications. Graduate studies in mathematics. American Mathematical Society (2010)

  32. 32.

    Vapnik, V.N.: The nature of statistical learning theory. Springer, New York (2013)

    MATH  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to D. P. Kouri.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This research was sponsored by the U.S. Air Force Office of Scientific Research, Optimization Program under Award NO: F4FGA09135G001.

Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC., a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kouri, D.P. A matrix-free trust-region newton algorithm for convex-constrained optimization. Optim Lett (2021). https://doi.org/10.1007/s11590-021-01794-1

Download citation

Keywords

  • Nonconvex optimization
  • Convex constraints
  • Trust regions
  • Spectral projected gradient
  • Large-scale optimization
  • Newton’s method