Skip to main content
SpringerLink
Log in

On solving trust-region and other regularised subproblems in optimization

  • Full Length Paper
  • Published:
Mathematical Programming Computation Aims and scope Submit manuscript

Abstract

The solution of trust-region and regularisation subproblems that arise in unconstrained optimization is considered. Building on the pioneering work of Gay, Moré and Sorensen, methods that obtain the solution of a sequence of parametrized linear systems by factorization are used. Enhancements using high-order polynomial approximation and inverse iteration ensure that the resulting method is both globally and asymptotically at least superlinearly convergent in all cases, including the notorious hard case. Numerical experiments validate the effectiveness of our approach. The resulting software is available as packages \({\tt TRS}\) and \({\tt RQS}\) as part of the GALAHAD optimization library, and is especially designed for large-scale problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Absil P.-A., Baker C.G., Gallivan K.A.: Trust-region methods on Riemannian manifolds. Found. Comput. Math. 7(3), 303–330 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  2. Absil P.-A., Mahony R., Sepulchre R.: Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton (2008)

    MATH  Google Scholar 

  3. Amestoy P., Duff I.S., Pralet S., Voemel C.: Adapting a parallel sparse direct solver to SMP architectures. Parallel Comput. 29(11–12), 1645–1668 (2003)

    Article  MathSciNet  Google Scholar 

  4. Apostol T.M.: Mathematical Analysis. 2nd edn. Addison-Wesley, Reading (1974)

    MATH  Google Scholar 

  5. Berkes P., Wiskott L.: Analysis and interpretation of quadratic models of receptive fields. Nat. Protoc. 2(2), 400–407 (2007)

    Article  Google Scholar 

  6. Busygin S., Ag C., Butenko S., Pardalos P.M.: A heuristic for the maximum independent set problem based on optimization of a quadratic over a sphere. J. Comb. Optim. 6(3), 287–297 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  7. Cartis, C., Gould, N.I.M., Toint, Ph.L.: Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results. Math. Program. Ser. A 51 pages (2009) doi:10.1007/s10107-009-0286-5

  8. Cartis C., Gould N.I.M., Toint Ph.L.: Trust-region and other regularisations of linear least-squares problems. BIT 49(1), 21–53 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  9. Chabrillac Y., Crouzeix J.-P.: Definiteness and semidefiniteness of quadratic forms revisited. Linear Algebra Appl. 63, 283–292 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  10. Cline A.K., Moler C.B., Stewart G.W., Wilkinson J.H.: An estimate for the condition number of a matrix. SIAM J. Numer. Anal. 16(2), 368–375 (1979)

    Article  MathSciNet  MATH  Google Scholar 

  11. Conn A.R., Gould N.I.M., Toint Ph.L.: Trust-Region Methods. SIAM, Philadelphia (2000)

    MATH  Google Scholar 

  12. Demmel J.W.: Applied Numerical Linear Algebra. SIAM, Philadelphia (1997)

    MATH  Google Scholar 

  13. Dolan E.D., Moré J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91(2), 201–213 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  14. Dollar, H.S.: On Taylor series approximations for trust-region and regularized subproblems in optimization. Internal Technical Report Internal-2009-1, Rutherford Appleton Laboratory, Chilton, Oxfordshire, England (2009)

  15. Dollar, H.S., Gould, N.I.M., Robinson, D.P.: On solving trust-region and other regularised subproblems in optimization. Technical Report RAL-TR-2009-003, Rutherford Appleton Laboratory (2009)

  16. Duff I.S.: MA57—a code for the solution of sparse symmetric definite and indefinite systems. ACM Trans. Math. Softw. 30(2), 118–144 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  17. Duff I.S., Reid J.K.: The multifrontal solution of indefinite sparse symmetric linear equations. ACM Trans. Math. Softw. 9(3), 302–325 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  18. Erway J.B., Gill P.E.: A subspace minimization method for the trust-region step. SIAM J. Optim. 20(3), 1439–1461 (2009)

    Article  MathSciNet  Google Scholar 

  19. Erway J.B., Gill P.E., Griffin J.D.: Iterative methods for finding a trust-region step. SIAM J. Optim. 20(2), 1110–1131 (2009)

    Article  MathSciNet  Google Scholar 

  20. Gander W.: On the Linear Least Squares Problem with a Quadratic Constraint. Technical Report STAN-CS-78-697. Computer Science Department, Stanford University, California (1978)

    Google Scholar 

  21. Gay D.M.: Computing optimal locally constrained steps. SIAM J. Sci. Stat. Comput. 2(2), 186–197 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  22. Gertz E.M., Gill P.E.: A primal-dual trust region algorithm for nonlinear optimization. Math. Program. Ser. B 100(1), 49–94 (2004)

    MathSciNet  MATH  Google Scholar 

  23. Gould N.I.M.: On practical conditions for the existence and uniqueness of solutions to the general equality quadratic-programming problem. Math. Program. 32(1), 90–99 (1985)

    Article  MATH  Google Scholar 

  24. Gould, N.I.M., Hu, Y., Scott, J.A.: A numerical evaluation of sparse direct solvers for the solution of large sparse symmetric linear systems of equations. ACM Trans. Math. Softw. 32(2), Article 10 (2007)

    Google Scholar 

  25. Gould N.I.M., Lucidi S., Roma M., Toint Ph.L.: Solving the trust-region subproblem using the Lanczos method. SIAM J. Optim. 9(2), 504–525 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  26. Gould N.I.M., Orban D., Toint Ph.L.: CUTEr (and SifDec), a Constrained and Unconstrained Testing Environment, revisited. ACM Trans. Math. Softw. 29(4), 373–394 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  27. Gould N.I.M., Orban D., Toint Ph.L.: GALAHAD—a library of thread-safe fortran 90 packages for large-scale nonlinear optimization. ACM Trans. Math. Softw. 29(4), 353–372 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  28. Gould N.I.M., Scott J.A.: A numerical evaluation of HSL packages for the direct solution of large sparse, symmetric linear systems of equations. ACM Trans. Math. Softw. 30(3), 300–325 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  29. Griewank A.: The Modification of Newton’s Method for Unconstrained Optimization by Bounding Cubic Terms. Technical Report DAMTP/NA12. Department of Applied Mathematics and Theoretical Physics, Cambridge University, Cambridge (1981)

    Google Scholar 

  30. Hager W.W.: Condition estimates. SIAM J. Sci. Stat. Comput. 5(2), 311–316 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  31. Hager W.W.: Minimizing a quadratic over a sphere. SIAM J. Optim. 12(1), 188–208 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  32. Hebden M.D.: An Algorithm for Minimization Using Exact Second Derivatives. Technical Report T.P. 515. AERE, Harwell Laboratory, Harwell (1973)

    Google Scholar 

  33. Higham N.J.: Fortran codes for estimating the one-norm of a real or complex matrix, with applications to condition estimation. ACM Trans. Math. Softw. 14(4), 381–396 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  34. Hogg J.D.: A DAG-Based Parallel Cholesky Factorization for Multicore Systems. Technical Report RAL-TR-2008-029. Rutherford Appleton Laboratory, Chilton (2008)

    Google Scholar 

  35. Hogg J.D., Reid J.K., Scott J.A.: A DAG-Based Sparse Cholesky Solver for Multicore Architectures. Technical Report RAL-TR-2009-004. Rutherford Appleton Laboratory, Chilton (2009)

    Google Scholar 

  36. Montero A.: Study of SU(3) vortex-like configurations with a new maximal center gauge fixing method. Phys. Lett. B467, 106–111 (1999)

    Google Scholar 

  37. Moré J.J.: Recent developments in algorithms and software for trust region methods. In: Bachem, A., Grötschel, M., Korte, B. (eds) Mathematical Programming: The State of the Art, pp. 258–287. Springer, Heidelberg (1983)

    Google Scholar 

  38. Moré J.J., Sorensen D.C.: Computing a trust region step. SIAM J. Sci. Stat. Comput. 4(3), 553–572 (1983)

    Article  MATH  Google Scholar 

  39. Nesterov Yu., Polyak B.T.: Cubic regularization of Newton method and its global performance. Math. Program. 108(1), 177–205 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  40. Parlett, B.N.: The Symmetric Eigenvalue Problem. Prentice-Hall, Englewood Cliffs, New Jersey, USA, 1980. Reprinted as Classics in Applied Mathematics 20, SIAM, Philadelphia, USA (1998)

  41. Poljack S., Wolkowicz H.: Convex relaxations of (0,1)–quadratic programming. Math. Oper. Res. 20(3), 550–561 (1995)

    Article  MathSciNet  Google Scholar 

  42. Reid J.K., Scott J.A.: An Out-of-Core Sparse Cholesky Solver. Technical Report RAL-TR-2006-013. Rutherford Appleton Laboratory, Chilton (2006)

    Google Scholar 

  43. Reinsch C.: Smoothing by spline functions II. Numerische Mathematik 16(5), 451–454 (1971)

    Article  MathSciNet  Google Scholar 

  44. Schenk O., Christen M., Burkhart H.: Algorithmic performance studies on graphics processing units. J. Parallel Distrib. Comput. 68, 1360–1369 (2008)

    Article  Google Scholar 

  45. Steihaug T.: The conjugate gradient method and trust regions in large scale optimization. SIAM J. Numer. Anal. 20(3), 626–637 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  46. Toint Ph.L.: Towards an efficient sparsity exploiting Newton method for minimization. In: Duff, I.S. (eds) Sparse Matrices and Their Uses, pp. 57–88. Academic Press, London (1981)

    Google Scholar 

  47. Traub J.F.: Iterative Methods for the Solution of Equations. Prentice-Hall, Englewood Cliffs (1964)

    MATH  Google Scholar 

  48. Trefethen L.N., Bai D.: Numerical Linear Algebra. SIAM, Philadelphia (1997)

    MATH  Google Scholar 

  49. Weiser M., Deuflhard P., Erdmann B.: Affine conjugate adaptive Newton methods for nonlinear elastomechanics. Optim. Methods Softw. 22(3), 413–431 (2007)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computational Science and Engineering Department, Rutherford Appleton Laboratory, Chilton, Oxfordshire, OX11 0QX, UK

    Nicholas I. M. Gould & H. Sue Thorne

  2. Numerical Analysis Group, Mathematical Institute, Oxford University, 24–29 St Giles’, Oxford, OX1 3LB, UK

    Daniel P. Robinson

Authors
  1. Nicholas I. M. Gould
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel P. Robinson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. H. Sue Thorne
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicholas I. M. Gould.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gould, N.I.M., Robinson, D.P. & Thorne, H.S. On solving trust-region and other regularised subproblems in optimization. Math. Prog. Comp. 2, 21–57 (2010). https://doi.org/10.1007/s12532-010-0011-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12532-010-0011-7

Keywords

Mathematics Subject Classification (2000)

Search

Navigation