Skip to main content

Recent advances in trust region algorithms

Mathematical Programming volume 151, pages 249–281 (2015)Cite this article

Abstract

Trust region methods are a class of numerical methods for optimization. Unlike line search type methods where a line search is carried out in each iteration, trust region methods compute a trial step by solving a trust region subproblem where a model function is minimized within a trust region. Due to the trust region constraint, nonconvex models can be used in trust region subproblems, and trust region algorithms can be applied to nonconvex and ill-conditioned problems. Normally it is easier to establish the global convergence of a trust region algorithm than that of its line search counterpart. In the paper, we review recent results on trust region methods for unconstrained optimization, constrained optimization, nonlinear equations and nonlinear least squares, nonsmooth optimization and optimization without derivatives. Results on trust region subproblems and regularization methods are also discussed.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

References

  1. Absil, P.-A., Baker, C.G., Gallivan, K.A.: Trust-region methods on Riemannian manifolds. Found. Comput. Math. 7, 303–330 (2007)

    MATH  MathSciNet  Google Scholar 

  2. Ai, W.B., Zhang, S.Z.: Strong duality for the CDT subproblem: a necessary and sufficient condition. SIAM J. Optim. 19, 1735–1756 (2009)

    MATH  MathSciNet  Google Scholar 

  3. Bandeira, A.S., Scheinberg, K., Vicente, L.N.: Computation of sparse low degree interpolating polynomials and their application to derivative-free optimization. Math. Program. 134, 223–257 (2012)

    MATH  MathSciNet  Google Scholar 

  4. Bandeira, A.S., Scheinberg, K., Vicente, L.N.: Convergence of trust region methods based on probabilistic models. SIAM J. Optim. 24, 1238–1264 (2014)

    MATH  MathSciNet  Google Scholar 

  5. Bastin, F., Malmedy, F., Mouffe, M., Toint, PhL, Tomanos, D.: A retrospective trust-region method for unconstrained optimization. Math. Program. 123, 395–418 (2010)

    MATH  MathSciNet  Google Scholar 

  6. Bienstock, D.: A Note on Polynomial Solvability of the CDT Problem. Technical Report, Columbia University, USA, arXiv:1406.6429v2 (2013)

  7. Billups, S.C., Larson, J., Graf, P.: Derivative-free optimization of expensive functions with computational error using weighted regression. SIAM J. Optim. 23, 27–53 (2013)

    MATH  MathSciNet  Google Scholar 

  8. Boumal, N., Absil, P.-A.: RTRMC: A Riemannian Trust-Region Method for Low-Rank Matrix Completion. Technical Report, ICTEAM Institute Universit e catholique de Louvain B-1348 Louvain-la-Neuve (2014) (http://www.optimization-online.org/DB_HTML/2012/09/3619.html)

  9. Burdakov, O., Gong, L.J., Yuan, Y., Zikrin, S.: On Efficiently Combining Limited Memory and Trust-Region Techniques. Techinical Report, Linkoping University, Sweden (2014)

  10. Burke, J.: Descent methods for composite nondifferentiable optimization problem. Math. Program. 33, 260–279 (1985)

    MATH  MathSciNet  Google Scholar 

  11. Byrd, R.H., Nocedal, J., Waltz, R.A.: An Integrated Package for Nonlinear Optimization. In: Di Pillo, G., Roma, M. (eds.) Large-scale nonlinear optimization, pp. 35–59. Springer, Berlin (2006)

    Google Scholar 

  12. Byrd, R., Schnabel, R.B., Shultz, G.A.: A trust region algorithm for nonlinear constrained optimization. SIAM J. Numer. Anal. 24, 1152–1170 (1987)

    MATH  MathSciNet  Google Scholar 

  13. Cartis, C., Gould, N.I.M., Toint, Ph.L.: On the evaluation complexity of composite function minimization with applications to nonconvex nonlinear programming. SIAM J. Optim. 21, 1721–1739 (2011)

  14. Cartis, C., Gould, N.I.M., Toint, Ph.L.: Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results. Math. Program. 127, 245–295 (2011)

  15. Cartis, C., Gould, N.I.M., Toint, Ph.L.: Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity. Math. Program. 130, 295–319 (2011)

  16. Cartis, C., Gould, N.I.M., Toint, Ph.L.: An adaptive cubic regularization algorithm for nonconvex optimization with convex constraints and its function-evaluation complexity. IMA J. Numer. Anal. 32(4), 1662–1695 (2012)

  17. Cartis, C., Gould, N.I.M., Toint, Ph.L.: On the Evaluation Complexity of Constrained Nonlinear Least-Squares and General Constrained Nonlinear Optimization Using Second-Order Methods. Report naXys-01-2013, Dept of Mathematics, FUNDP, Namur (B) (2013)

  18. Celis, M.R., Dennis, J.E., Tapia, R.A.: A trust region algorithm for nonlinear equality constrained optimization. In: Boggs, P.T., Byrd, R.H., Schnabel, R.B. (eds.) Numerical Optimization, pp. 71–82. SIAM, Philadelphia (1985)

    Google Scholar 

  19. Chen, X.D., Yuan, Y.: On local solutions of the CDT subproblem. SIAM J. Optim. 10, 359–383 (1999)

    MATH  MathSciNet  Google Scholar 

  20. Chen, X.J., Niu, L.F., Yuan, Y.: Optimality conditions and smoothing trust region Newton method for non-Lipschitz optimization. SIAM J. Optim. 23(3), 1528–1552 (2013)

    MATH  MathSciNet  Google Scholar 

  21. Clarke, F.H.: Optimization and Nonsmooth Analysis. SIAM, Philadelphia (1980)

    Google Scholar 

  22. Coleman, T.F., Li, Y.: On the convergence of interior-reflective Newton methods for nonlinear minimization subject to bounds. Math. Program. 67, 189–224 (1994)

    MATH  MathSciNet  Google Scholar 

  23. Coleman, T.F., Li, Y.: An interior trust region approach for nonlinear minimization subject to bounds. SIAM J. Opt. 6, 418–445 (1996)

    MATH  MathSciNet  Google Scholar 

  24. Conn, A.R., Gould, N.I.M., Toint, Ph.L.: Trust-Region Methods. MPS-SIAM Series on Optimization. SIAM, Philedalphia (2000)

  25. Conn, A.R., Scheinberg, K., Toint, Ph.L.: Recent progress in unconstrained nonlinear optimization without derivatives. Math. Program. 79, 397–414 (1997)

  26. Conn, A.R., Scheinberg, K., Toint, Ph.L.: On the convergence of derivative-free methods for unconstrained optimization. In: Buhmann, M.D., Iserles, A. (eds.) Approximation Theory and Optimization Tributes to M.J.D. Powell, pp. 83–108. Cambridge University Press, Cambridge (1997)

  27. Conn, A.R., Scheinberg, K., Vicente, L.N.: Geometry of interpolation sets in derivative free optimization. Math. Program. 111, 141–172 (2008)

    MATH  MathSciNet  Google Scholar 

  28. Conn, A.R., Scheinberg, K., Vicente, L.N.: Geometry of sample sets in derivative-free optimization: polynomial regression and underdetermined interpolation. IMA J. Numer. Anal. 28, 721–748 (2008)

    MATH  MathSciNet  Google Scholar 

  29. Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. MPS-SIAM Series on Optimization. SIAM, Philadelphia (2009)

    Google Scholar 

  30. Conn, A.R., Scheinberg, K., Vicente, L.N.: Global convergence of general derivative-free trust-region algorithms to first- and second-order critical points. SIAM J. Optim. 20, 387–415 (2009)

    MATH  MathSciNet  Google Scholar 

  31. Conn, A.R., Toint, Ph.L.: An algorithm using quadratic interpolation for unconstrained derivative free optimization. In: Di Pillo, G., Gianessi, F. (eds.) Nonlinear Optimization and Applications, pp. 27–47. Plenum, New York (1996)

  32. Conn, A.R., Vicente, L.N.: Bilevel derivative-free optimization its application to robust optimization. Optim. Methods Softw. 27(3), 559–575 (2012)

    MathSciNet  Google Scholar 

  33. Curtis, F., Gould, N.I.M., Robinson, D., Toint, Ph.L.: An Interior-Point Trust-Funnel Algorithm for Nonlinear Optimization. Report naXys-02-2014, Dept. of Mathematics, UNamur, Namur (B), (2014)

  34. Curtis, F., Jiang, H., Robinson, D.P.: An adaptive augmented Lagrangian method for large-scale constrained optimization. Math. Program. Ser. A (2014). doi:10.1007/s10107-014-0784-y

  35. Custodio, A.L., Rocha, H., Vicente, L.N.: Incorporating minimum Frobenius norm models in direct search. Comput. Optim. Appl. 46, 265–278 (2010)

    MATH  MathSciNet  Google Scholar 

  36. Dennis, J.E., Li, S.-B., Tapia, R.A.: A unified approach to global convergence of trust region methods for nonsmooth optimization. Math. Program. 68, 319–346 (1995)

    MATH  MathSciNet  Google Scholar 

  37. Dennis, J.E., Mei, H.H.: Two new unconstrained optimization algorithms which use function and gradient values. J. Optim. Theory Appl. 28, 453–482 (1979)

    MATH  MathSciNet  Google Scholar 

  38. Di, S., Sun, W.Y.: Trust region method for conic model to solve unconstrained optimization problems. Optim. Methods Softw. 6, 237–263 (1996)

    Google Scholar 

  39. Diouanne, Y., Gratton, S., Vicente, L.N.: Globally Convergent Evolutionary Strategies for Constrained Optimization. Technical Report TR-PA-14-50, CERFACS, Toulouse, France (2014)

  40. Duff, I.S., Nocedal, J., Reid, J.K.: The use of linear programming for the solution of sparse sets of nonlinear equations. SIAM J. Sci. Stat. Comput. 8, 99–108 (1987)

    MATH  MathSciNet  Google Scholar 

  41. El Hallabi, M., Tapia, R.A.: A Global Convergence Theory for Arbitrary Norm Trust Region Methods for Nonlinear Equations. Technical Report 87–25, Dept. Math. Sciences, Rice University, USA (1987)

  42. Fan, J.Y.: Convergence rate of the trust region method for nonlinear equations under local error bound condition. Comput. Optim. Appl. 34, 215–227 (2006)

    MATH  MathSciNet  Google Scholar 

  43. Fan, J.Y., Lu, N.: On the modified trust region algorithm. Optim. Methods Softw. (2014). doi:10.1080/10556788.2014.932943

    Google Scholar 

  44. Fan, J.Y., Pan, J.Y.: An improved trust region algorithm for nonlinear equations. Comput. Optim. Appl. 48, 59–70 (2011)

    MATH  MathSciNet  Google Scholar 

  45. Fan, J.Y., Yuan, Y.: A new trust region algorithm with trust region radius converging to zero. in: D. Li ed. Proceedings of the 5th International Conference on Optimization: Techniques and Applications (December 2001, Hongkong), pp. 786–794 (2001)

  46. Fasano, G., Morales, J.L., Nocedal, J.: On the geometry phase in model-based algorithms for derivative-free optimization. Optim. Methods Softw. 24, 145–154 (2009)

    MATH  MathSciNet  Google Scholar 

  47. Fletcher, R.: An Efficient, Global Convergent Algorithm for Unconstrained and Linearly Constrained Optimization Problems. Technical Report TP 431, AERE Harwell Laboratory, Oxfordshire, England (1970)

  48. Fletcher, R.: Practical Methods of Optimization, Volume 1: Unconstrained Optimization. Wiley, Chichester (1980)

    Google Scholar 

  49. Fletcher, R.: Practical Methods of Optimization, Volume 2: Constrained Optimization. Wiley, Chichester (1981)

    Google Scholar 

  50. Fletcher, R.: A model algorithm for composite NDO problem. Math. Prog. Study 17, 67–76 (1982)

    MATH  MathSciNet  Google Scholar 

  51. Fletcher, R.: Second order correction for nondifferentiable optimization. In: Watson, G.A. (ed.) Numerical Analysis, pp. 85–115. Springer, Berlin (1982)

    Google Scholar 

  52. Fletcher, R., Leyffer, S.: Nonlinear programming without a penalty function. Math. Program. 91, 239C269 (2002)

    MathSciNet  Google Scholar 

  53. Gay, D.M.: Computing optimal local constrained steps. SIAM J. Sci. Comput. 2, 186–197 (1981)

    MATH  MathSciNet  Google Scholar 

  54. Gong, L.J.: Trust Region Algorithms for Unconstrained Optimization. Ph.D. thesis, Academy of Mathematics and Systems Science, Chinese Academy of Sciences (2011)

  55. Gould, N.I.M., Orban, D., Sartenar, A., Toint, PhL: Sensitive of trust-region algorithms on their parameters. 4OR, Q. J. Italian Fr. Belgian OR Soc 3, 227–241 (2005)

    MATH  Google Scholar 

  56. Gould, N.I.M., Orban, D., Toint, PhL: CUTEr and SifDec: a constrained and unconstrained testing environment, revisited. ACM Trans. Math. Softw. (TOMS) 29(4), 373–394 (2003)

    MATH  MathSciNet  Google Scholar 

  57. Gould, N.I.M., Robinson, D.P., Thorne, H.S.: On solving trust-region and other regularised subproblems in optimization. Math. Program. Comput. 2, 21–57 (2010)

    MATH  MathSciNet  Google Scholar 

  58. Gould, N.I.M., Toint, PhL: Nonlinear programming without a penalty function or a filter. Math. Program. 122, 155–196 (2010)

    MATH  MathSciNet  Google Scholar 

  59. Grapiglia, G.N., Yuan, J.Y., Yuan, Y.: A subspace version of the Powell–Yuan trust region algorithm for equality constrained optimization. J. Oper. Res. Soc. China 1(4), 425–451 (2013)

    MATH  Google Scholar 

  60. Grapiglia, G.N., Yuan, J.Y., Yuan, Y.: On the convergence and worst-case complexity of trust-region and regularization methods for unconstrained optimization. Math. Program. (2014). doi:10.1007/s10107-014-0794-9

  61. Grapiglia, G.N., Yuan, J.Y., Yuan, Y.: A derivative-free trust-region algorithm for composite nonsmooth optimization. Comput. Appl. Math. (2014). doi:10.1007/s40314-014-0201-4

  62. Gratton, S., Royer, C.W., Vicente, L.N., Zhang, Z.: Direct Search Based on Probabilistic Descent. Technical Report 14–11, Dept. Math., University of Coimbra, Portugal (2014)

  63. Gratton, S., Sartenaer, A., Toint, Ph.L.: Recursive trust-region methods for multiscale nonlinear optimization. SIAM J. Optim. 19, 414–444 (2008)

  64. Gratton, S., Toint, PhL, Troeltzsch, A.: An active-set trust region method for derivative-free nonlinear bound-constrained optimization. Optim. Methods Softw. 26, 873–894 (2011)

    MATH  MathSciNet  Google Scholar 

  65. Gratton, S., Vicente, L.N.: A surrogate management framework using rigorous trust-region steps. Optim. Methods Softw. 29, 10–23 (2014)

    MATH  MathSciNet  Google Scholar 

  66. Griewank, A.: The Modification of Newtons Method for Unconstrained Optimization by Bounding Cubic Terms. Technical Report NA/12, DAMTP, University of Cambridge, England (1981)

  67. Hebden, M.D.: An Algorithm for Minimization Using Exact Second Order Derivatives. Technical Report, T.P. 515, AERE Harwell Laboratory, Harwell, Oxfordshire, England (1973)

  68. Hei, L.: A self-adaptive trust region algorithm. J. Comput. Math. 21, 229–236 (2003)

    MATH  MathSciNet  Google Scholar 

  69. Heinkenschloss, M., Vicente, L.N.: Analysis of inexact trust-region SQP algorithms. SIAM J. Optim. 12, 283–302 (2001)

    MATH  MathSciNet  Google Scholar 

  70. Hsia, Y., Sheu, R.L., Yuan, Y.: On the p-Regularized Trust Region Subproblem. arXiv:1409.4665 (2014)

  71. Kanzow, C., Klug, A.: An interior-point affine-scaling trust-region method for semismooth equations with box constraints. Comput. Optim. Appl. 37, 329C353 (2007)

    MathSciNet  Google Scholar 

  72. Levenberg, K.: A method for solution of certain problems in least squares. Q. J. Appl. Math. 2, 164–168 (1944)

    MATH  MathSciNet  Google Scholar 

  73. Liu, D.C., Nocedal, J.: On the limited memory BFGS method for large scale optimization. Math. Program. 45, 503–528 (1989)

    MATH  MathSciNet  Google Scholar 

  74. Madsen, K.: An algorithm for the minimax solution of overdetermined systems of nonlinear equations. J. Inst. Math. Its Appl. 16, 321–328 (1975)

    MATH  MathSciNet  Google Scholar 

  75. Marazzi, M., Nocedal, J.: Wedge trust region methods for derivative free optimization. Math. Program. 91, 289–305 (2002)

    MATH  MathSciNet  Google Scholar 

  76. Marquardt, D.: An algorithm for least-squares estimation on nonlinear parameters. SIAM J. Appl. Math. 11, 431–441 (1963)

    MATH  MathSciNet  Google Scholar 

  77. Martinez, J.M.: Local minimizers of quadratic functions on Euclidean balls and spheres. SIAM J. Optim. 4, 159–176 (1994)

    MATH  MathSciNet  Google Scholar 

  78. Moré, J.: The Levenberg–Marquardt algorithm: implementation and theory. In: Watson, G.A. (ed.) Numerical Analysis, pp. 105–116. Springer, Berlin (1978)

    Google Scholar 

  79. Moré, J.J.: Recent developments in algorithms and software for trust region methods. In: Bachem, A., Grötschel, M., Korte, B. (eds.) Mathematical Programming: The State of the Art, pp. 258–287. Springer, Berlin (1983)

    Google Scholar 

  80. Morrison, D.D.: Methods for nonlinear least squares problems and convergence proofs. In: Lorell, J., Yagi, F. (eds.) Proceedings of the Seminar on Tracking Programs and Orbit Determination, pp. 1–9. Jet Propulsion Laboratory, Pasadena (1960)

    Google Scholar 

  81. Nesterov, Yu., Polyak, B.T.: Cubic regularization of Newtons method and its global performance. Math. Program. 108, 177–205 (2006)

    MATH  MathSciNet  Google Scholar 

  82. Niu, L.F., Yuan, Y.: A new trust region algorithm for nonlinear constrained optimization. J. Comput. Math. 28, 72–86 (2010)

    MATH  MathSciNet  Google Scholar 

  83. Nocedal, J., Yuan, Y.: Combining trust region and line search techniques. In: Yuan, Y. (ed.) Advances in Nonlinear Programming, pp. 153–175. Kluwer, Netherlands (1998)

    Google Scholar 

  84. Oeuvray, R., Bierlaire, M.: BOOSTER: a derivative-free algorithm based on radial basis functions. Inter. J. Model. Simul. 29, 349–371 (2009)

    Google Scholar 

  85. Omojokun, E.O.: Trust Region Algorithms for Optimization with Nonlinear Equality and Inequality Constraints. Ph.D. Thesis, University of Colorado at Boulder, USA (1989)

  86. Osborne, M.R.: Nonlinear least squares—the Levenberg–Marquardt algorithm revisited. J. Aust. Math. Soc. B 19, 343–357 (1976)

    MATH  MathSciNet  Google Scholar 

  87. Peng, J., Yuan, Y.: Optimality conditions for the minimization of a quadratic with two quadratic constraints. SIAM J. Optim. 7, 579–594 (1997)

    MATH  MathSciNet  Google Scholar 

  88. Powell, M.J.D.: A hybrid method for nonlinear equations. In: Rabinowitz, P. (ed.) Numerical Methods for Nonlinear Algebraic Equations, pp. 87–114. Gordon and Breach, London (1970)

    Google Scholar 

  89. Powell, M.J.D.: A Fortran subroutine for solving systems of nonlinear algebraic equations. In: Robinowitz, P. (ed.) Numerical Methods for Nonlinear Algebraic Equations, pp. 115–161. Gordon and Breach, London (1970)

    Google Scholar 

  90. Powell, M.J.D.: A new algorithm for unconstrained optimization. In: Rosen, J.B., Mangasarian, O.L., Ritter, K. (eds.) Nonlinear Programming, pp. 31–65. Academic Press, London (1970)

    Google Scholar 

  91. Powell, M.J.D.: Convergence properties of a class of minimization algorithms. In: Mangasarian, O.L., Meyer, R.R., Robinson, S.M. (eds.) Nonlinear Programming 2, pp. 1–27. Academic Press, New York (1975)

    Google Scholar 

  92. Powell, M.J.D.: Some global convergence properties of a variable metric algorithm for minimization without exact line searches. In: Cottle, R.W., Lemke, C.E. (eds.) Nonlinear Programming, SIAM-AMS Proceedings, vol. IX, pp. 53–72. SIAM, Philadelphia (1976)

    Google Scholar 

  93. Powell, M.J.D.: General algorithms for discrete nonlinear approximation calculations. In: Schumaker, L.L. (ed.) Approximation Theory IV, pp. 187–218. Academy Press, New York (1984)

    Google Scholar 

  94. Powell, M.J.D.: A direct search optimization method that models the objective and constraint functions by linear interpolation. In: Gomez, S., Hennart, J.P. (eds.) Advances in Optimization and Numercial Analysis. Mathematics and Its Applications, vol. 275, pp. 51–67. Springer, Netherland (1994)

    Google Scholar 

  95. Powell, M.J.D.: Direct search algorithms for optimization calculations. Acta Numer. 7, 287–336 (1998)

    Google Scholar 

  96. Powell, M.J.D.: UOBYQA: unconstrained optimization by quadratic approximation. Math. Program. 92, 555–582 (2002)

    MATH  MathSciNet  Google Scholar 

  97. Powell, M.J.D.: On trust region methods for unconstrained minimization without derivatives. Math. Program. 97, 605–623 (2003)

    MATH  MathSciNet  Google Scholar 

  98. Powell, M.J.D.: Least Frobenius norm updating of quadratic models that satisfy interpolation conditions. Math. Program. 100, 183–215 (2004)

    MATH  MathSciNet  Google Scholar 

  99. Powell, M.J.D.: On the use of quadratic models in unconstrained minimization without derivatives. Optim. Methods Softw. 19, 399–411 (2004)

    MATH  MathSciNet  Google Scholar 

  100. Powell, M.J.D.: The NEWOUA software for unconstrained optimization without derivatives. In: Di Pillo, G., Roma, M. (eds.) Large Scale Nonlinear Optimization. Nonconvex Optimization and Its Applications, vol. 83, pp. 255–297. Springer, Berlin (2006)

    Google Scholar 

  101. Powell, M.J.D.: Developments of NEWUOA for minimization without derivatives. IMA J. Numer. Anal. 28, 649–664 (2008)

    MATH  MathSciNet  Google Scholar 

  102. Powell, M.J.D.: The BOBYQA Algorithym for Bound Constrained Optimization Without Derivatives. Technical Report DAMTP 2009/NA06, CMS, University of Cambridge, UK (2009)

  103. Powell, M.J.D.: On the convergence of trust region algorithm for unconstrained minimization without derivatives. Comput. Optim. Appl. 53, 527–555 (2012)

    MATH  MathSciNet  Google Scholar 

  104. Powell, M.J.D.: Beyond symetric Broyden for updating quadratic models in minimization without derivatives. Math. Program. 138, 475–500 (2013)

    MATH  MathSciNet  Google Scholar 

  105. Powell, M.J.D.: On fast trust region methods for quadratic models with linear constraints. DAMTP 2014/NA02, CMS, University of Cambridge (2014)

  106. Powell, M.J.D., Yuan, Y.: Conditions for superlinear convergence in \(l_1\) and \(l_\infty \) solutions of overdetermined non-linear equations. IMA J. Numer. Anal. 4, 241–251 (1984)

    MATH  MathSciNet  Google Scholar 

  107. Powell, M.J.D., Yuan, Y.: A trust region algorithm for equality constrained optimization. Math. Program. 49, 189–211 (1991)

    MathSciNet  Google Scholar 

  108. Qi, L., Sun, J.: A trust region algorithm for minimization of locally Lipschitzian functions. Math. Program. 66, 25–43 (1994)

    MATH  MathSciNet  Google Scholar 

  109. Sampaio, Ph.R., Toint, Ph.L.: A Derivative-Free Trust-Funnel Method for Equality-Constrained Nonlinear Optimization. Report naXys-08-2014, Dept of Mathematics, UNamur, Namur (B), (2014)

  110. Sampaio, R.J.B., Yuan, J.Y., Sun, W.Y.: Trust region algorithm for nonsmooth optimization. Appl. Math. Comput. 85, 109–116 (1997)

    MATH  MathSciNet  Google Scholar 

  111. Scheinberg, K., Toint, Ph.L.: Self-correcting geometry in model-based algorithms for derivative-free unconstrained optimization. SIAM J. Optim. 20, 3512–3532 (2010)

  112. Siegel, D.: Implementing and Modifying Broyden Class Updates for Large Scale Optimization. Report DAMPT1992/NA12, University of Cambridge, Department ofApplied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, England (1992)

  113. Sorensen, D.C.: Newton’s method with a model trust region modification. SIAM J. Numer. Anal. 19, 409–426 (1982)

    MATH  MathSciNet  Google Scholar 

  114. Sorensen, D.C.: Trust region methods for unconstrained optimization. In: Powell, M.J.D. (ed.) Nonlinear Optimization 1981, pp. 29–39. Academic Press, London (1982)

    Google Scholar 

  115. Steihaug, T.: The conjugate gradient method and trust regions in large scale optimization. SIAM J. Numer. Anal. 20, 626–637 (1983)

    MATH  MathSciNet  Google Scholar 

  116. Sturm, J.F., Zhang, S.Z.: On cones of nonnegative quadratic functions. Math. Oper. Res. 28, 246–267 (2003)

    MATH  MathSciNet  Google Scholar 

  117. Sun, W.Y., Yuan, Y.: A conic trust-region method for nonlinearly constrained optimization. Ann. Oper. Res. 103, 175–191 (2001)

    MATH  MathSciNet  Google Scholar 

  118. Toint, Ph.L.: Some numerical result using a sparse matrix updating formula in unconstrainded optimization. Math. Comput. 32, 839–851 (1978)

  119. Toint, Ph.L.: On the superlinear convergence of an algorithm for solving a sparse minimization problem. SIAM J. Numer. Anal. 16, 1036–1045 (1979)

  120. Toint, Ph.L.: Sparsity exploiting quasi-Newton methods for unconstrained optimization. In: Dixon, L.C.W., Spedicato, E., Szego, G.P. (eds.) Nonlinear Optimization: Theory and Algorithms, pp. 65–90. Birkhauser, Belgium (1980)

  121. Toint, Ph.L.: Convergence Properties of a Class of Minimization Algorithms That Use a Possibly Unbounded Sequences of Quadratic Approximation. Technical Report 81/1, Dept. Math., University of Namur, Belgium (1981)

  122. Toint, Ph.L.: Towards an efficient sparsity exploiting Newton method for minimization. In: Duff, I.S. (ed.) Sparse Matrices and Their Uses, pp. 57–88. Academic Press, Lodon (1978)

  123. Toint, Ph.L.: A non-monotone trust region algorithm for nonlinear optimization subject to convex constraints. Math. Program. 77, 69–94 (1997)

  124. Toint, Ph.L.: Nonlinear stepsize control, trust regions and regularizations for unconstrained optimization. Optim. Methods Sofw. 28, 82–95 (2013)

  125. Troeltzsch, A.: A sequential quadratic programming algorithm for equality-constrained optimization without derivatives. Optim. Lett. (2014). doi:10.1007/s11590-014-0830-y

  126. Vardi, A.: A trust region algorithm for equality constrained minimization: convergence properties and implementation. SIAM J. Numer. Anal. 22, 575–591 (1985)

    MATH  MathSciNet  Google Scholar 

  127. Villacorta, K.D.V., Oliveira, P.R., Soubeyran, A.: A trust-region method for unconstrained multiobjective problems with applications in satisficing processes. J. Opt. Theory Appl. 160, 865–889 (2014)

    MATH  MathSciNet  Google Scholar 

  128. Vicente, L.N.: A comparison between line searches and trust regions for nonlinear optimization. Investigacao Operacional 16, 173–179 (1996)

    Google Scholar 

  129. Waltz, R.A., Morales, J.L., Nocedal, J., Orban, D.: An interior algorithm for nonlinear optimization that combines line search and trust region steps. Math. Program. 107, 391C408 (2006)

    MathSciNet  Google Scholar 

  130. Wang, Z.H., Yuan, Y.: A subspace implementation of quasi-Newton trust region methods for unconstrained optimization. Numer. Math. 104, 241–269 (2006)

    MATH  MathSciNet  Google Scholar 

  131. Wang, X., Yuan, Y.: A trust region method based on a new affine scaling technique for simple bounded optimization. Optim. Methods Soft. 28, 871–888 (2013)

    MATH  MathSciNet  Google Scholar 

  132. Wang, X., Yuan, Y.: An augmented Lagrangian trust region method for equality constrained optimization. Optim. Methods Softw. (2014). doi:10.1080/10556788.2014.940947

  133. Wang X., Zhang H.: An augmented Lagrangian affine scaling method for nonlinear programming. https://www.math.lsu.edu/hozhang/papers/ALAS, (2014)

  134. Weiser, M., Deuflhard, P., Erdmann, B.: Affine conjugate adaptive Newton methods for nonlinear elastomechanics. Optim. Methods Softw. 22(3), 413–431 (2007)

    MATH  MathSciNet  Google Scholar 

  135. Wild, S.M., Regis, R.G., Shoemaker, C.A.S.: ORBIT: Optimization by radial basis function interpolation in trust-regions. SIAM J. Sci. Comput. 30, 3197–3219 (2008)

    MATH  MathSciNet  Google Scholar 

  136. Wild, S.M., Shoemaker, C.A.S.: Global convergence of radial basis function trust-region algorithms for derivative-free optimization. SIAM Rev. 55, 349–371 (2013)

    MATH  MathSciNet  Google Scholar 

  137. Winfield, D.H.: Function and Functional Minimization by Interpolation in Data Tables. Ph.D. Thesis, Harvard Unviersity, Cambridge, MA, USA (1969)

  138. Winfield, D.H.: Function minimization by interpolation in date table. IMA J. Numer. Anal. 12, 339–347 (1973)

    MATH  MathSciNet  Google Scholar 

  139. Yamashita, H., Yabe, H., Tanabe, T.: A globally and superlinearly convergent primal-dual interior point trust region method for large scale constrained optimization. Math. Program. 102, 111–151 (2005)

    MATH  MathSciNet  Google Scholar 

  140. Yuan, Y.: An example of only linearly convergence of trust region algorithms for nonsmooth optimization. IMA J. Numer. Anal. 4, 327–335 (1984)

    MATH  MathSciNet  Google Scholar 

  141. Yuan, Y.: Conditions for convergence of trust region algorithms for nonsmooth optimization. Math. Program. 31(2), 220–228 (1985)

    MATH  Google Scholar 

  142. Yuan, Y.: On the superlinear convergence of a trust region algorithm for nonsmooth optimization. Math. Program. 31(3), 269–285 (1985)

    MATH  Google Scholar 

  143. Yuan, Y.: Some Theories and Algorithms in Nonlinear Programming. Ph.D. Thesis, University of Cambridge, UK (1985)

  144. Yuan, Y.: On a subproblem of trust region algorithms for constrained optimization. Math. Program. 47, 53–63 (1990)

    MATH  Google Scholar 

  145. Yuan, Y.: A dual algorithm for minimizing a quadratic function with two quadratic constraints. J. Comput. Math. 9, 348–359 (1991)

    MATH  MathSciNet  Google Scholar 

  146. Yuan, Y.: On the convergence of a new trust region algorithm. Numer. Math. 70, 515–539 (1995)

    MATH  MathSciNet  Google Scholar 

  147. Yuan, Y.: Trust region algorithms for nonlinear equations. Information 1, 7–20 (1998)

    MATH  MathSciNet  Google Scholar 

  148. Yuan, Y.: A review of trust region algorithms for optimization. In: Ball, J.M., Hunt, J.C.R. (eds.) ICIAM 99, Proceedingss of the Fourth International Cogress on Industrial and Applied Mathematics, pp. 271–282. Oxford University Press, Oxford (2000)

    Google Scholar 

  149. Yuan, Y.: On the truncated conjugate gradient method. Math. Program. 87, 561–571 (2000)

    MATH  MathSciNet  Google Scholar 

  150. Yuan, Y.: A trust region algorithm for Nash equilibrium problems. Pac. J. Optim. 7, 125–138 (2011)

    MATH  MathSciNet  Google Scholar 

  151. Yuan, Y.: A review on subspace methods for nonlinear optimization. In: Proceedings of the International Congress of Mathematics 2014, Seoul, Korea, pp. 807–827 (2014)

  152. Zhang, H.C., Conn, A.R.: On the local convergence of a derivative-free algorithm for least-squares minimization. Comput. Optim. Appl. 51(2), 481–507 (2012)

    MATH  MathSciNet  Google Scholar 

  153. Zhang, H.C., Conn, A.R., Scheinberg, K.: A derivative-free algorithm for least squares minimization. SIAM J. Optim. 20(6), 3555–3576 (2012)

    MathSciNet  Google Scholar 

  154. Zhang, J.: A new trust region method for nonlinear equations. Math. Methods Oper. Res. 58, 283–298 (2003)

    MATH  MathSciNet  Google Scholar 

  155. Zhang, L.P.: A new trust region algorithm for nonsmooth convex minimization. Appl. Math. Comput. 193, 135–142 (2007)

    MATH  MathSciNet  Google Scholar 

  156. Zhang, Y.: Computing a Celis–Dennis–Tapia trust region step for equality constrained optimization. Math. Program. 55, 109–124 (1992)

    MATH  Google Scholar 

  157. Zhang, Z.: Sobolev seminorm of quadratic functions with applications to derivative-free optimization. Math. Program. 146, 77–96 (2014)

    MATH  MathSciNet  Google Scholar 

  158. Zikrin, S.: Large-Scale Optimization Mehods with Applications to Design of Filter Networks. Linkoping Studies in Science and Technology, Dissertation No. 151, Likoping University, Sweden (2014)

Download references

Acknowledgments

I am very grateful to my former students Jinyan Fan, Yong Xia, Zaikun Zhang and Xiao Wang for their helps during my preparation of this paper. I would like to thank my long-term colleagues and dear friends Philippe Toint and Andew Conn, and two anonymous referees for their valuable comments which help to improve the paper. This paper is supported in partial by Grants 11331012, 11321061 and 11461161005 of the National Natural Science Foundation of China.

Author information

Authors and Affiliations

  1. State Key Laboratory of Scientific/Engineering Computing, Institute of Computational Mathematics and Scientific/Engineering Computing, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Zhong Guan Cun Donglu 55, Beijing, 100190, China

    Ya-xiang Yuan

Authors
  1. Ya-xiang Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ya-xiang Yuan.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yuan, Yx. Recent advances in trust region algorithms. Math. Program. 151, 249–281 (2015). https://doi.org/10.1007/s10107-015-0893-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10107-015-0893-2

Keywords

  • Trust region algorithms
  • Nonlinear optimization
  • Subproblem
  • Complexity
  • Convergence

Mathematics Subject Classification

  • 65K05
  • 90C30