Constrained Optimization: Local Methods

  • Alexey F. Izmailov
  • Mikhail V. Solodov
Part of the Springer Series in Operations Research and Financial Engineering book series (ORFE)


This chapter is the central part of the book; it is devoted to local convergence analysis of Newton-type methods for optimization with constraints. The perturbed sequential quadratic programming (SQP) framework is introduced, which allows to treat, in addition and in a unified manner, not only various modifications of SQP itself (truncated, augmented Lagrangian based versions, and second-order corrections) but also a number of other algorithms even though their iterative subproblems are very different from SQP (linearly constrained Lagragian algorithm, sequential quadratically constrained quadratic programming, inexact restoration, and a certain interior feasible directions method).


  1. 3.
    L.T.H. An, An efficient algorithm for globally minimizing a quadratic function under convex quadratic constraints. Math. Program. 87, 401–426 (2000)CrossRefzbMATHMathSciNetGoogle Scholar
  2. 5.
    R. Andreani, E.G. Birgin, J.M. Martínez, M.L. Schuverdt, Augmented Lagrangian methods under the constant positive linear dependence constraint qualification. Math. Program. 111, 5–32 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  3. 9.
    M. Anitescu, A superlinearly convergent sequential quadratically constrained quadratic programming algorithm for degenerate nonlinear programming. SIAM J. Optim. 12, 949–978 (2002)CrossRefzbMATHMathSciNetGoogle Scholar
  4. 15.
    C. Audet, P. Hansen, B. Jaumard, G. Savard, A branch and cut algorithm for nonconvex quadratically constrained quadratic programming. Math. Program. 87, 131–152 (2000)zbMATHMathSciNetGoogle Scholar
  5. 18.
    D.P. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods (Academic Press, New York, 1982)zbMATHGoogle Scholar
  6. 21.
    E.G. Birgin, J.M. Martínez, Local convergence of an inexact-restoration method and numerical experiments. J. Optim. Theory Appl. 127, 229–247 (2005)CrossRefzbMATHMathSciNetGoogle Scholar
  7. 22.
    E.G. Birgin, J.M. Martínez, Improving ultimate convergence of an augmented Lagrangian method. Optim. Method. Softw. 23, 177–195 (2008)CrossRefzbMATHGoogle Scholar
  8. 23.
    P. Boggs, J. Tolle, Sequential quadratic programming. Acta Numer. 4, 1–51 (1995)CrossRefMathSciNetGoogle Scholar
  9. 26.
    J.F. Bonnans, Local analysis of Newton-type methods for variational inequalities and nonlinear programming. Appl. Math. Optim. 29, 161–186 (1994)CrossRefzbMATHMathSciNetGoogle Scholar
  10. 29.
    J.F. Bonnans, J.Ch. Gilbert, C. Lemaréchal, C. Sagastizábal, Numerical Optimization: Theoretical and Practical Aspects, 2nd edn. (Springer, Berlin, 2006)Google Scholar
  11. 34.
    R.H. Byrd, N.I.M. Gould, J. Nocedal, R.A. Waltz, An algorithm for nonlinear optimization using linear programming and equality constrained subproblems. Math. Program. 100, 27–48 (2004)zbMATHMathSciNetGoogle Scholar
  12. 36.
    R.H. Byrd, N.I.M. Gould, J. Nocedal, R.A. Waltz, On the convergence of successive linear-quadratic programming algorithms. SIAM J. Optim. 16, 471–489 (2006)CrossRefMathSciNetGoogle Scholar
  13. 38.
    R.H. Byrd, J. Nocedal, R.A. Waltz, Steering exact penalty methods for nonlinear programming. Optim. Method. Softw. 23, 197–213 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  14. 41.
    C. Chin, R. Fletcher, On the global convergence of an SLP-filter algorithm that takes EQP steps. Math. Program. 96, 161–177 (2003)CrossRefzbMATHMathSciNetGoogle Scholar
  15. 74.
    D. Fernández, M.V. Solodov, On local convergence of sequential quadratically-constrained quadratic-programming type methods, with an extension to variational problems. Comput. Optim. Appl. 39, 143–160 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 76.
    D. Fernández, M. Solodov, Local convergence of exact and inexact augmented Lagrangian methods under the second-order sufficient optimality condition. SIAM J. Optim. 22, 384–407 (2012)CrossRefzbMATHMathSciNetGoogle Scholar
  17. 77.
    D. Fernández, A.F. Izmailov, M.V. Solodov, Sharp primal superlinear convergence results for some Newtonian methods for constrained optimization. SIAM J. Optim. 20, 3312–3334 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  18. 78.
    D. Fernández, E.A. Pilotta, G.A. Torres, An inexact restoration strategy for the globalization of the sSQP method. Comput. Optim. Appl. 54, 595–617 (2013)CrossRefzbMATHMathSciNetGoogle Scholar
  19. 86.
    A. Fischer, A. Friedlander, A new line search inexact restoration approach for nonlinear programming. Comput. Optim. Appl. 46, 333–346 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  20. 88.
    R. Fletcher, Second order corrections for non-differentiable optimization, in Numerical Analysis, ed. by D. Griffiths (Springer, Berlin, 1982), pp. 85–114CrossRefGoogle Scholar
  21. 90.
    R. Fletcher, E.S. de la Maza, Nonlinear programming and nonsmooth optimization by successive linear programming. Math. Program. 43, 235–256 (1989)CrossRefzbMATHGoogle Scholar
  22. 96.
    M.P. Friedlander, M.A. Saunders, A globally convergent linearly constrained Lagrangian method for nonlinear optimization. SIAM J. Optim. 15, 863–897 (2005)CrossRefzbMATHMathSciNetGoogle Scholar
  23. 97.
    M. Fukushima, Z.-Q. Luo, P. Tseng, A sequential quadratically constrained quadratic programming method for differentiable convex minimization. SIAM J. Optim. 13, 1098–1119 (2003)CrossRefzbMATHMathSciNetGoogle Scholar
  24. 106.
    N.I.M. Gould, Some reflections on the current state of active-set and interior-point methods for constrained optimization. Numerical Analysis Group Internal Report 2003-1. Computational Science and Engineering Department, Rutherford Appleton Laboratory, Oxfordshire, 2003Google Scholar
  25. 107.
    N.I.M. Gould, D.P. Robinson, A second derivative SQP method: global convergence and practical issues. SIAM J. Optim. 20, 2023–2048 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  26. 108.
    N.I.M. Gould, D.P. Robinson, A second derivative SQP method: local convergence and practical issues. SIAM J. Optim. 20, 2049–2079 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  27. 109.
    N.I.M. Gould, D. Orban, Ph.L. Toint, Numerical methods for large-scale nonlinear optimization. Acta Numer. 14, 299–361 (2005)CrossRefzbMATHMathSciNetGoogle Scholar
  28. 115.
    J. Han, D. Sun, Superlinear convergence of approximate Newton methods for LC1 optimization problems without strict complementarity, in Recent Advances in Nonsmooth Optimization, vol 58, ed. by D.-Z. Du, L. Qi, R.S. Womersley (World Scientific, Singapore, 1993), pp. 353–367Google Scholar
  29. 116.
    J. Herskovits, A Two-Stage Feasible Direction Algorithm Including Variable Metric Techniques for Nonlinear Optimization Problems. Rapport de Recherche 118 (INRIA, Rocqencourt, 1982)Google Scholar
  30. 117.
    J. Herskovits, A two-stage feasible directions algorithm for nonlinear constrained optimization. Math. Program. 36, 19–38 (1986)CrossRefzbMATHMathSciNetGoogle Scholar
  31. 118.
    J. Herskovits, Feasible direction interior-point technique for nonlinear optimization. J. Optim. Theory Appl. 99, 121–146 (1998)CrossRefzbMATHMathSciNetGoogle Scholar
  32. 124.
    Z.-H. Huang, D. Sun, G. Zhao, A smoothing Newton-type algorithm of stronger convergence for the quadratically constrained convex quadratic programming. Comput. Optim. Appl. 35, 199–237 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  33. 133.
    A.F. Izmailov, A.S. Kurennoy, Partial Clarke generalized Jacobian and other generalized differentials, in Theoretical and Applied Problems of Nonlinear Analysis, ed. by V.A. Bereznyov (Computing Center RAS, Moscow, 2010, in Russian), pp. 77–90Google Scholar
  34. 147.
    A.F. Izmailov, M.V. Solodov, A truncated SQP method based on inexact interior-point solutions of subproblems. SIAM J. Optim. 20, 2584–2613 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  35. 154.
    A.F. Izmailov, A.L. Pogosyan, M.V. Solodov, Semismooth SQP method for equality-constrained optimization problems with an application to the lifted reformulation of mathematical programs with complementarity constraints. Optim. Method. Softw. 26, 847–872 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  36. 158.
    A.F. Izmailov, A.S. Kurennoy, M.V. Solodov, The Josephy–Newton method for semismooth generalized equations and semismooth SQP for optimization. Set-Valued Variational Anal. 21, 17–45 (2013)CrossRefzbMATHMathSciNetGoogle Scholar
  37. 159.
    A.F. Izmailov, A.S. Kurennoy, M.V. Solodov, Composite-step constrained optimization methods interpreted via the perturbed sequential quadratic programming framework (2013). IMPA preprint A2405Google Scholar
  38. 173.
    S. Kruk, H. Wolkowicz, Sequential, quadratically constrained, quadratic programming for general nonlinear programming, in Handbook of Semidefinite Programming, ed. by H. Wolkowicz, R. Saigal, L. Vandenberghe (Kluwer Academic, Dordrecht, 2000), pp. 563–575CrossRefGoogle Scholar
  39. 177.
    F. Leibfritz, E.W. Sachs, Inexact SQP interior point methods and large scale optimal control problems. SIAM J. Control Optim. 38, 272–293 (1999)CrossRefzbMATHMathSciNetGoogle Scholar
  40. 182.
    M.S. Lobo, L. Vandenberghe, S. Boyd, H. Lebret, Applications of second-order cone programming. Linear Algebra Appl. 284, 193–228 (1998)CrossRefzbMATHMathSciNetGoogle Scholar
  41. 192.
    J.M. Martínez, Inexact restoration method with Lagrangian tangent decrease and new merit function for nonlinear programming. J. Optim. Theory Appl. 111, 39–58 (2001)CrossRefzbMATHMathSciNetGoogle Scholar
  42. 193.
    J.M. Martínez, E.A. Pilotta, Inexact restoration algorithms for constrained optimization. J. Optim. Theory Appl. 104, 135–163 (2000)CrossRefzbMATHMathSciNetGoogle Scholar
  43. 194.
    J.M. Martínez, E.A. Pilotta, Inexact restoration methods for nonlinear programming: advances and perspectives, in Optimization and Control with Applications, ed. by L.Q. Qi, K.L. Teo, X.Q. Yang (Springer, New York, 2005), pp. 271–292CrossRefGoogle Scholar
  44. 197.
    D.Q. Mayne, E. Polak, Feasible direction algorithms for optimization problems with equality and inequality constraints. Math. Program. 11, 67–80 (1976)CrossRefzbMATHMathSciNetGoogle Scholar
  45. 200.
    J.L. Morales, J. Nocedal, Y. Wu, A sequential quadratic programming algorithm with an additional equality constrained phase. IMA J. Numer. Anal. 32, 553–579 (2012)CrossRefzbMATHMathSciNetGoogle Scholar
  46. 205.
    B.A. Murtagh, M.A. Saunders, A projected Lagrangian algorithm and its implementation for sparse nonlinear constraints. Math. Program. Study. 16, 84–117 (1982)CrossRefzbMATHMathSciNetGoogle Scholar
  47. 206.
    B.A. Murtagh, M.A. Saunders, MINOS 5.0 user’s guide. Technical Report SOL 83.20. Stanford University, 1983Google Scholar
  48. 207.
    Y.E. Nesterov, A.S. Nemirovskii, Interior Point Polynomial Methods in Convex Programming: Theory and Applications (SIAM Publications, Philadelphia, 1993)Google Scholar
  49. 208.
    J. Nocedal, S.J. Wright, Numerical Optimization, 2nd edn. (Springer, New York, 2006)zbMATHGoogle Scholar
  50. 214.
    E.R. Panier, A.L. Tits, J. Herskovits, A QP-free, globally convergent, locally superlinearly convergent algorithm for inequality constrained optimization. SIAM J. Control Optim. 26, 788–811 (1988)CrossRefzbMATHMathSciNetGoogle Scholar
  51. 215.
    V.M. Panin, A second-order method for discrete min-max problem. USSR Comput. Math. Math. Phys. 19, 90–100 (1979)CrossRefzbMATHMathSciNetGoogle Scholar
  52. 219.
    M.J.D. Powell, A fast algorithm for nonlinearly constrained optimization calculations, in Numerical Analysis Dundee 1977, ed. by G.A. Watson. Lecture Notes in Mathematics, vol 630 (Springer, Berlin, 1978), pp. 144–157Google Scholar
  53. 223.
    L. Qi, Superlinearly convergent approximate Newton methods for LC1 optimization problems. Math. Program. 64, 277–294 (1994)CrossRefzbMATHGoogle Scholar
  54. 226.
    H.-D. Qi, L. Qi, A new QP-free, globally convergent, locally superlinearlly convergent algorithm for inequality constrained optimization. SIAM J. Optim. 11, 113–132 (2000)CrossRefzbMATHMathSciNetGoogle Scholar
  55. 231.
    S.M. Robinson, A quadratically convergent algorithm for general nonlinear programming problems. Math. Program. 3, 145–156 (1972)CrossRefzbMATHGoogle Scholar
  56. 232.
    S.M. Robinson, Perturbed Kuhn–Tucker points and rates of convergence for a class of nonlinear-programming algorithms. Math. Program. 7, 1–16 (1974)CrossRefzbMATHGoogle Scholar
  57. 248.
    M.V. Solodov, On the sequential quadratically constrained quadratic programming methods. Math. Oper. Res. 29, 64–79 (2004)CrossRefzbMATHMathSciNetGoogle Scholar
  58. 260.
    A.L. Tits, A. Wächter, S. Bakhtiari, T.J. Urban, C.T. Lawrence, A primal-dual interior-point method for nonlinear programming with strong global and local convergence properties. SIAM J. Optim. 14, 173–199 (2003)CrossRefzbMATHMathSciNetGoogle Scholar
  59. 266.
    E.J. Wiest, E. Polak, A generalized quadratic programming-based phase-I–phase-II method for inequality-constrained optimization. Appl. Math. Optim. 26, 223–252 (1992)CrossRefzbMATHMathSciNetGoogle Scholar
  60. 268.
    R.B. Wilson, A simplicial algorithm for concave programming. Ph.D. thesis, Graduate School of Business Administration, Harvard University, U.S.A., 1963Google Scholar
  61. 270.
    S.J. Wright, Modifying SQP for degenerate problems. SIAM J. Optim. 13, 470–497 (2002)CrossRefzbMATHMathSciNetGoogle Scholar
  62. 273.
    Y.-F. Yang, D.-H. Li, L. Qi, A feasible sequential linear equation method for inequality constrained optimization. SIAM J. Optim. 13, 1222–1244 (2003)CrossRefzbMATHMathSciNetGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Alexey F. Izmailov
    • 1
  • Mikhail V. Solodov
    • 2
  1. 1.Moscow State UniversityMoscowRussia
  2. 2.Instituto de Matemática Pura e AplicadaRio de JaneiroBrazil

Personalised recommendations