Journal of Optimization Theory and Applications

, Volume 79, Issue 3, pp 427–462 | Cite as

Analysis and implementation of a dual algorithm for constrained optimization

  • W. W. Hager
Contributed Papers

Abstract

This paper analyzes a constrained optimization algorithm that combines an unconstrained minimization scheme like the conjugate gradient method, an augmented Lagrangian, and multiplier updates to obtain global quadratic convergence. Some of the issues that we focus on are the treatment of rigid constraints that must be satisfied during the iterations and techniques for balancing the error associated with constraint violation with the error associated with optimality. A preconditioner is constructed with the property that the rigid constraints are satisfied while ill-conditioning due to penalty terms is alleviated. Various numerical linear algebra techniques required for the efficient implementation of the algorithm are presented, and convergence behavior is illustrated in a series of numerical experiments.

Key Words

Constrained optimization multiplier methods preconditioning global convergence quadratic convergence 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Hager, W. W.,Dual Techniques for Constrained Optimization, Journal of Optimization Theory and Applications, Vol. 55, pp. 37–71, 1987.Google Scholar
  2. 2.
    Arrow, K. J., andSolow, R. M.,Gradient Methods for Constrained Maxima, with Weakened Assumptions, Studies in Linear and Nonlinear Programming, Edited by K. Arrow, L. Hurwicz, and H. Uzawa. Stanford University Press, Stanford, California, 1958.Google Scholar
  3. 3.
    Hestenes, M. R.,Multiplier and Gradient Methods, Journal of Optimization Theory and Applications, Vol. 4, pp. 303–320, 1969.Google Scholar
  4. 4.
    Powell, M. J. D.,A Method for Nonlinear Constraints in Minimization Problems, Optimization, Edited by R. Fletcher, Academic Press, New York, New York, pp. 283–298, 1972.Google Scholar
  5. 5.
    Rockafellar, R. T.,The Multiplier Method of Hestenes and Powell Applied to Convex Programming, Journal of Optimization Theory and Applications, Vol. 12, pp. 555–562, 1973.Google Scholar
  6. 6.
    Rockafellar, R. T.,A Dual Approach to Solving Nonlinear Programming Problems by Unconstrained Optimization, Mathematical Programming, Vol. 5, pp. 354–373, 1973.Google Scholar
  7. 7.
    Rockafellar, R. T.,Augmented Lagrange Multiplier Functions and Duality in Nonconvex Programming, SIAM Journal on Control, Vol. 12, pp. 268–285, 1974.Google Scholar
  8. 8.
    Bertsekas, D. P.,Constrained Optimization and Lagrange Multiplier Methods, Academic Press, New York, New York, 1982.Google Scholar
  9. 9.
    Hager, W. W.,Approximations to the Multiplier Method, SIAM Journal on Numerical Analysis, Vol. 22, pp. 16–46, 1985.Google Scholar
  10. 10.
    Polyak, B. T., andTret'yakov, N. V.,The Method of Penalty Estimates for Conditional Extremum Problems, Akademija Nauk SSSR, Žurnal Vyčislitel'noi Matematiki i Matematičeskoi Fiziki, Vol. 13, pp. 34–46, 1973.Google Scholar
  11. 11.
    Fortin, R., andGlowinski, R.,Augmented Lagrangian Methods: Applications to the Numerical Solution of Boundary-Value Problems, North-Holland, Amsterdam, Holland, 1983.Google Scholar
  12. 12.
    Rosen, J. B., andKreuser, J. L.,A Gradient Projection Algorithm for Nonlinear Constraints, Numerical Methods for Nonlinear Optimization, Edited by F. A. Lootsma, Academic Press, London, England, pp. 297–300, 1972.Google Scholar
  13. 13.
    Robinson, S. M.,A Quadratically-Convergent Algorithm for General Nonlinear Programming Problems, Mathematical Programming, Vol. 3, pp. 145–156, 1972.Google Scholar
  14. 14.
    Murtagh, B. A., andSaunders, M. A.,MINOS 5.0 User's Guide, Report 83-20R, Department of Operations Research, Stanford University, Standford, California, 1987.Google Scholar
  15. 15.
    Coleman, T. F., andConn, A. R.,Nonlinear Programming via an Exact Penalty Function: Asymptotic Analysis, Mathematical Programming, Vol. 24, pp. 123–136, 1982.Google Scholar
  16. 16.
    Coleman, T. F., andConn, A. R.,Nonlinear Programming via an Exact Penalty Function: Global Analysis, Mathematical Programming, Vol. 24, pp. 137–161, 1982.Google Scholar
  17. 17.
    Armijo, L.,Minimization of Functions Having Lipschitz Continuous First-Partial Derivatives, Pacific Journal of Mathematics, Vol. 16, pp. 1–3, 1966.Google Scholar
  18. 18.
    Burke, J., andHan, S. P.,A Gauss-Newton Approach to Solving Generalized Inequalities, Mathematics of Operations Research, Vol. 11, pp. 632–643, 1986.Google Scholar
  19. 19.
    Daniel, J. W.,Newton's Method for Nonlinear Inequalities, Numerische Mathematik, Vol. 21, pp. 381–387, 1973.Google Scholar
  20. 20.
    Gardia-Palomares, U., andRestuccia, A.,Application of the Armijo Stepsize Rule to the Solution of a Nonlinear System of Equalities and Inequalities, Journal of Optimization Theory and Applications, Vol. 41, pp. 405–415, 1983.Google Scholar
  21. 21.
    Pshenichnyi, B. N.,Newton's Method for the Solution of Systems of Equalities and Inequalities, Mathematical Notes of the Academy of Sciences of the USSR, Vol. 8, pp. 827–830, 1970.Google Scholar
  22. 22.
    Robinson, S. M.,Extension of Newton's Method to Nonlinear Functions with Values in a Cone, Numerische Mathematik, Vol. 19, pp. 341–347, 1972.Google Scholar
  23. 23.
    Valentine, F. A.,The Problem of Lagrange with Differential Inequalities as Added Side Conditions, Contributions to the Calculus of Variations, University of Chicago Press, Chicago, Illinois, pp. 407–448, 1937.Google Scholar
  24. 24.
    Ortega, J. M., andRheinboldt, W. C.,Iterative Solution of Nonlinear Equations in Several Variables, Academic Press, New York, New York, 1970.Google Scholar
  25. 25.
    Burke, J. V.,A Sequential Quadratic Programming Method for Potentially Infeasible Mathematical Programs, Journal of Mathematical Analysis and Applications, Vol. 139, pp. 319–351, 1989.Google Scholar
  26. 26.
    Luenberger, D. G.,Introduction to Linear and Nonlinear Programming, Addison-Wesley, Reading, Massachusetts, 1973.Google Scholar
  27. 27.
    Hager, W. W.,A Derivative-Based Bracketing Scheme for Univariate Minimization and the Conjugate Gradient Method, Computers and Mathematics with Applications, Vol. 18, pp. 779–795, 1989.Google Scholar
  28. 28.
    Hager, W. W.,Updating the Inverse of a Matrix, SIAM Review, Vol. 31, pp. 221–239, 1989.Google Scholar
  29. 29.
    Bartels, R. H., Golub, G. H., andSaunders, M. A.,Numerical Techniques in Mathematical Programming, Nonlinear Programming, Edited by R. B. Rosen, O. L. Mangasarian, and K. Ritter, Academic Press, New York, New York, pp. 123–176, 1970.Google Scholar
  30. 30.
    Gill, P. E., Golub, G. H., Murray, W., andSaunders, M. A.,Methods for Modifying Matrix Factorizations, Mathematics of Computation, Vol. 28, pp. 505–535, 1974.Google Scholar
  31. 31.
    Gill, P. E., andMurray, W.,Modification of Matrix Factorizations after a Rank-One Change, The State of the Art in Numerical Analysis, Edited by D. Jacobs, Academic Press, New York, New York, pp. 55–83, 1977.Google Scholar
  32. 32.
    Gill, P. E., Murray, W., andSaunders, M. A.,Methods for Computing and Modifying the LDV Factors of a Matrix, Mathematics of Computation, Vol. 29, pp. 1051–1077, 1975.Google Scholar
  33. 33.
    Duff, I. S., Erisman, A. M., andReid, J. K.,Direct Methods for Sparse Matrices, Oxford University Press, Oxford, England, 1986.Google Scholar
  34. 34.
    George, A., andLiu, J. W. H.,Computer Solution of Large Sparse Positive-Definite Systems, Prentice-Hall, Englewood Cliffs, New Jersey, 1981.Google Scholar
  35. 35.
    Tapia, R. A.,Diagonalized Multiplier Methods and Quasi-Newton Methods for Constrained Optimization, Journal of Optimization Theory and Applications, Vol. 22, pp. 135–194, 1977.Google Scholar
  36. 36.
    Djang, A.,Algorithmic Equivalence in Quadratic Programming, PhD Dissertation, Department of Operations Research, Stanford University, Stanford, California, 1979.Google Scholar
  37. 37.
    Du, D. Z., andZhang, X. S.,Global Convergence of Rosen's Gradient Projection Method, Mathematical Programming, Vol. 44, pp. 357–366, 1989.Google Scholar
  38. 38.
    Hirst, H. P.,N-Step Quadratic Convergence in the Conjugate Gradient Method, PhD Dissertation, Pennsylvania State University, University Park, Pennsylvania, 1989.Google Scholar
  39. 39.
    Hager, W. W.,Applied Numerical Linear Algebra, Prentice-Hall, Englewood Cliffs, New Jersey, 1988.Google Scholar
  40. 40.
    Polak, E., andRibière, G.,Note sur la Convergence de Methodes de Directions Conjugées, Revue Française d'Informatique et Recherche Opérationnelle, Vol. 16, pp. 35–43, 1969.Google Scholar
  41. 41.
    Conn, A. R., Gould, N. I. M., andTonit, P. L.,A Globally Convergent Augmented Lagrangian Algorithm for Optimization with General Constraints and Simple Bounds, SIAM Journal on Numerical Analysis, Vol. 28, pp. 545–572, 1991.Google Scholar
  42. 42.
    Polak, E.,On the Global Stabilization of Locally Convergent Algorithms, Automatica, Vol. 12, pp. 337–342, 1976.Google Scholar
  43. 43.
    Burke, J., andHan, S. P.,A Robust Sequential Quadratic Programming Method, Mathematical Programming, Vol. 43, pp. 277–303, 1989.Google Scholar
  44. 44.
    Colville, A. R.,A Comparative Study on Nonlinear Programming Codes, Report No. 320-2949, IBM New York Scientific Center, 1968.Google Scholar
  45. 45.
    Himmelblau, D. M.,Applied Nonlinear Programming, McGraw-Hill, New York, New York, 1972.Google Scholar
  46. 46.
    Hock, W., andSchittkowski, K.,Test Examples for Nonlinear Programming Codes, Springer-Verlag, Berlin, Germany, 1980.Google Scholar
  47. 47.
    Gill, P. E., Murray, W., Saunders, M. A., andWright, M. H.,User's Guide for NPSOL (Version 4.0): A Fortran Package for Nonlinear Programming, Report 86-2, Department of Operations Research, Stanford University, Stanford, California, 1986.Google Scholar
  48. 48.
    Wolfe, P.,Methods of Nonlinear Programming, Nonlinear Programming, Edited by J. Abadie, Interscience, John Wiley, New York, New York, pp. 97–131, 1967.Google Scholar
  49. 49.
    Davidon, W. C.,Variable Metric Method for Minimization, SIAM Journal on Optimization, Vol. 1, pp. 1–17, 1991.Google Scholar
  50. 50.
    Murtagh, B. A., andSaunders, M. A.,Large-Scale Linearly Constrained Optimization, Mathematical Programming, Vol. 14, pp. 41–72, 1978.Google Scholar
  51. 51.
    Murtagh, B. A., andSaunders, M. A.,A Projected Lagrangian Algorithm and Its Implementation for Sparse Nonlinear Constraints, Mathematical Programming Study, Vol. 16, pp. 84–117, 1982.Google Scholar
  52. 52.
    Robinson, S. M.,Strongly Regular Generalized Equations, Mathematics of Operations Research, Vol. 5, pp. 43–62, 1980.Google Scholar

Copyright information

© Plenum Publishing Corporation 1993

Authors and Affiliations

  • W. W. Hager
    • 1
  1. 1.Department of MathematicsUniversity of FloridaGainesville

Personalised recommendations