Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Analysis and implementation of a dual algorithm for constrained optimization

  • 106 Accesses

  • 26 Citations

Abstract

This paper analyzes a constrained optimization algorithm that combines an unconstrained minimization scheme like the conjugate gradient method, an augmented Lagrangian, and multiplier updates to obtain global quadratic convergence. Some of the issues that we focus on are the treatment of rigid constraints that must be satisfied during the iterations and techniques for balancing the error associated with constraint violation with the error associated with optimality. A preconditioner is constructed with the property that the rigid constraints are satisfied while ill-conditioning due to penalty terms is alleviated. Various numerical linear algebra techniques required for the efficient implementation of the algorithm are presented, and convergence behavior is illustrated in a series of numerical experiments.

This is a preview of subscription content, log in to check access.

References

  1. 1.

    Hager, W. W.,Dual Techniques for Constrained Optimization, Journal of Optimization Theory and Applications, Vol. 55, pp. 37–71, 1987.

  2. 2.

    Arrow, K. J., andSolow, R. M.,Gradient Methods for Constrained Maxima, with Weakened Assumptions, Studies in Linear and Nonlinear Programming, Edited by K. Arrow, L. Hurwicz, and H. Uzawa. Stanford University Press, Stanford, California, 1958.

  3. 3.

    Hestenes, M. R.,Multiplier and Gradient Methods, Journal of Optimization Theory and Applications, Vol. 4, pp. 303–320, 1969.

  4. 4.

    Powell, M. J. D.,A Method for Nonlinear Constraints in Minimization Problems, Optimization, Edited by R. Fletcher, Academic Press, New York, New York, pp. 283–298, 1972.

  5. 5.

    Rockafellar, R. T.,The Multiplier Method of Hestenes and Powell Applied to Convex Programming, Journal of Optimization Theory and Applications, Vol. 12, pp. 555–562, 1973.

  6. 6.

    Rockafellar, R. T.,A Dual Approach to Solving Nonlinear Programming Problems by Unconstrained Optimization, Mathematical Programming, Vol. 5, pp. 354–373, 1973.

  7. 7.

    Rockafellar, R. T.,Augmented Lagrange Multiplier Functions and Duality in Nonconvex Programming, SIAM Journal on Control, Vol. 12, pp. 268–285, 1974.

  8. 8.

    Bertsekas, D. P.,Constrained Optimization and Lagrange Multiplier Methods, Academic Press, New York, New York, 1982.

  9. 9.

    Hager, W. W.,Approximations to the Multiplier Method, SIAM Journal on Numerical Analysis, Vol. 22, pp. 16–46, 1985.

  10. 10.

    Polyak, B. T., andTret'yakov, N. V.,The Method of Penalty Estimates for Conditional Extremum Problems, Akademija Nauk SSSR, Žurnal Vyčislitel'noi Matematiki i Matematičeskoi Fiziki, Vol. 13, pp. 34–46, 1973.

  11. 11.

    Fortin, R., andGlowinski, R.,Augmented Lagrangian Methods: Applications to the Numerical Solution of Boundary-Value Problems, North-Holland, Amsterdam, Holland, 1983.

  12. 12.

    Rosen, J. B., andKreuser, J. L.,A Gradient Projection Algorithm for Nonlinear Constraints, Numerical Methods for Nonlinear Optimization, Edited by F. A. Lootsma, Academic Press, London, England, pp. 297–300, 1972.

  13. 13.

    Robinson, S. M.,A Quadratically-Convergent Algorithm for General Nonlinear Programming Problems, Mathematical Programming, Vol. 3, pp. 145–156, 1972.

  14. 14.

    Murtagh, B. A., andSaunders, M. A.,MINOS 5.0 User's Guide, Report 83-20R, Department of Operations Research, Stanford University, Standford, California, 1987.

  15. 15.

    Coleman, T. F., andConn, A. R.,Nonlinear Programming via an Exact Penalty Function: Asymptotic Analysis, Mathematical Programming, Vol. 24, pp. 123–136, 1982.

  16. 16.

    Coleman, T. F., andConn, A. R.,Nonlinear Programming via an Exact Penalty Function: Global Analysis, Mathematical Programming, Vol. 24, pp. 137–161, 1982.

  17. 17.

    Armijo, L.,Minimization of Functions Having Lipschitz Continuous First-Partial Derivatives, Pacific Journal of Mathematics, Vol. 16, pp. 1–3, 1966.

  18. 18.

    Burke, J., andHan, S. P.,A Gauss-Newton Approach to Solving Generalized Inequalities, Mathematics of Operations Research, Vol. 11, pp. 632–643, 1986.

  19. 19.

    Daniel, J. W.,Newton's Method for Nonlinear Inequalities, Numerische Mathematik, Vol. 21, pp. 381–387, 1973.

  20. 20.

    Gardia-Palomares, U., andRestuccia, A.,Application of the Armijo Stepsize Rule to the Solution of a Nonlinear System of Equalities and Inequalities, Journal of Optimization Theory and Applications, Vol. 41, pp. 405–415, 1983.

  21. 21.

    Pshenichnyi, B. N.,Newton's Method for the Solution of Systems of Equalities and Inequalities, Mathematical Notes of the Academy of Sciences of the USSR, Vol. 8, pp. 827–830, 1970.

  22. 22.

    Robinson, S. M.,Extension of Newton's Method to Nonlinear Functions with Values in a Cone, Numerische Mathematik, Vol. 19, pp. 341–347, 1972.

  23. 23.

    Valentine, F. A.,The Problem of Lagrange with Differential Inequalities as Added Side Conditions, Contributions to the Calculus of Variations, University of Chicago Press, Chicago, Illinois, pp. 407–448, 1937.

  24. 24.

    Ortega, J. M., andRheinboldt, W. C.,Iterative Solution of Nonlinear Equations in Several Variables, Academic Press, New York, New York, 1970.

  25. 25.

    Burke, J. V.,A Sequential Quadratic Programming Method for Potentially Infeasible Mathematical Programs, Journal of Mathematical Analysis and Applications, Vol. 139, pp. 319–351, 1989.

  26. 26.

    Luenberger, D. G.,Introduction to Linear and Nonlinear Programming, Addison-Wesley, Reading, Massachusetts, 1973.

  27. 27.

    Hager, W. W.,A Derivative-Based Bracketing Scheme for Univariate Minimization and the Conjugate Gradient Method, Computers and Mathematics with Applications, Vol. 18, pp. 779–795, 1989.

  28. 28.

    Hager, W. W.,Updating the Inverse of a Matrix, SIAM Review, Vol. 31, pp. 221–239, 1989.

  29. 29.

    Bartels, R. H., Golub, G. H., andSaunders, M. A.,Numerical Techniques in Mathematical Programming, Nonlinear Programming, Edited by R. B. Rosen, O. L. Mangasarian, and K. Ritter, Academic Press, New York, New York, pp. 123–176, 1970.

  30. 30.

    Gill, P. E., Golub, G. H., Murray, W., andSaunders, M. A.,Methods for Modifying Matrix Factorizations, Mathematics of Computation, Vol. 28, pp. 505–535, 1974.

  31. 31.

    Gill, P. E., andMurray, W.,Modification of Matrix Factorizations after a Rank-One Change, The State of the Art in Numerical Analysis, Edited by D. Jacobs, Academic Press, New York, New York, pp. 55–83, 1977.

  32. 32.

    Gill, P. E., Murray, W., andSaunders, M. A.,Methods for Computing and Modifying the LDV Factors of a Matrix, Mathematics of Computation, Vol. 29, pp. 1051–1077, 1975.

  33. 33.

    Duff, I. S., Erisman, A. M., andReid, J. K.,Direct Methods for Sparse Matrices, Oxford University Press, Oxford, England, 1986.

  34. 34.

    George, A., andLiu, J. W. H.,Computer Solution of Large Sparse Positive-Definite Systems, Prentice-Hall, Englewood Cliffs, New Jersey, 1981.

  35. 35.

    Tapia, R. A.,Diagonalized Multiplier Methods and Quasi-Newton Methods for Constrained Optimization, Journal of Optimization Theory and Applications, Vol. 22, pp. 135–194, 1977.

  36. 36.

    Djang, A.,Algorithmic Equivalence in Quadratic Programming, PhD Dissertation, Department of Operations Research, Stanford University, Stanford, California, 1979.

  37. 37.

    Du, D. Z., andZhang, X. S.,Global Convergence of Rosen's Gradient Projection Method, Mathematical Programming, Vol. 44, pp. 357–366, 1989.

  38. 38.

    Hirst, H. P.,N-Step Quadratic Convergence in the Conjugate Gradient Method, PhD Dissertation, Pennsylvania State University, University Park, Pennsylvania, 1989.

  39. 39.

    Hager, W. W.,Applied Numerical Linear Algebra, Prentice-Hall, Englewood Cliffs, New Jersey, 1988.

  40. 40.

    Polak, E., andRibière, G.,Note sur la Convergence de Methodes de Directions Conjugées, Revue Française d'Informatique et Recherche Opérationnelle, Vol. 16, pp. 35–43, 1969.

  41. 41.

    Conn, A. R., Gould, N. I. M., andTonit, P. L.,A Globally Convergent Augmented Lagrangian Algorithm for Optimization with General Constraints and Simple Bounds, SIAM Journal on Numerical Analysis, Vol. 28, pp. 545–572, 1991.

  42. 42.

    Polak, E.,On the Global Stabilization of Locally Convergent Algorithms, Automatica, Vol. 12, pp. 337–342, 1976.

  43. 43.

    Burke, J., andHan, S. P.,A Robust Sequential Quadratic Programming Method, Mathematical Programming, Vol. 43, pp. 277–303, 1989.

  44. 44.

    Colville, A. R.,A Comparative Study on Nonlinear Programming Codes, Report No. 320-2949, IBM New York Scientific Center, 1968.

  45. 45.

    Himmelblau, D. M.,Applied Nonlinear Programming, McGraw-Hill, New York, New York, 1972.

  46. 46.

    Hock, W., andSchittkowski, K.,Test Examples for Nonlinear Programming Codes, Springer-Verlag, Berlin, Germany, 1980.

  47. 47.

    Gill, P. E., Murray, W., Saunders, M. A., andWright, M. H.,User's Guide for NPSOL (Version 4.0): A Fortran Package for Nonlinear Programming, Report 86-2, Department of Operations Research, Stanford University, Stanford, California, 1986.

  48. 48.

    Wolfe, P.,Methods of Nonlinear Programming, Nonlinear Programming, Edited by J. Abadie, Interscience, John Wiley, New York, New York, pp. 97–131, 1967.

  49. 49.

    Davidon, W. C.,Variable Metric Method for Minimization, SIAM Journal on Optimization, Vol. 1, pp. 1–17, 1991.

  50. 50.

    Murtagh, B. A., andSaunders, M. A.,Large-Scale Linearly Constrained Optimization, Mathematical Programming, Vol. 14, pp. 41–72, 1978.

  51. 51.

    Murtagh, B. A., andSaunders, M. A.,A Projected Lagrangian Algorithm and Its Implementation for Sparse Nonlinear Constraints, Mathematical Programming Study, Vol. 16, pp. 84–117, 1982.

  52. 52.

    Robinson, S. M.,Strongly Regular Generalized Equations, Mathematics of Operations Research, Vol. 5, pp. 43–62, 1980.

Download references

Author information

Additional information

This research was supported by the National Science Foundation Grant DMS-89-03226 and by the U.S. Army Research Office Contract DAA03-89-M-0314.

We thank the referees for their many perceptive comments which led to substantial improvements in the presentation of this paper.

Communicated by E. Polak

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Hager, W.W. Analysis and implementation of a dual algorithm for constrained optimization. J Optim Theory Appl 79, 427–462 (1993). https://doi.org/10.1007/BF00940552

Download citation

Key Words

  • Constrained optimization
  • multiplier methods
  • preconditioning
  • global convergence
  • quadratic convergence