Skip to main content
SpringerLink
Log in

On the convergence of the coordinate descent method for convex differentiable minimization

  • Contributed Papers
  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

The coordinate descent method enjoys a long history in convex differentiable minimization. Surprisingly, very little is known about the convergence of the iterates generated by this method. Convergence typically requires restrictive assumptions such as that the cost function has bounded level sets and is in some sense strictly convex. In a recent work, Luo and Tseng showed that the iterates are convergent for the symmetric monotone linear complementarity problem, for which the cost function is convex quadratic, but not necessarily strictly convex, and does not necessarily have bounded level sets. In this paper, we extend these results to problems for which the cost function is the composition of an affine mapping with a strictly convex function which is twice differentiable in its effective domain. In addition, we show that the convergence is at least linear. As a consequence of this result, we obtain, for the first time, that the dual iterates generated by a number of existing methods for matrix balancing and entropy optimization are linearly convergent.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Mangasarian, O. L.,Sparsity-Preserving SOR Algorithms for Separable Quadratic and Linear Programming, Computers and Operations Research, Vol. 11, pp. 105–112, 1984.

    Google Scholar 

  2. Mangasarian, O. L., andDe Leone, R.,Parallel Gradient Projection Successive Overrelaxation for Symmetric Linear Complementarity Problems and Linear Programs, Annals of Operations Research, Vol. 14, pp. 41–59, 1988.

    Google Scholar 

  3. Herman, G. T., andLent, A.,A Family of Iterative Quadratic Optimization Algorithms for Pairs of Inequalities, with Application in Diagnostic Radiology, Mathematical Programming Study, Vol. 9, pp. 15–29, 1978.

    Google Scholar 

  4. Cottle, R. W., andGoheen, M. S.,A Special Class of Large Quadratic Programs, Nonlinear Programming 3, Edited by O. L. Mangasarian, R. R. Meyer, and S. M. Robinson, Academic Press, New York, New York, pp. 361–390, 1978.

    Google Scholar 

  5. Cottle, R. W., Golub, G. H., andSacher, R. S.,On the Solution of Large, Structured Linear Complementarity Problems: The Block-Partitioned Case, Applied Mathematics and Optimization, Vol. 4, pp. 347–363, 1978.

    Google Scholar 

  6. Dembo, R. S., andTulowitzki, U.,On the Minimization of Quadratic Functions Subject to Box Constraints, Yale University, School of Organization and Management, Working Paper Series B, No. 78, 1983; revised 1984.

  7. Arimoto, S.,An Algorithm for Computing the Capacity of Arbitrary DMCs, IEEE Transactions on Information Theory, Vol. IT-18, pp. 14–20, 1972.

    Google Scholar 

  8. Blahut, R.,Computation of Channel Capacity and Rate Distortion Functions, IEEE Transactions on Information Theory, Vol. IT-18, pp. 460–473, 1972.

    Google Scholar 

  9. Kruithof, J.,Calculation of Telephone Traffic, De Ingenieur (E. Electrotechnik 3), Vol. 52, pp. E15–25, 1937.

    Google Scholar 

  10. Lamond, B., andStewart, N. F.,Bregman's Balancing Method, Transportation Research, Vol. 15B, pp. 239–248, 1981.

    Google Scholar 

  11. Censor, Y.,Parallel Application of Block-Iterative Methods in Medical Imaging and Radiation Therapy, Mathematical Programming, Series B, Vol. 42, pp. 307–325, 1988.

    Google Scholar 

  12. Lent, A.,A Convergent Algorithm for Maximum Entropy Image Restoration with a Medical X-Ray Application, Image Analysis and Evaluation, Edited by R. Shaw, Society of Photographic Scientists and Engineers, Washington, DC, pp. 249–257, 1977.

    Google Scholar 

  13. Powell, M. J. D.,An Algorithm for Maximizing Entropy Subject to Simple Bounds, Mathematical Programming, Vol. 42, pp. 171–180, 1988.

    Google Scholar 

  14. Frieden, B. R.,Image Enhancement and Restoration, Picture Processing and Digital Filtering, Edited by T. S. Huang, Springer-Verlag, Berlin, Germany, pp. 177–248, 1975.

    Google Scholar 

  15. Jaynes, E. T.,On the Rationale of Maximum Entropy Methods, Proceedings of the IEEE, Vol. 70, pp. 939–952, 1982.

    Google Scholar 

  16. Johnson, R., andShore, J. E.,Which Is the Better Entropy Expression for Speech Processing—S log S or log S?, IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. ASSP-32, pp. 129–136, 1984.

    Google Scholar 

  17. Darroch, J. N., andRatcliff, D.,Generalized Iterative Scaling for Log-Linear Models, Annals of Mathematical Statistics, Vol. 43, pp. 1470–1480, 1972.

    Google Scholar 

  18. Freund, R. M.,Projective Transformations for Interior Point Methods, Part 1: Basic Theory and Linear Programming, Massachusetts Institute of Technology, Operations Research Center, Working Paper No. 180–88, 1988.

  19. Huard, P.,Resolution of Mathematical Programming with Nonlinear Constraints by the Method of Centers, Nonlinear Programming, Edited by J. Abadie, North-Holland, Amsterdam, Holland, pp. 207–219, 1967.

    Google Scholar 

  20. Karmarkar, N.,A New Polynomial-Time Algorithm for Linear Programming, Combinatorica, Vol. 4, pp. 373–395, 1984.

    Google Scholar 

  21. Sonnevend, G.,New Algorithms in Convex Programming Based on a Notion of Center for Systems of Analytic Inequalities and on Rational Extrapolation, Trends in Mathematical Optimization, Edited by K. H. Hoffman, J. B. Hiriart-Urruty, J. Zowe, and C. Lemarechal, Birkhäuser-Verlag, Basel, Switzerland, pp. 311–326, 1988.

    Google Scholar 

  22. Hildreth, C.,A Quadratic Programming Procedure, Naval Research Logistics Quarterly, Vol. 4, pp. 79–85, 1957; see alsoErratum, Vol. 4, p. 361, 1957.

    Google Scholar 

  23. Cottle, R. W., andPang, J.-S.,On the Convergence of a Block Successive Overrelaxation Method for a Class of Linear Complementarity Problems, Mathematical Programming Study, Vol. 17, pp. 126–138, 1982.

    Google Scholar 

  24. Cryer, C. W.,The Solution of a Quadratic Programming Problem Using Systematic Overrelaxation, SIAM Journal on Control and Optimization, Vol. 9, pp. 385–392, 1971.

    Google Scholar 

  25. Bachem, A., andKorte, B.,On the RAS Algorithm, Computing, Vol. 23, pp. 189–198, 1979.

    Google Scholar 

  26. Bregman, L. M.,Proof of the Convergence of Sheleikhovskii's Method for a Problem with Transportation Constraints, USSR Computational Mathematics and Mathematical Physics, Vol. 1, pp. 191–204, 1967.

    Google Scholar 

  27. Schneider, M. H., andZenios, S. A.,A Comparative Study of Algorithms for Matrix Balancing, Operations Research, Vol. 38, pp. 439–455, 1990.

    Google Scholar 

  28. Zenios, S. A., andIu, S. L.,Vector and Parallel Computing for Matrix Balancing, Annals of Operations Research, Vol. 22, pp. 161–180, 1990.

    Google Scholar 

  29. Censor, Y., andLent, A.,Optimization of log x Entropy over Linear Equality Constraints, SIAM Journal on Control and Optimization, Vol. 25, pp. 921–933, 1987.

    Google Scholar 

  30. Auslender, A.,Optimisation: Méthodes Numériques, Masson, Paris, France, 1976.

    Google Scholar 

  31. Bertsekas, D. P., andTsitsiklis, J. N.,Parallel and Distributed Computation: Numerical Methods, Prentice-Hall, Englewood Cliffs, New Jersey, 1989.

    Google Scholar 

  32. D'esopo, D. A.,A Convex Programming Procedure, Naval Research Logistics Quarterly, Vol. 6, pp. 33–42, 1959.

    Google Scholar 

  33. Glowinski, R.,Numerical Methods for Nonlinear Variational Problems, Springer-Verlag, New York, New York, 1984.

    Google Scholar 

  34. Luenberger, D. G.,Linear and Nonlinear Programming, Addison-Wesley, Reading, Massachusetts, 1973.

    Google Scholar 

  35. Polak, E.,Computational Methods in Optimization: A Unified Approach, Academic Press, New York, New York, 1971.

    Google Scholar 

  36. Powell, M. J. D.,On Search Directions for Minimization Algorithms, Mathematical Programming, Vol. 4, pp. 193–201, 1973.

    Google Scholar 

  37. Sargent, R. W. H., andSebastian, D. J.,On the Convergence of Sequential Minimization Algorithms, Journal of Optimization Theory and Applications, Vol. 12, pp. 567–575, 1973.

    Google Scholar 

  38. Zangwill, W. I.,Nonlinear Programming: A Unified Approach, Prentice-Hall, Englewood Cliffs, New Jersey, 1969.

    Google Scholar 

  39. Tseng, P.,Dual Ascent Methods for Problems with Strictly Convex Costs and Linear Constraints: A Unified Approach, SIAM Journal on Control and Optimization, Vol. 28, pp. 214–242, 1990.

    Google Scholar 

  40. Tseng, P.,Descent Methods for Convex Essentially Smooth Optimization, Journal of Optimization Theory and Applications, Vol. 70, pp. 109–135, 1991.

    Google Scholar 

  41. Tseng, P., andBertsekas, D. P.,Relaxation Methods for Problems with Strictly Convex Costs and Linear Constraints, Mathematics of Operations Research, Vol. 16, pp. 462–481, 1991.

    Google Scholar 

  42. Bregman, L. M.,The Relaxation Method of Finding the Common Point Convex Sets and Its Application to the Solution of Problems in Convex Programming, USSR Computational Mathematics and Mathematical Physics, Vol. 7, pp. 200–217, 1967.

    Google Scholar 

  43. Pang, J. S.,On the Convergence of Dual Ascent Methods for Large-Scale Linearly Constrained Optimization, University of Texas at Dallas, School of Management, Technical Report, 1984.

  44. Bertsekas, D. P., Hosein, P. A., andTseng, P.,Relaxation Methods for Newtork flow Problems with Convex Arc Costs, SIAM Journal on Control and Optimization, Vol. 25, pp. 1219–1243, 1987.

    Google Scholar 

  45. Luo, Z. Q., andTseng, P.,On the Convergence of a Matrix-Splitting Algorithm for the Symmetric Monotone Linear Complementarity Problem, SIAM Journal on Control and Optimization, Vol. 29, pp. 1037–1060, 1991.

    Google Scholar 

  46. Rockafellar, R. T.,Convex Analysis, Princeton University Press, Princeton, New Jersey, 1970.

    Google Scholar 

  47. Ortega, J. M., andRheinboldt, W. C.,Iterative Solution of Nonlinear Equations in Several Variables, Academic Press, New York, New York, 1970.

    Google Scholar 

  48. Tseng, P., andBertsekas, D. P.,Relaxation Methods for Problems with Strictly Convex Separable Costs and Linear Constraints, Mathematical Programming, Vol. 38, pp. 303–321, 1987.

    Google Scholar 

  49. Hoffman, A. J.,On Approximate Solutions of Systems of Linear Inequalities, Journal of Research of the National Bureau of Standards, Vol. 49, pp. 263–265, 1952.

    Google Scholar 

  50. Robinson, S. M.,Bounds for Errors in the Solution Set of a Perturbed Linear Program, Linear Algebra and Its Applications, Vol. 6, pp. 69–81, 1973.

    Google Scholar 

  51. Mangasarian, O. L., andShiau, T. H.,Lipschitz Continuity of Solutions of Linear Inequalities: Programs and Complementarity Problems, SIAM Journal on Control and Optimization, Vol. 25, pp. 583–595, 1987.

    Google Scholar 

  52. Luo, Z. Q., andTseng, P.,On the Linear Convergence of Descent Methods for Convex Essentially Smooth Minimization, SIAM Journal on Control and Optimization (to appear).

  53. Lin, Y. Y., andPang, J. S.,Iterative Methods for Large Convex Quadratic Programs: A Survey, SIAM Journal on Control and Optimization, Vol. 18, pp. 383–411, 1987.

    Google Scholar 

  54. Iusem, A. N., andDe Pierro, A. R.,On the Convergence Properties of Hildreth's Quadratic Programming Algorithm, Mathematical Programming, Series A, Vol. 47, pp. 37–51, 1990.

    Google Scholar 

  55. De Pierro, A. R., andIusem, A. N.,A Relaxed Version of Bregman's Method for Convex Programming, Journal of Optimization Theory and Applications, Vol. 51, pp. 421–440, 1986.

    Google Scholar 

  56. Censor, Y., andLent, A.,An Iterative Row-Action Method for Interval Convex Programming, Journal of Optimization Theory and Applications, Vol. 34, pp. 321–352, 1981.

    Google Scholar 

Download references

Author information

Author notes

  1. P. Tseng (Research Associate)

    Present address: Department of Mathematics, University of Washington, Seattle, Washington

Authors and Affiliations

  1. Communications Research Laboratory, Department of Electrical and Computer Engineering, McMaster University, Hamilton, Ontario, Canada

    Z. Q. Luo (Assistant Professor)

  2. Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, Massachusetts

    P. Tseng (Research Associate)

Authors
  1. Z. Q. Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. P. Tseng
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

Communicated by O. L. Mangasarian

This work was partially supported by the U.S. Army Research Office, Contract No. DAAL03-86-K-0171, by the National Science Foundation, Grant No. ECS-8519058, and by the Science and Engineering Research Board of McMaster University.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Luo, Z.Q., Tseng, P. On the convergence of the coordinate descent method for convex differentiable minimization. J Optim Theory Appl 72, 7–35 (1992). https://doi.org/10.1007/BF00939948

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF00939948

Key Words

Search

Navigation