On the convergence of the coordinate descent method for convex differentiable minimization

  • Z. Q. Luo
  • P. Tseng
Contributed Papers


The coordinate descent method enjoys a long history in convex differentiable minimization. Surprisingly, very little is known about the convergence of the iterates generated by this method. Convergence typically requires restrictive assumptions such as that the cost function has bounded level sets and is in some sense strictly convex. In a recent work, Luo and Tseng showed that the iterates are convergent for the symmetric monotone linear complementarity problem, for which the cost function is convex quadratic, but not necessarily strictly convex, and does not necessarily have bounded level sets. In this paper, we extend these results to problems for which the cost function is the composition of an affine mapping with a strictly convex function which is twice differentiable in its effective domain. In addition, we show that the convergence is at least linear. As a consequence of this result, we obtain, for the first time, that the dual iterates generated by a number of existing methods for matrix balancing and entropy optimization are linearly convergent.

Key Words

Coordinate descent convex differentiable optimization symmetric linear complementarity problems 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Mangasarian, O. L.,Sparsity-Preserving SOR Algorithms for Separable Quadratic and Linear Programming, Computers and Operations Research, Vol. 11, pp. 105–112, 1984.Google Scholar
  2. 2.
    Mangasarian, O. L., andDe Leone, R.,Parallel Gradient Projection Successive Overrelaxation for Symmetric Linear Complementarity Problems and Linear Programs, Annals of Operations Research, Vol. 14, pp. 41–59, 1988.Google Scholar
  3. 3.
    Herman, G. T., andLent, A.,A Family of Iterative Quadratic Optimization Algorithms for Pairs of Inequalities, with Application in Diagnostic Radiology, Mathematical Programming Study, Vol. 9, pp. 15–29, 1978.Google Scholar
  4. 4.
    Cottle, R. W., andGoheen, M. S.,A Special Class of Large Quadratic Programs, Nonlinear Programming 3, Edited by O. L. Mangasarian, R. R. Meyer, and S. M. Robinson, Academic Press, New York, New York, pp. 361–390, 1978.Google Scholar
  5. 5.
    Cottle, R. W., Golub, G. H., andSacher, R. S.,On the Solution of Large, Structured Linear Complementarity Problems: The Block-Partitioned Case, Applied Mathematics and Optimization, Vol. 4, pp. 347–363, 1978.Google Scholar
  6. 6.
    Dembo, R. S., andTulowitzki, U.,On the Minimization of Quadratic Functions Subject to Box Constraints, Yale University, School of Organization and Management, Working Paper Series B, No. 78, 1983; revised 1984.Google Scholar
  7. 7.
    Arimoto, S.,An Algorithm for Computing the Capacity of Arbitrary DMCs, IEEE Transactions on Information Theory, Vol. IT-18, pp. 14–20, 1972.Google Scholar
  8. 8.
    Blahut, R.,Computation of Channel Capacity and Rate Distortion Functions, IEEE Transactions on Information Theory, Vol. IT-18, pp. 460–473, 1972.Google Scholar
  9. 9.
    Kruithof, J.,Calculation of Telephone Traffic, De Ingenieur (E. Electrotechnik 3), Vol. 52, pp. E15–25, 1937.Google Scholar
  10. 10.
    Lamond, B., andStewart, N. F.,Bregman's Balancing Method, Transportation Research, Vol. 15B, pp. 239–248, 1981.Google Scholar
  11. 11.
    Censor, Y.,Parallel Application of Block-Iterative Methods in Medical Imaging and Radiation Therapy, Mathematical Programming, Series B, Vol. 42, pp. 307–325, 1988.Google Scholar
  12. 12.
    Lent, A.,A Convergent Algorithm for Maximum Entropy Image Restoration with a Medical X-Ray Application, Image Analysis and Evaluation, Edited by R. Shaw, Society of Photographic Scientists and Engineers, Washington, DC, pp. 249–257, 1977.Google Scholar
  13. 13.
    Powell, M. J. D.,An Algorithm for Maximizing Entropy Subject to Simple Bounds, Mathematical Programming, Vol. 42, pp. 171–180, 1988.Google Scholar
  14. 14.
    Frieden, B. R.,Image Enhancement and Restoration, Picture Processing and Digital Filtering, Edited by T. S. Huang, Springer-Verlag, Berlin, Germany, pp. 177–248, 1975.Google Scholar
  15. 15.
    Jaynes, E. T.,On the Rationale of Maximum Entropy Methods, Proceedings of the IEEE, Vol. 70, pp. 939–952, 1982.Google Scholar
  16. 16.
    Johnson, R., andShore, J. E.,Which Is the Better Entropy Expression for Speech Processing—S log S or log S?, IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. ASSP-32, pp. 129–136, 1984.Google Scholar
  17. 17.
    Darroch, J. N., andRatcliff, D.,Generalized Iterative Scaling for Log-Linear Models, Annals of Mathematical Statistics, Vol. 43, pp. 1470–1480, 1972.Google Scholar
  18. 18.
    Freund, R. M.,Projective Transformations for Interior Point Methods, Part 1: Basic Theory and Linear Programming, Massachusetts Institute of Technology, Operations Research Center, Working Paper No. 180–88, 1988.Google Scholar
  19. 19.
    Huard, P.,Resolution of Mathematical Programming with Nonlinear Constraints by the Method of Centers, Nonlinear Programming, Edited by J. Abadie, North-Holland, Amsterdam, Holland, pp. 207–219, 1967.Google Scholar
  20. 20.
    Karmarkar, N.,A New Polynomial-Time Algorithm for Linear Programming, Combinatorica, Vol. 4, pp. 373–395, 1984.Google Scholar
  21. 21.
    Sonnevend, G.,New Algorithms in Convex Programming Based on a Notion of Center for Systems of Analytic Inequalities and on Rational Extrapolation, Trends in Mathematical Optimization, Edited by K. H. Hoffman, J. B. Hiriart-Urruty, J. Zowe, and C. Lemarechal, Birkhäuser-Verlag, Basel, Switzerland, pp. 311–326, 1988.Google Scholar
  22. 22.
    Hildreth, C.,A Quadratic Programming Procedure, Naval Research Logistics Quarterly, Vol. 4, pp. 79–85, 1957; see alsoErratum, Vol. 4, p. 361, 1957.Google Scholar
  23. 23.
    Cottle, R. W., andPang, J.-S.,On the Convergence of a Block Successive Overrelaxation Method for a Class of Linear Complementarity Problems, Mathematical Programming Study, Vol. 17, pp. 126–138, 1982.Google Scholar
  24. 24.
    Cryer, C. W.,The Solution of a Quadratic Programming Problem Using Systematic Overrelaxation, SIAM Journal on Control and Optimization, Vol. 9, pp. 385–392, 1971.Google Scholar
  25. 25.
    Bachem, A., andKorte, B.,On the RAS Algorithm, Computing, Vol. 23, pp. 189–198, 1979.Google Scholar
  26. 26.
    Bregman, L. M.,Proof of the Convergence of Sheleikhovskii's Method for a Problem with Transportation Constraints, USSR Computational Mathematics and Mathematical Physics, Vol. 1, pp. 191–204, 1967.Google Scholar
  27. 27.
    Schneider, M. H., andZenios, S. A.,A Comparative Study of Algorithms for Matrix Balancing, Operations Research, Vol. 38, pp. 439–455, 1990.Google Scholar
  28. 28.
    Zenios, S. A., andIu, S. L.,Vector and Parallel Computing for Matrix Balancing, Annals of Operations Research, Vol. 22, pp. 161–180, 1990.Google Scholar
  29. 29.
    Censor, Y., andLent, A.,Optimization of log x Entropy over Linear Equality Constraints, SIAM Journal on Control and Optimization, Vol. 25, pp. 921–933, 1987.Google Scholar
  30. 30.
    Auslender, A.,Optimisation: Méthodes Numériques, Masson, Paris, France, 1976.Google Scholar
  31. 31.
    Bertsekas, D. P., andTsitsiklis, J. N.,Parallel and Distributed Computation: Numerical Methods, Prentice-Hall, Englewood Cliffs, New Jersey, 1989.Google Scholar
  32. 32.
    D'esopo, D. A.,A Convex Programming Procedure, Naval Research Logistics Quarterly, Vol. 6, pp. 33–42, 1959.Google Scholar
  33. 33.
    Glowinski, R.,Numerical Methods for Nonlinear Variational Problems, Springer-Verlag, New York, New York, 1984.Google Scholar
  34. 34.
    Luenberger, D. G.,Linear and Nonlinear Programming, Addison-Wesley, Reading, Massachusetts, 1973.Google Scholar
  35. 35.
    Polak, E.,Computational Methods in Optimization: A Unified Approach, Academic Press, New York, New York, 1971.Google Scholar
  36. 36.
    Powell, M. J. D.,On Search Directions for Minimization Algorithms, Mathematical Programming, Vol. 4, pp. 193–201, 1973.Google Scholar
  37. 37.
    Sargent, R. W. H., andSebastian, D. J.,On the Convergence of Sequential Minimization Algorithms, Journal of Optimization Theory and Applications, Vol. 12, pp. 567–575, 1973.Google Scholar
  38. 38.
    Zangwill, W. I.,Nonlinear Programming: A Unified Approach, Prentice-Hall, Englewood Cliffs, New Jersey, 1969.Google Scholar
  39. 39.
    Tseng, P.,Dual Ascent Methods for Problems with Strictly Convex Costs and Linear Constraints: A Unified Approach, SIAM Journal on Control and Optimization, Vol. 28, pp. 214–242, 1990.Google Scholar
  40. 40.
    Tseng, P.,Descent Methods for Convex Essentially Smooth Optimization, Journal of Optimization Theory and Applications, Vol. 70, pp. 109–135, 1991.Google Scholar
  41. 41.
    Tseng, P., andBertsekas, D. P.,Relaxation Methods for Problems with Strictly Convex Costs and Linear Constraints, Mathematics of Operations Research, Vol. 16, pp. 462–481, 1991.Google Scholar
  42. 42.
    Bregman, L. M.,The Relaxation Method of Finding the Common Point Convex Sets and Its Application to the Solution of Problems in Convex Programming, USSR Computational Mathematics and Mathematical Physics, Vol. 7, pp. 200–217, 1967.Google Scholar
  43. 43.
    Pang, J. S.,On the Convergence of Dual Ascent Methods for Large-Scale Linearly Constrained Optimization, University of Texas at Dallas, School of Management, Technical Report, 1984.Google Scholar
  44. 44.
    Bertsekas, D. P., Hosein, P. A., andTseng, P.,Relaxation Methods for Newtork flow Problems with Convex Arc Costs, SIAM Journal on Control and Optimization, Vol. 25, pp. 1219–1243, 1987.Google Scholar
  45. 45.
    Luo, Z. Q., andTseng, P.,On the Convergence of a Matrix-Splitting Algorithm for the Symmetric Monotone Linear Complementarity Problem, SIAM Journal on Control and Optimization, Vol. 29, pp. 1037–1060, 1991.Google Scholar
  46. 46.
    Rockafellar, R. T.,Convex Analysis, Princeton University Press, Princeton, New Jersey, 1970.Google Scholar
  47. 47.
    Ortega, J. M., andRheinboldt, W. C.,Iterative Solution of Nonlinear Equations in Several Variables, Academic Press, New York, New York, 1970.Google Scholar
  48. 48.
    Tseng, P., andBertsekas, D. P.,Relaxation Methods for Problems with Strictly Convex Separable Costs and Linear Constraints, Mathematical Programming, Vol. 38, pp. 303–321, 1987.Google Scholar
  49. 49.
    Hoffman, A. J.,On Approximate Solutions of Systems of Linear Inequalities, Journal of Research of the National Bureau of Standards, Vol. 49, pp. 263–265, 1952.Google Scholar
  50. 50.
    Robinson, S. M.,Bounds for Errors in the Solution Set of a Perturbed Linear Program, Linear Algebra and Its Applications, Vol. 6, pp. 69–81, 1973.Google Scholar
  51. 51.
    Mangasarian, O. L., andShiau, T. H.,Lipschitz Continuity of Solutions of Linear Inequalities: Programs and Complementarity Problems, SIAM Journal on Control and Optimization, Vol. 25, pp. 583–595, 1987.Google Scholar
  52. 52.
    Luo, Z. Q., andTseng, P.,On the Linear Convergence of Descent Methods for Convex Essentially Smooth Minimization, SIAM Journal on Control and Optimization (to appear).Google Scholar
  53. 53.
    Lin, Y. Y., andPang, J. S.,Iterative Methods for Large Convex Quadratic Programs: A Survey, SIAM Journal on Control and Optimization, Vol. 18, pp. 383–411, 1987.Google Scholar
  54. 54.
    Iusem, A. N., andDe Pierro, A. R.,On the Convergence Properties of Hildreth's Quadratic Programming Algorithm, Mathematical Programming, Series A, Vol. 47, pp. 37–51, 1990.Google Scholar
  55. 55.
    De Pierro, A. R., andIusem, A. N.,A Relaxed Version of Bregman's Method for Convex Programming, Journal of Optimization Theory and Applications, Vol. 51, pp. 421–440, 1986.Google Scholar
  56. 56.
    Censor, Y., andLent, A.,An Iterative Row-Action Method for Interval Convex Programming, Journal of Optimization Theory and Applications, Vol. 34, pp. 321–352, 1981.Google Scholar

Copyright information

© Plenum Publishing Corporation 1992

Authors and Affiliations

  • Z. Q. Luo
    • 1
  • P. Tseng
    • 2
  1. 1.Communications Research Laboratory, Department of Electrical and Computer EngineeringMcMaster UniversityHamiltonCanada
  2. 2.Laboratory for Information and Decision SystemsMassachusetts Institute of TechnologyCambridge

Personalised recommendations