# On the convergence of the coordinate descent method for convex differentiable minimization

- 2.6k Downloads
- 195 Citations

## Abstract

The coordinate descent method enjoys a long history in convex differentiable minimization. Surprisingly, very little is known about the convergence of the iterates generated by this method. Convergence typically requires restrictive assumptions such as that the cost function has bounded level sets and is in some sense strictly convex. In a recent work, Luo and Tseng showed that the iterates are convergent for the symmetric monotone linear complementarity problem, for which the cost function is convex quadratic, but not necessarily strictly convex, and does not necessarily have bounded level sets. In this paper, we extend these results to problems for which the cost function is the composition of an affine mapping with a strictly convex function which is twice differentiable in its effective domain. In addition, we show that the convergence is at least linear. As a consequence of this result, we obtain, for the first time, that the dual iterates generated by a number of existing methods for matrix balancing and entropy optimization are linearly convergent.

## Key Words

Coordinate descent convex differentiable optimization symmetric linear complementarity problems## Preview

Unable to display preview. Download preview PDF.

## References

- 1.Mangasarian, O. L.,
*Sparsity-Preserving SOR Algorithms for Separable Quadratic and Linear Programming*, Computers and Operations Research, Vol. 11, pp. 105–112, 1984.Google Scholar - 2.Mangasarian, O. L., andDe Leone, R.,
*Parallel Gradient Projection Successive Overrelaxation for Symmetric Linear Complementarity Problems and Linear Programs*, Annals of Operations Research, Vol. 14, pp. 41–59, 1988.Google Scholar - 3.Herman, G. T., andLent, A.,
*A Family of Iterative Quadratic Optimization Algorithms for Pairs of Inequalities, with Application in Diagnostic Radiology*, Mathematical Programming Study, Vol. 9, pp. 15–29, 1978.Google Scholar - 4.Cottle, R. W., andGoheen, M. S.,
*A Special Class of Large Quadratic Programs*, Nonlinear Programming 3, Edited by O. L. Mangasarian, R. R. Meyer, and S. M. Robinson, Academic Press, New York, New York, pp. 361–390, 1978.Google Scholar - 5.Cottle, R. W., Golub, G. H., andSacher, R. S.,
*On the Solution of Large, Structured Linear Complementarity Problems: The Block-Partitioned Case*, Applied Mathematics and Optimization, Vol. 4, pp. 347–363, 1978.Google Scholar - 6.Dembo, R. S., andTulowitzki, U.,
*On the Minimization of Quadratic Functions Subject to Box Constraints*, Yale University, School of Organization and Management, Working Paper Series B, No. 78, 1983; revised 1984.Google Scholar - 7.Arimoto, S.,
*An Algorithm for Computing the Capacity of Arbitrary DMCs*, IEEE Transactions on Information Theory, Vol. IT-18, pp. 14–20, 1972.Google Scholar - 8.Blahut, R.,
*Computation of Channel Capacity and Rate Distortion Functions*, IEEE Transactions on Information Theory, Vol. IT-18, pp. 460–473, 1972.Google Scholar - 9.Kruithof, J.,
*Calculation of Telephone Traffic*, De Ingenieur (E. Electrotechnik 3), Vol. 52, pp. E15–25, 1937.Google Scholar - 10.Lamond, B., andStewart, N. F.,
*Bregman's Balancing Method*, Transportation Research, Vol. 15B, pp. 239–248, 1981.Google Scholar - 11.Censor, Y.,
*Parallel Application of Block-Iterative Methods in Medical Imaging and Radiation Therapy*, Mathematical Programming, Series B, Vol. 42, pp. 307–325, 1988.Google Scholar - 12.Lent, A.,
*A Convergent Algorithm for Maximum Entropy Image Restoration with a Medical X-Ray Application*, Image Analysis and Evaluation, Edited by R. Shaw, Society of Photographic Scientists and Engineers, Washington, DC, pp. 249–257, 1977.Google Scholar - 13.Powell, M. J. D.,
*An Algorithm for Maximizing Entropy Subject to Simple Bounds*, Mathematical Programming, Vol. 42, pp. 171–180, 1988.Google Scholar - 14.Frieden, B. R.,
*Image Enhancement and Restoration*, Picture Processing and Digital Filtering, Edited by T. S. Huang, Springer-Verlag, Berlin, Germany, pp. 177–248, 1975.Google Scholar - 15.Jaynes, E. T.,
*On the Rationale of Maximum Entropy Methods*, Proceedings of the IEEE, Vol. 70, pp. 939–952, 1982.Google Scholar - 16.Johnson, R., andShore, J. E.,
*Which Is the Better Entropy Expression for Speech Processing—S log S or log S?*, IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. ASSP-32, pp. 129–136, 1984.Google Scholar - 17.Darroch, J. N., andRatcliff, D.,
*Generalized Iterative Scaling for Log-Linear Models*, Annals of Mathematical Statistics, Vol. 43, pp. 1470–1480, 1972.Google Scholar - 18.Freund, R. M.,
*Projective Transformations for Interior Point Methods, Part 1: Basic Theory and Linear Programming*, Massachusetts Institute of Technology, Operations Research Center, Working Paper No. 180–88, 1988.Google Scholar - 19.Huard, P.,
*Resolution of Mathematical Programming with Nonlinear Constraints by the Method of Centers*, Nonlinear Programming, Edited by J. Abadie, North-Holland, Amsterdam, Holland, pp. 207–219, 1967.Google Scholar - 20.Karmarkar, N.,
*A New Polynomial-Time Algorithm for Linear Programming*, Combinatorica, Vol. 4, pp. 373–395, 1984.Google Scholar - 21.Sonnevend, G.,
*New Algorithms in Convex Programming Based on a Notion of Center for Systems of Analytic Inequalities and on Rational Extrapolation*, Trends in Mathematical Optimization, Edited by K. H. Hoffman, J. B. Hiriart-Urruty, J. Zowe, and C. Lemarechal, Birkhäuser-Verlag, Basel, Switzerland, pp. 311–326, 1988.Google Scholar - 22.Hildreth, C.,
*A Quadratic Programming Procedure*, Naval Research Logistics Quarterly, Vol. 4, pp. 79–85, 1957; see also*Erratum*, Vol. 4, p. 361, 1957.Google Scholar - 23.Cottle, R. W., andPang, J.-S.,
*On the Convergence of a Block Successive Overrelaxation Method for a Class of Linear Complementarity Problems*, Mathematical Programming Study, Vol. 17, pp. 126–138, 1982.Google Scholar - 24.Cryer, C. W.,
*The Solution of a Quadratic Programming Problem Using Systematic Overrelaxation*, SIAM Journal on Control and Optimization, Vol. 9, pp. 385–392, 1971.Google Scholar - 25.
- 26.Bregman, L. M.,
*Proof of the Convergence of Sheleikhovskii's Method for a Problem with Transportation Constraints*, USSR Computational Mathematics and Mathematical Physics, Vol. 1, pp. 191–204, 1967.Google Scholar - 27.Schneider, M. H., andZenios, S. A.,
*A Comparative Study of Algorithms for Matrix Balancing*, Operations Research, Vol. 38, pp. 439–455, 1990.Google Scholar - 28.Zenios, S. A., andIu, S. L.,
*Vector and Parallel Computing for Matrix Balancing*, Annals of Operations Research, Vol. 22, pp. 161–180, 1990.Google Scholar - 29.Censor, Y., andLent, A.,
*Optimization of log x Entropy over Linear Equality Constraints*, SIAM Journal on Control and Optimization, Vol. 25, pp. 921–933, 1987.Google Scholar - 30.
- 31.Bertsekas, D. P., andTsitsiklis, J. N.,
*Parallel and Distributed Computation: Numerical Methods*, Prentice-Hall, Englewood Cliffs, New Jersey, 1989.Google Scholar - 32.D'esopo, D. A.,
*A Convex Programming Procedure*, Naval Research Logistics Quarterly, Vol. 6, pp. 33–42, 1959.Google Scholar - 33.Glowinski, R.,
*Numerical Methods for Nonlinear Variational Problems*, Springer-Verlag, New York, New York, 1984.Google Scholar - 34.Luenberger, D. G.,
*Linear and Nonlinear Programming*, Addison-Wesley, Reading, Massachusetts, 1973.Google Scholar - 35.Polak, E.,
*Computational Methods in Optimization: A Unified Approach*, Academic Press, New York, New York, 1971.Google Scholar - 36.Powell, M. J. D.,
*On Search Directions for Minimization Algorithms*, Mathematical Programming, Vol. 4, pp. 193–201, 1973.Google Scholar - 37.Sargent, R. W. H., andSebastian, D. J.,
*On the Convergence of Sequential Minimization Algorithms*, Journal of Optimization Theory and Applications, Vol. 12, pp. 567–575, 1973.Google Scholar - 38.Zangwill, W. I.,
*Nonlinear Programming: A Unified Approach*, Prentice-Hall, Englewood Cliffs, New Jersey, 1969.Google Scholar - 39.Tseng, P.,
*Dual Ascent Methods for Problems with Strictly Convex Costs and Linear Constraints: A Unified Approach*, SIAM Journal on Control and Optimization, Vol. 28, pp. 214–242, 1990.Google Scholar - 40.Tseng, P.,
*Descent Methods for Convex Essentially Smooth Optimization*, Journal of Optimization Theory and Applications, Vol. 70, pp. 109–135, 1991.Google Scholar - 41.Tseng, P., andBertsekas, D. P.,
*Relaxation Methods for Problems with Strictly Convex Costs and Linear Constraints*, Mathematics of Operations Research, Vol. 16, pp. 462–481, 1991.Google Scholar - 42.Bregman, L. M.,
*The Relaxation Method of Finding the Common Point Convex Sets and Its Application to the Solution of Problems in Convex Programming*, USSR Computational Mathematics and Mathematical Physics, Vol. 7, pp. 200–217, 1967.Google Scholar - 43.Pang, J. S.,
*On the Convergence of Dual Ascent Methods for Large-Scale Linearly Constrained Optimization*, University of Texas at Dallas, School of Management, Technical Report, 1984.Google Scholar - 44.Bertsekas, D. P., Hosein, P. A., andTseng, P.,
*Relaxation Methods for Newtork flow Problems with Convex Arc Costs*, SIAM Journal on Control and Optimization, Vol. 25, pp. 1219–1243, 1987.Google Scholar - 45.Luo, Z. Q., andTseng, P.,
*On the Convergence of a Matrix-Splitting Algorithm for the Symmetric Monotone Linear Complementarity Problem*, SIAM Journal on Control and Optimization, Vol. 29, pp. 1037–1060, 1991.Google Scholar - 46.Rockafellar, R. T.,
*Convex Analysis*, Princeton University Press, Princeton, New Jersey, 1970.Google Scholar - 47.Ortega, J. M., andRheinboldt, W. C.,
*Iterative Solution of Nonlinear Equations in Several Variables*, Academic Press, New York, New York, 1970.Google Scholar - 48.Tseng, P., andBertsekas, D. P.,
*Relaxation Methods for Problems with Strictly Convex Separable Costs and Linear Constraints*, Mathematical Programming, Vol. 38, pp. 303–321, 1987.Google Scholar - 49.Hoffman, A. J.,
*On Approximate Solutions of Systems of Linear Inequalities*, Journal of Research of the National Bureau of Standards, Vol. 49, pp. 263–265, 1952.Google Scholar - 50.Robinson, S. M.,
*Bounds for Errors in the Solution Set of a Perturbed Linear Program*, Linear Algebra and Its Applications, Vol. 6, pp. 69–81, 1973.Google Scholar - 51.Mangasarian, O. L., andShiau, T. H.,
*Lipschitz Continuity of Solutions of Linear Inequalities: Programs and Complementarity Problems*, SIAM Journal on Control and Optimization, Vol. 25, pp. 583–595, 1987.Google Scholar - 52.Luo, Z. Q., andTseng, P.,
*On the Linear Convergence of Descent Methods for Convex Essentially Smooth Minimization*, SIAM Journal on Control and Optimization (to appear).Google Scholar - 53.Lin, Y. Y., andPang, J. S.,
*Iterative Methods for Large Convex Quadratic Programs: A Survey*, SIAM Journal on Control and Optimization, Vol. 18, pp. 383–411, 1987.Google Scholar - 54.Iusem, A. N., andDe Pierro, A. R.,
*On the Convergence Properties of Hildreth's Quadratic Programming Algorithm*, Mathematical Programming, Series A, Vol. 47, pp. 37–51, 1990.Google Scholar - 55.De Pierro, A. R., andIusem, A. N.,
*A Relaxed Version of Bregman's Method for Convex Programming*, Journal of Optimization Theory and Applications, Vol. 51, pp. 421–440, 1986.Google Scholar - 56.Censor, Y., andLent, A.,
*An Iterative Row-Action Method for Interval Convex Programming*, Journal of Optimization Theory and Applications, Vol. 34, pp. 321–352, 1981.Google Scholar