Mathematical Programming

, Volume 155, Issue 1–2, pp 57–79 | Cite as

The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent

  • Caihua ChenEmail author
  • Bingsheng He
  • Yinyu Ye
  • Xiaoming Yuan
Full Length Paper Series A


The alternating direction method of multipliers (ADMM) is now widely used in many fields, and its convergence was proved when two blocks of variables are alternatively updated. It is strongly desirable and practically valuable to extend the ADMM directly to the case of a multi-block convex minimization problem where its objective function is the sum of more than two separable convex functions. However, the convergence of this extension has been missing for a long time—neither an affirmative convergence proof nor an example showing its divergence is known in the literature. In this paper we give a negative answer to this long-standing open question: The direct extension of ADMM is not necessarily convergent. We present a sufficient condition to ensure the convergence of the direct extension of ADMM, and give an example to show its divergence.


Alternating direction method of multipliers Convergence analysis Convex programming Splitting methods 

Mathematics Subject Classification

90C25 90C30 65K13 


  1. 1.
    Bertsekas, D.P.: Constrained Optimization and Lagrange Multiplier Methods. Academic Press, London (1982)zbMATHGoogle Scholar
  2. 2.
    Blum, E., Oettli, W.: Mathematische Optimierung. Grundlagen und Verfahren. Ökonometrie und Unternehmensforschung. Springer, Berlin-Heidelberg-New York (1975)Google Scholar
  3. 3.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3, 1–122 (2010)zbMATHCrossRefGoogle Scholar
  4. 4.
    Chan, T.F., Glowinski, R.: Finite Element Approximation and Iterative Solution of a Class of Mildly Non-linear Elliptic Equations, Technical Report. Stanford University, Stanford, CA (1978)Google Scholar
  5. 5.
    Chandrasekaran, V., Parrilo, P.A., Willsky, A.S.: Latent variable graphical model selection via convex optimization. Ann. Stat. 40, 1935–1967 (2012)zbMATHMathSciNetCrossRefGoogle Scholar
  6. 6.
    Eckstein, J., Yao, W.: Augmented Lagrangian and alternating direction methods for convex optimization: A tutorial and some illustrative computational results, manuscript (2012)Google Scholar
  7. 7.
    Eckstein, J., Bertsekas, D.P.: On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992)zbMATHMathSciNetCrossRefGoogle Scholar
  8. 8.
    Fortin, M., Glowinski, R.: On decomposition-coordination methods using an augmented Lagrangian. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary Problems. North- Holland, Amsterdam (1983)Google Scholar
  9. 9.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximations. Comput. Math. Appl. 2, 17–40 (1976)zbMATHCrossRefGoogle Scholar
  10. 10.
    Glowinski, R.: Numerical Methods for Nonlinear Variational Problems. Springer, Berlin (1984)zbMATHCrossRefGoogle Scholar
  11. 11.
    Glowinski, R.: On alternating directon methods of multipliers: a historical perspective. In: Springer Proceedings of a Conference Dedicated to J. Periaux (to appear)Google Scholar
  12. 12.
    Glowinski, R., Marrocco, A.: Approximation par èlèments finis d’ordre un et rèsolution par pènalisation-dualitè d’une classe de problémes non linèaires. R.A.I.R.O. R2, 41–76 (1975)MathSciNetGoogle Scholar
  13. 13.
    Gol’shtein, E.G., Tret’yakov, N.V.: Modified Lagrangian in convex programming and their generalizations. Math. Program. Studies 10, 86–97 (1979)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Han, D.R., Yuan, X.M.: A note on the alternating direction method of multipliers. J. Optim. Theory Appl. 155, 227–238 (2012)zbMATHMathSciNetCrossRefGoogle Scholar
  15. 15.
    He, B.S., Tao, M., Yuan, X.M.: Alternating direction method with Gaussian back substitution for separable convex programming. SIAM J. Optim. 22, 313–340 (2012)zbMATHMathSciNetCrossRefGoogle Scholar
  16. 16.
    He, B. S., Tao, M., Yuan, X. M.: A splitting method for separable convex programming. IMA J. Numer. Anal. (to appear)Google Scholar
  17. 17.
    He, B.S., Tao, M., Yuan, X.M.: Convergence rate and iteration complexity on the alternating direction method of multipliers with a substitution procedure for separable convex programming. Math. Oper. Res. (under revision)Google Scholar
  18. 18.
    He, B.S., Yuan, X.M.: On the \(O(1/n)\) convergence rate of the Douglas–Rachford alternating direction method. SIAM J. Num. Anal. 50, 700–709 (2012)zbMATHMathSciNetCrossRefGoogle Scholar
  19. 19.
    Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)zbMATHMathSciNetCrossRefGoogle Scholar
  20. 20.
    Hong, M., Luo, Z. Q.: On the linear convergence of the alternating direction method of multipliers, manuscript (August 2012)Google Scholar
  21. 21.
    McLachlan, G.J.: Discriminant Analysis and Statistical Pattern Recognition, vol. 544. Wiley-Interscience, New York (2004)zbMATHGoogle Scholar
  22. 22.
    Mohan, K., London, P., Fazel, M., Witten, D., Lee, S.: Node-based learning of multiple gaussian graphical models. arXiv:1303.5145 (2013)
  23. 23.
    Martinet, B.: Regularization d’inequations variationelles par approximations successives. Revue Francaise d’Informatique et de Recherche Opérationelle 4, 154–159 (1970)zbMATHMathSciNetGoogle Scholar
  24. 24.
    Peng, Y.G., Ganesh, A., Wright, J., Xu, W.L., Ma, Y.: Robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Trans. Pattern Anal. Mach. Intel. 34, 2233–2246 (2012)CrossRefGoogle Scholar
  25. 25.
    Powell, M.J.D.: A method for nonlinear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization, pp. 283–298. Academic Press, New York (1969)Google Scholar
  26. 26.
    Rockafellar, R.T.: Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res. 1, 97–116 (1976)zbMATHMathSciNetCrossRefGoogle Scholar
  27. 27.
    Tao, M., Yuan, X.M.: Recovering low-rank and sparse components of matrices from incomplete and noisy observations. SIAM J. Optim. 21, 57–81 (2011)zbMATHMathSciNetCrossRefGoogle Scholar
  28. 28.
    Wen, Z., Goldfarb, D., Yin, W.: Alternating direction augmented lagrangian methods for semidefinite programming. Math. Program. Comput. 2, 203–230 (2010)zbMATHMathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg and Mathematical Optimization Society 2014

Authors and Affiliations

  • Caihua Chen
    • 1
    Email author
  • Bingsheng He
    • 1
    • 2
  • Yinyu Ye
    • 1
    • 3
  • Xiaoming Yuan
    • 4
  1. 1.International Centre of Management Science and Engineering, School of Management and EngineeringNanjing UniversityNanjingChina
  2. 2.Department of MathematicsNanjing UniversityNanjingChina
  3. 3.Department of Management Science and Engineering, School of EngineeringStanford UniversityStanfordUSA
  4. 4.Department of MathematicsHong Kong Baptist UniversityHong KongChina

Personalised recommendations