Mathematical Programming

, Volume 155, Issue 1–2, pp 57–79

# The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent

Full Length Paper Series A

## Abstract

The alternating direction method of multipliers (ADMM) is now widely used in many fields, and its convergence was proved when two blocks of variables are alternatively updated. It is strongly desirable and practically valuable to extend the ADMM directly to the case of a multi-block convex minimization problem where its objective function is the sum of more than two separable convex functions. However, the convergence of this extension has been missing for a long time—neither an affirmative convergence proof nor an example showing its divergence is known in the literature. In this paper we give a negative answer to this long-standing open question: The direct extension of ADMM is not necessarily convergent. We present a sufficient condition to ensure the convergence of the direct extension of ADMM, and give an example to show its divergence.

## Keywords

Alternating direction method of multipliers Convergence analysis Convex programming Splitting methods

## Mathematics Subject Classification

90C25 90C30 65K13

## References

1. 1.
Bertsekas, D.P.: Constrained Optimization and Lagrange Multiplier Methods. Academic Press, London (1982)
2. 2.
Blum, E., Oettli, W.: Mathematische Optimierung. Grundlagen und Verfahren. Ökonometrie und Unternehmensforschung. Springer, Berlin-Heidelberg-New York (1975)Google Scholar
3. 3.
Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3, 1–122 (2010)
4. 4.
Chan, T.F., Glowinski, R.: Finite Element Approximation and Iterative Solution of a Class of Mildly Non-linear Elliptic Equations, Technical Report. Stanford University, Stanford, CA (1978)Google Scholar
5. 5.
Chandrasekaran, V., Parrilo, P.A., Willsky, A.S.: Latent variable graphical model selection via convex optimization. Ann. Stat. 40, 1935–1967 (2012)
6. 6.
Eckstein, J., Yao, W.: Augmented Lagrangian and alternating direction methods for convex optimization: A tutorial and some illustrative computational results, manuscript (2012)Google Scholar
7. 7.
Eckstein, J., Bertsekas, D.P.: On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992)
8. 8.
Fortin, M., Glowinski, R.: On decomposition-coordination methods using an augmented Lagrangian. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary Problems. North- Holland, Amsterdam (1983)Google Scholar
9. 9.
Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximations. Comput. Math. Appl. 2, 17–40 (1976)
10. 10.
Glowinski, R.: Numerical Methods for Nonlinear Variational Problems. Springer, Berlin (1984)
11. 11.
Glowinski, R.: On alternating directon methods of multipliers: a historical perspective. In: Springer Proceedings of a Conference Dedicated to J. Periaux (to appear)Google Scholar
12. 12.
Glowinski, R., Marrocco, A.: Approximation par èlèments finis d’ordre un et rèsolution par pènalisation-dualitè d’une classe de problémes non linèaires. R.A.I.R.O. R2, 41–76 (1975)
13. 13.
Gol’shtein, E.G., Tret’yakov, N.V.: Modified Lagrangian in convex programming and their generalizations. Math. Program. Studies 10, 86–97 (1979)
14. 14.
Han, D.R., Yuan, X.M.: A note on the alternating direction method of multipliers. J. Optim. Theory Appl. 155, 227–238 (2012)
15. 15.
He, B.S., Tao, M., Yuan, X.M.: Alternating direction method with Gaussian back substitution for separable convex programming. SIAM J. Optim. 22, 313–340 (2012)
16. 16.
He, B. S., Tao, M., Yuan, X. M.: A splitting method for separable convex programming. IMA J. Numer. Anal. (to appear)Google Scholar
17. 17.
He, B.S., Tao, M., Yuan, X.M.: Convergence rate and iteration complexity on the alternating direction method of multipliers with a substitution procedure for separable convex programming. Math. Oper. Res. (under revision)Google Scholar
18. 18.
He, B.S., Yuan, X.M.: On the $$O(1/n)$$ convergence rate of the Douglas–Rachford alternating direction method. SIAM J. Num. Anal. 50, 700–709 (2012)
19. 19.
Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)
20. 20.
Hong, M., Luo, Z. Q.: On the linear convergence of the alternating direction method of multipliers, manuscript (August 2012)Google Scholar
21. 21.
McLachlan, G.J.: Discriminant Analysis and Statistical Pattern Recognition, vol. 544. Wiley-Interscience, New York (2004)
22. 22.
Mohan, K., London, P., Fazel, M., Witten, D., Lee, S.: Node-based learning of multiple gaussian graphical models. arXiv:1303.5145 (2013)
23. 23.
Martinet, B.: Regularization d’inequations variationelles par approximations successives. Revue Francaise d’Informatique et de Recherche Opérationelle 4, 154–159 (1970)
24. 24.
Peng, Y.G., Ganesh, A., Wright, J., Xu, W.L., Ma, Y.: Robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Trans. Pattern Anal. Mach. Intel. 34, 2233–2246 (2012)
25. 25.
Powell, M.J.D.: A method for nonlinear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization, pp. 283–298. Academic Press, New York (1969)Google Scholar
26. 26.
Rockafellar, R.T.: Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res. 1, 97–116 (1976)
27. 27.
Tao, M., Yuan, X.M.: Recovering low-rank and sparse components of matrices from incomplete and noisy observations. SIAM J. Optim. 21, 57–81 (2011)
28. 28.
Wen, Z., Goldfarb, D., Yin, W.: Alternating direction augmented lagrangian methods for semidefinite programming. Math. Program. Comput. 2, 203–230 (2010)

© Springer-Verlag Berlin Heidelberg and Mathematical Optimization Society 2014

## Authors and Affiliations

• Caihua Chen
• 1
Email author
• Bingsheng He
• 1
• 2
• Yinyu Ye
• 1
• 3
• Xiaoming Yuan
• 4
1. 1.International Centre of Management Science and Engineering, School of Management and EngineeringNanjing UniversityNanjingChina
2. 2.Department of MathematicsNanjing UniversityNanjingChina
3. 3.Department of Management Science and Engineering, School of EngineeringStanford UniversityStanfordUSA
4. 4.Department of MathematicsHong Kong Baptist UniversityHong KongChina