Skip to main content
SpringerLink
Log in

Convergence analysis of the direct extension of ADMM for multiple-block separable convex minimization

  • Published:
Advances in Computational Mathematics Aims and scope Submit manuscript

Abstract

Recently, the alternating direction method of multipliers (ADMM) has found many efficient applications in various areas; and it has been shown that the convergence is not guaranteed when it is directly extended to the multiple-block case of separable convex minimization problems where there are m ≥ 3 functions without coupled variables in the objective. This fact has given great impetus to investigate various conditions on both the model and the algorithm’s parameter that can ensure the convergence of the direct extension of ADMM (abbreviated as “e-ADMM”). Despite some results under very strong conditions (e.g., at least (m − 1) functions should be strongly convex) that are applicable to the generic case with a general m, some others concentrate on the special case of m = 3 under the relatively milder condition that only one function is assumed to be strongly convex. We focus on extending the convergence analysis from the case of m = 3 to the more general case of m ≥ 3. That is, we show the convergence of e-ADMM for the case of m ≥ 3 with the assumption of only (m − 2) functions being strongly convex; and establish its convergence rates in different scenarios such as the worst-case convergence rates measured by iteration complexity and the globally linear convergence rate under stronger assumptions. Thus the convergence of e-ADMM for the general case of m ≥ 4 is proved; this result seems to be still unknown even though it is intuitive given the known result of the case of m = 3. Even for the special case of m = 3, our convergence results turn out to be more general than the existing results that are derived specifically for the case of m = 3.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ash, J.M., Berele, A., Catoiu, S.: Plausible and Genuine Extensions of L’Hospital’s Rule. Math. Mag. 85, 52–60 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  2. Bhatia, R., Elsner, L., Krause, G.: Bounds for the Variation of the Roots of a Polynomial and the Eigenvalues of a Matrix. Linear Algebra Appl. 142, 195–209 (1990)

    Article  MathSciNet  MATH  Google Scholar 

  3. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach Learn. 3, 1–122 (2010)

    Article  MATH  Google Scholar 

  4. Cai, X.J., Han, D.R., Yuan, X.M.: The direct extension of ADMM for three-block separable convex minimization models is convergent when one function is strongly convex. Comput. Optim. Appl. 66, 39–73 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  5. Chen, C.H., Shen, Y., You, Y.F.: On the convergence analysis of the alternating direction method of multipliers with three blocks. Abstr. Appl. Anal., 2013, Article ID 183961, 7 pages

  6. Chen, C.H., He, B.S., Ye, Y.Y., Yuan, X.M.: The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent. Math. Program., Ser. A 155, 57–79 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  7. Davis, D., Yin, W.T.: A three-operator splitting scheme and its optimization applications, available at https://arxiv.org/pdf/1504.01032v1.pdf

  8. Eckstein, J., Yao, W.: Understanding the convergence of the alternating direction method of multipliers: Theoretical and computational perspectives. Pacific. J. Optim. 11, 619–644 (2015)

    MathSciNet  MATH  Google Scholar 

  9. Facchinei, F., Pang, J.-S.: Finite-Dimensional Variational Inequalities and Complementarity problems, Volume I, Springer Series in Operations Research, Springer-Verlag (2003)

  10. Gabay, D.: Applications of the method of multipliers to variational inequalities. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary-Valued Problems, pp 299–331. North-Holland Publishing Co., Amsterdam (1983)

  11. Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput. Math. Appl. 2, 17–40 (1976)

    Article  MATH  Google Scholar 

  12. Garling, D.J.H.: Inequalities: A Journey into Linear Analysis. Cambridge University, United Kindom (2007)

    Book  MATH  Google Scholar 

  13. Glowinski, R., Marrocco, A.: Sur l’approximation paréléments finis d’ordre un et larésolution par pénalisation-dualité d’une classe de problèmes de Dirichlet non linéaires. Revue Fr. Autom. Inform. Rech. Opér., Anal Numér 2, 41–76 (1975)

    Google Scholar 

  14. Glowinski, R.: On alternating direction methods of multipliers: A historical perspective. In: Fitzgibbon, W., Kuznetsov, Y.A., Neittaanm ki, P., Pironneau, O. (eds.) Modeling, Simulation and Optimization for Science and Technology, pp 59–82. Springer, Netherlands (2014)

  15. Han, D.R., Yuan, X.M.: A note on the alternating direction method of multipliers. J. Optim. Theory Appli. 155, 227–238 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  16. He, B.S., Yuan, X.M.: On the O(1/n) convergence rate of Douglas-Rachford alternating direction method. SIAM J. Numer. Anal. 50, 700–709 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  17. He, B.S., Yuan, X.M.: On nonergodic convergence rate of Douglas-Rachford alternating direction method of multipliers. Numer. Math. 130, 567–577 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  18. Hong, M., Luo, Z.Q.: On the linear convergence of alternating direction method of multipliers. Math. Program. 162, 165–199 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  19. Li, M., Sun, D.F., Toh, K.C.: A convergent 3-block semi-proximal ADMM for convex minimization problems with one strongly convex block. Asia Pac. J. Oper. Res. 32(1550024), 19 (2015)

    MathSciNet  MATH  Google Scholar 

  20. Lin, T.Y., Ma, S.Q., Zhang, S.Z.: On the sublinear convergence rate of multi-block ADMM. J. Oper. Res. Soc. China 3, 251–274 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  21. Lin, T.Y., Ma, S.Q., Zhang, S.Z.: On the global linear convergence of the ADMM with multi-block variables. SIAM J. Optim. 25, 1478–1497 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  22. Peng, Y.G., Ganesh, A., Wright, J., Xu, W.L., Ma, Y.: Robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Tran. Pattern Anal. Mach. Intel. 34, 2233–2246 (2012)

    Article  Google Scholar 

  23. Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)

    Book  MATH  Google Scholar 

  24. Sun, D.F., Toh, K.C., Yang, L.Q.: A convergent 3-block semi-proximal alternating direction method of multipliers for conic programming with 4-type constraints. SIAM J. Optim. 25, 882–915 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  25. Tao, M., Yuan, X.M.: Recovering low-rank and sparse components of matrices from incomplete and noisy observations. SIAM J. Optim. 21, 57–81 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  26. Wang, X.F., Hong, M.Y., Chang, T.H., Razaviyayn, M., Luo, Z. Q.: Joint day-ahead power procurement and load scheduling using stochastic alternating direction method of multipliers. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp 7754–7758 (2014)

  27. Zhou, X., Yang, C., Yu, W.: Moving object detection by detecting contiguous outliers in the Low-Rank representation. IEEE Trans. Pattern Anal. Mach. Intell. 35, 597–610 (2013)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, 210093, China

    Min Tao

  2. Department of Mathematics, Hong Kong Baptist University, Hong Kong, China

    Xiaoming Yuan

Authors
  1. Min Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoming Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Min Tao.

Additional information

Communicated by: Karsten Urban

M. Tao: This author was supported by the Natural Science Foundation of China: NSFC-11301280 and the sponsorship of Jiangsu overseas research and training program for university prominent young and middle-aged teachers and presidents.

X. Yuan: This author was supported by the General Research Fund from Hong Kong Research Grants Council: 12313516.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tao, M., Yuan, X. Convergence analysis of the direct extension of ADMM for multiple-block separable convex minimization. Adv Comput Math 44, 773–813 (2018). https://doi.org/10.1007/s10444-017-9560-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10444-017-9560-x

Keywords

Mathematics Subject Classification (2010)

Search

Navigation