On the Sublinear Convergence Rate of Multi-block ADMM

Abstract

The alternating direction method of multipliers (ADMM) is widely used in solving structured convex optimization problems. Despite its success in practice, the convergence of the standard ADMM for minimizing the sum of \(N\,(N\geqslant 3)\) convex functions, whose variables are linked by linear constraints, has remained unclear for a very long time. Recently, Chen et al. (Math Program, doi:10.1007/s10107-014-0826-5, 2014) provided a counter-example showing that the ADMM for \(N\geqslant 3\) may fail to converge without further conditions. Since the ADMM for \(N\geqslant 3\) has been very successful when applied to many problems arising from real practice, it is worth further investigating under what kind of sufficient conditions it can be guaranteed to converge. In this paper, we present such sufficient conditions that can guarantee the sublinear convergence rate for the ADMM for \(N\geqslant 3\). Specifically, we show that if one of the functions is convex (not necessarily strongly convex) and the other N-1 functions are strongly convex, and the penalty parameter lies in a certain region, the ADMM converges with rate O(1 / t) in a certain ergodic sense and o(1 / t) in a certain non-ergodic sense, where t denotes the number of iterations. As a by-product, we also provide a simple proof for the O(1 / t) convergence rate of two-block ADMM in terms of both objective error and constraint violation, without assuming any condition on the penalty parameter and strong convexity on the functions.

This is a preview of subscription content, log in to check access.

Notes

  1. 1.

    Preprint available at http://arxiv.org/abs/1408.4265.

References

  1. 1.

    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite-element approximations. Comp. Math. Appl. 2, 17–40 (1976)

    Article  MATH  Google Scholar 

  2. 2.

    Glowinski, R., Marrocco, A.: Sur l’approximation par éléments finis et la résolution par pénalisation-dualité d’une classe de problèmes de Dirichlet non linéaires. Revue Française d’Automatique, Informatique, Recherche Operationnelle, Serie Rouge (AnalyseNumérique), R-2,pp. 41–76 (1975)

  3. 3.

    Douglas, J., Rachford, H.H.: On the numerical solution of the heat conduction problem in 2 and 3 space variables. Trans. Am. Math. Soc. 82, 421–439 (1956)

    MathSciNet  Article  MATH  Google Scholar 

  4. 4.

    Peaceman, D.H., Rachford, H.H.: The numerical solution of parabolic elliptic differential equations. SIAM J. Appl. Math. 3, 28–41 (1955)

    MathSciNet  Article  MATH  Google Scholar 

  5. 5.

    Eckstein, J.: Splitting methods for monotone operators with applications to parallel optimization. PhD thesis, Massachusetts Institute of Technology (1989)

  6. 6.

    Fortin, M., Glowinski, R.: Augmented Lagrangian Methods: Applications to the Numerical Solution of Boundary-Value Problems. North-Holland Pub, Co, Amsterdam (1983)

    Google Scholar 

  7. 7.

    Glowinski, R., Le Tallec, P.: Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics. SIAM, Philadelphia (1989)

    Google Scholar 

  8. 8.

    Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964–979 (1979)

    MathSciNet  Article  MATH  Google Scholar 

  9. 9.

    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)

    Article  Google Scholar 

  10. 10.

    Eckstein, J.: Augmented Lagrangian and alternating direction methods for convex optimization: A tutorial and some illustrative computational results (2012). Preprint http://www.optimization-online.org/DB_HTML/2012/12/3704.html

  11. 11.

    Eckstein, J., Bertsekas, D.P.: On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992)

    MathSciNet  Article  MATH  Google Scholar 

  12. 12.

    Gabay, D.: Applications of the method of multipliers to variational inequalities. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary Value Problems. North-Holland, Amsterdam (1983)

    Google Scholar 

  13. 13.

    Boley, D.: Local linear convergence of the alternating direction method of multipliers on quadratic or linear programs. SIAM J. Optim. 23(4), 2183–2207 (2013)

    MathSciNet  Article  MATH  Google Scholar 

  14. 14.

    Davis, D., Yin, W.: Faster convergence rates of relaxed Peaceman–Rachford and ADMM under regularity assumptions. Technical report, UCLA CAM Report 14–58 (2014)

  15. 15.

    Deng, W., Yin, W.: On the global and linear convergence of the generalized alternating direction method of multipliers. J. Sci. Comput. (2015). doi:10.1007/s10915-015-0048-x

  16. 16.

    He, B., Yuan, X.: On the \({O}(1/n)\) convergence rate of Douglas–Rachford alternating direction method. SIAM J. Numer. Anal. 50, 700–709 (2012)

    MathSciNet  Article  MATH  Google Scholar 

  17. 17.

    He, B., Yuan, X.: On nonergodic convergence rate of Douglas–Rachford alternating direction method of multipliers. Numerische Mathematik 130(3), 567–577 (2015)

    MathSciNet  Article  Google Scholar 

  18. 18.

    Monteiro, R.D.C., Svaiter, B.F.: Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers. SIAM J. Optim. 23, 475–507 (2013)

    MathSciNet  Article  MATH  Google Scholar 

  19. 19.

    Chen, C., He, B., Ye, Y., Yuan, X.: The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent. Math. Program. (2014). doi:10.1007/s10107-014-0826-5

  20. 20.

    Peng, Y., Ganesh, A., Wright, J., Xu, W., Ma, Y.: RASL: Robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2233–2246 (2012)

    Article  Google Scholar 

  21. 21.

    Tao, M., Yuan, X.: Recovering low-rank and sparse components of matrices from incomplete and noisy observations. SIAM J. Optim. 21, 57–81 (2011)

    MathSciNet  Article  MATH  Google Scholar 

  22. 22.

    Sun, D., Toh, K.-C., Yang, L.: A convergent 3-block semiproximal alternating direction method of multipliers for conic programming with 4-type constraints. SIAM J. Optim. 25, 882–915 (2015)

    MathSciNet  Article  Google Scholar 

  23. 23.

    Wang, X., Hong, M., Ma, S., Luo, Z.-Q.: Solving multiple-block separable convex minimization problems using two-block alternating direction method of multipliers (2013). Preprint arXiv:1308.5294

  24. 24.

    Han, D., Yuan, X.: A note on the alternating direction method of multipliers. J. Optim. Theory Appl. 155(1), 227–238 (2012)

    MathSciNet  Article  MATH  Google Scholar 

  25. 25.

    Chen, C., Shen, Y., You, Y.: On the convergence analysis of the alternating direction method of multipliers with three blocks. Abstract and Applied Analysis, Article ID 183961 (2013). doi:10.1155/2013/183961

  26. 26.

    Cai, X., Han, D., Yuan, X.: The direct extension of ADMM for three-block separable convex minimization models is convergent when one function is strongly convex (2014). Preprint http://www.optimization-online.org/DB_HTML/2014/11/4644.html

  27. 27.

    Li, M., Sun, D., Toh, K.C.: A convergent 3-block semi-proximal ADMM for convex minimization problems with one strongly convex block. Asia-Pacific J. Oper. Res. 32(3), 1550024 (2015)

    MathSciNet  Article  Google Scholar 

  28. 28.

    Davis, D., Yin, W.: A three-operator splitting scheme and its optimization applications. Technical report, UCLA CAM Report 15–13 (2015)

  29. 29.

    Lin, T., Ma, S., Zhang, S.: Iteration complexity analysis of multi-block ADMM for a family of convex minimization without strong convexity (2015). Preprint arXiv:1504.03087

  30. 30.

    Lin, T., Ma, S., Zhang, S.: Global convergence of unmodified 3-block ADMM for a class of convex minimization problems (2015). Preprint arXiv:1505.04252

  31. 31.

    Hong, M., Luo, Z.: On the linear convergence of the alternating direction method of multipliers (2012). Preprint arXiv:1208.3922

  32. 32.

    Deng, W., Lai, M., Peng, Z., Yin, W.: Parallel multi-block ADMM with \(o(1/k)\) convergence (2013). Preprint arXiv:1312.3040

  33. 33.

    He, B., Hou, L., Yuan, X.: On full Jacobian decomposition of the augmented Lagrangian method for separable convex programming (2013). Preprint http://www.optimization-online.org/DB_HTML/2013/05/3894.html

  34. 34.

    He, B., Tao, M., Yuan, X.: Alternating direction method with Gaussian back substitution for separable convex programming. SIAM J. Optim. 22, 313–340 (2012)

    MathSciNet  Article  MATH  Google Scholar 

  35. 35.

    B. He, M. Tao, and X. Yuan. Convergence rate and iteration complexity on the alternating direction method of multipliers with a substitution procedure for separable convex programming (2013). Preprint http://www.optimization-online.org/DB_FILE/2012/09/3611.pdf

  36. 36.

    Hong, M., Chang, T.-H., Wang, X., Razaviyayn, M., Ma, S., Luo, Z.-Q.: A block successive upper bound minimization method of multipliers for linearly constrained convex optimization (2014). Preprint arXiv:1401.7079

  37. 37.

    Lin, T., Ma, S., Zhang, S.: On the global linear convergence of the ADMM with multi-block variables. SIAM J. Optim., to appear (2015)

Download references

Acknowledgments

We would like to thank the editor and the anonymous referees for carefully reading this paper and for insightful comments.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Shi-Qian Ma.

Additional information

The research of S.-Q. Ma was supported in part by the Hong Kong Research Grants Council General Research Fund Early Career Scheme (No. CUHK 439513). The research of S.-Z. Zhang was supported in part by the National Natural Science Foundation (No. CMMI 1161242).

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lin, T., Ma, S. & Zhang, S. On the Sublinear Convergence Rate of Multi-block ADMM. J. Oper. Res. Soc. China 3, 251–274 (2015). https://doi.org/10.1007/s40305-015-0092-0

Download citation

Keywords

  • Alternating direction method of multipliers
  • Sublinear convergence rate
  • Convex optimization

Mathematics Subject Classification

  • 90C25
  • 90C30