Skip to main content

Global Convergence of Unmodified 3-Block ADMM for a Class of Convex Minimization Problems

Journal of Scientific Computing volume 76, pages 69–88 (2018)Cite this article

Abstract

The alternating direction method of multipliers (ADMM) has been successfully applied to solve structured convex optimization problems due to its superior practical performance. The convergence properties of the 2-block ADMM have been studied extensively in the literature. Specifically, it has been proven that the 2-block ADMM globally converges for any penalty parameter \(\gamma >0\). In this sense, the 2-block ADMM allows the parameter to be free, i.e., there is no need to restrict the value for the parameter when implementing this algorithm in order to ensure convergence. However, for the 3-block ADMM, Chen et al. (Math Program 155:57–79, 2016) recently constructed a counter-example showing that it can diverge if no further condition is imposed. The existing results on studying further sufficient conditions on guaranteeing the convergence of the 3-block ADMM usually require \(\gamma \) to be smaller than a certain bound, which is usually either difficult to compute or too small to make it a practical algorithm. In this paper, we show that the 3-block ADMM still globally converges with any penalty parameter \(\gamma >0\) if the third function \(f_3\) in the objective is smooth and strongly convex, and its condition number is in [1, 1.0798), besides some other mild conditions. This requirement covers an important class of problems to be called regularized least squares decomposition (RLSD) in this paper.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

References

  1. Boley, D.: Local linear convergence of the alternating direction method of multipliers on quadratic or linear programs. SIAM J. Optim. 23(4), 2183–2207 (2013)

    MathSciNet  Article  MATH  Google Scholar 

  2. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)

    Article  MATH  Google Scholar 

  3. Cai, X., Han, D., Yuan, X.: On the convergence of the direct extension of ADMM for three-block separable convex minimization models with one strongly convex function. Comput. Optim. Appl. 66(1), 39–73 (2017)

    MathSciNet  Article  MATH  Google Scholar 

  4. Chen, C., He, B., Ye, Y., Yuan, X.: The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent. Math. Program. 155, 57–79 (2016)

    MathSciNet  Article  MATH  Google Scholar 

  5. Chen, C., Shen, Y., You, Y.: On the convergence analysis of the alternating direction method of multipliers with three blocks. Abstr. Appl. Anal. Article ID 183961 (2013)

  6. Davis, D., Yin, W.: A three-operator splitting scheme and its optimization applications, pp. 15–13. Technical Report, UCLA CAM Report (2015)

  7. Deng, W., Lai, M., Peng, Z., Yin, W.: Parallel multi-block ADMM with \(o(1/k)\) convergence. J. Sci. Comput. 71(2), 712–736 (2017)

    MathSciNet  Article  MATH  Google Scholar 

  8. Deng, W., Yin, W.: On the global and linear convergence of the generalized alternating direction method of multipliers. J. Sci. Comput. 66(3), 889–916 (2016)

    MathSciNet  Article  MATH  Google Scholar 

  9. Eckstein, J., Bertsekas, D.P.: On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992)

    MathSciNet  Article  MATH  Google Scholar 

  10. Fortin, M., Glowinski, R.: Augmented Lagrangian Methods: Applications to the Numerical Solution of Boundary-Value Problems. North-Holland, Amsterdam (1983)

    MATH  Google Scholar 

  11. Gabay, D.: Applications of the method of multipliers to variational inequalities. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary Value Problems. North-Holland, Amsterdam (1983)

    Google Scholar 

  12. Glowinski, R., Le Tallec, P.: Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics. SIAM, Philadelphia (1989)

    Book  MATH  Google Scholar 

  13. Han, D., Yuan, X.: A note on the alternating direction method of multipliers. J. Optim. Theory Appl. 155(1), 227–238 (2012)

    MathSciNet  Article  MATH  Google Scholar 

  14. He, B., Hou, L., Yuan, X.: On full Jacobian decomposition of the augmented Lagrangian method for separable convex programming. SIAM J. Optim. 25(4), 2274–2312 (2015)

    MathSciNet  Article  MATH  Google Scholar 

  15. He, B., Tao, M., Yuan, X.: Alternating direction method with Gaussian back substitution for separable convex programming. SIAM J. Optim. 22, 313–340 (2012)

    MathSciNet  Article  MATH  Google Scholar 

  16. He, B., Tao, M., Yuan, X.: A splitting method for separable convex programming. IMA J. Numer. Anal. 35(1), 394–426 (2015)

    MathSciNet  Article  MATH  Google Scholar 

  17. He, B., Tao, M., Yuan, X.: Convergence rate and iteration complexity on the alternating direction method of multipliers with a substitution procedure for separable convex programming. Math. Oper. Res. 42(3), 662–691 (2017)

    MathSciNet  Article  MATH  Google Scholar 

  18. He, B., Yuan, X.: On the \({O}(1/n)\) convergence rate of Douglas-Rachford alternating direction method. SIAM J. Numer. Anal. 50, 700–709 (2012)

    MathSciNet  Article  MATH  Google Scholar 

  19. Hong, M., Chang, T.-H., Wang, X., Razaviyayn, M., Ma, S., Luo, Z.-Q.: A block successive upper bound minimization method of multipliers for linearly constrained convex optimization. arXiv preprint arXiv:1401.7079 (2014)

  20. Hong, M., Luo, Z.-Q.: On the linear convergence of the alternating direction method of multipliers. Math. Program. 162(1), 165–199 (2017)

    MathSciNet  Article  MATH  Google Scholar 

  21. Hong, M., Luo, Z.-Q., Razaviyayn, M.: Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems. SIAM J. Optim. 26(1), 337–364 (2016)

    MathSciNet  Article  MATH  Google Scholar 

  22. Li, G., Pong, T.: Global convergence of splitting methods for nonconvex composite optimization. SIAM J. Optim. 25, 2434–2460 (2015)

    MathSciNet  Article  MATH  Google Scholar 

  23. Li, M., Sun, D., Toh, K.-C.: A convergent 3-block semi-proximal ADMM for convex minimization problems with one strongly convex block. Asia Pac. J. Oper. Res. 32(3), 1550024 (2015)

    MathSciNet  Article  MATH  Google Scholar 

  24. Li, X., Ng, M.K., Yuan, X.: Median filtering-based methods for static background extraction from surveillance video. Numer. Linear Algebra Appl. 22(5), 845–865 (2015)

    MathSciNet  Article  MATH  Google Scholar 

  25. Lin, T., Ma, S., Zhang, S.: On the global linear convergence of the ADMM with multiblock variables. SIAM J. Optim. 25(3), 1478–1497 (2015)

    MathSciNet  Article  MATH  Google Scholar 

  26. Lin, T., Ma, S., Zhang, S.: On the sublinear convergence rate of multi-block ADMM. J. Oper. Res. Soc. China 3(3), 251–274 (2015)

    MathSciNet  Article  MATH  Google Scholar 

  27. Lin, T., Ma, S., Zhang, S.: Iteration complexity analysis of multi-block ADMM for a family of convex minimization without strong convexity. J. Sci. Comput. 69, 52–81 (2016)

    MathSciNet  Article  MATH  Google Scholar 

  28. Lin, Z., Chen, M., Wu, L., Ma, Y.: The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. UIUC Technical Report UILU-ENG-09-2215. arXiv:1009.5055v2 (2009)

  29. Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964–979 (1979)

    MathSciNet  Article  MATH  Google Scholar 

  30. Ma, S., Johnson, D., Ashby, C., Xiong, D., Cramer, C.L., Moore, J.H., Zhang, S., Huang, X.: SPARCoC: a new framework for molecular pattern discovery and cancer gene identification. PLoS ONE 10(3), e0117135 (2015)

    Article  Google Scholar 

  31. Monteiro, R.D.C., Svaiter, B.F.: Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers. SIAM J. Optim. 23, 475–507 (2013)

    MathSciNet  Article  MATH  Google Scholar 

  32. Peng, Y., Ganesh, A., Wright, J., Xu, W., Ma, Y.: RASL: robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2233–2246 (2012)

    Article  Google Scholar 

  33. Sun, D., Toh, K.-C., Yang, L.: A convergent 3-block semiproximal alternating direction method of multipliers for conic programming with 4-type constraints. SIAM J. Optim. 25, 882–915 (2015)

    MathSciNet  Article  MATH  Google Scholar 

  34. Sun, R., Luo, Z.-Q., Ye, Y.: On the expected convergence of randomly permuted ADMM. https://arxiv.org/abs/1503.06387 (2015)

  35. Tao, M., Yuan, X.: Recovering low-rank and sparse components of matrices from incomplete and noisy observations. SIAM J. Optim. 21, 57–81 (2011)

    MathSciNet  Article  MATH  Google Scholar 

  36. Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc B 58(1), 267–288 (1996)

    MathSciNet  MATH  Google Scholar 

  37. Wang, X., Hong, M., Ma, S., Luo, Z.-Q.: Solving multiple-block separable convex minimization problems using two-block alternating direction method of multipliers. Pac. J. Optim. 11(4), 645–667 (2015)

    MathSciNet  MATH  Google Scholar 

  38. Waters, A., Sankaranarayanan, A., Baraniuk, R.: Sparcs: recovering low-rank and sparse matrices from compressive measurements. In: NIPS (2011)

  39. Wen, Z., Goldfarb, D., Yin, W.: Alternating direction augmented Lagrangian methods for semidefinite programming. Math. Program. Comput. 2, 203–230 (2010)

    MathSciNet  Article  MATH  Google Scholar 

  40. Wright, J., Ganesh, A., Min, K., Ma, Y.: Compressive principal component pursuit. Inf. Inference 2(1), 32–68 (2013)

    Google Scholar 

  41. Yang, J., Zhang, Y.: Alternating direction algorithms for \(\ell _1\) problems in compressive sensing. SIAM J. Sci. Comput. 33(1), 250–278 (2011)

    MathSciNet  Article  Google Scholar 

  42. Zhou, Z., Li, X., Wright, J., Candès, E.J., Ma, Y.: Stable principal component pursuit. In: Proceedings of International Symposium on Information Theory (2010)

Download references

Acknowledgements

The research of Shiqian Ma is supported in part by a startup package in Department of Mathematics at UC Davis. The research of Shuzhong Zhang is supported in part by NSF Grant with Grant No. CMMI-1462408.

Author information

Authors and Affiliations

  1. Department of Industrial Engineering and Operations Research, UC Berkeley, Berkeley, CA, 94704, USA

    Tianyi Lin

  2. Department of Mathematics, UC Davis, Davis, CA, 95616, USA

    Shiqian Ma

  3. Department of Industrial and Systems Engineering, University of Minnesota, Minneapolis, MN, 55455, USA

    Shuzhong Zhang

Authors
  1. Tianyi Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shiqian Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shuzhong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shiqian Ma.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lin, T., Ma, S. & Zhang, S. Global Convergence of Unmodified 3-Block ADMM for a Class of Convex Minimization Problems. J Sci Comput 76, 69–88 (2018). https://doi.org/10.1007/s10915-017-0612-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10915-017-0612-7

Keywords

  • ADMM
  • Global convergence
  • Convex minimization
  • Regularized least squares decomposition