Advertisement

Journal of Scientific Computing

, Volume 66, Issue 3, pp 1204–1217 | Cite as

On the Proximal Jacobian Decomposition of ALM for Multiple-Block Separable Convex Minimization Problems and Its Relationship to ADMM

  • Bingsheng He
  • Hong-Kun Xu
  • Xiaoming YuanEmail author
Article

Abstract

The augmented Lagrangian method (ALM) is a benchmark for solving convex minimization problems with linear constraints. When the objective function of the model under consideration is representable as the sum of some functions without coupled variables, a Jacobian or Gauss–Seidel decomposition is often implemented to decompose the ALM subproblems so that the functions’ properties could be used more effectively in algorithmic design. The Gauss–Seidel decomposition of ALM has resulted in the very popular alternating direction method of multipliers (ADMM) for two-block separable convex minimization models and recently it was shown in He et al. (Optimization Online, 2013) that the Jacobian decomposition of ALM is not necessarily convergent. In this paper, we show that if each subproblem of the Jacobian decomposition of ALM is regularized by a proximal term and the proximal coefficient is sufficiently large, the resulting scheme to be called the proximal Jacobian decomposition of ALM, is convergent. We also show that an interesting application of the ADMM in Wang et al. (Pac J Optim, to appear), which reformulates a multiple-block separable convex minimization model as a two-block counterpart first and then applies the original ADMM directly, is closely related to the proximal Jacobian decomposition of ALM. Our analysis is conducted in the variational inequality context and is rooted in a good understanding of the proximal point algorithm.

Keywords

Convex optimization Alternating direction method of multipliers Augmented Lagrangian method Jacobian decomposition Parallel computation Proximal point algorithm Variational inequality problem 

References

  1. 1.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3, 1–122 (2010)CrossRefzbMATHGoogle Scholar
  2. 2.
    Chen, C.H., He, B.S., Ye, Y.Y., Yuan, X.M.: The direct extension of ADMM for multi-block minimization problems is not necessarily convergent. Math. Program. Ser. A. doi: 10.1007/s10107-014-0826-5
  3. 3.
    Eckstein, J., Yao, W.: Augmented Lagrangian and alternating direction methods for convex optimization: a tutorial and some illustrative computational results. Pac. J. Optim. (to appear)Google Scholar
  4. 4.
    Glowinski, R.: On alternating directon methods of multipliers: a historical perspective. In: Springer Proceedings of a Conference Dedicated to J. Periaux (to appear)Google Scholar
  5. 5.
    Glowinski, R., Marrocco, A.: Approximation par éléments finis d’ordre un et résolution par pénalisation-dualité d’une classe de problèmes non linéaires, R.A.I.R.O., R2 (1975), pp. 41–76Google Scholar
  6. 6.
    Gu, G.Y., He, B.S., Yuan, X.M.: Customized proximal point algorithms for linearly constrained convex minimization and saddle-point problems: a unified approach. Comput. Optim. Appl. 59, 135–161 (2014)CrossRefMathSciNetzbMATHGoogle Scholar
  7. 7.
    Güler, O.: On the convergence of the proximal point algorithm for convex minimization. SIAM J. Control Optim. 29(2), 403–419 (1991)CrossRefMathSciNetzbMATHGoogle Scholar
  8. 8.
    Han, D.R., Yuan, X.M., Zhang, W.X.: An augmented-Lagrangian-based parallel splitting method for separable convex programming with applications to image processing. Math. Comput. 83, 2263–2291 (2014)CrossRefMathSciNetzbMATHGoogle Scholar
  9. 9.
    He, B.S.: Parallel splitting augmented Lagrangian methods for monotone structured variational inequalities. Comput. Optim. Appl. 42, 195–212 (2009)CrossRefMathSciNetzbMATHGoogle Scholar
  10. 10.
    He, B.S., Hou, L.S., Yuan, X.M.: On full Jacobian decomposition of the augmented Lagrangian method for separable convex programming. Optimization Online (2013). http://www.optimization-online.org/
  11. 11.
    He, B.S., Tao, M., Yuan, X.M.: Alternating direction method with Gaussian back substitution for separable convex programming. SIAM J. Optim. 22, 313–340 (2012)CrossRefMathSciNetzbMATHGoogle Scholar
  12. 12.
    He, B.S., Tao, M., Yuan, X.M.: A splitting method for separable convex programming. IMA J. Numer. Anal. 35, 394–426 (2015)CrossRefMathSciNetzbMATHGoogle Scholar
  13. 13.
    He, B.S., Tao, M., Yuan, X.M.: Convergence rate and iteration complexity on the alternating direction method of multipliers with a substitution procedure for separable convex programming. Math. Oper. Res. (under revision)Google Scholar
  14. 14.
    Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)CrossRefMathSciNetzbMATHGoogle Scholar
  15. 15.
    Martinet, B.: Regularision d’inéquations variationnelles par approximations successive. Revue Fr d’Autom Inf Rech Opér 126, 154–159 (1970)MathSciNetGoogle Scholar
  16. 16.
    Ng, M.K., Yuan, X.M., Zhang, W.X.: A coupled variational image decomposition and restoration model for blurred cartoon-plus-texture images with missing pixels. IEEE Trans. Image Sci. 22(6), 2233–2246 (2013)CrossRefMathSciNetGoogle Scholar
  17. 17.
    Powell, M.J.D.: A method for nonlinear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization, pp. 283–298. Academic Press, New York (1969)Google Scholar
  18. 18.
    Rockafellar, R.T.: Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res. 1, 97–116 (1976)CrossRefMathSciNetzbMATHGoogle Scholar
  19. 19.
    Tao, M., Yuan, X.M.: Recovering low-rank and sparse components of matrices from incomplete and noisy observations. SIAM J. Optim. 21, 57–81 (2011)CrossRefMathSciNetzbMATHGoogle Scholar
  20. 20.
    Wang, X.F., Hong, M.Y., Ma, S.Q., Luo, Z.Q.: Solving multiple-block separable convex minimizaion problems using two-block alternating direction method of multipliers. Pac. J. Optim. (to appear)Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Department of MathematicsSouth University of Science and Technology of ChinaShenzhenChina
  2. 2.Department of MathematicsNanjing UniversityNanjingChina
  3. 3.Department of Mathematics, School of ScienceHangzhou Dianzi UniversityHangzhouChina
  4. 4.Department of Applied MathematicsNational Sun Yet-sen UniversityKaohsiungTaiwan
  5. 5.Department of MathematicsHong Kong Baptist UniversityHong KongChina

Personalised recommendations