Skip to main content
SpringerLink
Log in

Analysis of Fully Preconditioned Alternating Direction Method of Multipliers with Relaxation in Hilbert Spaces

  • Regular Paper
  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

Alternating direction method of multipliers is a powerful first-order method for non-smooth optimization problems including various applications in inverse problems and imaging. However, there is no clear result on the weak convergence of alternating direction method of multipliers in infinite-dimensional Hilbert spaces with relaxation. In this paper, by employing a kind of partial gap analysis, we prove the weak convergence of a general preconditioned and relaxed version in infinite-dimensional Hilbert spaces, with preconditioning for solving all the involved implicit equations under mild conditions. We also give the corresponding ergodic convergence rates respecting to the partial gap function. Furthermore, the connections between certain preconditioned and relaxed alternating direction method of multipliers and the corresponding Douglas–Rachford splitting methods are discussed. Numerical tests show the efficiency of the proposed overrelaxation variants with preconditioning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)

    Article  MATH  Google Scholar 

  2. Chambolle, A., Pock, T.: An introduction to continuous optimization for imaging. Acta Numerica 25, 161–319 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  3. Glowinski, R., Marrocco, A.: Sur l’approximation, par éléments finis d’ordre un, et la résolution, par pénalisation-dualité d’une classe de problèmes de Dirichlet non linéaires. ESAIM Math. Model. Numér. 9(2), 41–76 (1975)

    MATH  Google Scholar 

  4. Chan, T., Glowinski, R.: Finite Element Approximation and Iterative Solution of a Class of Mildly Non-linear Elliptic Equations. Stanford University, Stanford (1978)

    Google Scholar 

  5. Glowinski, R., Osher, S., Yin, W. (eds.): Splitting Methods in Communication, Imaging, Science, and Engineering. Springer, Berlin (2016)

    MATH  Google Scholar 

  6. Glowinski, R.: On alternating direction methods of multipliers: a historical perspective. In: Fitzgibbon, W., Kuznetsov, Y.A., Neittaanmäki, P., Pironneau, O. (eds.) Modeling, Simulation and Optimization for Science and Technology. Computational Methods in Applied Sciences, vol. 34, pp. 59–82. Springer, Berlin (2014)

    Google Scholar 

  7. Glowinski, R.: Variational Methods for the Numerical Solution of Nonlinear Elliptic Problems. SIAM, Philadelphia (2015)

    Book  MATH  Google Scholar 

  8. Deng, W., Yin, W.: On the global and linear convergence of the generalized alternating direction method of multipliers. J. Sci. Comput. 66(3), 889–916 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  9. Eckstein, J., Bertsekas, D.P.: On the Douglas–Rachford splitting method and the proximal algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  10. Li, M., Sun, D., Toh, K.C.: A majorized ADMM with indefinite proximal terms for linearly constrained convex composite optimization. SIAM J. Optim. 26(2), 922–950 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  11. Zhang, X., Burger, M., Osher, S.: A unified primal-dual algorithm framework based on Bregman iteration. J. Sci. Comput. 46(1), 20–46 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  12. Monteiro, R.D.C., Svaiter, B.F.: Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers. SIAM J. Optim. 23(1), 475–507 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  13. Fang, E.X., He, B., Liu, H., Yuan, X.: Generalized alternating direction method of multipliers: new theoretical insights and applications. Math. Program. Comput. 7(2), 149–187 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  14. Glowinski, R., Le Tallec, P.: Augmented Lagrangians and Operator-Splitting Methods in Nonlinear Mechanics. SIAM, Philadelphia (1989)

    Book  MATH  Google Scholar 

  15. Glowinski, R.: Numerical Methods for Nonlinear Variational Problems. Springer, New York (1984)

    Book  MATH  Google Scholar 

  16. Bredies, K., Sun, H.: A proximal-point analysis of preconditioned alternating direction method of multipliers. J. Optim. Theory Appl. 173(3), 878–907 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  17. Bourgat, J.F., Dumay, J.M., Glowinski, R.: Large displacement calculations of flexible pipelines by finite element and nonlinear programming methods. SIAM J. Sci. Stat. Comput. 1(1), 34–81 (1980)

    Article  MathSciNet  MATH  Google Scholar 

  18. Bermudez, A., Moreno, C.: Duality methods for solving variational inequalities. Comput. Math. Appl. 7(1), 43–85 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  19. Fortin, M., Glowinski, R.: On decomposition-coordination methods using an augmented Lagrangian. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary Value Problems, pp. 97–146. North-Holland, Amsterdam (1983)

    Chapter  Google Scholar 

  20. Bredies, K., Sun, H.: Accelerated Douglas–Rachford methods for the solution of convex-concave saddle-point problems. arXiv eprints, arXiv:1604.06282

  21. Chambolle, A., Pock, T.: On the ergodic convergence rates of a first-order primal-dual algorithm. Math. Program. 159(1), 253–287 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  22. Attouch, H., Soueycatt, M.: Augmented Lagrangian and proximal alternating direction methods of multipliers in Hilbert spaces. Applications to games PDE’s and control. Pac. J. Optim. 5(1), 17–37 (2008)

    MathSciNet  MATH  Google Scholar 

  23. Gabay, D.: Applications of the method of multipliers to variational inequalities. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary Value Problems, pp. 299–331. North-Holland, Amsterdam (1983)

    Chapter  Google Scholar 

  24. Bredies, K., Sun, H.: Preconditioned Douglas–Rachford algorithms for TV- and TGV-regularized variational imaging problems. J. Math. Imaging Vis. 52(3), 317–344 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  25. Nien, H., Fessler, J.A.: Relaxed linearized algorithms for faster X-ray CT image reconstruction. IEEE Trans. Med. Imaging 35(4), 1090–8 (2016)

    Article  Google Scholar 

  26. Yuan, J., Bae, E., Tai, X.: A study on continuous max-flow and min-cut approaches. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2217–2224 (2010)

  27. Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  28. Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, New York (2011)

    Book  MATH  Google Scholar 

  29. Ito, K., Kunisch, K.: Lagrange Multiplier Approach to Variational Problems and Applications. SIAM, Philadelphia (2008)

    Book  MATH  Google Scholar 

  30. Chambolle A., Pock T.: Diagonal preconditioning for first order primal-dual algorithms in convex optimization. In: Proceedings of the 2011 International Conference on Computer Vision (ICCV), pp. 1762–1769 (2011)

  31. Chen, L., Sun, D., Toh, K.C.: An efficient inexact symmetric Gauss-Seidel based majorized ADMM for high-dimensional convex composite conic programming. Math. Program. 161(1–2), 237–270 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  32. Ouyang, Y., Chen, Y., Lan Jr., G.E.P.: An accelerated linearized alternating direction method of multipliers. SIAM J. Imaging Sci. 8(1), 644–681 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  33. Lions, P., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16(6), 964–979 (1979)

    Article  MathSciNet  MATH  Google Scholar 

  34. Setzer, S.: Split Bregman Algorithm, Douglas–Rachford Splitting and Frame Shrinkage. Scale Space and Variational Methods in Computer Vision, pp. 464–476. Springer, Berlin (2009)

    Google Scholar 

  35. Reed, M., Simon, B.: Methods of Modern Mathematical Physics I: Functional Analysis. Academic Press, San Diego (1980)

    MATH  Google Scholar 

  36. Sun, H.: Analysis of fully preconditioned ADMM with relaxation in Hilbert spaces. arXiv eprints, arXiv:1611.04801 (2016)

  37. Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM J. Imaging Sci. 3(3), 492–526 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  38. Xu, Y.: Accelerated first-order primal-dual proximal methods for linearly constrained composite convex programming. SIAM J. Optim. 27(3), 1459–1484 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  39. Hintermüller, M., Rautenberg, C. N., Hahn, J.: Functional-analytic and numerical issues in splitting methods for total variation-based image reconstruction, Inverse Problems 30, 055014 (34pp) (2014)

Download references

Acknowledgements

The author is very grateful to Kristian Bredies, who suggested numerous improvements during preparing the draft of this article. He also acknowledges the support of NSF of China under grant No.  11701563 and the Fundamental Research Funds for the Central Universities, and the research funds of Renmin University of China (15XNLF20). He also acknowledges the support of Alexander von Humboldt Foundation.

Author information

Authors and Affiliations

  1. Institute for Mathematical Sciences, Renmin University of China, No. 59, Zhongguancun Street, Beijing, China

    Hongpeng Sun

Authors
  1. Hongpeng Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongpeng Sun.

Additional information

Communicated by Roland Glowinski.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, H. Analysis of Fully Preconditioned Alternating Direction Method of Multipliers with Relaxation in Hilbert Spaces. J Optim Theory Appl 183, 199–229 (2019). https://doi.org/10.1007/s10957-019-01535-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-019-01535-6

Keywords

Mathematics Subject Classification

Search

Navigation