Advertisement

A Proximal Point Analysis of the Preconditioned Alternating Direction Method of Multipliers

  • Kristian Bredies
  • Hongpeng SunEmail author
Article

Abstract

We study preconditioned algorithms of alternating direction method of multipliers type for nonsmooth optimization problems. The alternating direction method of multipliers is a popular first-order method for general constrained optimization problems. However, one of its drawbacks is the need to solve implicit subproblems. In various applications, these subproblems are either easily solvable or linear, but nevertheless challenging. We derive a preconditioned version that allows for flexible and efficient preconditioning for these linear subproblems. The original and preconditioned version is written as a new kind of proximal point method for the primal problem, and the weak (strong) convergence in infinite (finite) dimensional Hilbert spaces is proved. Various efficient preconditioners with any number of inner iterations may be used in this preconditioned framework. Furthermore, connections between the preconditioned version and the recently introduced preconditioned Douglas–Rachford method for general nonsmooth problems involving quadratic–linear terms are established. The methods are applied to total variation denoising problems, and their benefits are shown in numerical experiments.

Keywords

Alternating direction method of multipliers Proximal point algorithm Weak convergence analysis Linear preconditioning techniques 

Mathematics Subject Classification

65K10 90C25 65F08 

Notes

Acknowledgements

Kristian Bredies and Hongpeng Sun acknowledge support of the Austrian Science Fund (FWF) under grant SFB32 (SFB “Mathematical Optimization and Applications in the Biomedical Sciences”) during the preparation of this manuscript. Hongpeng Sun also acknowledges the support of Fundamental Research Funds for the Central Universities, and the research funds of Renmin University of China (15XNLF20).

References

  1. 1.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)CrossRefzbMATHGoogle Scholar
  2. 2.
    Eckstein, J., Yao, W.: Understanding the convergence of the alternating direction method of multipliers: theoretical and computational perspectives. Pac. J. Optim. 11(4), 619–644 (2015)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Esser, E.: Application of Lagrangian-based alternating direction methods and connections to split Bregman. CAM Report 09-31, UCLA, Center for Applied Math. (2009)Google Scholar
  4. 4.
    Ramani, S., Fessler, J.A.: A splitting-based iterative algorithm for accelerated statistical X-ray CT reconstruction. IEEE Trans. Med. Imag. 31(3), 677–688 (2012)CrossRefGoogle Scholar
  5. 5.
    Wu, C., Tai, X.C.: Augmented Lagrangian method, dual methods, and split Bregman iteration for ROF, vectorial TV, and high order models. SIAM J. Imaging Sci. 3(3), 300–339 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Fortin, M., Glowinski, R.: On decomposition-coordination methods using an augmented Lagrangian. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary Value Problems, pp. 97–146. North-Holland, Amsterdam (1983)CrossRefGoogle Scholar
  7. 7.
    Glowinski, R., Marrocco, A.: Sur l’approximation, par éléments finis d’ordre un, et la résolution, par pénalisation-dualité d’une classe de problèmes de Dirichlet non linéaires. ESAIM Math. Model. Num. 9(2), 41–76 (1975)zbMATHGoogle Scholar
  8. 8.
    Gabay, D.: Applications of the method of multipliers to variational inequalities. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary Value Problems, pp. 299–331. North-Holland, Amsterdam (1983)CrossRefGoogle Scholar
  9. 9.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput. Math. Appl. 2(1), 17–40 (1976)CrossRefzbMATHGoogle Scholar
  10. 10.
    Mercier, B.: Approximation par éléments finis et résolution par un algorithme de pénalisation-dualité d’un problème d’élasto-plasticité. C.R. Acad. Sci. Math. 280, 287–290 (1975)zbMATHGoogle Scholar
  11. 11.
    Glowinski, R.: Variational Methods for the Numerical Solution of Nonlinear Elliptic Problems. SIAM, Philadelphia, PA (2015)CrossRefzbMATHGoogle Scholar
  12. 12.
    Glowinski, R.: On alternating direction methods of multipliers: a historical perspective. In: Fitzgibbon, W., Kuznetsov, Y.A., Neittaanmäki, P., Pironneau, O. (eds.) Modeling, Simulation and Optimization for Science and Technology, vol. 34 of the series Computational Methods in Applied Sciences, pp. 59–82. Springer, Berlin (2014)Google Scholar
  13. 13.
    Chan, T., Glowinski, R.: Finite Element Approximation and Iterative Solution of a Class of Mildly Non-Linear Elliptic Equations. Computer Science Department, Stanford University, Stanford (1978)Google Scholar
  14. 14.
    Glowinski, R., Osher, S., Yin, W. (eds.): Splitting Methods in Communication, Imaging, Science, and Engineering. Springer, Berlin (2016)zbMATHGoogle Scholar
  15. 15.
    Glowinski, R., Le Tallec, P.: Augmented Lagrangians and Operator-Splitting Methods in Nonlinear Mechanics. SIAM, Philadelphia, PA (1989)CrossRefzbMATHGoogle Scholar
  16. 16.
    Eckstein, J., Bertsekas, D.P.: On the Douglas-Rachford splitting method and the proximal algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Lions, P., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16(6), 964–979 (1979)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40, 120–145 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Li, M., Sun, D., Toh, K.C.: A majorized ADMM with indefinite proximal terms for linearly constrained convex composite optimization. SIAM J. Optim. 26(2), 922–950 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Zhang, X., Burger, M., Osher, S.: A unified primal-dual algorithm framework based on Bregman iteration. J. Sci. Comput. 46(1), 20–46 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Bredies, K., Sun, H.: Preconditioned Douglas–Rachford algorithms for TV- and TGV-regularized variational imaging problems. J. Math. Imaging Vis. 52(3), 317–344 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Bredies, K., Sun, H.: Preconditioned Douglas–Rachford splitting methods for convex–concave saddle-point problems. SIAM J. Numer. Anal. 53(1), 421–444 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Deng, W., Yin, W.: On the global and linear convergence of the generalized alternating direction method of multipliers. J. Sci. Comput. 66(3), 889–916 (2016)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Fang, E.X., He, B., Liu, H., Yuan, X.: Generalized alternating direction method of multipliers: new theoretical insights and applications. Math. Prog. Comp. 7(2), 149–187 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Monteiro, R.D.C., Svaiter, B.F.: Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers. SIAM J. Optim. 23(1), 475–507 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Shefi, R., Teboulle, M.: Rate of convergence analysis of decomposition methods based on the proximal method of multipliers for convex minimization. SIAM J. Optim. 24(1), 269–297 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Attouch, H., Soueycatt, M.: Augmented Lagrangian and proximal alternating direction methods of multipliers in Hilbert spaces. Applications to games, PDE’s and control. Pac. J. Optim. 5(1), 17–37 (2008)MathSciNetzbMATHGoogle Scholar
  28. 28.
    Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, New York (2011)CrossRefzbMATHGoogle Scholar
  29. 29.
    Ito, K., Kunisch, K.: Lagrange Multiplier Approach to Variational Problems and Applications. SIAM, Philadelphia (2008)CrossRefzbMATHGoogle Scholar
  30. 30.
    Reed, M., Simon, B.: Methods of Modern Mathematical Physics I: Functional Analysis. Academic Press, San Diego (1980)zbMATHGoogle Scholar
  31. 31.
    Opial, Z.: Weak convergence of the sequence of successive approximations for nonexpansive mappings. Bull. Amer. Math. Soc. 73(4), 591–597 (1967)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Esser, E., Zhang, X., Chan, T.: A general framework for a class of first order primal–dual algorithms for convex optimization in imaging science. SIAM J. Imaging Sci. 3(4), 1015–1046 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Goldstein, T., Osher, S.: The split Bregman method for \(L^1\) regularized problems. SIAM J. Imaging Sci. 2(2), 323–343 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Setzer, S.: Split Bregman algorithm, Douglas–Rachford splitting and frame shrinkage. In: Tai, X.C., Mørken, K., Lysaker, M., Lie, K.A. (eds.) Scale Space and Variational Methods in Computer Vision, pp. 464–476. Springer, Berlin (2009)CrossRefGoogle Scholar
  35. 35.
    Yan, M., Yin, W.: Self equivalence of the alternating direction method of multipliers. arXiv eprints, 1407:7400 (2014)Google Scholar
  36. 36.
    Chambolle A., Pock T.: Diagonal preconditioning for first order primal-dual algorithms in convex optimization. In: Proceedings of the 2011 International Conference on Computer Vision (ICCV), pp. 1762–1769 (2011)Google Scholar
  37. 37.
    Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Ekeland, I., Témam, R.: Convex Analysis and Variational Problems. SIAM, Philadelphia (1999)CrossRefzbMATHGoogle Scholar
  39. 39.
    Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward–backward splitting. Multiscale Model. Sim. 4(4), 1168–1200 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  40. 40.
    Bredies, K., Sun, H.: Accelerated Douglas–Rachford methods for the solution of convex-concave saddle-point problems. arXiv eprints, 1604:06282 (2016)Google Scholar
  41. 41.
    Xu, Y.: Accelerated first-order primal-dual proximal methods for linearly constrained composite convex programming. arXiv eprints, 1606:09155 (2016)Google Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Institute for Mathematics and Scientific ComputingUniversity of GrazGrazAustria
  2. 2.Institute for Mathematical SciencesRenmin University of ChinaBeijingChina

Personalised recommendations