A Proximal Point Analysis of the Preconditioned Alternating Direction Method of Multipliers
- 474 Downloads
We study preconditioned algorithms of alternating direction method of multipliers type for nonsmooth optimization problems. The alternating direction method of multipliers is a popular first-order method for general constrained optimization problems. However, one of its drawbacks is the need to solve implicit subproblems. In various applications, these subproblems are either easily solvable or linear, but nevertheless challenging. We derive a preconditioned version that allows for flexible and efficient preconditioning for these linear subproblems. The original and preconditioned version is written as a new kind of proximal point method for the primal problem, and the weak (strong) convergence in infinite (finite) dimensional Hilbert spaces is proved. Various efficient preconditioners with any number of inner iterations may be used in this preconditioned framework. Furthermore, connections between the preconditioned version and the recently introduced preconditioned Douglas–Rachford method for general nonsmooth problems involving quadratic–linear terms are established. The methods are applied to total variation denoising problems, and their benefits are shown in numerical experiments.
KeywordsAlternating direction method of multipliers Proximal point algorithm Weak convergence analysis Linear preconditioning techniques
Mathematics Subject Classification65K10 90C25 65F08
Kristian Bredies and Hongpeng Sun acknowledge support of the Austrian Science Fund (FWF) under grant SFB32 (SFB “Mathematical Optimization and Applications in the Biomedical Sciences”) during the preparation of this manuscript. Hongpeng Sun also acknowledges the support of Fundamental Research Funds for the Central Universities, and the research funds of Renmin University of China (15XNLF20).
- 3.Esser, E.: Application of Lagrangian-based alternating direction methods and connections to split Bregman. CAM Report 09-31, UCLA, Center for Applied Math. (2009)Google Scholar
- 12.Glowinski, R.: On alternating direction methods of multipliers: a historical perspective. In: Fitzgibbon, W., Kuznetsov, Y.A., Neittaanmäki, P., Pironneau, O. (eds.) Modeling, Simulation and Optimization for Science and Technology, vol. 34 of the series Computational Methods in Applied Sciences, pp. 59–82. Springer, Berlin (2014)Google Scholar
- 13.Chan, T., Glowinski, R.: Finite Element Approximation and Iterative Solution of a Class of Mildly Non-Linear Elliptic Equations. Computer Science Department, Stanford University, Stanford (1978)Google Scholar
- 35.Yan, M., Yin, W.: Self equivalence of the alternating direction method of multipliers. arXiv eprints, 1407:7400 (2014)Google Scholar
- 36.Chambolle A., Pock T.: Diagonal preconditioning for first order primal-dual algorithms in convex optimization. In: Proceedings of the 2011 International Conference on Computer Vision (ICCV), pp. 1762–1769 (2011)Google Scholar
- 40.Bredies, K., Sun, H.: Accelerated Douglas–Rachford methods for the solution of convex-concave saddle-point problems. arXiv eprints, 1604:06282 (2016)Google Scholar
- 41.Xu, Y.: Accelerated first-order primal-dual proximal methods for linearly constrained composite convex programming. arXiv eprints, 1606:09155 (2016)Google Scholar