# Self Equivalence of the Alternating Direction Method of Multipliers

• 3473 Accesses

Part of the Scientific Computation book series (SCIENTCOMP)

## Abstract

The alternating direction method of multipliers (ADM or ADMM) breaks a complex optimization problem into much simpler subproblems. The ADM algorithms are typically short and easy to implement yet exhibit (nearly) state-of-the-art performance for large-scale optimization problems.

To apply ADM, we first formulate a given problem into the “ADM-ready” form, so the final algorithm depends on the formulation. A problem like $$\mathop{\mathrm{minimize}}\limits _{\mathbf{x}}u(\mathbf{x}) + v(\mathbf{C}\mathbf{x})$$ has six different “ADM-ready” formulations. They can be in the primal or dual forms, and they differ by how dummy variables are introduced. To each “ADM-ready” formulation, ADM can be applied in two different orders depending on how the primal variables are updated. Finally, we get twelve different ADM algorithms! How do they compare to each other? Which algorithm should one choose? In this chapter, we show that many of the different ways of applying ADM are equivalent. Specifically, we show that ADM applied to a primal formulation is equivalent to ADM applied to its Lagrange dual; ADM is equivalent to a primal-dual algorithm applied to the saddle-point formulation of the same problem. These results are surprising since the primal and dual variables in ADM are seemingly treated very differently, and some previous work exhibit preferences in one over the other on specific problems.

In addition, when one of the two objective functions is quadratic, possibly subject to an affine constraint, we show that swapping the update order of the two primal variables in ADM gives the same algorithm. These results identify the few truly different ADM algorithms for a problem, which generally have different forms of subproblems from which it is easy to pick one with the most computationally friendly subproblems.

### Keywords

• Dual Problem
• Dual Variable
• Master Problem
• Augmented Lagrangian Method
• Matrix Vector Multiplication

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
• DOI: 10.1007/978-3-319-41589-5_5
• Chapter length: 30 pages
• Own it forever
• Exclusive offer for individuals only
• Tax calculation will be finalised during checkout
eBook
USD   169.00
Price excludes VAT (USA)
• ISBN: 978-3-319-41589-5
• Own it forever
• Exclusive offer for individuals only
• Tax calculation will be finalised during checkout
Softcover Book
USD   219.99
Price excludes VAT (USA)
Hardcover Book
USD   219.99
Price excludes VAT (USA)

## References

1. Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer (2011)

2. Chambolle, A.: An algorithm for total variation minimization and applications. Journal of Mathematical Imaging and Vision 20 (1-2), 89–97 (2004)

3. Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision 40 (1), 120–145 (2011)

4. Davis, D., Yin, W.: Faster convergence rates of relaxed Peaceman-Rachford and ADMM under regularity assumptions. arXiv preprint arXiv:1407.5210 (2014)

5. Davis, D., Yin, W.: Convergence rate analysis of several splitting schemes. In: R. Glowinski, S. Osher, W. Yin (eds.) Splitting Methods in Communication and Imaging, Science and Engineering, Chapter 4 Springer (2016)

6. Deng, W., Yin, W.: On the global and linear convergence of the generalized alternating direction method of multipliers. Journal of Scientific Computing 66 (3), 889–916 (2015)

7. Douglas, J., Rachford, H.H.: On the numerical solution of heat conduction problems in two and three space variables. Transactions of the American Mathematical Society 82 (2), 421–439 (1956)

8. Eckstein, J.: Splitting methods for monotone operators with applications to parallel optimization. Ph.D. thesis, Massachusetts Institute of Technology (1989)

9. Eckstein, J., Fukushima, M.: Some reformulations and applications of the alternating direction method of multipliers. In: Large Scale Optimization, pp. 115–134. Springer (1994)

10. Esser, E., Zhang, X., Chan, T.: A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science. SIAM Journal on Imaging Sciences 3 (4), 1015–1046 (2010)

11. Esser, J.: Primal dual algorithms for convex models and applications to image restoration, registration and nonlocal inpainting. Ph.D. thesis, University of California, Los Angeles (2010)

12. Fukushima, M.: The primal Douglas-Rachford splitting algorithm for a class of monotone mappings with application to the traffic equilibrium problem. Mathematical Programming 72 (1), 1–15 (1996)

13. Gabay, D.: Applications of the method of multipliers to variational inequalities. In: M. Fortin, R. Glowinski (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary-Value Problems. North-Holland: Amsterdam, Amsterdam (1983)

14. Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Computers & Mathematics with Applications 2 (1), 17–40 (1976)

15. Glowinski, R., Marroco, A.: Sur l’approximation, par éléments finis d’ordre un, et la résolution, par pénalisation-dualité d’une classe de problèmes de dirichlet non linéaires. Rev. Française d’Automat. Inf. Recherche Opérationelle 9 (2), 41–76 (1975)

16. Goldstein, T., Osher, S.: The split Bregman method for l1-regularized problems. SIAM Journal on Imaging Sciences 2 (2), 323–343 (2009)

17. Hestenes, M.: Multiplier and gradient methods. Journal of Optimization Theory and Applications 4 (5), 303–320 (1969)

18. Lions, P., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM Journal on Numerical Analysis 16 (6), 964–979 (1979)

19. Peaceman, D.W., Rachford, H.H.: The numerical solution of parabolic and elliptic differential equations. Journal of the Society for Industrial and Applied Mathematics 3 (1), 28–41 (1955)

20. Rockafellar, R.T.: A dual approach to solving nonlinear programming problems by unconstrained optimization. Mathematical Programming 5 (1), 354–373 (1973)

21. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena 60 (1–4), 259–268 (1992)

22. Wang, Y., Yang, J., Yin, W., Zhang, Y.: A new alternating minimization algorithm for total variation image reconstruction. SIAM Journal on Imaging Sciences 1 (3), 248–272 (2008)

23. Xiao, Y., Zhu, H., Wu, S.Y.: Primal and dual alternating direction algorithms for 1- 1-norm minimization problems in compressive sensing. Computational Optimization and Applications 54 (2), 441–459 (2013)

24. Yang, J., Zhang, Y.: Alternating direction algorithms for 1-problems in compressive sensing. SIAM Journal on Scientific Computing 33 (1), 250–278 (2011)

25. Yang, Y., Möller, M., Osher, S.: A dual split Bregman method for fast 1 minimization. Mathematics of Computation 82 (284), 2061–2085 (2013)

## Acknowledgements

This work is supported by NSF Grants DMS-1349855 and DMS-1317602 and ARO MURI Grant W911NF-09-1-0383. We thank Jonathan Eckstein for bringing his early work [8, Chapter 3.5] and [9] to our attention and anonymous reviewers for their helpful comments and suggestions.

## Author information

Authors

### Corresponding author

Correspondence to Ming Yan .

## Rights and permissions

Reprints and Permissions