Advertisement

Journal of Scientific Computing

, Volume 76, Issue 3, pp 1698–1717 | Cite as

A New Primal–Dual Algorithm for Minimizing the Sum of Three Functions with a Linear Operator

  • Ming Yan
Article

Abstract

In this paper, we propose a new primal–dual algorithm for minimizing \(f({\mathbf {x}})+g({\mathbf {x}})+h({\mathbf {A}}{\mathbf {x}})\), where f, g, and h are proper lower semi-continuous convex functions, f is differentiable with a Lipschitz continuous gradient, and \({\mathbf {A}}\) is a bounded linear operator. The proposed algorithm has some famous primal–dual algorithms for minimizing the sum of two functions as special cases. E.g., it reduces to the Chambolle–Pock algorithm when \(f=0\) and the proximal alternating predictor–corrector when \(g=0\). For the general convex case, we prove the convergence of this new algorithm in terms of the distance to a fixed point by showing that the iteration is a nonexpansive operator. In addition, we prove the O(1 / k) ergodic convergence rate in the primal–dual gap. With additional assumptions, we derive the linear convergence rate in terms of the distance to the fixed point. Comparing to other primal–dual algorithms for solving the same problem, this algorithm extends the range of acceptable parameters to ensure its convergence and has a smaller per-iteration cost. The numerical experiments show the efficiency of this algorithm.

Keywords

Fixed-point iteration Nonexpansive operator Chambolle–Pock Primal–dual Three-operator splitting 

Notes

Acknowledgements

The Authors would like to thank the anonymous reviewers for their helpful comments.

References

  1. 1.
    Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, Berlin (2011)CrossRefzbMATHGoogle Scholar
  2. 2.
    Baytas, I.M., Yan, M., Jain, A.K., Zhou, J.: Asynchronous multi-task learning. In: IEEE International Conference on Data Mining (ICDM), pp. 11–20. IEEE (2016)Google Scholar
  3. 3.
    Briceno-Arias, L.M.: Forward-Douglas–Rachford splitting and forward-partial inverse method for solving monotone inclusions. Optimization 64(5), 1239–1261 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Chambolle, A., Pock, T.: A first-order primal–dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Chambolle, A., Pock, T.: An introduction to continuous optimization for imaging. Acta Numer. 25, 161–319 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Chambolle, A., Pock, T.: On the ergodic convergence rates of a first-order primal–dual algorithm. Math. Program. 159(1–2), 253–287 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Chen, P., Huang, J., Zhang, X.: A primal–dual fixed point algorithm for convex separable minimization with applications to image restoration. Inverse Probl. 29(2), 025011 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Chen, P., Huang, J., Zhang, X.: A primal–dual fixed point algorithm for minimization of the sum of three convex separable functions. Fixed Point Theory Appl. 2016(1), 1–18 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Combettes, P.L., Condat, L., Pesquet, J.C., Vu, B.C.: A forward–backward view of some primal–dual optimization methods in image recovery. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 4141–4145 (2014)Google Scholar
  10. 10.
    Combettes, P.L., Pesquet, J.C.: Primal–dual splitting algorithm for solving inclusions with mixtures of composite, lipschitzian, and parallel-sum type monotone operators. Set Valued Var. Anal. 20(2), 307–330 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Condat, L.: A direct algorithm for 1-d total variation denoising. IEEE Signal Process. Lett. 20(11), 1054–1057 (2013)CrossRefGoogle Scholar
  12. 12.
    Condat, L.: A primal–dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms. J. Optim. Theory Appl. 158(2), 460–479 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Davis, D.: Convergence rate analysis of primal–dual splitting schemes. SIAM J. Optim. 25(3), 1912–1943 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Davis, D., Yin, W.: Convergence rate analysis of several splitting schemes. In: Glowinski, R., Osher, S.J., Yin, W. (eds.) Splitting Methods in Communication, Imaging, Science, and Engineering, pp. 115–163. Springer, Cham (2016)CrossRefGoogle Scholar
  15. 15.
    Davis, D., Yin, W.: A three-operator splitting scheme and its optimization applications. Set Valued Var. Anal. 25(4), 829–858 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Douglas, J., Rachford, H.H.: On the numerical solution of heat conduction problems in two and three space variables. Trans. Am. Math. Soc. 82, 421–489 (1956)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Drori, Y., Sabach, S., Teboulle, M.: A simple algorithm for a class of nonsmooth convexconcave saddle-point problems. Oper. Res. Lett. 43(2), 209–214 (2015)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Esser, E., Zhang, X., Chan, T.F.: A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science. SIAM J. Imaging Sci. 3(4), 1015–1046 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Gabay, D.: Applications of the method of multipliers to variational inequalities. Stud. Math. Appl. 15, 299–331 (1983)Google Scholar
  20. 20.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput. Math. Appl. 2(1), 17–40 (1976)CrossRefzbMATHGoogle Scholar
  21. 21.
    Glowinski, R., Marroco, A.: Sur l’approximation, par éléments finis d’ordre un, et la résolution, par pénalisation-dualité d’une classe de problèmes de dirichlet non linéaires. Revue française d’automatique, informatique, recherche opérationnelle. Analyse numérique 9(2), 41–76 (1975)Google Scholar
  22. 22.
    He, B., You, Y., Yuan, X.: On the convergence of primal–dual hybrid gradient algorithm. SIAM J. Imaging Sci. 7(4), 2526–2537 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Komodakis, N., Pesquet, J.C.: Playing with duality: an overview of recent primal-dual approaches for solving large-scale optimization problems. IEEE Signal Process. Mag. 32(6), 31–54 (2015)CrossRefGoogle Scholar
  24. 24.
    Krol, A., Li, S., Shen, L., Xu, Y.: Preconditioned alternating projection algorithms for maximum a posteriori ECT reconstruction. Inverse Probl. 28(11), 115005 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Latafat, P., Patrinos, P.: Asymmetric forward–backward-adjoint splitting for solving monotone inclusions involving three operators. Comput. Optim. Appl. 68(1), 57–93 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Lorenz, D.A., Pock, T.: An inertial forward–backward algorithm for monotone inclusions. J. Math. Imaging Vis. 51(2), 311–325 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Loris, I., Verhoeven, C.: On a generalization of the iterative soft-thresholding algorithm for the case of non-separable penalty. Inverse Probl. 27(12), 125007 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    O’Connor, D., Vandenberghe, L.: On the equivalence of the primal-dual hybrid gradient method and Douglas–Rachford splitting. optimization-online.org (2017)Google Scholar
  29. 29.
    Passty, G.B.: Ergodic convergence to a zero of the sum of monotone operators in Hilbert space. J. Math. Anal. Appl. 72(2), 383–390 (1979)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Peaceman, D.W., Rachford, H.H.: The numerical solution of parabolic and elliptic differential equations. J. Soc. Ind. Appl. Math. 3(1), 28–41 (1955)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Peng, Z., Wu, T., Xu, Y., Yan, M., Yin, W.: Coordinate friendly structures, algorithms and applications. Ann. Math. Sci. Appl. 1(1), 57–119 (2016)zbMATHGoogle Scholar
  32. 32.
    Pock, T., Cremers, D., Bischof, H., Chambolle, A.: An algorithm for minimizing the Mumford–Shah functional. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 1133–1140. IEEE (2009)Google Scholar
  33. 33.
    Tibshirani, R., Saunders, M., Rosset, S., Zhu, J., Knight, K.: Sparsity and smoothness via the fused lasso. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 67(1), 91–108 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Vũ, B.C.: A splitting algorithm for dual monotone inclusions involving cocoercive operators. Adv. Comput. Math. 38(3), 667–681 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Yan, M., Yin, W.: Self equivalence of the alternating direction method of multipliers. In: Glowinski, R., Osher, S.J., Yin, W. (eds.) Splitting Methods in Communication, Imaging, Science, and Engineering, pp. 165–194. Springer, Cham (2016)CrossRefGoogle Scholar
  36. 36.
    Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 67(2), 301–320 (2005)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computational Mathematics, Science and EngineeringMichigan State UniversityEast LansingUSA
  2. 2.Department of MathematicsMichigan State UniversityEast LansingUSA

Personalised recommendations