Journal of Mathematical Imaging and Vision

, Volume 53, Issue 2, pp 171–181 | Cite as

iPiasco: Inertial Proximal Algorithm for Strongly Convex Optimization

Article

Abstract

In this paper, we present a forward–backward splitting algorithm with additional inertial term for solving a strongly convex optimization problem of a certain type. The strongly convex objective function is assumed to be a sum of a non-smooth convex and a smooth convex function. This additional knowledge is used for deriving a worst-case convergence rate for the proposed algorithm. It is proved to be an optimal algorithm with linear rate of convergence. For certain problems this linear rate of convergence is better than the provably optimal worst-case rate of convergence for smooth strongly convex functions. We demonstrate the efficiency of the proposed algorithm in numerical experiments and examples from image processing.

Keywords

Heavy-ball method Strongly convex optimization Inertial proximal method Convergence analysis 

References

  1. 1.
    Alvarez, F.: Weak convergence of a relaxed and inertial hybrid projection-proximal point algorithm for maximal monotone operators in Hilbert space. SIAM J. Optim. 14(3), 773–782 (2003)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Alvarez, F., Attouch, H.: An inertial proximal method for maximal monotone operators via discretization of a nonlinear oscillator with damping. Set-Valued Anal. 9(1–2), 3–11 (2001)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Attouch, H., Peypouquet, J., Redont, P.: A dynamical approach to an inertial forward-backward algorithm for convex minimization. SIAM J. Optim. 24(1), 232–256 (2014)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. CMS Books in Mathematics, Springer (2011)Google Scholar
  5. 5.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Appl. Math. 2(1), 183–202 (2009)MathSciNetGoogle Scholar
  6. 6.
    Bioucas-Dias, J.M., Figueiredo, M.: A new twist: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process. 16(12), 2992–3004 (2007)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vision 40(1), 120–145 (2011)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Chambolle, A., Pock, T.: On the ergodic convergence rates of a first-order primal-dual algorithm (2014). to appearGoogle Scholar
  9. 9.
    Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward-backward splitting. Multiscale Modeling Simul. 4(4), 1168–1200 (2005)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Combettes, P.L., Dũng, D., Vũ, B.C.: Dualization of signal recovery problems. Set-Valued Var. Anal. 18(3–4), 373–404 (2010)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004)CrossRefGoogle Scholar
  12. 12.
    Drori, Y., Teboulle, M.: Performance of first-order methods for smooth convex minimization: a novel approach. Math. Program. 145(1–2), 451–482 (2014)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Gelfand, I.: Normierte Ringe. Rec. Math. [Mat. Sbornik] N.S. 9(51), 3–24 (1941)MathSciNetGoogle Scholar
  14. 14.
    Goldfarb, D., Ma, S.: Fast multiple-splitting algorithms for convex optimization. SIAM J. Optim. 22(2), 533–556 (2012)MathSciNetCrossRefGoogle Scholar
  15. 15.
    He, B., Yuan, X.: On the \(O(1/n)\) convergence rate of the Douglas-Rachford alternating direction method. SIAM J. Numer. Anal. 50(2), 700–709 (2012)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Hong, M., Luo, Z.-Q.: On the linear convergence of the alternating direction method of multipliers. ArXiv e-prints, August (2012)Google Scholar
  17. 17.
    Mainberger, M., Weickert, J.: Edge-based image compression with homogeneous diffusion. In: Jiang, Xiaoyi, Petkov, Nicolai (eds.) Computer Analysis of Images and Patterns. Lecture Notes in Computer Science, vol. 5702, pp. 476–483. Springer, Berlin (2009)Google Scholar
  18. 18.
    Moudafi, A., Oliny, M.: Convergence of a splitting inertial proximal method for monotone operators. J. Comput. Appl. Math. 155, 447–454 (2003)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Nesterov, Y.: A method of solving a convex programming problem with convergence rate \(O(1/k^2)\). Soviet Math. Doklady 27, 372–376 (1983)Google Scholar
  20. 20.
    Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course. Applied Optimization, vol. 87. Kluwer Academic Publishers, Boston (2004)Google Scholar
  21. 21.
    Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127–152 (2005)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Nesterov, Y.: Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. Optim. 22(2), 341–362 (2012)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Nesterov, Y.: Gradient methods for minimizing composite functions. Math. Program. 140(1), 125–161 (2013)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Ochs, P., Chen, Y., Brox, T., Pock, T.: ipiano: Inertial proximal algorithm for non-convex optimization. SIAM J. Imaging Sci. 7(2), 1388–1419 (2014)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Poljak, B.T.: Introduction to optimization. Optimization Software (1987)Google Scholar
  26. 26.
    Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4(5), 1–17 (1964)CrossRefGoogle Scholar
  27. 27.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D 60, 259–268 (1992)CrossRefGoogle Scholar
  28. 28.
    Shor, N.Z.: Minimization Methods for Non-differentiable Functions. Springer-Verlag New York Inc, New York (1985)CrossRefGoogle Scholar
  29. 29.
  30. 30.
    Zavriev, S.K., Kostyuk, F.V.: Heavy-ball method in nonconvex optimization problems. Comput. Math. Model. 4(4), 336–341 (1993)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.University of FreiburgFreiburgGermany
  2. 2.Graz University of TechnologyGrazAustria
  3. 3.Digital Safety & Security DepartmentAIT Austrian Institute of Technology GmbHViennaAustria

Personalised recommendations