Variable Metric Forward–Backward Algorithm for Minimizing the Sum of a Differentiable Function and a Convex Function

  • Emilie Chouzenoux
  • Jean-Christophe Pesquet
  • Audrey Repetti


We consider the minimization of a function G defined on \({ \mathbb{R} } ^{N}\), which is the sum of a (not necessarily convex) differentiable function and a (not necessarily differentiable) convex function. Moreover, we assume that G satisfies the Kurdyka–Łojasiewicz property. Such a problem can be solved with the Forward–Backward algorithm. However, the latter algorithm may suffer from slow convergence. We propose an acceleration strategy based on the use of variable metrics and of the Majorize–Minimize principle. We give conditions under which the sequence generated by the resulting Variable Metric Forward–Backward algorithm converges to a critical point of G. Numerical results illustrate the performance of the proposed algorithm in an image reconstruction application.


Nonconvex optimization Nonsmooth optimization Majorize–Minimize algorithms Forward–Backward algorithm Image reconstruction Proximity operator 


  1. 1.
    Chen, G.H.G., Rockafellar, R.T.: Convergence rates in forward–backward splitting. SIAM J. Optim. 7, 421–444 (1997) CrossRefzbMATHMathSciNetGoogle Scholar
  2. 2.
    Tseng, P.: A modified forward–backward splitting method for maximal monotone mappings. SIAM J. Control Optim. 38(2), 431–446 (1998) CrossRefGoogle Scholar
  3. 3.
    Attouch, H., Bolte, J.: On the convergence of the proximal algorithm for nonsmooth functions involving analytic features. Math. Program. 116, 5–16 (2008) CrossRefMathSciNetGoogle Scholar
  4. 4.
    Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods. Math. Program. 137, 91–129 (2011) CrossRefMathSciNetGoogle Scholar
  5. 5.
    Łojasiewicz, S.: Une propriété topologique des sous-ensembles analytiques réels, pp. 87–89. Editions du centre National de la Recherche Scientifique (1963) Google Scholar
  6. 6.
    Beck, A., Teboulle, M.: Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems. IEEE Trans. Image Process. 18(11), 2419–2434 (2009) CrossRefMathSciNetGoogle Scholar
  7. 7.
    Bioucas-Dias, J.M., Figueiredo, M.A.: A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process. 16, 2992–3004 (2007) CrossRefMathSciNetGoogle Scholar
  8. 8.
    Zibulevsky, M., Elad, M.: 2 1 optimization in signal and image processing. IEEE Signal Process. Mag. 27(3), 76–88 (2010) CrossRefMathSciNetGoogle Scholar
  9. 9.
    Kowalski, M.: Proximal algorithm meets a conjugate descent. Tech. Rep. (2010).
  10. 10.
    Nesterov, Y.: Gradient methods for minimizing composite objective function (2007).
  11. 11.
    Bonnans, J.F., Gilbert, J.C., Lemaréchal, C., Sagastizábal, C.A.: A family of variable metric proximal methods. Math. Program. 68, 15–47 (1995) zbMATHGoogle Scholar
  12. 12.
    Burke, J.V., Qian, M.: A variable metric proximal point algorithm for monotone operators. SIAM J. Control Optim. 37, 353–375 (1999) CrossRefzbMATHMathSciNetGoogle Scholar
  13. 13.
    Parente, L.A., Lotito, P.A., Solodov, M.V.: A class of inexact variable metric proximal point algorithms. SIAM J. Optim. 19, 240–260 (2008) CrossRefMathSciNetGoogle Scholar
  14. 14.
    Lotito, P.A., Parente, L.A., Solodov, M.V.: A class of variable metric decomposition methods for monotone variational inclusions. J. Convex Anal. 16, 857–880 (2009) zbMATHMathSciNetGoogle Scholar
  15. 15.
    Combettes, P.L., Vũ, B.C.: Variable metric forward–backward splitting with applications to monotone inclusions in duality. Optimization (2013, to appear). doi: 10.1080/02331934.2012.733883. arXiv:1206.6791 [math.OC]
  16. 16.
    Becker, S., Fadili, J.: A quasi-Newton proximal splitting method. In: Bartlett, P., Pereira, F., Burges, C., Bottou, L., Weinberger, K. (eds.) Advances in Neural Information Processing Systems (NIPS), vol. 25, pp. 2618–2626 (2012) Google Scholar
  17. 17.
    Lee, J.D., Sun, Y., Saunders, M.A.: Proximal Newton-type methods for convex optimization. In: Bartlett, P., Pereira, F., Burges, C., Bottou, L., Weinberger, K. (eds.) Advances in Neural Information Processing Systems (NIPS), vol. 25, pp. 827–835 (2012) Google Scholar
  18. 18.
    Tran-Dinh, Q., Kyrillidis, A., Cevher, V.: Composite self-concordant minimization. Tech. Rep. (2013).
  19. 19.
    Hunter, D.R., Lange, K.: A tutorial on MM algorithms. Am. Stat. 58(1), 30–37 (2004) CrossRefMathSciNetGoogle Scholar
  20. 20.
    Hiriart-Urruty, J.B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms. Springer, New York (1993) Google Scholar
  21. 21.
    Combettes, P.L., Vũ, B.C.: Variable metric quasi-Fejér monotonicity. Nonlinear Anal. 78, 17–31 (2013) CrossRefzbMATHMathSciNetGoogle Scholar
  22. 22.
    Moreau, J.J.: Proximité et dualité dans un espace Hilbertien. Bull. Soc. Math. Fr. 93, 273–299 (1965) zbMATHMathSciNetGoogle Scholar
  23. 23.
    Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward–backward splitting. Multiscale Model. Simul. 4(4), 1168–1200 (2005) CrossRefzbMATHMathSciNetGoogle Scholar
  24. 24.
    Combettes, P.L., Pesquet, J.C.: Proximal thresholding algorithm for minimization over orthonormal bases. SIAM J. Optim. 18(4), 1351–1376 (2007) CrossRefzbMATHMathSciNetGoogle Scholar
  25. 25.
    Bolte, J., Daniilidis, A., Lewis, A.: The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM J. Optim. 17, 1205–1223 (2006) CrossRefMathSciNetGoogle Scholar
  26. 26.
    Bolte, J., Daniilidis, A., Lewis, A., Shiota, M.: Clarke subgradients of stratifiable functions. SIAM J. Optim. 18(2), 556–572 (2007) CrossRefzbMATHMathSciNetGoogle Scholar
  27. 27.
    Kurdyka, K., Parusinski, A.: w f-stratification of subanalytic functions and the Łojasiewicz inequality. C. R. Acad. Sci., Ser. 1 Math. 318(2), 129–133 (1994) zbMATHMathSciNetGoogle Scholar
  28. 28.
    Bertsekas, D.P.: Nonlinear Programming, 2nd edn. Athena Scientific, Belmont (1999) zbMATHGoogle Scholar
  29. 29.
    Chen, K.: Matrix Preconditioning Techniques and Applications. Cambridge Monogr. Appl. Comput. Math., vol. 19. Cambridge University Press, Cambridge (2005) CrossRefzbMATHGoogle Scholar
  30. 30.
    Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (1999) CrossRefzbMATHGoogle Scholar
  31. 31.
    Bertsekas, D.P.: Projected Newton methods for optimization problems with simple constraints. SIAM J. Control Optim. 20, 221–246 (1982) CrossRefzbMATHMathSciNetGoogle Scholar
  32. 32.
    Birgin, E.G., Martínez, J.M., Raydan, M.: Nonmonotone spectral projected gradient methods on convex sets. SIAM J. Optim. 10(4), 1196–1211 (2000) CrossRefzbMATHMathSciNetGoogle Scholar
  33. 33.
    Bonettini, S., Zanella, R., Zanni, L.: A scaled gradient projection method for constrained image deblurring. Inverse Probl. 25(1), 015002+ (2009) CrossRefMathSciNetGoogle Scholar
  34. 34.
    Rockafellar, R.T., Wets, R.J.B.: Variational Analysis, 1st edn. Grundlehren der Mathematischen Wissenschaften, vol. 317. Springer, Berlin (1998) CrossRefzbMATHGoogle Scholar
  35. 35.
    Mordukhovich, B.S.: Variational Analysis and Generalized Differentiation. I. Basic Theory. Grundlehren der Mathematischen Wissenschaften, vol. 330. Springer, Berlin (2006) Google Scholar
  36. 36.
    Bolte, J., Daniilidis, A., Ley, O., Mazet, L.: Characterizations of Łojasiewicz inequalities and applications. Transl. Am. Math. Soc. 362(6), 3319–3363 (2010) CrossRefzbMATHMathSciNetGoogle Scholar
  37. 37.
    Erdogan, H., Fessler, J.A.: Monotonic algorithms for transmission tomography. IEEE Trans. Med. Imaging 18(9), 801–814 (1999) CrossRefGoogle Scholar
  38. 38.
    Healey, G.E., Kondepudy, R.: Radiometric CCD camera calibration and noise estimation. IEEE Trans. Pattern Anal. Mach. Intell. 16(3), 267–276 (1994) CrossRefGoogle Scholar
  39. 39.
    Tian, H., Fowler, B., Gamal, A.E.: Analysis of temporal noise in CMOS photodiode active pixel sensor. IEEE Solid State Circ. Mag. 36(1), 92–101 (2001) CrossRefGoogle Scholar
  40. 40.
    Janesick, J.R.: Photon Transfer, vol. PM170. SPIE, Bellingham (2007) CrossRefGoogle Scholar
  41. 41.
    Foi, A., Trimeche, M., Katkovnik, V., Egiazarian, K.: Practical Poissonian–Gaussian noise modeling and fitting for single-image raw-data. IEEE Trans. Image Process. 17(10), 1737–1754 (2008) CrossRefMathSciNetGoogle Scholar
  42. 42.
    Li, J., Shen, Z., Jin, R., Zhang, X.: A reweighted 2 method for image restoration with Poisson and mixed Poisson–Gaussian noise. Tech. rep. (2012). UCLA Preprint.
  43. 43.
    Sorenson, H.W.: Parameter Estimation: Principles and Problems, Control and Systems Theory vol. 9, 1st edn. Dekker, New York (1980) Google Scholar
  44. 44.
    Dahlquist, G., Bjorck, A.: Numerical Methods. Dover, Mineola (2003) zbMATHGoogle Scholar
  45. 45.
    Chaâri, L., Pustelnik, N., Chaux, C., Pesquet, J.C.: Solving inverse problems with overcomplete transforms and convex optimization techniques. In: Wavelets XIII, San Diego, CA, USA, vol. 7446. SPIE, Bellingham (2009). 14 pp. CrossRefGoogle Scholar
  46. 46.
    Elad, M., Milanfar, P., Ron, R.: Analysis versus synthesis in signal priors. Inverse Probl. 23(3), 947–968 (2007) CrossRefzbMATHGoogle Scholar
  47. 47.
    Pustelnik, N., Pesquet, J.C., Chaux, C.: Relaxing tight frame condition in parallel proximal methods for signal restoration. IEEE Trans. Signal Process. 60(2), 968–973 (2012) CrossRefMathSciNetGoogle Scholar
  48. 48.
    Combettes, P.L., Dũng, D., Vũ, B.C.: Proximity for sums of composite functions. J. Math. Anal. Appl. 380(2), 680–688 (2011) CrossRefzbMATHMathSciNetGoogle Scholar
  49. 49.
    Zubal, I.G., Harrell, C.R., Smith, E.O., Rattner, Z., Gindi, G., Hoffer, P.B.: Computerized three-dimensional segmented human anatomy. Med. Phys. 21, 299–302 (1994) CrossRefGoogle Scholar
  50. 50.
    Slaney, M., Kak, A.: Principles of Computerized Tomographic Imaging. SIAM, Philadelphia (1988) zbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Emilie Chouzenoux
    • 1
  • Jean-Christophe Pesquet
    • 1
  • Audrey Repetti
    • 1
  1. 1.Laboratoire d’Informatique Gaspard Monge and CNRS UMR 8049Université Paris-Est Marne-la-ValléeMarne-la-ValléeFrance

Personalised recommendations