Numerische Mathematik

, Volume 136, Issue 2, pp 343–381 | Cite as

Nonconvex nonsmooth optimization via convex–nonconvex majorization–minimization



The class of majorization–minimization algorithms is based on the principle of successively minimizing upper bounds of the objective function. Each upper bound, or surrogate function, is locally tight at the current estimate, and each minimization step decreases the value of the objective function. We present a majorization–minimization approach based on a novel convex–nonconvex upper bounding strategy for the solution of a certain class of nonconvex nonsmooth optimization problems. We propose an efficient algorithm for minimizing the (convex) surrogate function based on the alternating direction method of multipliers. A preliminary convergence analysis for the proposed approach is provided. Numerical experiments show the effectiveness of the proposed method for the solution of nonconvex nonsmooth minimization problems.

Mathematics Subject Classification

65K05 Mathematical programming 65K10 Optimization and variational techniques 90C26 Nonconvex programming,  global optimization 49J52 Nonsmooth analysis 


  1. 1.
    Afonso, M.V., Bioucas-Dias, J.M., Figueiredo, M.A.T.: Fast image recovery using variable splitting and constrained optimization. IEEE Trans. Image Process. 19(9), 2345–2356 (2010)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Asif, M.S., Romberg, J.: Fast and accurate algorithms for re-weighted \(l_1\)-norm minimization. IEEE Trans. Signal Process. 61(23), 5905–5916 (2013)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Baglama, J., Reichel, L.: Augmented implicitly restarted lanczos bidiagonalization methods. SIAM J. Sci. Comput. 27(1), 19–42 (2005)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, Berlin (2011)Google Scholar
  5. 5.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imag. Sci. 2(1), 183–202 (2009)CrossRefMATHGoogle Scholar
  6. 6.
    Blake, A., Zisserman, A.: Visual Reconstruction. MIT Press, New York (1987)Google Scholar
  7. 7.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)CrossRefMATHGoogle Scholar
  8. 8.
    Candès, E., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inform. Theory 52(2), 489–509 (2006)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Candès, E.J., Wakin, M.B., Boyd, S.: Enhancing sparsity by reweighted l1 minimization. J. Fourier Anal. Appl. 14(5), 877–905 (2008)MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Charbonnier, P., Blanc-Feraud, L., Aubert, G., Barlaud, M.: Deterministic edge-preserving regularization in computed imaging. IEEE Trans. Image Process. 6(2), 298–311 (1997)CrossRefGoogle Scholar
  11. 11.
    Chartrand, R.: Fast algorithms for nonconvex compressive sensing:MRI reconstruction from very few data. In: Proceedings of the 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro—ISBI’09, pp. 262–265. Boston, MA, USA (2009)Google Scholar
  12. 12.
    Chen, P.-Y., Selesnick, I.W.: Group-sparse signal denoising: non-convex regularization, convex optimization. IEEE Trans. Signal Process. 62(13), 3464–3478 (2014)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Chouzenoux, E., Jezierska, A., Pesquet, J., Talbot, H.: A majorize-minimize subspace approach for \(\ell _2-\ell _0\) image regularization. SIAM J. Imag. Sci. 6(1), 563–591 (2013)CrossRefMATHGoogle Scholar
  14. 14.
    Chouzenoux, E., Pesquet, J.-C., Repetti, A.: Variable metric forward-backward algorithm for minimizing the sum of a differentiable function and a convex function. J. Optim. Theory Appl. 162(1), 107–132 (2014)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Combettes, P.L., Pesquet, J.-C.: Proximal thresholding algorithm for minimization over orthonormal bases. SIAM J. Optim. 18(4), 1351–1376 (2008)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Combettes, P.L., Pesquet, J.-C.: Proximal splitting methods in signal processing. In: Bauschke, H.H., et al., editors. Fixed-Point Algorithms for Inverse Problems in Science and Engineering, pp. 185–212. Springer, Berlin (2011)Google Scholar
  17. 17.
    Daubechies, I., DeVore, R., Fornasier, M., Gunturk, C.: Iteratively reweighted least squares minimization for sparse recovery. Commun. Pure Appl. Math. 63(1), 1–38 (2010)MathSciNetCrossRefMATHGoogle Scholar
  18. 18.
    Ding, Y., Selesnick, I.W.: Artifact-free wavelet denoising: Non-convex sparse regularization, convex optimization. IEEE Signal Process. Lett. 22(9), 1364–1368 (2015)CrossRefGoogle Scholar
  19. 19.
    Do, T.-M.-T., Artières, T.: Regularized bundle methods for convex and non-convex risks. J. Mach. Learn. Res. 13(1), 3539–3583 (2012)MathSciNetMATHGoogle Scholar
  20. 20.
    Donoho, D.L.: Compressed sensing. IEEE Trans. Inform. Theory 52(4), 1289–1306 (2006)MathSciNetCrossRefMATHGoogle Scholar
  21. 21.
    Figueiredo, M., Bioucas-Dias, J., Nowak, R.: Majorization–minimization algorithms for wavelet-based image restoration. IEEE Trans. Image Process. 16(12), 2980–2991 (2007)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Figueiredo, M., Nowak, R.: An EM algorithm for wavelet-based image restoration. IEEE Trans. Image Process. 12(8), 906–916 (2003)MathSciNetCrossRefMATHGoogle Scholar
  23. 23.
    Geman, D., Chengda, Y.: Nonlinear image recovery with half-quadratic regularization. IEEE Trans. Image Process. 4(7), 932–946 (1995)CrossRefGoogle Scholar
  24. 24.
    Geman, D., Reynolds, G.: Constrained restoration and the recovery of discontinuities. IEEE Trans. Pattern Anal. Mach. Intel. 14(3), 367–383 (1992)Google Scholar
  25. 25.
    Goldstein, T., Osher, S.: The split Bregman method for L1-regularized problems. SIAM J. Imag. Sci. 2(2), 323–343 (2009)CrossRefMATHGoogle Scholar
  26. 26.
    Jacobson, M.W., Fessler, J.A.: Properties of mm algorithms on convex feasible sets: extended version. Technical Report, Comm. and Sign. Proc. Lab., Dept. of EECS, Univ. of Michigan, Ann Arbor, 48109-2122, p. 353 (2004)Google Scholar
  27. 27.
    Lange, K., Chi, E.C., Zhou, H.: A brief survey of modern optimization for statisticians. Int. Stat. Rev. 82(1), 46–70 (2014)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Lanza, A., Morigi, S., Reichel, L., Sgallari, F.: A generalized Krylov subspace method for \(\ell _p-\ell _q \) minimization. SIAM J. Sci. Comput. 37(5), S30–S50 (2015)CrossRefMATHGoogle Scholar
  29. 29.
    Lanza, A., Morigi, S., Sgallari, F.: Convex image denoising via non-convex regularization. In: Aujol, J.-F., Nikolova, M., Papadakis, N., editors. Scale Space and Variational Methods in Computer Vision, vol. 9087. Lecture Notes in Computer Science, pp. 666–677. Springer, Berlin (2015)Google Scholar
  30. 30.
    Lanza, A., Morigi, S., Sgallari, F.: Constrained \({TV}_p-\ell _2\) model for image restoration. J. Sci. Comput. 68(1), 64–91 (2016)MathSciNetCrossRefMATHGoogle Scholar
  31. 31.
    Lanza, A., Morigi, S., Sgallari, F.: Convex image denoising via non-convex regularization with parameter selection. J. Math. Imag. Vis. 56(2), 195–220 (2016). doi:10.1007/s10851-016-0655-7
  32. 32.
    Laporte, L., Flamary, R., Canu, S., Déjean, S., Mothe, J.: Nonconvex regularizations for feature selection in ranking with sparse SVM. IEEE Trans. Neural Netw. Learn. Syst. 25(6), 1118–1130 (2014)Google Scholar
  33. 33.
    Mairal, J.: Incremental majorization–minimization optimization with application to large-scale machine learning. SIAM J. Optim. 25(2), 829–855 (2015)MathSciNetCrossRefMATHGoogle Scholar
  34. 34.
    Mourad, N., Reilly, J.P.: Minimizing nonconvex functions for sparse vector reconstruction. IEEE Trans. Signal Process. 58(7), 3485–3496 (2010)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Nesterov, Y., et al.: Gradient methods for minimizing composite objective function (2012)Google Scholar
  36. 36.
    Nikolova, M.: Estimation of binary images by minimizing convex criteria. Proc. IEEE Int. Conf. Image Process. (ICIP), vol. 2, pp. 108–112 (1998)Google Scholar
  37. 37.
    Nikolova, M.: Markovian reconstruction using a GNC approach. IEEE Trans. Image Process. 8(9), 1204–1220 (1999)CrossRefGoogle Scholar
  38. 38.
    Nikolova, M.: Energy minimization methods. In: Scherzer, O., editor. Handbook of Mathematical Methods in Imaging, chap. 5, pp. 138–186. Springer, Berlin (2011)Google Scholar
  39. 39.
    Nikolova, M., Ng, M.K., Tam, C.-P.: Fast nonconvex nonsmooth minimization methods for image restoration and reconstruction. IEEE Trans. Image Process. 19(12), 3073–3088 (2010)MathSciNetCrossRefGoogle Scholar
  40. 40.
    Parekh, A., Selesnick, I.W.: Convex denoising using non-convex tight frame regularization. IEEE Signal Process. Lett. 22(10), 1786–1790 (2015)CrossRefGoogle Scholar
  41. 41.
    Rao, B.D., Engan, K., Cotter, S.F., Palmer, J., Kreutz-Delgado, K.: Subset selection in noise based on diversity measure minimization. IEEE Trans. Signal Process. 51(3), 760–770 (2003)CrossRefGoogle Scholar
  42. 42.
    Repetti, A., Pham, M.Q., Duval, L., Chouzenoux, E., Pesquet, J.-C.: Euclid in a taxicab: sparse blind deconvolution with smoothed l1/l2 regularization. IEEE Signal Process. Lett. 22(5), 539–543 (2015)CrossRefGoogle Scholar
  43. 43.
    Rodriguez, P., Wohlberg, B.: Efficient minimization method for a generalized total variation functional. IEEE Trans. Image Process. 18(2), 322–332 (2009)MathSciNetCrossRefGoogle Scholar
  44. 44.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D 60(1–4), 259–268 (1992)MathSciNetCrossRefMATHGoogle Scholar
  45. 45.
    Selesnick, I.W., Parekh, A., Bayram, I.: Convex 1-D total variation denoising with non-convex regularization. IEEE Signal Process. Lett. 22(2), 141–144 (2015)CrossRefGoogle Scholar
  46. 46.
    Selesnick, I.W., Bayram, I.: Sparse signal estimation by maximally sparse convex optimization. IEEE Trans. Signal. Process. 62(5), 1078–1092 (2014)MathSciNetCrossRefGoogle Scholar
  47. 47.
    Voronin, S., Chartrand, R.: A new generalized thresholding algorithm for inverse problems with sparsity constraints. In: Proc. IEEE Int. Conf. Acoust, Speech, Signal Processing (ICASSP), pp. 1636–1640 (2013)Google Scholar
  48. 48.
    Wang, Y., Yang, J., Yin, W., Zhang, Y.: A new alternating minimization algorithm for total variation image reconstruction. SIAM J. Imag. Sci. 1(3), 248–272 (2008)MathSciNetCrossRefMATHGoogle Scholar
  49. 49.
    Zhao, Y.-B., Li, D.: Reweighted \( \ell _1 \)-minimization for sparse solutions to underdetermined linear systems. SIAM J. Optim. 22(3), 1065–1088 (2012)MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • A. Lanza
    • 1
  • S. Morigi
    • 1
  • I. Selesnick
    • 2
  • F. Sgallari
    • 1
  1. 1.Department of MathematicsUniversity of BolognaBolognaItaly
  2. 2.Department of Electrical and Computer EngineeringNew York UniversityBrooklynUSA

Personalised recommendations