Dictionary learning with the \({{\ell }_{1/2}}\)-regularizer and the coherence penalty and its convergence analysis

  • Zhenni LiEmail author
  • Takafumi Hayashi
  • Shuxue Ding
  • Yujie Li
Original Article


The \({{\ell }_{1/2}}\)-regularizer has been studied widely in compressed sensing, but there have been few studies about dictionary learning problems. The dictionary learning method with the \({{\ell }_{1/2}}\)-regularizer aims to learn a dictionary, which requires solving a very challenging nonconvex and nonsmooth optimization problem. In addition, the low mutual coherence of a dictionary is an important property that ensures the optimality of the sparse representation in the dictionary. In this paper, we address a dictionary learning problem involving the \({{\ell }_{1/2}}\)-regularizer and the coherence penalty, which is difficult to solve quickly and efficiently. We employ a decomposition scheme and an alternating optimization, which transforms the overall problem into a set of minimizations of single-vector-variable subproblems. Although the subproblems are nonsmooth and even nonconvex, we propose the use of proximal operator technology to conquer them, which leads to a rapid and efficient dictionary learning algorithm. In a theoretical analysis, we establish the algorithm’s global convergence. Experiments were performed for dictionary learning using both synthetic data and real-world data. For the synthetic data, we demonstrated that our algorithm performed better than state-of-the-art algorithms. Using real-world data, the learned dictionaries were shown to be more efficient than algorithms using \({{\ell }_{1}}\)-norm for sparsity.


Sparse representation Dictionary learning Proximal operator \({{\ell }_{1/2}}\)-regularizer  Incoherence 


  1. 1.
    Elad M (2010) Sparse and redundant representation. Springer, BerlinCrossRefzbMATHGoogle Scholar
  2. 2.
    Elad M, Figueiredo M, Ma Y (2010) On the role of sparse and redundant representations in image processing. Proc IEEE 98(6):972–982CrossRefGoogle Scholar
  3. 3.
    Huang K, Aviyente S (2006) Sparse representation for signal classification. Proc Conf Neur Inf Process Syst 19:609–616Google Scholar
  4. 4.
    Engan K, Aase S, Husoy J (1999). Method of optimal directions for frame design. Proc IEEE Int Conf Acoust Speech Signal Process (ICASSP) 5:2443–2446Google Scholar
  5. 5.
    Aharon M, Elad M, Bruckstein A (2006) K-svd: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans Signal Process 54(11):4311–4322CrossRefzbMATHGoogle Scholar
  6. 6.
    Dai W, Xu T, Wang W (2012) Simultaneous codeword optimization (simco) for dictionary update and learning. IEEE Trans Signal Process 60(12):6340–6353MathSciNetCrossRefGoogle Scholar
  7. 7.
    Li Z, Ding S, Li Y (2015) A fast algorithm for learning overcomplete dictionary for sparse representation based on proximal operators. Neural Comput 27(9):1951–1982CrossRefGoogle Scholar
  8. 8.
    Bao C, Ji H, Quan Y, Shen Z (2014) \({{\ell }_{o}}\)-norm-based dictionary learning by proximal methods with global convergence. IEEE Conf Comput Vis Pattern Recognit (CVPR) 3858–3865Google Scholar
  9. 9.
    Yaghoobi M, Blumensath T, Davies M (2013) Dictionary learning for sparse approximations with the majorization method. IEEE Trans Signal Process 57(6):2178–2191MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Tropp JA (2004) Greed is good: algorithmic results for sparse approximation. IEEE Trans Inf Theory 50(10):2231–2242MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Rakotomamonjy A (2013) Direct optimization of the dictionary learning problem. IEEE Trans Signal Process 61(22):5495–5506MathSciNetCrossRefGoogle Scholar
  12. 12.
    Li Z, Tang Z, Ding S (2013) Dictionary learning by nonnegative matrix factorization with \({{\ell }_{1/2}}\)-norm sparsity constraint. IEEE Int Conf Cybern (CYBCONF2) Lausanne Switz 63–67Google Scholar
  13. 13.
    Mailhe B, Barchiesi D, Plumbley MD (2012) INK-SVD: Learning incoherent dictionaries for sparse representations. IEEE Int Conf Acoust Speech Signal Process (ICASSP) 3573–3576Google Scholar
  14. 14.
    Barchiesi D, Plumbley MD (2013) Learning incoherent dictionaries for sparse approximation using iterative projections and rotations. IEEE Trans Signal Process 61(8):2055–2065CrossRefGoogle Scholar
  15. 15.
    Lin T, Liu S, Zha H (2012) Incoherent dictionary learning for sparse representation. IEEE 21st International Conference on Pattern Recognition (ICPR), pp 1237–1240Google Scholar
  16. 16.
    Moreau JJ (1962) Fonctions convexes duales et points proximaux dans un espace Hilbertien. Comptes Rendues de lAcademie des Sciences de Paris 255:2897–2899MathSciNetzbMATHGoogle Scholar
  17. 17.
    Combettes PL, Pesquet J (2010) Proximal splitting methods in signal processing. arXiv:0912.3522v4Google Scholar
  18. 18.
    Mallat SG, Zhang Z (1993) Matching pursuits with time–frequency dictionaries. IEEE Trans Signal Process 41(12):3397–3415CrossRefzbMATHGoogle Scholar
  19. 19.
    Chen SS, Donoho DL, Saunders MA (2001) Atomic decomposition by basis pursuit. SIAM Rev 43(1):129–159MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Chartrand R, Yin W (2008) Iteratively reweighted algorithms for compressive sensing. Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), pp 3869–3872Google Scholar
  21. 21.
    Daubechies I, Defrise M, Mol CD (2004) An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Comm Pure Appl Math 57(11):1413–1457MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Beck A, Teboulle M (2009) A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imag Sci 2(1):183–202MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Xu ZB, Guo HL, Wang Y, Zhang H (2012) Representative of \({{\ell }_{1/2}}\) regularization among (0 < q ≤ 1) \({{\ell }_{q}}\) regularizations: an experimental study based on phase diagram. Acta Automatica Sinica 38:1225–1228MathSciNetGoogle Scholar
  24. 24.
    Lin J, Lin S, Wang Y, Xu ZB (2014) \({{\ell }_{1/2}}\) Regularization: convergence of iterative half thresholding algorithm. IEEE Trans Signal Process 62(1):2317–2329MathSciNetGoogle Scholar
  25. 25.
    Xu ZB, Chang X, Xu F, Zhang H (2012) \({{\ell }_{1/2}}\) Regularization: a thresholding representation theory and a fast solver. IEEE Trans Neur Networks Learning Syst 23(7):1013–1027CrossRefGoogle Scholar
  26. 26.
    Fazel M, Hindi H, Boyd SP (2003) Log-det heuristic for matrix rank minimization with applications to Hankel and Euclidean distance matrices. Proc Am Control Conf 3:2156–2162Google Scholar
  27. 27.
    Hoyer PO (2004) Nonnegative matrix factorization with sparseness constraints. J Mach Learn Res 5:1457–1469zbMATHGoogle Scholar
  28. 28.
    Attouch H, Bolte J, Svaiter BF (2013) Convergence of descent methods for semialgebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods. Math Program Ser A 137(1–2):91–129CrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  • Zhenni Li
    • 1
    Email author
  • Takafumi Hayashi
    • 2
  • Shuxue Ding
    • 1
  • Yujie Li
    • 3
  1. 1.School of Computer Science and EngineeringThe University of AizuAizu-WakamatsuJapan
  2. 2.Graduate School of Science and TechnologyNiigata UniversityNiigataJapan
  3. 3.Artificial Intelligence CenterAIST TsukubaIbarakiJapan

Personalised recommendations