# Regularization and Incoherence

## Abstract

A dictionary should be faithful to the signals it represents, in the sense that the sparse representation error in learning is small, but also must be reliable when recovering sparse representations. A direct way to obtain good recovery guarantees is to modify the objective of the DL optimization such that the resulted dictionary is incoherent, meaning that the atoms are generally far from each other. Alternatively, we can explicitly impose mutual coherence bounds. A related modification of the objective is regularization, in the usual form encountered in least squares problems. Regularization has the additional benefit of making the DL process avoid bottlenecks generated by ill-conditioned dictionaries. We present several algorithms for regularization and promoting incoherence and illustrate their benefits. The K-SVD family can be adapted to such approaches, with good results in the case of regularization. Other methods for obtaining incoherence are based on the gradient of a combined objective including the frame potential or by including a decorrelation step in the standard algorithms. Such a step is based on successive projections on two sets whose properties are shared by the Gram matrix corresponding to the dictionary, which reduce mutual coherence, and rotations of the dictionary, which regain the adequacy to the training signals. We also give a glimpse on the currently most efficient methods that aim at the minimization of the mutual coherence of a frame, regardless of the training signals.

## References

- 10.D. Barchiesi, M.D. Plumbley, Learning incoherent dictionaries for sparse approximation using iterative projections and rotations. IEEE Trans. Signal Process.
**61**(8), 2055–2065 (2013)Google Scholar - 27.P.G. Casazza, R.G. Lynch, A brief introduction to Hilbert space frame theory and its applications, in
*Proceedings of Symposia in Applied Mathematics*, ed. by K.A. Okoudjou, vol. 73 (American Mathematical Society, Providence, 2016), pp. 1–52Google Scholar - 38.W. Chen, M.R.D. Rodrigues, I.J. Wassell, On the use of unit-norm tight frames to improve the average MSE performance in compressive sensing applications. IEEE Signal Process. Lett.
**19**(1), 8–11 (2012)CrossRefGoogle Scholar - 44.W. Dai, T. Xu, W. Wang, Simultaneous codeword optimization (SimCO) for dictionary update and learning. IEEE Trans. Signal Process.
**60**(12), 6340–6353 (2012)Google Scholar - 49.P. Davies, N.J. Higham, Numerically stable generation of correlation matrices and their factors. BIT Numer. Math.
**40**(4), 640–651 (2000)Google Scholar - 58.B. Dumitrescu, Designing incoherent frames with only matrix-vector multiplications. IEEE Signal Process. Lett.
**24**(9), 1265–1269 (2017)CrossRefGoogle Scholar - 60.B. Dumitrescu, P. Irofti, Regularized K-SVD. IEEE Signal Process. Lett.
**24**(3), 309–313 (2017)CrossRefGoogle Scholar - 100.P. Irofti, B. Dumitrescu, Regularized algorithms for dictionary learning, in
*Proceedings of International Conference Communications*, Bucharest (2016), pp. 439–442Google Scholar - 120.T. Lin, S. Liu, H. Zha, Incoherent dictionary learning for sparse representation, in
*21st International Conference on Pattern Recognition (ICPR)*, Tsukuba, November 2012, pp. 1237–1240Google Scholar - 133.J.L. Massey, T. Mittelholzer, Welch’s bound and sequence sets for code-division multiple-access systems, in
*Sequences II*, ed. by R. Capocelli, A. De Santis, U. Vaccaro (Springer, Berlin, 1993), pp. 63–78CrossRefGoogle Scholar - 138.D. Needell R. Vershynin, Uniform uncertainty principle and signal recovery via regularized orthogonal matching pursuit. Found. Comput. Math.
**9**(3), 317–334 (2009)MathSciNetCrossRefGoogle Scholar - 139.M. Nejati, S. Samavi, N. Karimi, S.M.R. Soroushmehr, K. Najarian, Boosted dictionary learning for image compression. IEEE Trans. Image Process.
**25**(10), 4900–4915 (2016)MathSciNetCrossRefGoogle Scholar - 163.C. Rusu, N. Gonzalez-Prelcic, Designing incoherent frames through convex techniques for optimized compressed sensing. IEEE Trans. Signal Process.
**64**(9), 2334–2344 (2016)MathSciNetCrossRefGoogle Scholar - 166.M. Sadeghi, M. Babaie-Zadeh, Incoherent unit-norm frame design via an alternating minimization penalty method. IEEE Signal Process. Lett.
**24**(1), 32–36 (2017)CrossRefGoogle Scholar - 176.C.D. Sigg, T. Dikk, J.D. Buhmann, Learning dictionaries with bounded self-coherence. IEEE Signal Process. Lett.
**19**(19), 861–865 (2012)CrossRefGoogle Scholar - 191.J.A. Tropp, I.S. Dhillon, R.W. Heath Jr., T. Strohmer, Designing structured tight frames via an alternating projection method. IEEE Trans. Inf. Theory
**51**(1), 188–209 (2005)MathSciNetCrossRefGoogle Scholar - 192.E.V. Tsiligianni, L.P. Kondi, A.K. Katsaggelos, Construction of incoherent unit norm tight frames with application to compressed sensing. IEEE Trans. Info. Theory
**60**(4), 2319–2330 (2014)MathSciNetCrossRefGoogle Scholar - 197.S. Waldron, Generalized Welch bound equality sequences are tight frames. IEEE Trans. Inf. Theory
**49**(9), 2307–2309 (2003)MathSciNetCrossRefGoogle Scholar - 203.L. Welch, Lower bounds on the maximum cross correlation of signals. IEEE Trans. Inf. Theory
**20**(3), 397–399 (1974)CrossRefGoogle Scholar - 211.M. Yaghoobi, L. Daudet, M.E. Davies, Parametric dictionary design for sparse coding. IEEE Trans. Signal Process.
**57**(12), 4800–4810 (2009)MathSciNetCrossRefGoogle Scholar