International Journal of Computer Vision

, Volume 114, Issue 2–3, pp 137–167 | Cite as

Structured Overcomplete Sparsifying Transform Learning with Convergence Guarantees and Applications

Article

Abstract

In recent years, sparse signal modeling, especially using the synthesis model has been popular. Sparse coding in the synthesis model is however, NP-hard. Recently, interest has turned to the sparsifying transform model, for which sparse coding is cheap. However, natural images typically contain diverse textures that cannot be sparsified well by a single transform. Hence, in this work, we propose a union of sparsifying transforms model. Sparse coding in this model reduces to a form of clustering. The proposed model is also equivalent to a structured overcomplete sparsifying transform model with block cosparsity, dubbed OCTOBOS. The alternating algorithm introduced for learning such transforms involves simple closed-form solutions. A theoretical analysis provides a convergence guarantee for this algorithm. It is shown to be globally convergent to the set of partial minimizers of the non-convex learning problem. We also show that under certain conditions, the algorithm converges to the set of stationary points of the overall objective. When applied to images, the algorithm learns a collection of well-conditioned square transforms, and a good clustering of patches or textures. The resulting sparse representations for the images are much better than those obtained with a single learned transform, or with analytical transforms. We show the promising performance of the proposed approach in image denoising, which compares quite favorably with approaches involving a single learned square transform or an overcomplete synthesis dictionary, or gaussian mixture models. The proposed denoising method is also faster than the synthesis dictionary based approach.

Keywords

Sparsifying transform learning Dictionary learning  Convergence guarantees Overcomplete representation Clustering Image representation Sparse representation Image denoising Machine learning 

References

  1. Agarwal, A., Anandkumar, A., Jain, P., Netrapalli, P., & Tandon, R. (2013). Learning sparsely used overcomplete dictionaries via alternating minimization, arXiv:1310.7991, Preprint.
  2. Agarwal, A., Anandkumar, A., Jain, P., Netrapalli, P., & Tandon, R. (2014). Learning sparsely used overcomplete dictionaries. Journal of Machine Learning Research, 35, 1–15.Google Scholar
  3. Aharon, M., & Elad, M. (2008). Sparse and redundant modeling of image content using an image-signature-dictionary. SIAM Journal on Imaging Sciences, 1(3), 228–247.CrossRefMathSciNetMATHGoogle Scholar
  4. Aharon, M., Elad, M., & Bruckstein, A. (2006). K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing, 54(11), 4311–4322.CrossRefGoogle Scholar
  5. Arora, S., Ge, R., & Moitra, A. (2013). New algorithms for learning incoherent and overcomplete dictionaries. arXiv:1308.6273v5.pdf, Preprint
  6. Bao, C., Ji, H., Quan, Y., & Shen, Z. (2014). \(\ell _{0}\) Norm based dictionary learning by proximal methods with global convergence. In IEEE Conference on Computer Vision and Pattern Recognition. Online: http://www.math.nus.edu.sg/~matzuows/BJQS.pdf, to appear
  7. Brodatz, P. (1966). Textures: A photographic album for artists and designers. New York: Dover.Google Scholar
  8. Bruckstein, A. M., Donoho, D. L., & Elad, M. (2009). From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Review, 51(1), 34–81.CrossRefMathSciNetMATHGoogle Scholar
  9. Candès, E. J., Donoho, D. L. (1999). Curvelets: A surprisingly effective nonadaptive representation for objects with edges. In Curves and surfaces (pp. 105–120). Nashville: Vanderbilt University Press.Google Scholar
  10. Candès, E. J., & Donoho, D. L. (1999). Ridgelets: A key to higher-dimensional intermittency? Philosophical Transactions of the Royal Society of London Series A: Mathematical, Physical and Engineering Sciences, 357(1760), 2495–2509.CrossRefMATHGoogle Scholar
  11. Candès, E. J., Eldar, Y. C., Needell, D., & Randall, P. (2011). Compressed sensing with coherent and redundant dictionaries. Applied and Computational Harmonic Analysis, 31(1), 59–73.CrossRefMathSciNetMATHGoogle Scholar
  12. Chambolle, A. (2004). An algorithm for total variation minimization and applications. Journal of Mathematical Imaging and Vision, 20(1–2), 89–97.MathSciNetGoogle Scholar
  13. Chen, S. S., Donoho, D. L., & Saunders, M. A. (1998). Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing, 20(1), 33–61.CrossRefMathSciNetGoogle Scholar
  14. Chen, Y., Pock, T., & Bischof, H. (2012a). Learning \(\ell _{1}\)-based analysis and synthesis sparsity priors using bi-level optimization. In Proceedings of the Workshop on Analysis Operator Learning vs. Dictionary Learning, NIPS. arXiv:1401.4105
  15. Chen, Y. C., Sastry, C. S., Patel, V. M., Phillips, P. J., & Chellappa, R. (2012b). Rotation invariant simultaneous clustering and dictionary learning. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1053–1056).Google Scholar
  16. Chi, Y. T., Ali, M., Rajwade, A., & Ho, J. (2013). Block and group regularized sparse modeling for dictionary learning. In CVPR (pp. 377–382).Google Scholar
  17. Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Image denoising by sparse 3D transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8), 2080–2095.CrossRefMathSciNetGoogle Scholar
  18. Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2011). BM3D web page. Retrieved 2014, from http://www.cs.tut.fi/~foi/GCF-BM3D/
  19. Dai, W., & Milenkovic, O. (2009). Subspace pursuit for compressive sensing signal reconstruction. IEEE Transactions on Information Theory, 55(5), 2230–2249.CrossRefMathSciNetGoogle Scholar
  20. Davis, G., Mallat, S., & Avellaneda, M. (1997). Adaptive greedy approximations. Journal of Constructive Approximation, 13(1), 57–98.CrossRefMathSciNetMATHGoogle Scholar
  21. Do, M. N., & Vetterli, M. (2005). The contourlet transform: An efficient directional multiresolution image representation. IEEE Transactions on Image Processing, 14(12), 2091–2106.CrossRefMathSciNetGoogle Scholar
  22. Donoho, D. L., & Elad, M. (2003). Optimally sparse representation in general (nonorthogonal) dictionaries via \(\ell ^1\) minimization. Proceedings of the National Academy of Sciences, 100(5), 2197–2202.Google Scholar
  23. Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004). Least angle regression. Annals of Statistics, 32, 407–499.CrossRefMathSciNetMATHGoogle Scholar
  24. Elad, M. (2009). Michael Elad personal page. http://www.cs.technion.ac.il/~elad/Various/KSVD_Matlab_ToolBox.zip. Accessed 2014.
  25. Elad, M., & Aharon, M. (2006). Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing, 15(12), 3736–3745.CrossRefMathSciNetGoogle Scholar
  26. Elad, M., Milanfar, P., & Rubinstein, R. (2007). Analysis versus synthesis in signal priors. Inverse Problems, 23(3), 947–968.CrossRefMathSciNetMATHGoogle Scholar
  27. Engan, K., Aase, S., & Hakon-Husoy, J. (1999). Method of optimal directions for frame design. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (pp. 2443–2446).Google Scholar
  28. Giryes, R., Nam, S., Elad, M., Gribonval, R., & Davies, M. (2014). Greedy-like algorithms for the cosparse analysis model. Linear Algebra and its Applications, 441:22–60, Special Issue on Sparse Approximate Solution of Linear Systems.Google Scholar
  29. Gorodnitsky, I. F., George, J., & Rao, B. D. (1995). Neuromagnetic source imaging with FOCUSS: A recursive weighted minimum norm algorithm. Electrocephalography and Clinical Neurophysiology, 95, 231–251.CrossRefGoogle Scholar
  30. Harikumar, G., & Bresler, Y. (1996). A new algorithm for computing sparse solutions to linear inverse problems. In ICASSP (pp. 1331–1334).Google Scholar
  31. Hawe, S., Kleinsteuber, M., & Diepold, K. (2013). Analysis operator learning and its application to image reconstruction. IEEE Transactions on Image Processing, 22(6), 2138–2150.CrossRefMathSciNetGoogle Scholar
  32. He, D.-C., & Safia, A. (2013). Multiband Texture Database. http://multibandtexture.recherche.usherbrooke.ca/original_brodatz.html. Accessed 2014.
  33. Kong, S., & Wang, D. (2012). A dictionary learning approach for classification: Separating the particularity and the commonality. In Proceedings of the 12th European Conference on Computer Vision (pp 186–199).Google Scholar
  34. Liao, H. Y., & Sapiro, G. (2008). Sparse representations for limited data tomography. In Proceedings of the IEEE International Symposium on Biomedical Imaging (ISBI) (pp 1375–1378).Google Scholar
  35. Liu, Y., Tiebin, M., & Li, S. (2012). Compressed sensing with general frames via optimal-dual-based \(\ell _{1}\)-analysis. IEEE Transactions on Information Theory, 58(7), 4201–4214.CrossRefGoogle Scholar
  36. Mairal, J., Elad, M., & Sapiro, G. (2008a). Sparse representation for color image restoration. IEEE Transactions on Image Processing, 17(1), 53–69.CrossRefMathSciNetGoogle Scholar
  37. Mairal, J., Sapiro, G., & Elad, M. (2008b). Learning multiscale sparse representations for image and video restoration. SIAM, Multiscale Modeling and Simulation, 7(1), 214–241.CrossRefMathSciNetMATHGoogle Scholar
  38. Mairal, J., Bach, F., Ponce, J., Sapiro, G, & Zisserman, A. (2009). Non-local sparse models for image restoration. In IEEE International Conference on Computer Vision (pp. 2272–2279).Google Scholar
  39. Mairal, J., Bach, F., Ponce, J., & Sapiro, G. (2010). Online learning for matrix factorization and sparse coding. Journal of Machine Learning Research, 11, 19–60.MathSciNetMATHGoogle Scholar
  40. Mallat, S. (1999). A wavelet tour of signal processing. Boston: Academic Press.MATHGoogle Scholar
  41. Mallat, S. G., & Zhang, Z. (1993). Matching pursuits with time-frequency dictionaries. IEEE Transactions on Signal Processing, 41(12), 3397–3415.CrossRefMATHGoogle Scholar
  42. Marcellin, M. W., Gormish, M. J., Bilgin, A., & Boliek, M. P. (2000). An overview of JPEG-2000. In Proceedings of the Data Compression Conference (pp. 523–541).Google Scholar
  43. Nam, S., Davies, M. E., Elad, M., & Gribonval, R. (2011). Cosparse analysis modeling: Uniqueness and algorithms. In ICASSP (pp. 5804–5807).Google Scholar
  44. Natarajan, B. K. (1995). Sparse approximate solutions to linear systems. SIAM Journal on Computing, 24(2), 227–234.CrossRefMathSciNetMATHGoogle Scholar
  45. Olshausen, B. A., & Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583), 607–609.CrossRefGoogle Scholar
  46. Ophir, B., Elad, M., Bertin, N., & Plumbley, M. (2011). Sequential minimal eigenvalues: An approach to analysis dictionary learning. In Proceedings of the European Signal Processing Conference (EUSIPCO) (pp. 1465–1469).Google Scholar
  47. Pati, Y., Rezaiifar, R., & Krishnaprasad, P. (1993). Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In Asilomar Conference on Signals, Systems and Computers (Vol. 1, pp. 40–44).Google Scholar
  48. Peleg, T., & Elad, M. (2014). A statistical prediction model based on sparse representations for single image super-resolution. IEEE Transactions on Image Processing, 23(6), 2569–2582.Google Scholar
  49. Peyré, G., & Fadili, J. (2011). Learning analysis sparsity priors. In Proceedings of the International Conference on Sampling Theory and Applications (SampTA), Singapore. http://hal.archives-ouvertes.fr/hal-00542016/. Accessed 2014.
  50. Pfister, L. (2013). Tomographic reconstruction with adaptive sparsifying transforms. Master’s Thesis, University of Illinois at Urbana-Champaign.Google Scholar
  51. Pfister, L., & Bresler, Y. (2014). Model-based iterative tomographic reconstruction with adaptive sparsifying transforms. In S. P. I. E. International (Ed.), Symposium on Electronic Imaging: Computational Imaging XII, to appear.Google Scholar
  52. Pratt, W. K., Kane, J., & Andrews, H. C. (1969). Hadamard transform image coding. Proceedings of the IEEE, 57(1), 58–68.CrossRefGoogle Scholar
  53. Ramirez, I., Sprechmann, P., & Sapiro, G. (2010). Classification and clustering via dictionary learning with structured incoherence and shared features. In 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 3501–3508).Google Scholar
  54. Ravishankar, S., & Bresler, Y. (2011a). MR image reconstruction from highly undersampled k-space data by dictionary learning. IEEE Transactions on Medical Imaging, 30(5), 1028–1041.CrossRefGoogle Scholar
  55. Ravishankar, S., & Bresler, Y. (2011b). Multiscale dictionary learning for MRI. In Proceedings of ISMRM (p. 2830).Google Scholar
  56. Ravishankar, S., & Bresler, Y. (2012a). Learning doubly sparse transforms for image representation. In IEEE International Conference on Image Processing (pp 685–688).Google Scholar
  57. Ravishankar, S., & Bresler, Y. (2012b). Learning sparsifying transforms for signal and image processing. In SIAM Conference on Imaging Science (p. 51).Google Scholar
  58. Ravishankar, S., & Bresler, Y. (2013a). Closed-form solutions within sparsifying transform learning. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE (pp. 5378–5382).Google Scholar
  59. Ravishankar, S., & Bresler, Y. (2013b). Learning doubly sparse transforms for images. IEEE Transactions on Image Processing, 22(12), 4598–4612.CrossRefMathSciNetGoogle Scholar
  60. Ravishankar, S., & Bresler, Y. (2013c). Learning sparsifying transforms. IEEE Transactions on Signal Processing, 61(5), 1072–1086.CrossRefMathSciNetGoogle Scholar
  61. Ravishankar, S., & Bresler, Y. (2013d). Sparsifying transform learning for compressed sensing MRI. In Proceedings of the IEEE International Symposium on Biomedical Imaging (pp. 17–20).Google Scholar
  62. Ravishankar, S., & Bresler, Y. (2014). \(\ell _0\) Sparsifying transform learning with efficient optimal updates and convergence guarantees. In IEEE Transactions on Signal Processing. (submitted). https://uofi.box.com/s/vrw0i13jbkj6n8xh9u9h. Accessed 2014.
  63. Ravishankar, S., & Bresler, Y. (2014). Online sparsifying transform learning: Part II: Convergence analysis. IEEE Journal of Selected Topics in Signal Process. (accepted). https://uofi.box.com/s/cmqme2avnz5pygobxj3u. Accessed 2014.
  64. Rubinstein, R., & Elad, M. (2011). K-SVD dictionary-learning for analysis sparse models. In Proceedings of SPARS11 (p. 73).Google Scholar
  65. Rubinstein, R., Bruckstein, A. M., & Elad, M. (2010). Dictionaries for sparse representation modeling. Proceedings of the IEEE, 98(6), 1045–1057.CrossRefGoogle Scholar
  66. Rubinstein, R., Faktor, T., & Elad, M. (2012). K-SVD dictionary-learning for the analysis sparse model. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 5405–5408).Google Scholar
  67. Sadeghi, M., Babaie-Zadeh, M., & Jutten, C. (2013). Dictionary learning for sparse representation: A novel approach. IEEE Signal Processing Letters, 20(12), 1195–1198.CrossRefGoogle Scholar
  68. Sahoo, S. K., & Makur, A. (2013). Dictionary training for sparse representation as generalization of k-means clustering. IEEE Signal Processing Letters, 20(6), 587–590.CrossRefGoogle Scholar
  69. Skretting, K., & Engan, K. (2010). Recursive least squares dictionary learning algorithm. IEEE Transactions on Signal Processing, 58(4), 2121–2130.CrossRefMathSciNetGoogle Scholar
  70. Smith, L. N., & Elad, M. (2013). Improving dictionary learning: Multiple dictionary updates and coefficient reuse. IEEE Signal Processing Letters, 20(1), 79–82.CrossRefGoogle Scholar
  71. Spielman, D. A., Wang, H., & Wright, J. (2012). Exact recovery of sparsely-used dictionaries. In Proceedings of the 25th Annual Conference on Learning Theory (pp. 37.1–37.18).Google Scholar
  72. Sprechmann, P., Bronstein, A., & Sapiro, G. (2012a). Learning efficient structured sparse models. In Proceedings of the 29th International Conference on Machine Learning (Vol. 1, pp. 615–622).Google Scholar
  73. Sprechmann, P., Bronstein, A. M., Sapiro, G. (2012b). Learning efficient sparse and low rank models. arXiv:1212.3631, Preprint.
  74. Wang, S., Zhang, L., Liang, Y., Pan, Q. (2012). Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In IEEE Conference on Computer Vision and Pattern Recognition (pp. 2216–2223).Google Scholar
  75. Weiss, Y. (2011). Yair Weiss home page. Retrieved 2014, from http://www.cs.huji.ac.il/~daniez/epllcode.zip.
  76. Xu, Y., Yin, W. (2013). A fast patch-dictionary method for whole-image recovery ftp://ftp.math.ucla.edu/pub/camreport/cam13-38.pdf, UCLA CAM report 13–38.
  77. Yaghoobi, M., Blumensath, T., & Davies, M. (2009). Dictionary learning for sparse approximations with the majorization method. IEEE Transaction on Signal Processing, 57(6), 2178–2191.CrossRefMathSciNetGoogle Scholar
  78. Yaghoobi, M., Nam, S., Gribonval, R., & Davies, M. (2011). Analysis operator learning for overcomplete cosparse representations. In European Signal Processing Conference (EUSIPCO) (pp. 1470–1474).Google Scholar
  79. Yaghoobi, M., Nam, S., Gribonval, R., & Davies, M. E. (2012). Noise aware analysis operator learning for approximately cosparse signals. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 5409–5412).Google Scholar
  80. Yaghoobi, M., Nam, S., Gribonval, R., & Davies, M. E. (2013). Constrained overcomplete analysis operator learning for cosparse signal modelling. IEEE Transactions on Signal Processing, 61(9), 2341–2355.CrossRefGoogle Scholar
  81. Yu, G., Sapiro, G., & Mallat, S. (2012). Solving inverse problems with piecewise linear estimators: From gaussian mixture models to structured sparsity. IEEE Transactions on Image Processing, 21(5), 2481–2499.CrossRefMathSciNetGoogle Scholar
  82. Zelnik-Manor, L., Rosenblum, K., & Eldar, Y. C. (2012). Dictionary optimization for block-sparse representations. IEEE Transactions on Signal Processing, 60(5), 2386–2395.CrossRefMathSciNetGoogle Scholar
  83. Zoran, D., & Weiss, Y. (2011). From learning models of natural image patches to whole image restoration. In IEEE International Conference on Computer Vision (pp. 479–486).Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Bihan Wen
    • 1
  • Saiprasad Ravishankar
    • 1
  • Yoram Bresler
    • 1
  1. 1.Department of Electrical and Computer Engineering and the Coordinated Science LaboratoryUniversity of IllinoisUrbana-ChampaignUSA

Personalised recommendations