Skip to main content
SpringerLink
Log in

Learning Sparse Overcomplete Codes for Images

  • Published:
Journal of VLSI signal processing systems for signal, image and video technology Aims and scope Submit manuscript

Abstract

Images can be coded accurately using a sparse set of vectors from a learned overcomplete dictionary, with potential applications in image compression and feature selection for pattern recognition. We present a survey of algorithms that perform dictionary learning and sparse coding and make three contributions. First, we compare our overcomplete dictionary learning algorithm (FOCUSS-CNDL) with overcomplete independent component analysis (ICA). Second, noting that once a dictionary has been learned in a given domain the problem becomes one of choosing the vectors to form an accurate, sparse representation, we compare a recently developed algorithm (sparse Bayesian learning with adjustable variance Gaussians, SBL-AVG) to well known methods of subset selection: matching pursuit and FOCUSS. Third, noting that in some cases it may be necessary to find a non-negative sparse coding, we present a modified version of the FOCUSS algorithm that can find such non-negative codings. Efficient parallel implementations in VLSI could make these algorithms more practical for many applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. R. C. Gonzales and R. E. Woods, “Digital Image Processing,” Addison-Wesley, Reading, Massachusetts, 1993.

    Google Scholar 

  2. E. Oja, “A Simplified Neuron Model as a Principal Component Analyzer,” J. Math. Biol., vol. 15, 1982, pp. 267–273.

    Article  MathSciNet  Google Scholar 

  3. T. T. Pham and R. J. P. deFigueiredo,“Maximum Likelihood Estimation of a Class of Non-Gaussian Densities with Application to ℓ p Deconvolution,” IEEE Trans. Acoust. Speech Signal Process, vol. 37, no. 1, January 1989, pp. 73–82.

    Article  MathSciNet  Google Scholar 

  4. C. Jutten and J. Hérault, “Blind Separation of Sources. Part I. An Adaptive Algorithm Based on Neuromimetic Architecture,” Signal Process, vol. 24, 1991, pp. 1–10.

    Article  Google Scholar 

  5. K. Kreutz-Delgado, J. F. Murray, B. D. Rao, K. Engan, T.-W. Lee, and T. J. Sejnowski, “Dictionary Learning Algorithms for Sparse Representation,” Neural Comput., vol. 15, no. 2, February 2003, pp. 349–396.

    Article  Google Scholar 

  6. M. S. Lewicki and T. J. Sejnowski, “Learning Overcomplete Representations,” Neural Comput., vol. 12, no. 2, February 2000, pp. 337–365.

    Article  Google Scholar 

  7. K. Engan, J. H. Husoy, and S. O. Aase, “Frame Based Representation and Compression of Still Images,” in Proc. ICIP, 2001, pp. 1–4.

  8. B. A. Olshausen and D. J. Field, “Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by V1?,” Vis. Res., vol. 37, 1997, pp. 3311–3325.

    Article  Google Scholar 

  9. D. M. Weber and D. Casasent, “Quadratic Gabor Filters for Object Detection,” IEEE Trans. Image Process, vol. 10, no. 2, February 2001, pp. 218–230.

    Article  Google Scholar 

  10. B. D. Rao and K. Kreutz-Delgado, “An Affine Scaling Methodology for Best Basis Selection,” IEEE Trans. Signal Process, vol. 47, 1999, pp. 187–200.

    Article  MathSciNet  Google Scholar 

  11. M. E. Tipping, “Sparse Bayesian Learning and the Relevance Vector Machine,” J. Mach. Learn. Res., vol. 1, 2001, pp. 211–244.

    Article  MathSciNet  Google Scholar 

  12. S. F. Cotter, J. Adler, B. D. Rao, and K. Kreutz-Delgado, “Forward Sequential Algorithms for Best Basis Selection,” IEE Proc., Vis. Image Signal Process, vol. 146, no. 5, October 1999, pp. 235–244.

    Article  Google Scholar 

  13. J. F. Murray and K. Kreutz-Delgado, “An Improved FOCUSS-based Learning Algorithm for Solving Sparse Linear Inverse Problems,” in Conference Record of the 35th Asilomar Conference on Signals, Systems and Computers (IEEE), vol. 1, Pacific Grove, California, November 2001, pp. 347–351.

  14. M. S. Lewicki and B. A. Olshausen, “A probabilistic Framework for the Adaptation and Comparison of Image Codes,” J. Opt. Soc. Am. A, vol. 16, no. 7, July 1999, pp. 1587–1601.

    Google Scholar 

  15. K. Engan, “Frame Based Signal Representation and Compression.” PhD thesis, Stavanger Universty College, Norway, 2000.

  16. P. Paatero and U. Tapper, “Positive Matrix Factorization: A Nonnegative Factor Model with Optimal Utilization of Error Estimates of Data Values,” Environmetrics, vol. 5, 1994, pp. 111–126.

    Google Scholar 

  17. D. D. Lee and H. S. Seung, “Learning the Parts of Objects by Non-negative Matrix Factorization,” Nature, vol. 401, October 1999, pp. 788–791.

    Article  Google Scholar 

  18. P. O. Hoyer, “Non-negative Sparse Coding,” in Proc. of the 12th IEEE Workshop on Neural Networks for Sig. Proc., 2002, pp. 557–565.

  19. M. D. Plumbley, “Algorithms for Nonnegative Independent Component Analysis,” IEEE Trans. Neural Netw., vol. 14, no. 3, May 2003, pp. 534–543.

    Article  Google Scholar 

  20. S. Chen and D. Donoho, “Atomic Decomposition by Basis Pursuit,” SIAM J. Sci. Comput., vol. 20, no. 1, 1998, pp. 33–61.

    Article  MathSciNet  Google Scholar 

  21. D. Donoho and M. Elad, “Optimally Sparse Representation in General (Nonorthogonal) Dictionaries via l 1 Minimization,” Proc. Natl. Acad. Sci. U. S. A., vol. 100, no. 5, March 2003, pp. 2197–2202.

    Article  MathSciNet  Google Scholar 

  22. D. P. Wipf and B. D. Rao, “Probabilistic Analysis for Basis Selection via ℓ p Diversity Measures,” in Proc. of the Intl. Conf. on Acoustics, Speech, and Signal Processing, 2004 (ICASSP ’04), vol. 2, May 2004.

  23. D. P. Wipf and B. D. Rao, “Sparse Bayesian Learning for Basis Selection,” IEEE Trans. Signal Process, vol. 52, no. 8, 2004, pp. 2153–2164.

    Article  MathSciNet  Google Scholar 

  24. I. F. Gorodnitsky, J. S. George, and B. D. Rao, “Neuromagnetic Source Imaging with FOCUSS: A Recursive Weighted Minimum Norm Algorithm,” Electroencephalogr. Clin. Neurophysiol., vol. 95, no. 4, 1995, pp. 231–251.

    Article  Google Scholar 

  25. R. Vigário and E. Oja, “Independence: A New Criterion for the Analysis of the Electromagnetic Fields in the Global Brain?,” Neural Netw., vol. 13, 2000, pp. 891–907.

    Article  Google Scholar 

  26. C. M. Bishop and M. E. Tipping, “Bayesian Regression and Classification,” in Advances in Learning Theory: Methods, Models and Applications, volume 190 of NATO Science Series III: Computer Systems Sciences, J. Suykens, G. Horvath, S. Basu, C. Micchelli, and J. Vandewalle (Eds.), IOS, Amsterdam, 2003, pp. 267–285.

  27. R. J. Muirhead, “Aspects of Multivariate Statistical Theory,” Wiley, New York, 1982.

    MATH  Google Scholar 

  28. S. G. Mallat and Z. Zhang, “Matching Pursuits with Time-frequency Dictionaries,” IEEE Trans. Signal Process, vol. 41, no. 12, 1993, pp. 3397–3415.

    Article  Google Scholar 

  29. S. F. Cotter, “Subset Selection Algorithms with Applications.” PhD thesis, University of California at San Diego, 2001.

  30. K. Engan, K. Skretting, and J. H. Husoy, “A Family of Iterative LS-based Dictionary Learning Algorithms, ILS-DLA, for Sparse Signal Representation,” to appear, IEEE Trans. on Signal Processing.

  31. M. Girolami, “A Variational Method for Learning Sparse and Overcomplete Representations,” Neural Comput., vol. 13, 2001, pp. 2517–2532.

    Article  Google Scholar 

  32. J. A. Palmer and K. Kreutz-Delgado, “A General Framework for Component Estimation,” in Proc. 4th Intl. Symp. on ICA, 2003.

  33. M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation,” to appear, IEEE Trans. on Signal Processing.

  34. S. A. Nene, S. K. Nayar, and H. Murase, “Columbia Object Image Library (COIL-100),” Technical Report CUCS-006-96, Columbia University, 1996.

  35. G. H. Golub and C. F. V. Loan, “Matrix Computations,” John Hopkins University Press, Baltimore, Maryland, 1983.

    MATH  Google Scholar 

  36. S. Peng and S. Sedukhin, “Parallel Algorithm and Architectures for Two-step Division-free Gaussian Elimination,” in Algorithms and Architectures for Parallel Processing, 3rd International Conference on (ICAPP 97), 1997, pp. 489–502.

  37. J.-C. Tsay and P.-Y. Chang, “Design of Efficient Regular Arrays for Matrix Multiplication by Two-step Regularization,” IEEE Trans. Parallel Distrib. Syst., vol. 6, no. 2, February 1995, pp. 215–222.

    Article  Google Scholar 

  38. K. Muhammad and K. Roy, “Reduced Computational Redundancy Implementation of DSP Algorithms Using Computation Sharing Vector Scaling,” IEEE Transactions on VLSI Systems, vol. 10, no. 3, 2002, pp. 292–300.

    Article  Google Scholar 

  39. D. Zhang, “Parallel VLSI Neural System Design,” Springer, Berlin Heidelberg New York, 1998.

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Brain and Cognitive Sciences Department, Massachusetts Institute of Technology, 77 Massachusetts Ave. 46-5065, Cambridge, MA, 02139, USA

    Joseph F. Murray

  2. Electrical and Computer Engineering Department, University of California, San Diego, 9500 Gilman Dr. Dept 0407, La Jolla, CA, 92093-0407, USA

    Kenneth Kreutz-Delgado

Authors
  1. Joseph F. Murray
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kenneth Kreutz-Delgado
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joseph F. Murray.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Murray, J.F., Kreutz-Delgado, K. Learning Sparse Overcomplete Codes for Images. J VLSI Sign Process Syst Sign Image Video Technol 45, 97–110 (2006). https://doi.org/10.1007/s11265-006-9774-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11265-006-9774-5

Keywords

Search

Navigation