Sparse Coding Neural Gas Applied to Image Recognition

Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 198)


A generalization of the Sparse Coding Neural Gas (SCNG) algorithm for feature learning is proposed and is then discussed in the context of modern classifier techniques for images. Two versions are presented. The latter obtains faster convergence by exploiting the nature of particular feature coding methods. The algorithm is then used as part of a larger image classification system, which is tested on the MNIST handwritten digit dataset and the NORB object dataset, obtaining results close to state-of-the-art methods.


Neural Gas Sparse Coding Sparse Coding Neural Gas Image Recognition Matching Pursuit 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Jarrett, K., Kavukcuoglu, K., Ranzato, M., Keen, Y.: What is the Best Multi-Stage Architecture for Object Recognition? In: Proc. International Conference on Computer Vision, ICCV 2009 (2009)Google Scholar
  2. 2.
    Olshausen, B., Field, D.: Emergence of Simple-Cell Receptive Field Properties by Learning a Sparse Code for Natural Images. Nature 381, 607–609 (1996)CrossRefGoogle Scholar
  3. 3.
    Olshausen, B., Field, D.: Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by V1? Vision Research 37, 3311–3325 (1998)CrossRefGoogle Scholar
  4. 4.
    Krizhevsky, A.: Learning Multiple Layers of Features from Tiny Images (2009)Google Scholar
  5. 5.
    Kavukcuoglu, K., Sermanet, P., Boureau, Y., Gregor, K., Mathieu, M., LeCun, Y.: Learning Convolutional Feature Hierarchies for Visual Recognition. In: Advances in Neural Information Processing Systems (NIPS 2010), vol. 23 (2010)Google Scholar
  6. 6.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-Based Learning Applied to Document Recognition. In: Intelligent Signal Processing, pp. 306–351. IEEE Press (2001)Google Scholar
  7. 7.
    LeCun, Y., Kavukcuoglu, K., Farabet, C.: Convolutional Networks and Applications in Vision. In: Proc. International Symposium on Circuits and Signals (ISCAS 2010). IEEE Press (2010)Google Scholar
  8. 8.
    Simard, P.Y., Steinkraus, D., Platt, J.C.: Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. In: Int’l Conference on Document Analysis and Recognition, pp. 958–963 (2003)Google Scholar
  9. 9.
    Labusch, K., Barth, E., Martinetz, T.: Simple Method for High-Performance Digit Recognition Based on Sparse Coding. IEEE Transactions on Neural Networks 19(11), 1985–1989 (2008)CrossRefGoogle Scholar
  10. 10.
    Henaff, M., Jarret, K., Kavukcuoglu, K., LeCun, Y.: Unsupervised Learning for Scalable Audio Classification. In: Proceedings of International Symposium on Music Information Retrieval, ISMIR 2011 (2011)Google Scholar
  11. 11.
    Labusch, K., Barth, E., Martinetz, T.: Sparse Coding Neural Gas: Learning of Overcomplete Data Representations. Neurocomputing 72(7-9), 1547–1555 (2009)CrossRefGoogle Scholar
  12. 12.
    Labusch, K., Barth, E., Martinetz, T.: Learning Data Representations with Sparse Coding Neural Gas. In: Proceedings of the 16th European Symposium on Artificial Neural Networks, pp. 233–238 (2008)Google Scholar
  13. 13.
    Labusch, K., Barth, E., Martinetz, T.: Demixing Jazz-Music: Sparse Coding Neural Gas for the Separation of Noisy Overcomplete Sources. Neural Network World 19(5), 561–579 (2009)Google Scholar
  14. 14.
    Labusch, K., Barth, E., Martinetz, T.: Sparse Coding Neural Gas for the Separation of Noisy Overcomplete Sources. In: Kůrková, V., Neruda, R., Koutník, J. (eds.) ICANN 2008, Part I. LNCS, vol. 5163, pp. 788–797. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  15. 15.
    Martinetz, T., Schulten, K.: A “Neural-Gas” Network Learns Toplogies. Artificial Neural Networks 1, 397–402 (1991)Google Scholar
  16. 16.
    Martinetz, T., Berkovich, S., Schulten, K.: “Neural-Gas” Network for Vector Quantization and its Application to Time-Series Prediction. IEEE Transactions on Neural Networks 4(4), 397–402 (1991)Google Scholar
  17. 17.
    Saxe, A., Koh, P.W., Chen, Z., Bahand, M., Suresh, B., Ng, A.: On Random Weights and Unsupervised Feature Learning. In: Proceedings of the Twenty-Eight International Conference on Machine Learning (2011)Google Scholar
  18. 18.
    Raina, R., Battle, A., Lee, H., Packer, B., Ng, A.: Self-taught Learning: Transfer Learning from Unlabeled Data. In: Proceedings of the 24th International Conference on Machine Learning (ICML 2007), pp. 759–766 (2007)Google Scholar
  19. 19.
    Donoho, D.: For Most Large Underdetermined Systems of Linear Equations the Minimal \(\mathcal{L}_1\)-norm Solution is also the Sparsest Solution. Communications on Pure and Applied Mathematics 59, 797–766 (2007)Google Scholar
  20. 20.
    Mallat, Z., Zhang, Z.: Matching Pursuits With Time-Frequency Dictionaries. IEEE Transactions on Signal Processing 41, 3397–3451 (1993)MATHCrossRefGoogle Scholar
  21. 21.
    Davis, G., Mallat, S., Zhang, Z.: Adaptive Time-Frequency Decomposition with Matching Pursuits. SPIE Journal of Optical Engineering 33, 2183–2191 (1994)Google Scholar
  22. 22.
    Pati, Y., Rezaiifar, R., Krishnaprasad, P.: Orthogonal Matching Pursuit: Recursive Function Approximation with Applications to Wavelet Decomposition. In: Asilomar Conference on Signals, Systems and Computers (1993)Google Scholar
  23. 23.
    Blumensath, T., Davies, M.: On the Difference Between Orthogonal Matching Pursuit and Orthogonal Least Squares (2007)Google Scholar
  24. 24.
    Oja, E.: Simplified Neuron Model as a Principal Component Analyzer. Journal of Mathematical Biology 15, 267–273 (1982)MathSciNetMATHCrossRefGoogle Scholar
  25. 25.
    LeCun, Y., Huang, F.-J., Bottou, L.: Learning Methods for Generic Object Recognition with Invariance to Pose and Lighting. In: Proceedings of CVPR 2004 (2004)Google Scholar
  26. 26.
    Fan, R.-E., Chang, K.-W., Hsieh, C.-J., Wang, X.-R., Lin, C.-J.: LIBLINEAR: A Library for Large Linear Classification. J. Mach. Learn. Res. 9, 1871–1874 (2008)MATHGoogle Scholar
  27. 27.
    Cireşan, D., Meier, U., Masci, J., Gambardella, L., Schmidhuber, J.: High-Performance Neural Networks for Visual Object Classification (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Institute for Neuro- and BioinformaticsUniversity of LübeckLübeckGermany
  2. 2.LAPIThe “POLITEHNICA” University of BucureştiBucureştiRomania

Personalised recommendations