Evaluating Sparse Codes on Handwritten Digits

  • Linda Main
  • Benjamin Cowley
  • Adam Kneller
  • John Thornton
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8272)


Sparse coding of visual information has been of interest to the neuroscientific community for many decades and it is widely recognised that sparse codes should exhibit a high degree of statistical independence, typically measured by the kurtosis of the response distributions. In this paper we extend work on the hierarchical temporal memory model by studying the suitability of the augmented spatial pooling (ASP) sparse coding algorithm in comparison with independent component analysis (ICA) when applied to the recognition of handwritten digits. We present an extension to the ASP algorithm that forms synaptic receptive fields located closer to their respective columns and show that this produces lower Naïve Bayes classification errors than both ICA and the original ASP algorithm. In evaluating kurtosis as a predictor of classification performance, we also show that additional measures of dispersion and mutual information are needed to reliably distinguish between competing approaches.


Mutual Information Independent Component Analysis Independent Component Analysis Sparse Code Code Unit 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Carlson, E.T., Rasquinha, R.J., Zhang, K., Connor, C.E.: A sparse object coding scheme in area v4. Current Biology 21(4), 288–293 (2011)CrossRefGoogle Scholar
  2. 2.
    Felleman, D.J., Van Essen, D.C.: Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex 1(1), 1–47 (1991)CrossRefGoogle Scholar
  3. 3.
    Friston, K.: Learning and inference in the brain. Neural Networks 16(9), 1325–1352 (2003)CrossRefGoogle Scholar
  4. 4.
    Fu, M., Yu, X., Lu, J., Zuo, Y.: Repetitive motor learning induces coordinated formation of clustered dendritic spines in vivo. Nature 483(7387), 92–95 (2012)CrossRefGoogle Scholar
  5. 5.
    George, D., Hawkins, J.: A hierarchical Bayesian model of invariant pattern recognition in the visual cortex. In: Proceedings of the International Joint Conference on Neural Networksm, IJCNN 2005, pp. 1812–1817 (2005)Google Scholar
  6. 6.
    Hawkins, J., Ahmad, S., Dubinsky, D.: Hierarchical temporal memory including HTM cortical learning algorithms. Tech. rep., Numenta, Inc, Palto Alto (2011),
  7. 7.
    Hawkins, J., Blakeslee, S.: On intelligence. Henry Holt, New York (2004)Google Scholar
  8. 8.
    Hawkins, J., George, D.: Hierarchical temporal memory: Concepts, theory and terminology. Tech. rep., Numenta, Inc, Palto Alto (2006),
  9. 9.
    Hyvärinen, A., Hurri, J., Hoyer, P.: Natural Image Statistics: A probabilistic approach to early computational vision. Springer-Verlag, New York Inc. (2009)CrossRefGoogle Scholar
  10. 10.
    Hyvärinen, A., Karhunen, J., Oja, E.: Independent Components Analysis. John Wiley and Sons, Inc., New York (2001)CrossRefGoogle Scholar
  11. 11.
    Le, Q.V., Zou, W.Y., Yeung, S.Y., Ng, A.Y.: Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3361–3368 (2011)Google Scholar
  12. 12.
    LeCun, Y., Cortes, C.: MNIST handwritten digit database. AT&T Labs (1998),
  13. 13.
    Lee, T.S., Mumford, D.: Hierarchical Bayesian inference in visual cortex. Journal of the Optical Society of America A 20(7), 1434–1448 (2003)CrossRefGoogle Scholar
  14. 14.
    Malone, B.J., Kumar, V.R., Ringach, D.L.: Dynamics of receptive field size in primary visual cortex. Journal of neurophysiology 97(1), 407–414 (2007)CrossRefGoogle Scholar
  15. 15.
    Mountcastle, V.B.: Introduction to the special issue on computation in cortical columns. Cerebral Cortex 13(1), 2–4 (2003)CrossRefGoogle Scholar
  16. 16.
    Olshausen, B.A., et al.: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381(6583), 607–609 (1996)CrossRefGoogle Scholar
  17. 17.
    Stone, J.V.: Independent component analysis. Wiley Online Library (2004)Google Scholar
  18. 18.
    Thornton, J., Main, L., Srbic, A.: Fixed frame temporal pooling. In: Thielscher, M., Zhang, D. (eds.) AI 2012. LNCS, vol. 7691, pp. 707–718. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  19. 19.
    Thornton, J., Srbic, A.: Spatial pooling for greyscale images. International Journal of Machine Learning and Cybernetics 4, 207–216 (2013)CrossRefGoogle Scholar
  20. 20.
    Thornton, J., Srbic, A., Main, L., Chitsaz, M.: Augmented spatial pooling. In: Wang, D., Reynolds, M. (eds.) AI 2011. LNCS, vol. 7106, pp. 261–270. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  21. 21.
    Willmore, B.D., Mazer, J.A., Gallant, J.L.: Sparse coding in striate and extrastriate visual cortex. Journal of Neurophysiology 105(6), 2907–2919 (2011)CrossRefGoogle Scholar
  22. 22.
    Willmore, B., Tolhurst, D.J.: Characterizing the sparseness of neural codes. Network: Computation in Neural Systems 12(3), 255–270 (2001)Google Scholar
  23. 23.
    Willmore, B., Watters, P.A., Tolhurst, D.J.: A comparison of natural-image-based models of simple-cell coding. Perception-London 29(9), 1017–1040 (2000)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2013

Authors and Affiliations

  • Linda Main
    • 1
  • Benjamin Cowley
    • 1
  • Adam Kneller
    • 1
  • John Thornton
    • 1
  1. 1.Institute for Integrated and Intelligent SystemsGriffith UniversityAustralia

Personalised recommendations