Evaluating Sparse Codes on Handwritten Digits
Sparse coding of visual information has been of interest to the neuroscientific community for many decades and it is widely recognised that sparse codes should exhibit a high degree of statistical independence, typically measured by the kurtosis of the response distributions. In this paper we extend work on the hierarchical temporal memory model by studying the suitability of the augmented spatial pooling (ASP) sparse coding algorithm in comparison with independent component analysis (ICA) when applied to the recognition of handwritten digits. We present an extension to the ASP algorithm that forms synaptic receptive fields located closer to their respective columns and show that this produces lower Naïve Bayes classification errors than both ICA and the original ASP algorithm. In evaluating kurtosis as a predictor of classification performance, we also show that additional measures of dispersion and mutual information are needed to reliably distinguish between competing approaches.
KeywordsMutual Information Independent Component Analysis Independent Component Analysis Sparse Code Code Unit
Unable to display preview. Download preview PDF.
- 5.George, D., Hawkins, J.: A hierarchical Bayesian model of invariant pattern recognition in the visual cortex. In: Proceedings of the International Joint Conference on Neural Networksm, IJCNN 2005, pp. 1812–1817 (2005)Google Scholar
- 6.Hawkins, J., Ahmad, S., Dubinsky, D.: Hierarchical temporal memory including HTM cortical learning algorithms. Tech. rep., Numenta, Inc, Palto Alto (2011), https://www.groksolutions.com/technology.html#cla-whitepaper
- 7.Hawkins, J., Blakeslee, S.: On intelligence. Henry Holt, New York (2004)Google Scholar
- 8.Hawkins, J., George, D.: Hierarchical temporal memory: Concepts, theory and terminology. Tech. rep., Numenta, Inc, Palto Alto (2006), http://www.numenta.com/htm-overview/education/Numenta_HTM_Concepts.pdf
- 11.Le, Q.V., Zou, W.Y., Yeung, S.Y., Ng, A.Y.: Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3361–3368 (2011)Google Scholar
- 12.LeCun, Y., Cortes, C.: MNIST handwritten digit database. AT&T Labs (1998), http://yann.lecun.com/exdb/mnist
- 17.Stone, J.V.: Independent component analysis. Wiley Online Library (2004)Google Scholar
- 22.Willmore, B., Tolhurst, D.J.: Characterizing the sparseness of neural codes. Network: Computation in Neural Systems 12(3), 255–270 (2001)Google Scholar