Biological Cybernetics

, Volume 108, Issue 1, pp 61–73 | Cite as

Classification using sparse representations: a biologically plausible approach

  • M. W. SpratlingEmail author
Original Paper


Representing signals as linear combinations of basis vectors sparsely selected from an overcomplete dictionary has proven to be advantageous for many applications in pattern recognition, machine learning, signal processing, and computer vision. While this approach was originally inspired by insights into cortical information processing, biologically plausible approaches have been limited to exploring the functionality of early sensory processing in the brain, while more practical applications have employed non-biologically plausible sparse coding algorithms. Here, a biologically plausible algorithm is proposed that can be applied to practical problems. This algorithm is evaluated using standard benchmark tasks in the domain of pattern classification, and its performance is compared to a wide range of alternative algorithms that are widely used in signal and image processing. The results show that for the classification tasks performed here, the proposed method is competitive with the best of the alternative algorithms that have been evaluated. This demonstrates that classification using sparse representations can be performed in a neurally plausible manner, and hence, that this mechanism of classification might be exploited by the brain.


Sparse coding Classification Predictive coding  Neural networks Pattern recognition 


Conflict of interest

The authors declare that they have no conflict of interest.


  1. Aharon M, Elad M, Bruckstein A (2006) K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans Signal Process 54(11):4311–4322CrossRefGoogle Scholar
  2. Bell AJ, Sejnowski TJ (1997) The ‘independent components’ of natural scenes are edge filters. Vis Res 37(23):3327–3338PubMedCentralPubMedCrossRefGoogle Scholar
  3. Berkes P, Turner RE, Sahani M (2009) A structured model of video reproduces primary visual cortical organisation. PLoS Comput Biol 5(9):e1000495Google Scholar
  4. Bociu I, Pitas I (2004) A new sparse image representation algorithm applied to facial expression recognition. In: Proceedings of the IEEE signal processing society workshop on machine learning for signal processing, pp 539–548Google Scholar
  5. Charles AS, Garrigues P, Rozell CJ (2012) A common network architecture efficiently implements a variety of sparsity-based inference problems. Neural Comput 24(12):3317–3339PubMedCentralPubMedCrossRefGoogle Scholar
  6. Damnjanovic I, Davies MEP, Plumbley MD (2010) Smallbox—an evaluation framework for sparse representations and dictionary learning algorithms. Signal Process 6365:418–425Google Scholar
  7. De Meyer K, Spratling MW (2011) Multiplicative gain modulation arises through unsupervised learning in a predictive coding model of cortical function. Neural Comput 23(6):1536–1567PubMedCrossRefGoogle Scholar
  8. De Meyer K, Spratling MW (2013) A model of partial reference frame transforms through pooling of gain-modulated responses. Cerebral Cortex 23(5):1230–1239PubMedCrossRefGoogle Scholar
  9. Elad M, Aharon M (2006) Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans Image Process 15(12):3736–3745PubMedCrossRefGoogle Scholar
  10. Elhamifar E, Vidal R (2011) Robust classification using structured sparse representation. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognitionGoogle Scholar
  11. Engan K, Skretting K, Husøy H (2007) Family of iterative LS-based dictionary learning algorithms, ILS-DLA, for sparse signal representation. Digit Signal Process 17:32–49CrossRefGoogle Scholar
  12. Falconbridge MS, Stamps RL, Badcock DR (2006) A simple Hebbian/anti-Hebbian network learns the sparse, independent components of natural images. Neural Comput 18(2):415–429PubMedCrossRefGoogle Scholar
  13. Fischer S, Cristóbal G, Redondo R (2006) Sparse overcomplete Gabor wavelet representation based on local competition. IEEE Trans Image Process 15(2):265–272Google Scholar
  14. Fischer S, Redondo R, Perrinet L, Cristóbal G (2007) Sparse approximation of images inspired from the functional architecture of the primary visual areas. EURASIP J Adv Signal Process 2007:90727CrossRefGoogle Scholar
  15. Földiák P (1990) Forming sparse representations by local anti-Hebbian learning. Biol Cybern 64:165–170PubMedCrossRefGoogle Scholar
  16. Hamker FH, Wiltschut J (2007) Hebbian learning in a model with dynamic rate-coded neurons: an alternative to the generative model approach for learning receptive fields from natural scenes. Netw Comput Neural Syst 18:249–266CrossRefGoogle Scholar
  17. Harpur GF (1997) Low entropy coding with unsupervised neural networks. PhD thesis, Department of Engineering. University of CambridgeGoogle Scholar
  18. Hoyer PO (2003) Modeling receptive fields with non-negative sparse coding. Neurocomputing 52–54:547–552CrossRefGoogle Scholar
  19. Hoyer PO (2004) Non-negative matrix factorization with sparseness constraints. J Mach Learn Res 5:1457–1469Google Scholar
  20. Hoyer PO, Hyvärinen A (2000) Independent component analysis applied to feature extraction from colour and stereo images. Netw Comput Neural Syst 11(3):191–210CrossRefGoogle Scholar
  21. Hyvarinen A, Hoyer P, Oja E (1998) Sparse code shrinkage for image denoising. In: Proceedings of the international joint conference on neural networks vol 2, pp 859–864Google Scholar
  22. Jehee JFM, Ballard DH (2009) Predictive feedback can account for biphasic responses in the lateral geniculate nucleus. PLoS Computat Biol 5(5):e1000373CrossRefGoogle Scholar
  23. Jiang Z, Lin Z, Davis LS (2011) Learning a discriminative dictionary for sparse coding via label consistent K-SVD. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognitionGoogle Scholar
  24. Jiang Z, Lin Z, Davis LS (2013) Label consistent K-SVD: learning a discriminative dictionary for recognition. IEEE Trans Pattern Anal Mach Intell 35(11):2651–2664Google Scholar
  25. Kang LW, Hsu CY, Chen HW, Lu CS, Lin CY, Pei SC (2011) Feature-based sparse representation for image similarity assessment. IEEE Trans Multimed 13(5):1019–1030CrossRefGoogle Scholar
  26. King PD, Zylberberg J, DeWeese MR (2013) Inhibitory interneurons decorrelate excitatory cells to drive sparse code formation in a spiking model of V1. J Neurosci 33(13):5475–5485PubMedCrossRefGoogle Scholar
  27. Lee H, Ekanadham C, Ng AY (2008) Sparse deep belief net model for visual area V2. In: Platt JC, Koller D, Singer Y, Roweis S (eds) Advances in Neural information processing systems, vol 20. MIT Press, Cambridge, MA, pp 873–880Google Scholar
  28. Lemme A, Reinhart RF, Steil JJ (2010) Efficient online learning of a non-negative sparse autoencoder. In: Proceedings of the European symposium on artificial neural networksGoogle Scholar
  29. Liu J, Jia Y (2012) A lateral inhibitory spiking neural network for sparse representation in visual cortex. Advances in brain inspired, cognitive systems. Springer, Berlin, pp 259–267Google Scholar
  30. Lücke J (2009) Receptive field self-organization in a model of the fine structure in V1 cortical columns. Neural Comput 21(10):2805–2845PubMedCrossRefGoogle Scholar
  31. Mairal J, Sapiro G, Elad M (2007) Multiscale sparse image representation with learned dictionaries. In: IEEE international conference on image processing (ICIP)Google Scholar
  32. Mairal J, Bach F, Ponce J, Sapiro G, Zisserman A (2008a) Discriminative learned dictionaries for local image analysis. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 1–8Google Scholar
  33. Mairal J, Bach F, Ponce J, Sapiro G, Zisserman A (2008b) Supervised dictionary learning. In: Advances in neural information processing systemsGoogle Scholar
  34. Mairal J, Bach F, Ponce J, Sapiro G, Zisserman A (2009) Non-local sparse models for image restoration. In: Proceedings of the international conference on computer vision, pp 2272–2279Google Scholar
  35. Mairal J, Bach F, Ponce J, Sapiro G (2010) Online learning for matrix factorization and sparse coding. J Mach Learn Res 11:19–60Google Scholar
  36. Mairal J, Bach F, Ponce J (2012) Task-driven dictionary learning. IEEE Trans Pattern Anal Mach Intell 32(4):791–804Google Scholar
  37. Murray JF, Kreutz-Delgado K (2006) Learning sparse overcomplete codes for images. J VLSI Signal Process 45:97–110Google Scholar
  38. Olshausen BA, Field DJ (1996) Emergence of simple-cell receptive properties by learning sparse code for natural images. Nature 381:607–609PubMedCrossRefGoogle Scholar
  39. Olshausen BA, Field DJ (1997) Sparse coding with an overcomplete basis set: a strategy employed by V1? Vis Res 37(23):3311–3325PubMedCrossRefGoogle Scholar
  40. Olshausen BA, Field DJ (2004) Sparse coding of sensory inputs. Curr Opin Neurobiol 14:481–487PubMedCrossRefGoogle Scholar
  41. Pece AEC (2002) The problem of sparse image coding. J Math Imaging Vis 17(2):89–108CrossRefGoogle Scholar
  42. Pece AEC, Petkov N (2000) Fast atomic decomposition by the inhibition method. In: Proceedings of the international conference on pattern recognition, pp 215–218Google Scholar
  43. Perrinet LU (2010) Role of homeostasis in learning sparse representations. Neural Comput 22(7):1812–1836PubMedCentralPubMedCrossRefGoogle Scholar
  44. Plumbley MD (2007) Dictionary learning for L1-exact sparse coding. Independent component analysis and signal separation. Springer, Berlin, pp 406–413CrossRefGoogle Scholar
  45. Ramirez I, Sprechmann P, Sapiro G (2010) Classification and clustering via dictionary learning with structured incoherence and shared features. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 3501–3508Google Scholar
  46. Rao RPN, Ballard DH (1999) Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neurosci 2(1):79–87PubMedCrossRefGoogle Scholar
  47. Rehn M, Sommer FT (2007) A network that uses few active neurons to code visual input predicts the diverse shapes of cortical receptive fields. J Comput Neurosci 22:135–146PubMedCrossRefGoogle Scholar
  48. Rozell CJ, Johnson D, Baraniuk R, Olshausen BA (2008) Sparse coding via thresholding and local competition in neural circuits. Neural Comput 20(10):2526–2563PubMedCrossRefGoogle Scholar
  49. Spratling MW (2006) Learning image components for object recognition. J Mach Learn Res 7:793–815Google Scholar
  50. Spratling MW (2008) Predictive coding as a model of biased competition in visual selective attention. Vis Res 48(12):1391–1408PubMedCrossRefGoogle Scholar
  51. Spratling MW (2010) Predictive coding as a model of response properties in cortical area V1. J Neurosci 30(9):3531–3543PubMedCrossRefGoogle Scholar
  52. Spratling MW (2011) A single functional model accounts for the distinct properties of suppression in cortical area V1. Vis Res 51(6):563–576PubMedCrossRefGoogle Scholar
  53. Spratling MW (2012a) Predictive coding accounts for V1 response properties recorded using reverse correlation. Biol Cybern 106(1):37–49PubMedCrossRefGoogle Scholar
  54. Spratling MW (2012b) Predictive coding as a model of the V1 saliency map hypothesis. Neural Netw 26:7–28PubMedCrossRefGoogle Scholar
  55. Spratling MW (2012c) Unsupervised learning of generative and discriminative weights encoding elementary image components in a predictive coding model of cortical function. Neural Comput 24(1):60–103PubMedCrossRefGoogle Scholar
  56. Spratling MW, Johnson MH (2004) Neural coding strategies and mechanisms of competition. Cogn Syst Res 5(2):93–117CrossRefGoogle Scholar
  57. Spratling MW, De Meyer K (2009) Kompass R (2009) Unsupervised learning of overlapping image components using divisive input modulation. Comput Intell Neurosci 381457:1–19CrossRefGoogle Scholar
  58. Sprechmann P, Sapiro G (2010) Dictionary learning and sparse coding for unsupervised clustering. In: IEEE international conference on acoustics speech and signal processing (ICASSP), pp 2042–2045Google Scholar
  59. Thiagarajan JJ, Spanias A (2011) Learning dictionaries for local sparse coding in image classification. In: Asilomar conference on signals, systems and computersGoogle Scholar
  60. Tošić I, Frossard P (2011) Dictionary learning. IEEE Signal Process Mag 28(2):27–38CrossRefGoogle Scholar
  61. Tropp JA, Wright SJ (2010) Computational methods for sparse solution of linear inverse problems. Proc IEEE 98(6):948–958CrossRefGoogle Scholar
  62. Van Hateren JH, van der Schaaf A (1998) Independent component filters of natural images compared with simple cells in primary visual cortex. Proc R Soc Lond Ser B Biol Sci 265:359–66CrossRefGoogle Scholar
  63. Weber C, Triesch J (2008) A sparse generative model of V1 simple cells with intrinsic plasticity. Neural Comput 20:1261–1284PubMedCrossRefGoogle Scholar
  64. Wei C-P, Chao Y-W, Yeh Y-R, Wang Y-CF (2013) Locally-sensitive dictionary learning for sparse representation based classification. Pattern Recogn 46:1277–1287CrossRefGoogle Scholar
  65. Wiltschut J, Hamker FH (2009) Efficient coding correlates with spatial frequency tuning in a model of V1 receptive field organization. Visual Neurosci 26:21–34CrossRefGoogle Scholar
  66. Wright J, Ma Y, Mairal J, Sapiro G, Huang T, Yan S (2009a) Sparse representation for computer vision and pattern recognition. Proc IEEE 98(6):1031–1044CrossRefGoogle Scholar
  67. Wright J, Yang A, Ganesh A, Sastry S, Ma Y (2009b) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31(2):27–210CrossRefGoogle Scholar
  68. Yang M, Zhang L (2010) Gabor feature based sparse representations for face recognition with gabor occlusion dictionary. In: Proceedings of the European conference on computer visionGoogle Scholar
  69. Yang A, Ganesh A, Zhou Z, Sastry S, Ma Y (2010) A review of fast \(\ell _1\)-minimization algorithms for robust face recognition. arXiv, 1007.3753Google Scholar
  70. Yang M, Zhang L, Feng X, Zhang D (2011) Fisher discrimination dictionary learning for sparse representation. In: Proceedings of the international conference on computer vision pp 543–550Google Scholar
  71. Zhang L, Yang M, Feng X (2011) Sparse representation or collaborative representation: Which helps face recognition? In: Proceedings of the international conference on computer vision, pp 471–478Google Scholar
  72. Zhang H, Zhang Y, Huang TS (2013) Simultaneous discriminative projection and dictionary learning for sparse representation based classification. Pattern Recogn 46:346–354Google Scholar
  73. Zhu M, Rozell CJ (2013) Visual nonclassical receptive field effects emerge from sparse coding in a dynamical system. PLoS Comput Biol 9(8):e1003191 Google Scholar
  74. Zylberberg J, Murphy JT, DeWeese MR (2011) A sparse coding model with synaptically local plasticity and spiking neurons can account for the diverse shapes of V1 simple cell receptive fields. PLoS Comput Biol 7(10):e1002250PubMedCentralPubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Department of InformaticsKing’s College LondonStrand, LondonUK

Personalised recommendations