Cognitive Computation

, Volume 6, Issue 2, pp 145–157 | Cite as

Decoding Word Information from Spatiotemporal Activity of Sensory Neurons

  • Kazuhisa FujitaEmail author
  • Yusuke Hara
  • Youichi Suzukawa
  • Yoshiki Kashimori


Spatiotemporal activity of neurons is ubiquitous in sensory coding in the CNS. It is a fundamental problem for sensory perception to understand how sensory information is decoded from the spatiotemporal activity. However, little is known about the decoding mechanism. To address this issue, we are concerned with auditory system as a model system exhibiting spatiotemporal activity. We present here a model of auditory cortex, which performs a hierarchical processing of auditory information. The model consists of three layers of two-dimensional networks. The first layer represents auditory stimulus as a spatiotemporal activity of neurons. The second layer consists of feature-detecting neurons, which extract the features of phonemes and their overlaps from the spatiotemporal activity of the first layer. The third layer combines information of the sound features encoded by the second layer and decodes word information about the sound stimulus as a temporal sequence of attractors. Using the model, we show how the information of phonemes and words emerge in the hierarchical processing of the auditory cortex. We also show that the overlap between phonemes plays a crucial role in linking the attractors of phonemes. The present study may provide a clue for understanding the mechanism by which word information is decoded from spatiotemporal activity of neurons.


Decoding mechanism Spatiotemporal activity Word information Auditory system Neural model 


  1. 1.
    Ahissar E, Zacksenhouse M. Temporal and spatial coding in the rat vibrissal system. Prog Brain Res. 2001;130:75–87.PubMedCrossRefGoogle Scholar
  2. 2.
    Barak O, Tsodyks M. Recognition by variance: learning rules for spatiotemporal patterns. Neural Comput. 2006;18:2343–58.PubMedCrossRefGoogle Scholar
  3. 3.
    Bregman AS, Campbell J. Primary auditory stream segregation and perception of order in rapid sequences of tones. J Exp Psychol. 1971;89:244–9.PubMedCrossRefGoogle Scholar
  4. 4.
    Bregman AS. Auditory scene analysis: the perceptual organization of sound. Cambridge: A Bradford Book; 1994.Google Scholar
  5. 5.
    Buonomano DV, Merzenich MM. Temporal information transformed into a spatial code by a neural network with realistic properties. Science. 1995;267:1028–30.PubMedCrossRefGoogle Scholar
  6. 6.
    Chang EF, Rieger JW, Johnson K, Berger MS, Barbaro NM, Knight RT. Categorical speech representation in human superior temporal gyrus. Nat Neurosci. 2010;13:1428–32.PubMedCentralPubMedCrossRefGoogle Scholar
  7. 7.
    Dahmen JC, Hartley DEH, King AJ. Stimulus-timing-dependent plasticity of cortical frequency representation. J Neurosci. 2008;28:13629–39.PubMedCentralPubMedCrossRefGoogle Scholar
  8. 8.
    DeAngelis GC, Ohzawa I, Freeman RD. Spatiotemporal organization of simple-cell receptive fields in the cat’s striate cortex II linearity of temporal and spatial summation. J Neurophysiol. 1993;69:1118–35.PubMedGoogle Scholar
  9. 9.
    deCharms R, Blake D, Merzenich M. Optimizing sound features for cortical neurons. Science. 1998;280:1439–43.PubMedCrossRefGoogle Scholar
  10. 10.
    DeWitt I, Rauschecker JP. Phoneme and word recognition in the auditory ventral stream. Proc Natl Acad Sci USA. 2012;109:E505–14.PubMedCentralPubMedCrossRefGoogle Scholar
  11. 11.
    Drullman R. Temporal envelope and fine structure cues for speech intelligibility. J Acoust Soc Am. 1995;97:585–92.PubMedCrossRefGoogle Scholar
  12. 12.
    Fishman YI, Resera DH, Arezzoa JC, Steinschneidera M. Neural correlates of auditory stream segregation in primary auditory cortex of the awake monkey. Hear Res. 2001;151:167–87.PubMedCrossRefGoogle Scholar
  13. 13.
    Freiwald WA, Tsao DY. Functional compartmentalization and viewpoint generalization within the macaque face-processing system. Science. 2010;330:845–51.PubMedCentralPubMedCrossRefGoogle Scholar
  14. 14.
    Fowler CA. Segmentation of coarticulated speech in perception. Percept Psychophys. 1984;36:359–68.PubMedCrossRefGoogle Scholar
  15. 15.
    Fujita K, Kashimori Y, Kambara T. Spatiotemporal burst coding for extracting features of spatiotemporally varying stimuli. Biol Cybern. 2007;97:293–305.PubMedCrossRefGoogle Scholar
  16. 16.
    Fukunishi K, Murai N, Uno H. Dynamic characteristics of the auditory cortex of guinea pigs observed with multichannel optical recording. Biol Cybern. 1992;67:501–9.PubMedCrossRefGoogle Scholar
  17. 17.
    Fukunishi K, Murai N. Temporal coding in the guinea-pig auditory cortex as revealed by optical imaging and its pattern-time-series analysis. Biol Cybern. 1995;72:463–73.PubMedCrossRefGoogle Scholar
  18. 18.
    Gerstner W, Kempter R, Hemmen JL, Wagner H. A neuronal learning rule for sub-millisecond temporal coding. Nature. 1996;383:76–81.PubMedCrossRefGoogle Scholar
  19. 19.
    Ghitza O. Linking speech perception and neurophysiology: speech decoding guided by cascaded oscillators locked to the input rhythm. Front Psychol. 2011;2:130.PubMedCentralPubMedCrossRefGoogle Scholar
  20. 20.
    Horikawa J, Hosokawa Y, Kubota M, Nasu M, Taniguchi I. Optical imaging of spatiotemporal patterns of glutamatergic excitation and GABAergic inhibition in the guinea-pig auditory cortex in vivo. J Physiol. 1996;497:629–38.PubMedCentralPubMedGoogle Scholar
  21. 21.
    Izhikevich EM. Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex. 2007;17:2443–52.PubMedCrossRefGoogle Scholar
  22. 22.
    Jaeger H, Haas H. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science. 2004;304:78–80.PubMedCrossRefGoogle Scholar
  23. 23.
    Karmarkar UR, Najarian MT, Buonomano DV. Mechanisms and significance of spike-timing dependent plasticity. Biol Cybern. 2002;87:373–382.PubMedCrossRefGoogle Scholar
  24. 24.
    Knüsel P, Wyss R, König P, Verschure PFMJ. Decoding a temporal population code. Neural Comput. 2004;16:2079–2100.Google Scholar
  25. 25.
    Koch C. Biophysics of computation: information processing in single neurons (computational neuroscience). 1st ed. Oxford: Oxford University Press; 1998.Google Scholar
  26. 26.
    Kohonen T. Self-organizing maps. 3rd ed. Berlin: Springer; 2001.CrossRefGoogle Scholar
  27. 27.
    Laurent G. Olfactory network dynamics and the coding of multidimensional signals. Nat Rev Neurosci. 2002;3:884–95.PubMedCrossRefGoogle Scholar
  28. 28.
    Legenstein R, Pecevski D, Maass W. A learning theory for reward-modulated spike-timing-dependent plasticity with application to biofeedback. PLoS Comput Biol. 2008;4:e1000180.PubMedCentralPubMedCrossRefGoogle Scholar
  29. 29.
    Liberman AM. The grammars of speech and language. Cogn Psychol. 1970;1:301–23.CrossRefGoogle Scholar
  30. 30.
    Maass W, Natschläger T, Markram H. Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput. 2002;14:2531–60.PubMedCrossRefGoogle Scholar
  31. 31.
    Mauk MD, Buonomano DV. The neural basis of temporal processing. Annu Rev Neurosci. 2004;27:307–40.PubMedCrossRefGoogle Scholar
  32. 32.
    Mazor O, Laurent G. Transient dynamics versus fixed points in odor representations by locust antennal lobe projection neurons. Neuron. 2005;48:661–73.PubMedCrossRefGoogle Scholar
  33. 33.
    McClelland JL, Elman JL. The TRACE model of speech perception. Cogn Psychol. 1986;18:1–86.PubMedCrossRefGoogle Scholar
  34. 34.
    Nikolic D, Häusler S, Singer W, Maass W. Temporal dynamics of information content carried by neurons in the primary visual cortex. In: Schölkopf B, Platt JC, Hoffman T, editors. Advances in neural information processing systems, vol 19. Vancouver: MIT Press; 2007. p.1041–8.Google Scholar
  35. 35.
    Rao RPN, Ballard DH. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat Neurosci. 1999;2:79–87.PubMedCrossRefGoogle Scholar
  36. 36.
    Riesenhuber M, Poggio T. Models of object recognition. Nat Neurosci. 1999;3:1199–203.CrossRefGoogle Scholar
  37. 37.
    Schnupp JWH, Hall TM, Kokelaar RF, Ahmed B. Plasticity of temporal pattern codes for vocalization stimuli in primary auditory cortex. J Neurosci. 2006;26:4785–95.PubMedCrossRefGoogle Scholar
  38. 38.
    Schnupp J, Nelken I, King A. Auditory neuroscience: making sense of sound. Cambridge: The MIT Press; 2011.Google Scholar
  39. 39.
    Shannon RV, Zeng F-G, Kamath V, Wygonski J, Ekelid M. Speech recognition with primarily temporal cues. Science. 1995;270:303–4.PubMedCrossRefGoogle Scholar
  40. 40.
    Song S, Miller KD, Abbott LF. Competitive Hebbian learning through spike-timing-dependent synaptic plasticity. Nat Neurosci. 2000;3:919–26.PubMedCrossRefGoogle Scholar
  41. 41.
    Sprekeler H, Michaelis C, Wiskott L. Slowness: an objective for spike-timing-dependent plasticity? PLoS Comput Biol. 2007;3:e112.PubMedCentralPubMedCrossRefGoogle Scholar
  42. 42.
    Stevens KN. Toward a model for lexical access based on acoustic landmarks and distinctive features. J Acoust Soc Am. 2002;111:1872–91.PubMedCrossRefGoogle Scholar
  43. 43.
    Tanaka K. Inferotemporal cortex and object vision. Annu Rev Neurosci. 1996;19:109–39.PubMedCrossRefGoogle Scholar
  44. 44.
    Taniguchi I, Horikawa J, Moriyama T, Nasu M. Spatio-temporal pattern of frequency representation in the auditory cortex of guinea pigs. Neurosci Lett. 1992;146:37–40.PubMedCrossRefGoogle Scholar
  45. 45.
    Taniguchi I, Nasu M. Spatio-temporal representation of sound intensity in the guinea pig auditory cortex observed by optical recording. Neurosci Lett. 1993;151:178–81.PubMedCrossRefGoogle Scholar
  46. 46.
    Theunissen FE, Sen K, Doupe AJ. Spectral-temporal receptive fields of nonlinear auditory neurons obtained using natural sounds. J Neurosci. 2000;20:2315–31.PubMedGoogle Scholar
  47. 47.
    Tzounopoulos T, Kim Y, Oertel D, Trussell LO. Cell-specific, spike timing-dependent plasticities in the dorsal cochlear nucleus. Nat Neurosci. 2004;7:719–25.PubMedCrossRefGoogle Scholar
  48. 48.
    Yamaguchi Y, Horikawa J, Taniguchi I. Neural dynamics of vocal processing in the auditory cortex. In: Poznanski RR, editor. Biophysical neural networks. New York: Mary Ann Liebert; 2001. p. 343–62.Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Kazuhisa Fujita
    • 1
    • 2
    Email author
  • Yusuke Hara
    • 3
  • Youichi Suzukawa
    • 3
  • Yoshiki Kashimori
    • 2
    • 3
  1. 1.Tsuyama National College of TechnologyOkayamaJapan
  2. 2.Department of Engineering ScienceUniversity of Electro-CommunicationsTokyoJapan
  3. 3.Graduate School of Information SystemsUniversity of Electro-CommunicationsTokyoJapan

Personalised recommendations