Cognitive Neurodynamics

, Volume 12, Issue 5, pp 481–499 | Cite as

An oscillatory neural network model that demonstrates the benefits of multisensory learning

  • A. Ravishankar Rao
Research Article


Since the world consists of objects that stimulate multiple senses, it is advantageous for a vertebrate to integrate all the sensory information available. However, the precise mechanisms governing the temporal dynamics of multisensory processing are not well understood. We develop a computational modeling approach to investigate these mechanisms. We present an oscillatory neural network model for multisensory learning based on sparse spatio-temporal encoding. Recently published results in cognitive science show that multisensory integration produces greater and more efficient learning. We apply our computational model to qualitatively replicate these results. We vary learning protocols and system dynamics, and measure the rate at which our model learns to distinguish superposed presentations of multisensory objects. We show that the use of multiple channels accelerates learning and recall by up to 80%. When a sensory channel becomes disabled, the performance degradation is less than that experienced during the presentation of non-congruent stimuli. This research furthers our understanding of fundamental brain processes, paving the way for multiple advances including the building of machines with more human-like capabilities.


Oscillatory neural networks Synchronization Binding Multisensory processing Learning Audio–visual processing 



The author greatly appreciates helpful comments from the reviewers, which improved this manuscript.


  1. Acebrón JA, Bonilla LL, Vicente CJP, Ritort F, Spigler R (2005) The kuramoto model: a simple paradigm for synchronization phenomena. Rev Mod Phys 77(1):137CrossRefGoogle Scholar
  2. Amedi A, von Kriegstein K, van Atteveldt NM, Beauchamp M, Naumer MJ (2005) Functional imaging of human crossmodal identification and object recognition. Exp Brain Res 166(3–4):559–571CrossRefGoogle Scholar
  3. Bahrick LE, Lickliter R (2012) The role of intersensory redundancy in early perceptual, cognitive, and social development. In: Bremner A, Lewkowicz DJ, Spence C (eds) Multisensory development. Oxford University Press, Oxford, pp 183–205CrossRefGoogle Scholar
  4. Balasubramaniam P, Banu LJ (2014) Synchronization criteria of discrete-time complex networks with time-varying delays and parameter uncertainties. Cognit Neurodyn 8(3):199–215CrossRefGoogle Scholar
  5. Bastiaansen M, Hagoort P (2006) Oscillatory neuronal dynamics during language comprehension. Prog Brain Res 159:179–196CrossRefGoogle Scholar
  6. Bavelier D, Neville HJ (2002) Cross-modal plasticity: Where and how? Nat Rev Neurosci 3(6):443CrossRefGoogle Scholar
  7. Coco M, Badino L, Cipresso P, Chirico A, Ferrari E, Riva G, Gaggioli A, D’Ausilio A (2016) Multilevel behavioral synchronisation in a joint tower-building task. IEEE Trans Cognit Dev Syst 99:1–1Google Scholar
  8. Darrell T, Fisher Iii JW, Viola P (2000) Audio-visual segmentation and the cocktail party effect. In: Advances in multimodal interfaces ICMI 2000. Springer, pp 32–40Google Scholar
  9. Davis ET, Scott K, Pair J, Hodges LF, Oliverio J (1999) Can audio enhance visual perception and performance in a virtual environment? In: Proceedings of the human factors and ergonomics society annual meeting, vol. 43, no. 22. SAGE Publications, pp 1197–1201Google Scholar
  10. Driver J, Noesselt T (2008) Multisensory interplay reveals crossmodal influences on sensory-specificbrain regions, neural responses, and judgments. Neuron 57(1):11–23CrossRefGoogle Scholar
  11. Falchier A, Clavagnier S, Barone P, Kennedy H (2002) Anatomical evidence of multimodal integration in primate striate cortex. J Neurosci 22(13):5749–5759CrossRefGoogle Scholar
  12. Feng Y, Lapata M (2010) Visual information in semantic representation. In: Human language technologies: the, (2010) annual conference of the north American chapter of the association for computational linguistics. Association for Computational Linguistics, pp 91–99Google Scholar
  13. Fetsch CR, DeAngelis GC, Angelaki DE (2013) Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons. Nat Rev Neurosci 14(6):429–442CrossRefGoogle Scholar
  14. Formisano E, Kim D, Di Salle F, van de Moortele P, Ugurbil K, Goebel R (2003) Mirror-symmetric tonotopic maps in human primary auditory cortex. Neuron 40(4):859–869CrossRefGoogle Scholar
  15. Garagnani M, Lucchese G, Tomasello R, Wennekers T, Pulvermüller F (2017) A spiking neurocomputational model of high-frequency oscillatory brain responses to words and pseudowords. Front Comput Neurosci. CrossRefPubMedPubMedCentralGoogle Scholar
  16. Gershman SJ, Horvitz EJ, Tenenbaum JB (2015) Computational rationality: a converging paradigm for intelligence in brains, minds, and machines. Science 349(6245):273–278CrossRefGoogle Scholar
  17. Ghazanfar AA, Schroeder CE (2006) Is neocortex essentially multisensory? Trends Cognit Sci 10(6):278–285CrossRefGoogle Scholar
  18. Gray C, König P, Engel A, Singer W (1989) Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature 338(6213):334–337CrossRefGoogle Scholar
  19. Grossberg S, Somers D (1991) Synchronized oscillations during cooperative feature linking in a cortical model of visual perception. Neural Netw 4(4):453–466CrossRefGoogle Scholar
  20. Guo D, Wang Q, Perc M (2012) Complex synchronous behavior in interneuronal networks with delayed inhibitory and fast electrical synapses. Phys Rev E 85(6):061905CrossRefGoogle Scholar
  21. Guo D, Perc M, Zhang Y, Xu P, Yao D (2017) Frequency-difference-dependent stochastic resonance in neural systems. Phys Rev E 96(2):022415CrossRefGoogle Scholar
  22. Haykin S, Chen Z (2005) The cocktail party problem. Neural Comput 17(9):1875–1902CrossRefGoogle Scholar
  23. Hikosaka O, Nakamura K, Sakai K, Nakahara H (2002) Central mechanisms of motor skill learning. Curr Opin Neurobiol 12(2):217–222CrossRefGoogle Scholar
  24. Jamone L, Ugur E, Cangelosi A, Fadiga L, Bernardino A, Piater J, Santos-Victor J (2016) Affordances in psychology, neuroscience and robotics: a survey. IEEE Trans Cognit Dev Syst 99:1–1Google Scholar
  25. Kazanovich Y, Borisyuk R (2017) Reaction times in visual search can be explained by a simple model of neural synchronization. Neural Netw 87:1–7CrossRefGoogle Scholar
  26. Khaleghi B, Khamis A, Karray FO, Razavi SN (2013) Multisensor data fusion: a review of the state-of-the-art. Inf Fusion 14(1):28–44CrossRefGoogle Scholar
  27. Kiewra KA (2002) How classroom teachers can help students learn and teach them how to learn. Theory Pract 41(2):71–80CrossRefGoogle Scholar
  28. Kopell NJ, Gritton HJ, Whittington MA, Kramer MA (2014) Beyond the connectome: the dynome. Neuron 83(6):1319–1328CrossRefGoogle Scholar
  29. Lee P-L, Wang C-L, Hamman D, Hsiao C-H, Huang C-H (2013) Notetaking instruction enhances students’ science learning. Child Dev Res. CrossRefGoogle Scholar
  30. Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cognit Brain Res 14(1):115–128CrossRefGoogle Scholar
  31. Mudrik L, Lamy D, Deouell LY (2010) Erp evidence for context congruity effects during simultaneous object-scene processing. Neuropsychologia 48(2):507–517CrossRefGoogle Scholar
  32. Murray MM, Thelen A, Thut G, Romei V, Martuzzi R, Matusz PJ (2016) The multisensory function of the human primary visual cortex. Neuropsychologia 83:161–169CrossRefGoogle Scholar
  33. Noda K, Arie H, Suga Y, Ogata T (2014) Multimodal integration learning of robot behavior using deep neural networks. Robot Auton Syst 62(6):721–736CrossRefGoogle Scholar
  34. Qu J, Wang R, Yan C, Du Y (2014) Oscillations and synchrony in a cortical neural network. Cognit Neurodyn 8(2):157–166CrossRefGoogle Scholar
  35. Quak M, London RE, Talsma D (2015) A multisensory perspective of working memory. Front Hum Neurosci. CrossRefPubMedPubMedCentralGoogle Scholar
  36. Rao A, Cecchi G (2010) An objective function utilizing complex sparsity for efficient segmentation. Int J Intell Comput Cybern 3(2):173–206CrossRefGoogle Scholar
  37. Rao AR, Cecchi G (2013) Multi-sensory integration using sparse spatio-temporal encoding. In: Neural networks (IJCNN), The 2013 international joint conference on. IEEE, pp 1–8Google Scholar
  38. Rao AR, Cecchi GA, Peck CC, Kozloski JR (2008) Unsupervised segmentation with dynamical units. IEEE Trans Neural Netw 19(1):168–182CrossRefGoogle Scholar
  39. Rohde M, van Dam LC, Ernst MO (2016) Statistically optimal multisensory cue integration: a practical tutorial. Multisens Res 29(4–5):279–317CrossRefGoogle Scholar
  40. Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117CrossRefGoogle Scholar
  41. Seitz AR, Kim R, Shams L (2006) Sound facilitates visual learning. Curr Biol 16(14):1422–1427CrossRefGoogle Scholar
  42. Shams L, Kim R (2010) Crossmodal influences on visual perception. Phys Life Rev 7(3):269–284CrossRefGoogle Scholar
  43. Shams L, Seitz AR (2008) Benefits of multisensory learning. Trends Cognit Sci 12(11):411–417CrossRefGoogle Scholar
  44. Socher R, Lin CC, Manning C, Ng AY, (2011) Parsing natural scenes and natural language with recursive neural networks. In: Proceedings of the 28th international conference on machine learning (ICML-11), pp 129–136Google Scholar
  45. Sompolinsky H, Golomb D, Kleinfeld D (1990) Global processing of visual stimuli in a neural network of coupled oscillators. Proc Natl Acad Sci 87(18):7200–7204CrossRefGoogle Scholar
  46. Spence C (2011) Crossmodal correspondences: a tutorial review. Atten Percept Psychophys 73(4):971–995CrossRefGoogle Scholar
  47. Thelen A, Talsma D, Murray MM (2015) Single-trial multisensory memories affect later auditory and visual object discrimination. Cognition 138:148–160CrossRefGoogle Scholar
  48. van Atteveldt N, Murray MM, Thut G, Schroeder CE (2014) Multisensory integration: flexible use of general operations. Neuron 81(6):1240–1253CrossRefGoogle Scholar
  49. van der Velde F, de Kamps M (2002) Synchrony in the eye of the beholder: an analysis of the role of neural synchronization in cognitive processes. Brain Mind 3(3):291–312CrossRefGoogle Scholar
  50. Van Rullen R (2017) Perception science in the age of deep neural networks, Frontiers in Psychology, vol. 8, p. 142, [Online]. Available:
  51. Von der Malsburg C (1999) The what and why of binding: the modelers perspective. Neuron 24(1):95–104CrossRefGoogle Scholar
  52. Wang L (1996) Oscillatory and chaotic dynamics in neural networks under varying operating conditions. IEEE Trans Neural Netw 7(6):1382–1388CrossRefGoogle Scholar
  53. Wang DL, Brown GJ (1999) Separation of speech from interfering sounds based on oscillatory correlation. IEEE Trans Neural Netw 10(3):684–697CrossRefGoogle Scholar
  54. Wang Q, Chen G, Perc M (2011) Synchronous bursts on scale-free neuronal networks with attractive and repulsive coupling. PLoS ONE 6(1):e15851CrossRefGoogle Scholar
  55. Yamashita I, Katahira K, Igarashi Y, Okanoya K, Okada M (2013) Recurrent network for multisensory integration-identification of common sources of audiovisual stimuli. Front Comput Neurosci. CrossRefPubMedPubMedCentralGoogle Scholar
  56. Yilmaz E, Uzuntarla M, Ozer M, Perc M (2013) Stochastic resonance in hybrid scale-free neuronal networks. Physica A Stat Mech Its Appl 392(22):5735–5741CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2018

Authors and Affiliations

  1. 1.Gildart Haase School of Computer Sciences and EngineeringFairleigh Dickinson UniversityTeaneckUSA

Personalised recommendations