Cognitive Neurodynamics

, Volume 12, Issue 5, pp 481–499 | Cite as

An oscillatory neural network model that demonstrates the benefits of multisensory learning

  • A. Ravishankar RaoEmail author
Research Article


Since the world consists of objects that stimulate multiple senses, it is advantageous for a vertebrate to integrate all the sensory information available. However, the precise mechanisms governing the temporal dynamics of multisensory processing are not well understood. We develop a computational modeling approach to investigate these mechanisms. We present an oscillatory neural network model for multisensory learning based on sparse spatio-temporal encoding. Recently published results in cognitive science show that multisensory integration produces greater and more efficient learning. We apply our computational model to qualitatively replicate these results. We vary learning protocols and system dynamics, and measure the rate at which our model learns to distinguish superposed presentations of multisensory objects. We show that the use of multiple channels accelerates learning and recall by up to 80%. When a sensory channel becomes disabled, the performance degradation is less than that experienced during the presentation of non-congruent stimuli. This research furthers our understanding of fundamental brain processes, paving the way for multiple advances including the building of machines with more human-like capabilities.


Oscillatory neural networks Synchronization Binding Multisensory processing Learning Audio–visual processing 



The author greatly appreciates helpful comments from the reviewers, which improved this manuscript.


  1. Acebrón JA, Bonilla LL, Vicente CJP, Ritort F, Spigler R (2005) The kuramoto model: a simple paradigm for synchronization phenomena. Rev Mod Phys 77(1):137CrossRefGoogle Scholar
  2. Amedi A, von Kriegstein K, van Atteveldt NM, Beauchamp M, Naumer MJ (2005) Functional imaging of human crossmodal identification and object recognition. Exp Brain Res 166(3–4):559–571CrossRefPubMedGoogle Scholar
  3. Bahrick LE, Lickliter R (2012) The role of intersensory redundancy in early perceptual, cognitive, and social development. In: Bremner A, Lewkowicz DJ, Spence C (eds) Multisensory development. Oxford University Press, Oxford, pp 183–205CrossRefGoogle Scholar
  4. Balasubramaniam P, Banu LJ (2014) Synchronization criteria of discrete-time complex networks with time-varying delays and parameter uncertainties. Cognit Neurodyn 8(3):199–215CrossRefGoogle Scholar
  5. Bastiaansen M, Hagoort P (2006) Oscillatory neuronal dynamics during language comprehension. Prog Brain Res 159:179–196CrossRefPubMedGoogle Scholar
  6. Bavelier D, Neville HJ (2002) Cross-modal plasticity: Where and how? Nat Rev Neurosci 3(6):443CrossRefPubMedGoogle Scholar
  7. Coco M, Badino L, Cipresso P, Chirico A, Ferrari E, Riva G, Gaggioli A, D’Ausilio A (2016) Multilevel behavioral synchronisation in a joint tower-building task. IEEE Trans Cognit Dev Syst 99:1–1Google Scholar
  8. Darrell T, Fisher Iii JW, Viola P (2000) Audio-visual segmentation and the cocktail party effect. In: Advances in multimodal interfaces ICMI 2000. Springer, pp 32–40Google Scholar
  9. Davis ET, Scott K, Pair J, Hodges LF, Oliverio J (1999) Can audio enhance visual perception and performance in a virtual environment? In: Proceedings of the human factors and ergonomics society annual meeting, vol. 43, no. 22. SAGE Publications, pp 1197–1201Google Scholar
  10. Driver J, Noesselt T (2008) Multisensory interplay reveals crossmodal influences on sensory-specificbrain regions, neural responses, and judgments. Neuron 57(1):11–23CrossRefPubMedPubMedCentralGoogle Scholar
  11. Falchier A, Clavagnier S, Barone P, Kennedy H (2002) Anatomical evidence of multimodal integration in primate striate cortex. J Neurosci 22(13):5749–5759CrossRefPubMedGoogle Scholar
  12. Feng Y, Lapata M (2010) Visual information in semantic representation. In: Human language technologies: the, (2010) annual conference of the north American chapter of the association for computational linguistics. Association for Computational Linguistics, pp 91–99Google Scholar
  13. Fetsch CR, DeAngelis GC, Angelaki DE (2013) Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons. Nat Rev Neurosci 14(6):429–442CrossRefPubMedGoogle Scholar
  14. Formisano E, Kim D, Di Salle F, van de Moortele P, Ugurbil K, Goebel R (2003) Mirror-symmetric tonotopic maps in human primary auditory cortex. Neuron 40(4):859–869CrossRefPubMedGoogle Scholar
  15. Garagnani M, Lucchese G, Tomasello R, Wennekers T, Pulvermüller F (2017) A spiking neurocomputational model of high-frequency oscillatory brain responses to words and pseudowords. Front Comput Neurosci. CrossRefPubMedPubMedCentralGoogle Scholar
  16. Gershman SJ, Horvitz EJ, Tenenbaum JB (2015) Computational rationality: a converging paradigm for intelligence in brains, minds, and machines. Science 349(6245):273–278CrossRefPubMedGoogle Scholar
  17. Ghazanfar AA, Schroeder CE (2006) Is neocortex essentially multisensory? Trends Cognit Sci 10(6):278–285CrossRefGoogle Scholar
  18. Gray C, König P, Engel A, Singer W (1989) Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature 338(6213):334–337CrossRefPubMedGoogle Scholar
  19. Grossberg S, Somers D (1991) Synchronized oscillations during cooperative feature linking in a cortical model of visual perception. Neural Netw 4(4):453–466CrossRefGoogle Scholar
  20. Guo D, Wang Q, Perc M (2012) Complex synchronous behavior in interneuronal networks with delayed inhibitory and fast electrical synapses. Phys Rev E 85(6):061905CrossRefGoogle Scholar
  21. Guo D, Perc M, Zhang Y, Xu P, Yao D (2017) Frequency-difference-dependent stochastic resonance in neural systems. Phys Rev E 96(2):022415CrossRefPubMedGoogle Scholar
  22. Haykin S, Chen Z (2005) The cocktail party problem. Neural Comput 17(9):1875–1902CrossRefPubMedGoogle Scholar
  23. Hikosaka O, Nakamura K, Sakai K, Nakahara H (2002) Central mechanisms of motor skill learning. Curr Opin Neurobiol 12(2):217–222CrossRefPubMedGoogle Scholar
  24. Jamone L, Ugur E, Cangelosi A, Fadiga L, Bernardino A, Piater J, Santos-Victor J (2016) Affordances in psychology, neuroscience and robotics: a survey. IEEE Trans Cognit Dev Syst 99:1–1Google Scholar
  25. Kazanovich Y, Borisyuk R (2017) Reaction times in visual search can be explained by a simple model of neural synchronization. Neural Netw 87:1–7CrossRefPubMedGoogle Scholar
  26. Khaleghi B, Khamis A, Karray FO, Razavi SN (2013) Multisensor data fusion: a review of the state-of-the-art. Inf Fusion 14(1):28–44CrossRefGoogle Scholar
  27. Kiewra KA (2002) How classroom teachers can help students learn and teach them how to learn. Theory Pract 41(2):71–80CrossRefGoogle Scholar
  28. Kopell NJ, Gritton HJ, Whittington MA, Kramer MA (2014) Beyond the connectome: the dynome. Neuron 83(6):1319–1328CrossRefPubMedPubMedCentralGoogle Scholar
  29. Lee P-L, Wang C-L, Hamman D, Hsiao C-H, Huang C-H (2013) Notetaking instruction enhances students’ science learning. Child Dev Res. CrossRefGoogle Scholar
  30. Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cognit Brain Res 14(1):115–128CrossRefGoogle Scholar
  31. Mudrik L, Lamy D, Deouell LY (2010) Erp evidence for context congruity effects during simultaneous object-scene processing. Neuropsychologia 48(2):507–517CrossRefPubMedGoogle Scholar
  32. Murray MM, Thelen A, Thut G, Romei V, Martuzzi R, Matusz PJ (2016) The multisensory function of the human primary visual cortex. Neuropsychologia 83:161–169CrossRefPubMedGoogle Scholar
  33. Noda K, Arie H, Suga Y, Ogata T (2014) Multimodal integration learning of robot behavior using deep neural networks. Robot Auton Syst 62(6):721–736CrossRefGoogle Scholar
  34. Qu J, Wang R, Yan C, Du Y (2014) Oscillations and synchrony in a cortical neural network. Cognit Neurodyn 8(2):157–166CrossRefGoogle Scholar
  35. Quak M, London RE, Talsma D (2015) A multisensory perspective of working memory. Front Hum Neurosci. CrossRefPubMedPubMedCentralGoogle Scholar
  36. Rao A, Cecchi G (2010) An objective function utilizing complex sparsity for efficient segmentation. Int J Intell Comput Cybern 3(2):173–206CrossRefGoogle Scholar
  37. Rao AR, Cecchi G (2013) Multi-sensory integration using sparse spatio-temporal encoding. In: Neural networks (IJCNN), The 2013 international joint conference on. IEEE, pp 1–8Google Scholar
  38. Rao AR, Cecchi GA, Peck CC, Kozloski JR (2008) Unsupervised segmentation with dynamical units. IEEE Trans Neural Netw 19(1):168–182CrossRefPubMedGoogle Scholar
  39. Rohde M, van Dam LC, Ernst MO (2016) Statistically optimal multisensory cue integration: a practical tutorial. Multisens Res 29(4–5):279–317CrossRefPubMedGoogle Scholar
  40. Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117CrossRefPubMedGoogle Scholar
  41. Seitz AR, Kim R, Shams L (2006) Sound facilitates visual learning. Curr Biol 16(14):1422–1427CrossRefPubMedGoogle Scholar
  42. Shams L, Kim R (2010) Crossmodal influences on visual perception. Phys Life Rev 7(3):269–284CrossRefPubMedGoogle Scholar
  43. Shams L, Seitz AR (2008) Benefits of multisensory learning. Trends Cognit Sci 12(11):411–417CrossRefGoogle Scholar
  44. Socher R, Lin CC, Manning C, Ng AY, (2011) Parsing natural scenes and natural language with recursive neural networks. In: Proceedings of the 28th international conference on machine learning (ICML-11), pp 129–136Google Scholar
  45. Sompolinsky H, Golomb D, Kleinfeld D (1990) Global processing of visual stimuli in a neural network of coupled oscillators. Proc Natl Acad Sci 87(18):7200–7204CrossRefPubMedGoogle Scholar
  46. Spence C (2011) Crossmodal correspondences: a tutorial review. Atten Percept Psychophys 73(4):971–995CrossRefPubMedGoogle Scholar
  47. Thelen A, Talsma D, Murray MM (2015) Single-trial multisensory memories affect later auditory and visual object discrimination. Cognition 138:148–160CrossRefPubMedGoogle Scholar
  48. van Atteveldt N, Murray MM, Thut G, Schroeder CE (2014) Multisensory integration: flexible use of general operations. Neuron 81(6):1240–1253CrossRefPubMedPubMedCentralGoogle Scholar
  49. van der Velde F, de Kamps M (2002) Synchrony in the eye of the beholder: an analysis of the role of neural synchronization in cognitive processes. Brain Mind 3(3):291–312CrossRefGoogle Scholar
  50. Van Rullen R (2017) Perception science in the age of deep neural networks, Frontiers in Psychology, vol. 8, p. 142, [Online]. Available:
  51. Von der Malsburg C (1999) The what and why of binding: the modelers perspective. Neuron 24(1):95–104CrossRefPubMedGoogle Scholar
  52. Wang L (1996) Oscillatory and chaotic dynamics in neural networks under varying operating conditions. IEEE Trans Neural Netw 7(6):1382–1388CrossRefPubMedGoogle Scholar
  53. Wang DL, Brown GJ (1999) Separation of speech from interfering sounds based on oscillatory correlation. IEEE Trans Neural Netw 10(3):684–697CrossRefPubMedGoogle Scholar
  54. Wang Q, Chen G, Perc M (2011) Synchronous bursts on scale-free neuronal networks with attractive and repulsive coupling. PLoS ONE 6(1):e15851CrossRefPubMedPubMedCentralGoogle Scholar
  55. Yamashita I, Katahira K, Igarashi Y, Okanoya K, Okada M (2013) Recurrent network for multisensory integration-identification of common sources of audiovisual stimuli. Front Comput Neurosci. CrossRefPubMedPubMedCentralGoogle Scholar
  56. Yilmaz E, Uzuntarla M, Ozer M, Perc M (2013) Stochastic resonance in hybrid scale-free neuronal networks. Physica A Stat Mech Its Appl 392(22):5735–5741CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2018

Authors and Affiliations

  1. 1.Gildart Haase School of Computer Sciences and EngineeringFairleigh Dickinson UniversityTeaneckUSA

Personalised recommendations