Multimodal System Based on Self-organizing Maps

  • Magnus Johnsson
  • Christian Balkenius
  • Germund Hesslow
Part of the Studies in Computational Intelligence book series (SCI, volume 343)


We present experiments with a multimodal system based on a novel variant of the Self-Organizing Map (SOM) called the Associative Self-Organizing Map (A-SOM). The A-SOM is similar to the SOM and develops a representation of its input space, but also learns to associate its activity with additional inputs, e.g. the activities of one or several external SOMs. This enables the modelling of expectations in one sensory modality due to the activity elicited in another modality. The paper presents the A-SOM algorithm generalized to an arbitrary number of (possibly delayed) associated activities together with simulation results with a multimodal sensory system and its extension to a system that also includes an action network. The simulation results were very encouraging and confirmed: The ability of the A-SOM to learn to associate the representations of its input space with the representations of the input spaces developed in two connected SOMs; The ability of the extended system to elicit proper activity in the action network; The simulations demonstrated good generalization ability.


Self-Organizing Map Neural network Associative Self-Organizing Map A-SOM SOM ANN Expectations Simulation hypothesis Cognitive modelling Cross-modal activation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Balkenius, C., Morén, J., Johansson, B., Johnsson, M.: Ikaros: Building cognitive models for robots. In: Hülse, M., Hild, M. (eds.) Workshop on current software frameworks in cognitive robotics integrating different computational paradigms (in conjunction with IROS 2008), Nice, France, pp. 47–54 (2008)Google Scholar
  2. 2.
    Bartolomeo, P.: The relationship between visual perception and visual mental imagery: a reappraisal of the neuropsychological evidence. Cortex 38, 357–378 (2002)CrossRefGoogle Scholar
  3. 3.
    Bishop, C.M.: Neural Networks for Pattern Recognition. Oxford University Press, Oxford (1995)Google Scholar
  4. 4.
    Carpenter, G.A., Grossberg, S., Markuzon, N., Reynolds, J.H., Rosen, D.B.: Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps. IEEE Transactions on Neural Networks 3, 698–713 (1992)CrossRefGoogle Scholar
  5. 5.
    Hesslow, G.: Conscious thought as simulation of behaviour and perception. Trends Cogn. Sci. 6, 242–247 (2002)CrossRefGoogle Scholar
  6. 6.
    Johnsson, M., Balkenius, C.: Associating SOM representations of haptic submodalities. In: Ramamoorthy, S., Hayes, G.M. (eds.) Towards Autonomous Robotic Systems 2008, pp. 124–129 (2008)Google Scholar
  7. 7.
    Johnsson, M., Balkenius, C., Hesslow, G.: Neural network architecture for crossmodal activation and perceptual sequences. In: Papers from the AAAI Fall Symposium Biologically Inspired Cognitive Architectures 2009, pp. 85–86 (2009)Google Scholar
  8. 8.
    Kayser, C., Petkov, C.I., Augath, M., Logothetis, N.K.: Functional imaging reveals visual modification of specific fields in auditory cortex. J. Neurosci. 27, 1824–1835 (2007)CrossRefGoogle Scholar
  9. 9.
    Kohonen, T.: Self-Organization and Associative Memory. Springer, Heidelberg (1988)zbMATHGoogle Scholar
  10. 10.
    Kosslyn, S.M., Ganis, G., Thompson, W.L.: Neural foundations of imagery. Nature Rev. Neurosci. 2, 635–642 (2001)CrossRefGoogle Scholar
  11. 11.
    McGurk, H., MacDonald, J.: Hearing lips and seeing voices. Nature 264, 746–748 (1976)CrossRefGoogle Scholar
  12. 12.
    Nguyen, L.D., Woon, K.Y., Tan, A.H.: A self-organizing neural model for multimedia information fusion. In: International Conference on Information Fusion 2008, pp. 1738–1744 (2008)Google Scholar
  13. 13.
    Tan, A.H.: Adaptive resonance associative map. Neural Networks 8, 437–446 (1995)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Magnus Johnsson
    • 1
  • Christian Balkenius
    • 1
  • Germund Hesslow
    • 2
  1. 1.Lund University Cognitive Science, KungshusetLundagårdSweden
  2. 2.Department of Experimental Medical ScienceLund, BMC F10Sweden

Personalised recommendations