Inputs, Outputs, and Multisensory Processing

Abstract

Humans, like other animals, have sensory structures allowing them to transduce diverse physical stimuli into signals that the nervous system can process. Sensory pathways, characteristic for each type of “sense,” process unisensory stimuli at various stages before they feed into multimodal and “higher” structures implementing eventually processes such as spatial orientation, object recognition, maintenance of homeostasis, or specific motivation. Those pathways, typically involving subcortical and cortical parts, establish a hierarchical structure. Hierarchical processing involves not only feed forward but also parallel and feedback processing so that progressively more complex properties of the signals are represented. Complex representations of the environmental and bodily signals are only partly accessible to our subjective experience. In this chapter, we introduce the processing pathways and hierarchies of the main sensory systems. Further, we put special emphasis on how sensory representations are integrated into coherent multisensory representations of our environment. Multisensory perception not only offers specific advantages over unisensory perception but also entails specific difficulties leading to multisensory illusions. We describe psychophysical and neural principles of how the nervous system generates our multisensory perception. Finally, we give an outlook on possible roles that multisensory perceptions may play in mental disorders.