Psychophysical Models of Sound Localisation with Audiovisual Interactions

Abstract

Visual signals can have an important impact on the perceived location of sound sources. Neurological mechanisms enable interactions between seeing and hearing to form a sense of space. The effect of vision on auditory localisation percepts is of fundamental importance. A sound source is either perceived at the location of the visual source or it is perceptually shifted toward it’s direction. This bias is one form of visual capture. The extent of the interactions depends on time and space constraints beyond which visual and auditory cues do not necessarily interact. These constraints and interactions vary for the localisation of sources along the horizontal and vertical planes, as well as with distance. While the traditional models of audiovisual interaction in space perception assume sensory integration, recent models allow for sensory cues to either interact or not. Models of visual dominance, modality appropriateness, and maximum likelihood estimation predict one combined percept. The newer models of causal inference allow for varied perceptual outcomes depending on the relationship of the different sensory cues. Finally, visual spatial cues can induce changes to how sounds are localised after the audiovisual experience. This notorious effect, known as the ventriloquism aftereffect, is possibly the main mechanism of auditory space learning and calibration. The ventriloquism aftereffect has been described with a causal inference model and with an inverse model. The current chapter discusses all of the above concepts, establishing a connection between psychophysical data and available models.