Sensory substitution devices convert information normally associated with one sense into another sense (e.g. converting vision into sound). This is often done to compensate for an impaired sense. The present research uses a multimodal approach in which both natural vision and sound-from-vision (‘soundscapes’) are simultaneously presented. Although there is a systematic correspondence between what is seen and what is heard, we introduce a local discrepancy between the signals (the presence of a target object that is heard but not seen) that the participant is required to locate. In addition to behavioural responses, the participants’ gaze is monitored with eye-tracking. Although the target object is only presented in the auditory channel, behavioural performance is enhanced when visual information relating to the non-target background is presented. In this instance, vision may be used to generate predictions about the soundscape that enhances the ability to detect the hidden auditory object. The eye-tracking data reveal that participants look for longer in the quadrant containing the auditory target even when they subsequently judge it to be located elsewhere. As such, eye movements generated by soundscapes reveal the knowledge of the target location that does not necessarily correspond to the actual judgment made. The results provide a proof of principle that multimodal sensory substitution may be of benefit to visually impaired people with some residual vision and, in normally sighted participants, for guiding search within complex scenes.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Arnott SR, Alain C (2011) The auditory dorsal pathway: orienting vision. Neurosci Biobehav Rev 35:2162–2173. doi:10.1016/j.neubiorev.2011.04.005
Brown DJ, Macpherson T, Ward J (2011) Seeing with sound? Exploring different characteristics of a visual-to-auditory sensory substitution device. Perception 40:1120–1135
Capelle C, Trullemans C, Arno P, Veraart C (1998) A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution. IEEE Trans Biomed Eng 45:1279–1293. doi:10.1109/10.720206
Colonius H, Arndt P (2001) A two-stage model for visual–auditory interaction in saccadic latencies. Percept Psychophys 63:126–147. doi:10.3758/BF03200508
Corneil BD, Wanrooij MV, Munoz DP, Opstal AJV (2002) Auditory–visual interactions subserving goal-directed saccades in a complex scene. J Neurophysiol 88:438–454
Deneve S, Pouget A (2004) Bayesian multisensory integration and cross-modal spatial links. J Physiol. doi:10.1016/j.jphysparis.2004.03.011
Frassinetti F, Pavani F, Ladavas E (2002) Acoustical vision of neglected stimuli: interaction among spatially converging audiovisual inputs in neglect patients. J Cogn Neurosci 14:62–69
Grothe B, Pecka M, McAlpine D (2010) Mechanisms of sound localization in mammals. Physiol Rev 90:983–1012. doi:10.1152/physrev.00026.2009
Haigh A, Brown DJ, Meijer P, Proulx MJ (2013) How well do you see what you hear? The acuity of visual-to-auditory sensory substitution. Front Psychol. doi:10.3389/fpsyg.2013.00330
Hamilton-Fletcher G, Ward J (2013) Representing colour through hearing and touch in sensory substitution devices. Multisens Res 26:503–532. doi:10.1163/22134808-00002434
Hancock PA, Mercado JE, Merlo J, Van Erp JBF (2013) Improving target detection in visual search through the augmenting multi-sensory cues. Ergonomics 56:729–738. doi:10.1080/00140139.2013.771219
Kastner S, De Weerd P, Pinsk MA, Elizondo MI, Desimone R, Ungerleider LG (2001) Modulation of sensory suppression: implications for receptive field sizes in the human visual cortex. J Neurophysiol 86:1398–1411
Kim J-K, Zatorre RJ (2008) Generalized learning of visual-to-auditory substitution in sighted individuals. Brain Res 242:263–275
Klapetek A, Ngo MK, Spence C (2012) Does crossmodal correspondence modulate the facilitatory effect of auditory cues on visual search? Atten Percept Psychophys 74:1154–1167. doi:10.3758/s13414-012-0317-9
Levy-Tzedek S, Hanassy S, Abboud S, Maidenbaum S, Amedi A (2012a) Fast, accurate reaching movements with a visual-to-auditory sensory substitution device. Restor Neurol Neurosci 30:313–323. doi:10.3233/rnn-2012-110219
Levy-Tzedek S, Novick I, Arbel R, Abboud S, Maidenbaum S, Vaadia E, Amedi A (2012b) Cross-sensory transfer of sensory-motor information: visuomotor learning affects performance on an audiomotor task, using sensory-substitution. Sci Rep. doi:10.1038/srep00949
Mannan SK, Pambakian ALM, Kennard C (2010) Compensatory strategies following visual search training in patients with homonymous hemianopia: an eye movement study. J Neurol 257:1812–1821
Matusz PJ, Eimer M (2013) Top-down control of audiovisual search by bimodal search templates. Psychophysiology 50:996–1009. doi:10.1111/psyp.12086
Meijer PBL (1992) An experimental system for auditory image representations. IEEE Trans Biomed Eng 39:112–121
Meredith MA, Stein BE (1986) Visual, auditory and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56:640–662
Orchard-Mills E, Van der Burg E, Alais D (2013) Amplitude-modulated auditory stimuli influence selection of visual spatial frequencies. J Vis. doi:10.1167/13.3.6
Pavani F, Ladavas E, Driver J (2003) Auditory and multisensory aspects of visuospatial neglect. Trends Cogn Sci 7:407–414
Poirier C, De Volder AG, Tranduy D, Scheiber C (2007) Pattern recognition using a device substituting audition for vision in blindfolded sighted subjects. Neuropsychologia 45:1108–1121
Proulx MJ, Ptito M, Amedi A (2014) Multisensory integration, sensory substitution and visual rehabilitation. Neurosci Biobehav Rev 41:1–2. doi:10.1016/j.neubiorev.2014.03.004
Renier L, Collignon O, Poirier C et al (2005) Cross-modal activation of visual cortex during depth perception using auditory substitution of vision. Neuroimage 26:573–580. doi:10.1016/j.neuroimage.2005.01.047
Spence C (2011) Crossmodal correspondences: a tutorial review. Atten Percept Psychophys 73:971–995. doi:10.3758/s13414-010-0073-7
Van der Burg E, Olivers CNL, Bronkhorst AW, Theeuwes J (2008) Pip and pop: nonspatial auditory signals improve spatial visual search. J Exp Psychol Hum Percep Perform 34:1053–1065. doi:10.1037/0096-15126.96.36.1993
Van der Burg E, Olivers CNL, Bronkhorst AW, Theeuwes J (2009) Poke and pop: tactile–visual synchrony increases visual saliency. Neurosci Lett 450:60–64. doi:10.1016/j.neulet.2008.11.002
Ward J, Meijer P (2010) Visual experiences in the blind induced by an auditory sensory substitution device. Conscious Cogn 19:492–500. doi:10.1016/j.concog.2009.10.006
Wright TD, Ward J (2013) The evolution of a visual-to-auditory sensory substitution device using interactive genetic algorithms. Q J Exp Psychol 66:1620–1638
Zambarbieri D, Schmid R, Magenes G, Prablanc C (1982) Saccadic responses evoked by presentation of visual and auditory targets. Exp Brain Res 47:417–427. doi:10.1007/BF00239359
Zihl J (1995) Visual scanning behavior in patients with homonymous hemianopia. Neuropsychologia 33:287–303
About this article
Cite this article
Wright, T.D., Margolis, A. & Ward, J. Using an auditory sensory substitution device to augment vision: evidence from eye movements. Exp Brain Res 233, 851–860 (2015). https://doi.org/10.1007/s00221-014-4160-1
- Sensory substitution