Using an auditory sensory substitution device to augment vision: evidence from eye movements

Abstract

Sensory substitution devices convert information normally associated with one sense into another sense (e.g. converting vision into sound). This is often done to compensate for an impaired sense. The present research uses a multimodal approach in which both natural vision and sound-from-vision (‘soundscapes’) are simultaneously presented. Although there is a systematic correspondence between what is seen and what is heard, we introduce a local discrepancy between the signals (the presence of a target object that is heard but not seen) that the participant is required to locate. In addition to behavioural responses, the participants’ gaze is monitored with eye-tracking. Although the target object is only presented in the auditory channel, behavioural performance is enhanced when visual information relating to the non-target background is presented. In this instance, vision may be used to generate predictions about the soundscape that enhances the ability to detect the hidden auditory object. The eye-tracking data reveal that participants look for longer in the quadrant containing the auditory target even when they subsequently judge it to be located elsewhere. As such, eye movements generated by soundscapes reveal the knowledge of the target location that does not necessarily correspond to the actual judgment made. The results provide a proof of principle that multimodal sensory substitution may be of benefit to visually impaired people with some residual vision and, in normally sighted participants, for guiding search within complex scenes.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

References

  1. Arnott SR, Alain C (2011) The auditory dorsal pathway: orienting vision. Neurosci Biobehav Rev 35:2162–2173. doi:10.1016/j.neubiorev.2011.04.005

    PubMed  Article  Google Scholar 

  2. Brown DJ, Macpherson T, Ward J (2011) Seeing with sound? Exploring different characteristics of a visual-to-auditory sensory substitution device. Perception 40:1120–1135

    PubMed  Article  Google Scholar 

  3. Capelle C, Trullemans C, Arno P, Veraart C (1998) A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution. IEEE Trans Biomed Eng 45:1279–1293. doi:10.1109/10.720206

    CAS  PubMed  Article  Google Scholar 

  4. Colonius H, Arndt P (2001) A two-stage model for visual–auditory interaction in saccadic latencies. Percept Psychophys 63:126–147. doi:10.3758/BF03200508

    CAS  PubMed  Article  Google Scholar 

  5. Corneil BD, Wanrooij MV, Munoz DP, Opstal AJV (2002) Auditory–visual interactions subserving goal-directed saccades in a complex scene. J Neurophysiol 88:438–454

    CAS  PubMed  Google Scholar 

  6. Deneve S, Pouget A (2004) Bayesian multisensory integration and cross-modal spatial links. J Physiol. doi:10.1016/j.jphysparis.2004.03.011

    Google Scholar 

  7. Frassinetti F, Pavani F, Ladavas E (2002) Acoustical vision of neglected stimuli: interaction among spatially converging audiovisual inputs in neglect patients. J Cogn Neurosci 14:62–69

    PubMed  Article  Google Scholar 

  8. Grothe B, Pecka M, McAlpine D (2010) Mechanisms of sound localization in mammals. Physiol Rev 90:983–1012. doi:10.1152/physrev.00026.2009

    CAS  PubMed  Article  Google Scholar 

  9. Haigh A, Brown DJ, Meijer P, Proulx MJ (2013) How well do you see what you hear? The acuity of visual-to-auditory sensory substitution. Front Psychol. doi:10.3389/fpsyg.2013.00330

    PubMed Central  PubMed  Google Scholar 

  10. Hamilton-Fletcher G, Ward J (2013) Representing colour through hearing and touch in sensory substitution devices. Multisens Res 26:503–532. doi:10.1163/22134808-00002434

    PubMed  Article  Google Scholar 

  11. Hancock PA, Mercado JE, Merlo J, Van Erp JBF (2013) Improving target detection in visual search through the augmenting multi-sensory cues. Ergonomics 56:729–738. doi:10.1080/00140139.2013.771219

    PubMed  Article  Google Scholar 

  12. Kastner S, De Weerd P, Pinsk MA, Elizondo MI, Desimone R, Ungerleider LG (2001) Modulation of sensory suppression: implications for receptive field sizes in the human visual cortex. J Neurophysiol 86:1398–1411

    CAS  PubMed  Google Scholar 

  13. Kim J-K, Zatorre RJ (2008) Generalized learning of visual-to-auditory substitution in sighted individuals. Brain Res 242:263–275

    Article  Google Scholar 

  14. Klapetek A, Ngo MK, Spence C (2012) Does crossmodal correspondence modulate the facilitatory effect of auditory cues on visual search? Atten Percept Psychophys 74:1154–1167. doi:10.3758/s13414-012-0317-9

    PubMed  Article  Google Scholar 

  15. Levy-Tzedek S, Hanassy S, Abboud S, Maidenbaum S, Amedi A (2012a) Fast, accurate reaching movements with a visual-to-auditory sensory substitution device. Restor Neurol Neurosci 30:313–323. doi:10.3233/rnn-2012-110219

    CAS  PubMed  Google Scholar 

  16. Levy-Tzedek S, Novick I, Arbel R, Abboud S, Maidenbaum S, Vaadia E, Amedi A (2012b) Cross-sensory transfer of sensory-motor information: visuomotor learning affects performance on an audiomotor task, using sensory-substitution. Sci Rep. doi:10.1038/srep00949

    PubMed Central  PubMed  Google Scholar 

  17. Mannan SK, Pambakian ALM, Kennard C (2010) Compensatory strategies following visual search training in patients with homonymous hemianopia: an eye movement study. J Neurol 257:1812–1821

    PubMed Central  PubMed  Article  Google Scholar 

  18. Matusz PJ, Eimer M (2013) Top-down control of audiovisual search by bimodal search templates. Psychophysiology 50:996–1009. doi:10.1111/psyp.12086

    Google Scholar 

  19. Meijer PBL (1992) An experimental system for auditory image representations. IEEE Trans Biomed Eng 39:112–121

    CAS  PubMed  Article  Google Scholar 

  20. Meredith MA, Stein BE (1986) Visual, auditory and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56:640–662

    CAS  PubMed  Google Scholar 

  21. Orchard-Mills E, Van der Burg E, Alais D (2013) Amplitude-modulated auditory stimuli influence selection of visual spatial frequencies. J Vis. doi:10.1167/13.3.6

    PubMed  Google Scholar 

  22. Pavani F, Ladavas E, Driver J (2003) Auditory and multisensory aspects of visuospatial neglect. Trends Cogn Sci 7:407–414

    PubMed  Article  Google Scholar 

  23. Poirier C, De Volder AG, Tranduy D, Scheiber C (2007) Pattern recognition using a device substituting audition for vision in blindfolded sighted subjects. Neuropsychologia 45:1108–1121

    CAS  PubMed  Article  Google Scholar 

  24. Proulx MJ, Ptito M, Amedi A (2014) Multisensory integration, sensory substitution and visual rehabilitation. Neurosci Biobehav Rev 41:1–2. doi:10.1016/j.neubiorev.2014.03.004

    PubMed  Article  Google Scholar 

  25. Renier L, Collignon O, Poirier C et al (2005) Cross-modal activation of visual cortex during depth perception using auditory substitution of vision. Neuroimage 26:573–580. doi:10.1016/j.neuroimage.2005.01.047

    PubMed  Article  Google Scholar 

  26. Spence C (2011) Crossmodal correspondences: a tutorial review. Atten Percept Psychophys 73:971–995. doi:10.3758/s13414-010-0073-7

    PubMed  Article  Google Scholar 

  27. Van der Burg E, Olivers CNL, Bronkhorst AW, Theeuwes J (2008) Pip and pop: nonspatial auditory signals improve spatial visual search. J Exp Psychol Hum Percep Perform 34:1053–1065. doi:10.1037/0096-1523.34.5.1053

    Article  Google Scholar 

  28. Van der Burg E, Olivers CNL, Bronkhorst AW, Theeuwes J (2009) Poke and pop: tactile–visual synchrony increases visual saliency. Neurosci Lett 450:60–64. doi:10.1016/j.neulet.2008.11.002

    PubMed  Article  Google Scholar 

  29. Ward J, Meijer P (2010) Visual experiences in the blind induced by an auditory sensory substitution device. Conscious Cogn 19:492–500. doi:10.1016/j.concog.2009.10.006

    PubMed  Article  Google Scholar 

  30. Wright TD, Ward J (2013) The evolution of a visual-to-auditory sensory substitution device using interactive genetic algorithms. Q J Exp Psychol 66:1620–1638

    Article  Google Scholar 

  31. Zambarbieri D, Schmid R, Magenes G, Prablanc C (1982) Saccadic responses evoked by presentation of visual and auditory targets. Exp Brain Res 47:417–427. doi:10.1007/BF00239359

    CAS  PubMed  Article  Google Scholar 

  32. Zihl J (1995) Visual scanning behavior in patients with homonymous hemianopia. Neuropsychologia 33:287–303

    CAS  PubMed  Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jamie Ward.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wright, T.D., Margolis, A. & Ward, J. Using an auditory sensory substitution device to augment vision: evidence from eye movements. Exp Brain Res 233, 851–860 (2015). https://doi.org/10.1007/s00221-014-4160-1

Download citation

Keywords

  • Sensory substitution
  • Blindness
  • Multisensory
  • Auditory
  • Eye-tracking
  • Spatial