Attention, Perception, & Psychophysics

, Volume 80, Issue 4, pp 999–1010 | Cite as

Visually induced gains in pitch discrimination: Linking audio-visual processing with auditory abilities

  • Cecilie MøllerEmail author
  • Andreas Højlund
  • Klaus B. Bærentsen
  • Niels Chr. Hansen
  • Joshua C. Skewes
  • Peter Vuust


Perception is fundamentally a multisensory experience. The principle of inverse effectiveness (PoIE) states how the multisensory gain is maximal when responses to the unisensory constituents of the stimuli are weak. It is one of the basic principles underlying multisensory processing of spatiotemporally corresponding crossmodal stimuli that are well established at behavioral as well as neural levels. It is not yet clear, however, how modality-specific stimulus features influence discrimination of subtle changes in a crossmodally corresponding feature belonging to another modality. Here, we tested the hypothesis that reliance on visual cues to pitch discrimination follow the PoIE at the interindividual level (i.e., varies with varying levels of auditory-only pitch discrimination abilities). Using an oddball pitch discrimination task, we measured the effect of varying visually perceived vertical position in participants exhibiting a wide range of pitch discrimination abilities (i.e., musicians and nonmusicians). Visual cues significantly enhanced pitch discrimination as measured by the sensitivity index d’, and more so in the crossmodally congruent than incongruent condition. The magnitude of gain caused by compatible visual cues was associated with individual pitch discrimination thresholds, as predicted by the PoIE. This was not the case for the magnitude of the congruence effect, which was unrelated to individual pitch discrimination thresholds, indicating that the pitch-height association is robust to variations in auditory skills. Our findings shed light on individual differences in multisensory processing by suggesting that relevant multisensory information that crucially aids some perceivers’ performance may be of less importance to others, depending on their unisensory abilities.


Multisensory processing Hearing 



We thank Zohar Eitan for much valued inputs in the design stage of the experiment, Sukhbinder Kumar and Victoria Williamson for sharing the original pitch threshold estimation scripts, and Signe Hagner for very competent help with data collection. This project has been supported by seed funding from the Interacting Minds Centre, AU, DK. Center for Music in the Brain is funded by the Danish National Research Foundation (DNRF117).

Supplementary material

13414_2017_1481_MOESM1_ESM.docx (626 kb)
ESM 1 (DOCX 625 kb)


  1. Abel, M. K., Li, H. C., Russo, F. A., Schlaug, G., & Loui, P. (2016). Audiovisual interval size estimation is associated with early musical training. PLOS ONE, 11(10), e0163589. CrossRefPubMedPubMedCentralGoogle Scholar
  2. Alais, D., Newell, F. N., & Mamassian, P. (2010). Multisensory processing in review: From physiology to behaviour. Seeing and Perceiving, 23(1), 3–38. CrossRefPubMedGoogle Scholar
  3. Albouy, P., Leveque, Y., Hyde, K. L., Bouchet, P., Tillmann, B., & Caclin, A. (2015). Boosting pitch encoding with audiovisual interactions in congenital amusia. Neuropsychologia, 67, 111–120. CrossRefPubMedGoogle Scholar
  4. Angelaki, D. E., Gu, Y., & DeAngelis, G. C. (2009). Multisensory integration: Psychophysics, neurophysiology, and computation. Current Opinion in Neurobiology, 19(4), 452–458. CrossRefPubMedPubMedCentralGoogle Scholar
  5. Auvray, M., & Spence, C. (2008). The multisensory perception of flavor. Consciousness and Cognition, 17(3), 1016–1031. CrossRefPubMedGoogle Scholar
  6. Bernstein, I. H., & Edelstein, B. A. (1971). Effects of some variations in auditory input upon visual choice reaction time. Journal of Experimental Psychology, 87(2), 241–247.CrossRefPubMedGoogle Scholar
  7. Bernstein, N. A. (1996). On dexterity and its development. In M. L. Latash & M. T. Turvey (Eds.), Dexterity and its development (pp. 3–237). Mahwah, NJ: Erlbaum.Google Scholar
  8. Bolognini, N., Frassinetti, F., Serino, A., & Làdavas, E. (2005). “Acoustical vision” of below threshold stimuli: Interaction among spatially converging audiovisual inputs. Experimental Brain Research, 160(3), 273–282. CrossRefPubMedGoogle Scholar
  9. Caclin, A., Bouchet, P., Djoulah, F., Pirat, E., Pernier, J., & Giard, M. H. (2011). Auditory enhancement of visual perception at threshold depends on visual abilities. Brain Research, 1396, 35–44. CrossRefPubMedGoogle Scholar
  10. Calvert, G., Spence, C., & Stein, B. E. (Eds.). (2004). The handbook of multisensory processes. Cambridge, MA: MIT Press.Google Scholar
  11. Chen, Y., & Spence, C. (2010). When hearing the bark helps to identify the dog: Semantically-congruent sounds modulate the identification of masked pictures. Cognition, 114(3), 389–404. CrossRefPubMedGoogle Scholar
  12. Deneve, S., & Pouget, A. (2004). Bayesian multisensory integration and cross-modal spatial links. Journal of Physiology–Paris, 98(1), 249–258. Google Scholar
  13. Diederich, A., & Colonius, H. (2004). Bimodal and trimodal multisensory enhancement: Effects of stimulus onset and intensity on reaction time. Perception & Psychophysics, 66(8), 1388–1404.CrossRefGoogle Scholar
  14. Doehrmann, O., & Naumer, M. J. (2008). Semantics and the multisensory brain: How meaning modulates processes of audio-visual integration. Brain Research, 1242, 136–150. CrossRefPubMedGoogle Scholar
  15. Edelman, G. M., & Tononi, G. (2000). A universe of consciousness: How matter becomes imagination. New York, NY: Basic Books.Google Scholar
  16. Eitan, Z., & Granot, R. Y. (2006). How music moves: Musical parameters and listeners images of motion. Music Perception: An Interdisciplinary Journal, 23(3), 221–248. CrossRefGoogle Scholar
  17. Eitan, Z., & Timmers, R. (2010). Beethoven’s last piano sonata and those who follow crocodiles: Cross-domain mappings of auditory pitch in a musical context. Cognition, 114(3), 405–422. CrossRefPubMedGoogle Scholar
  18. Erber, N. P. (1975). Auditory-visual perception of speech. Journal of Speech and Hearing Disorders, 40(4), 481. CrossRefPubMedGoogle Scholar
  19. Ernst, M. O. (2012). Optimal multisensory integration: Assumptions and limits. In B. E. Stein (Ed.), The new handbook of multisensory processing (pp. 527–543). Cambridge, MA: MIT Press.Google Scholar
  20. Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415(6870), 429–433. CrossRefPubMedGoogle Scholar
  21. Evans, K. K., & Treisman, A. (2010). Natural cross-modal mappings between visual and auditory features. Journal of Vision, 10(1), 6.1–6. Google Scholar
  22. Forster, B., Cavina-Pratesi, C., Aglioti, S. M., & Berlucchi, G. (2002). Redundant target effect and intersensory facilitation from visual–tactile interactions in simple reaction time. Experimental Brain Research, 143(4), 480–487. CrossRefPubMedGoogle Scholar
  23. Frassinetti, F., Bolognini, N., & Ladavas, E. (2002). Enhancement of visual perception by crossmodal visuo-auditory interaction. Experimental Brain Research, 147(3), 332–343. CrossRefPubMedGoogle Scholar
  24. Gescheider, G. A., Kane, M. J., Sager, L. C., & Ruffolo, L. J. (1974). The effect of auditory stimulation on responses to tactile stimuli. Bulletin of the Psychonomic Society, 3(3), 204–206.CrossRefGoogle Scholar
  25. Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York, NY: Wiley.Google Scholar
  26. Hansen, N. C., & Pearce, M. T. (2014). Predictive uncertainty in auditory sequence processing. Frontiers in Psychology, 5, 1052. CrossRefPubMedPubMedCentralGoogle Scholar
  27. Hansen, N. C., Vuust, P., & Pearce, M. (2016). “If you have to ask, you’ll never know”: Effects of specialised stylistic expertise on predictive processing of music. PLOS ONE, 11(10), e0163584. CrossRefPubMedPubMedCentralGoogle Scholar
  28. Hautus, M. J. (1995). Corrections for extreme proportions and their biasing effects on estimated values ofd′. Behavior Research Methods, Instruments, & Computers, 27(1), 46–51. CrossRefGoogle Scholar
  29. Laurienti, P. J., Kraft, R. A., Maldjian, J. A., Burdette, J. H., & Wallace, M. T. (2004). Semantic congruence is a critical factor in multisensory behavioral performance. Experimental Brain Research, 158(4), 405–414. CrossRefPubMedGoogle Scholar
  30. Laurienti, P. J., Burdette, J. H., Maldjian, J. A., & Wallace, M. T. (2006). Enhanced multisensory integration in older adults. Neurobiology of Aging, 27(8), 1155–1163. CrossRefPubMedGoogle Scholar
  31. Levitt, H. (1971). Transformed up-down methods in psychoacoustics. The Journal of the Acoustical Society of America, 49(2), 467–477. Google Scholar
  32. Lickliter, R., & Bahrick, L. E. (2004). Perceptual development and the origins of multisensory responsiveness. In G. A. Calvert, C. Spence, & B. E. Stein (Eds.), The handbook of multisensory processes (pp. 643–654). Cambridge, MA: MIT Press.Google Scholar
  33. Lovelace, C. T., Stein, B. E., & Wallace, M. T. (2003). An irrelevant light enhances auditory detection in humans: A psychophysical analysis of multisensory integration in stimulus detection. Cognitive Brain Research, 17(2), 447–453. CrossRefPubMedGoogle Scholar
  34. Lu, X., Ho, H. T., Sun, Y., Johnson, B. W., & Thompson, W. F. (2016). The influence of visual information on auditory processing in individuals with congenital amusia: An ERP study. NeuroImage, 135, 142–151. CrossRefPubMedGoogle Scholar
  35. Maxwell, S. E., & Delaney, H. D. (2004). Designing experiments and analyzing data: A model comparison perspective. Mahwah, NJ: Erlbaum.Google Scholar
  36. Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46(4), 806–834. CrossRefGoogle Scholar
  37. Melara, R. D., & O’Brien, T. P. (1987). Interaction between synesthetically corresponding dimensions. Journal of Experimental Psychology: General, 116(4), 323–336. CrossRefGoogle Scholar
  38. Meredith, A. M., & Stein, B. E. (1986). Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Research, 365(2), 350–354. CrossRefPubMedGoogle Scholar
  39. Molholm, S., Ritter, W., Javitt, D. C., & Foxe, J. J. (2004). Multisensory visual-auditory object recognition in humans: A high-density electrical mapping study. Cerebral Cortex (New York, NY: 1991), 14(4), 452–465.Google Scholar
  40. Nichols, E. S., & Grahn, J. A. (2016). Neural correlates of audiovisual integration in music reading. Neuropsychologia.
  41. Orchard-Mills, E., Van der Burg, E., & Alais, D. (2015). Crossmodal correspondence between auditory pitch and visual elevation affects temporal ventriloquism. Perception, 45(4), 409–424. CrossRefGoogle Scholar
  42. Oxenham, A. J. (2012). Pitch perception. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 32(39), 13335–13338. CrossRefGoogle Scholar
  43. Paraskevopoulos, E., Kraneburg, A., Herholz, S. C., Bamidis, P. D., & Pantev, C. (2015). Musical expertise is related to altered functional connectivity during audiovisual integration. Proceedings of the National Academy of Sciences of the United States of America, 112(40), 12522–12527. CrossRefPubMedPubMedCentralGoogle Scholar
  44. Paraskevopoulos, E., Kuchenbuch, A., Herholz, S. C., & Pantev, C. (2012). Musical expertise induces audiovisual integration of abstract congruency rules. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 32(50), 18196–18203. CrossRefGoogle Scholar
  45. Parise, C. V. (2016). Crossmodal correspondences: Standing issues and experimental guidelines. Multisensory Research, 29(1/3), 7–28.CrossRefPubMedGoogle Scholar
  46. Parise, C. V., Knorre, K., & Ernst, M. O. (2014). Natural auditory scene statistics shapes human spatial hearing. Proceedings of the National Academy of Sciences of the United States of America, 111(16), 6104–6108. CrossRefPubMedPubMedCentralGoogle Scholar
  47. Parise, C. V., Spence, C., & Deroy, O. (2016). Understanding the correspondences: Introduction to the special issue on crossmodal correspondences. Multisensory Research, 29(1/3), 1–6. CrossRefPubMedGoogle Scholar
  48. Patching, G. R., & Quinlan, P. T. (2002). Garner and congruence effects in the speeded classification of bimodal signals. Journal of Experimental Psychology: Human Perception and Performance, 28(4), 755–775. PubMedGoogle Scholar
  49. Pratt, C. C. (1930). The spatial character of high and low tones. Journal of Experimental Psychology, 13(3), 278.CrossRefGoogle Scholar
  50. Proulx, M. J., Brown, D. J., Pasqualotto, A., & Meijer, P. (2014). Multisensory perceptual learning and sensory substitution. Neuroscience and Biobehavioral Reviews, 41, 16–25. CrossRefPubMedGoogle Scholar
  51. Ross, L. A., Saint-Amour, D., Leavitt, V. M., Javitt, D. C., & Foxe, J. J. (2007). Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cerebral Cortex (New York, NY: 1991), 17(5), 1147–1153. Google Scholar
  52. Rowe, C. (1999). Receiver psychology and the evolution of multicomponent signals. Animal Behaviour, 58(5), 921–931. CrossRefPubMedGoogle Scholar
  53. Rowland, B. A. (2012). Commentary: Computational models of multisensory integration: Bayesian frameworks, development, and timing. In B. E. Stein (Ed.), The new handbook of multisensory processing (pp. 559–511). Cambridge, MA: MIT Press.Google Scholar
  54. Senkowski, D., Saint-Amour, D., Hofle, M., & Foxe, J. J. (2011). Multisensory interactions in early evoked brain activity follow the principle of inverse effectiveness. NeuroImage, 56(4), 2200–2208. CrossRefPubMedGoogle Scholar
  55. Spence, C. (2007). Audiovisual multisensory integration. Acoustical Science and Technology, 28(2), 61–70.CrossRefGoogle Scholar
  56. Spence, C. (2011). Crossmodal correspondences: A tutorial review. Attention, Perception & Psychophysics, 73(4), 971–995. CrossRefGoogle Scholar
  57. Stein, B. E., Laurienti, P. J., Wallace, M. T., & Stanford, T. R. (2002). Multisensory integration. In V. S. Ramachandran (Ed.), Encyclopedia of the human brain (pp. 227–241). New York, NY: Academic Press.CrossRefGoogle Scholar
  58. Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Cambridge, MA: MIT Press.Google Scholar
  59. Stumpf, C. (1883). Tonpsychologie, I. Leipzig, Germany: Hirzel.Google Scholar
  60. Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. The Journal of the Acoustical Society of America, 26(2), 212. CrossRefGoogle Scholar
  61. Tervaniemi, M., Just, V., Koelsch, S., Widmann, A., & Schroger, E. (2005). Pitch discrimination accuracy in musicians vs nonmusicians: An event-related potential and behavioral study. Experimental Brain Research, 161(1), 1–10. CrossRefPubMedGoogle Scholar
  62. Van Engen, K. J., Phelps, J. E., Smiljanic, R., & Chandrasekaran, B. (2014). Enhancing speech intelligibility: Interactions among context, modality, speech style, and masker. Journal of Speech, Language, and Hearing Research : JSLHR, 57(5), 1908–1918. CrossRefPubMedGoogle Scholar
  63. Vroomen, J., & Gelder, B. D. (2000). Sound enhances visual perception: Cross-modal effects of auditory organization on vision. Journal of Experimental Psychology: Human Perception and Performance, 26(5), 1583–1590. PubMedGoogle Scholar
  64. Vuust, P., Brattico, E., Seppanen, M., Naatanen, R., & Tervaniemi, M. (2012). The sound of music: Differentiating musicians using a fast, musical multi-feature mismatch negativity paradigm. Neuropsychologia, 50(7), 1432–1443. CrossRefPubMedGoogle Scholar
  65. Walker, P., Bremner, J. G., Mason, U., Spring, J., Mattock, K., Slater, A., & Johnson, S. P. (2010). Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences. Psychological Science, 21(1), 21–25. CrossRefPubMedGoogle Scholar
  66. Wallentin, M., Nielsen, A. H., Friis-Olivarius, M., Vuust, C., & Vuust, P. (2010). The musical ear test, a new reliable test for measuring musical competence. Learning and Individual Differences, 20(3), 188–196. CrossRefGoogle Scholar
  67. Williamson, V. J., Liu, F., Peryer, G., Grierson, M., & Stewart, L. (2012). Perception and action de-coupling in congenital amusia: Sensitivity to task demands. Neuropsychologia, 50(1), 172–180.Google Scholar

Copyright information

© The Psychonomic Society, Inc. 2018

Authors and Affiliations

  • Cecilie Møller
    • 1
    • 2
    Email author
  • Andreas Højlund
    • 3
    • 4
  • Klaus B. Bærentsen
    • 1
  • Niels Chr. Hansen
    • 2
    • 5
  • Joshua C. Skewes
    • 4
  • Peter Vuust
    • 2
  1. 1.Department of PsychologyAarhus UniversityAarhusDenmark
  2. 2.Center for Music in the BrainAarhus University & The Royal Academy of Music Aarhus/AalborgAarhus CDenmark
  3. 3.Center of Functionally Integrative NeuroscienceAarhus University HospitalAarhusDenmark
  4. 4.Interacting Minds CentreAarhus UniversityAarhusDenmark
  5. 5.Cognitive and Systematic Musicology Laboratory, School of MusicOhio State UniversityColumbusUSA

Personalised recommendations