Abstract
Congruency between auditory and visuospatial stimuli has been previously shown to affect responses to multisensory stimulus pairs, and congruency between stimuli and response devices may play a role in response speed and accuracy. Across two experiments, we tested whether the accuracy and speed of pitch judgments were affected by a congruent or incongruent paired visual stimulus, and whether the relationship was modulated by response orientation. In Experiment 1, participants using a vertically (transversely) oriented keyboard demonstrated a large crossmodal vertical effect, but a minimal crossmodal horizontal effect. In contrast, Experiment 2 used a horizontally oriented keyboard, while also examining whether musical training impacts pitch judgments. As in the first experiment, we found an effect of response mapping on pitch judgments; these results suggest that vertical visual stimuli are processed automatically, while the effects of horizontal visual stimuli are decisional and require a compatible response orientation. Based on these findings, we propose an effect we call the ROMPR effect: response orientation modulates pitch–space relationships. Unexpectedly, non-musicians demonstrated significant ROMPR effects while trained musicians did not. We suggest that non-musicians are more likely to use visual information when making spatial location judgments of pitch: unlike musicians, they have not been trained to rely exclusively on auditory information during pitch processing. We also discuss alternative explanations of the data: namely, that there is a need to disambiguate audiovisual congruency from visual–response congruency with modulations of experimental design.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The datasets generated and analysed in the current study are available in the Open Science Framework repository at the following URL: https://osf.io/b72h4/.
Notes
This effect is also known as the spatial–pitch association of response codes (SPARC; Lidji et al., 2007).
We thank an anonymous reviewer for this very helpful interpretation of our results.
References
Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14, 257–262. https://doi.org/10.1016/s0960-9822(04)00043-0.
Beecham, R., Reeve, R. A., & Wilson, S. J. (2009). Spatial representations are specific to different domains of knowledge. PLoS ONE, 4, 1–5. https://doi.org/10.1371/journal.pone.0005543.
Bizley, J. K., Jones, G. P., & Town, S. M. (2016). Where are multisensory signals combined for perceptual decision-making? Current Opinion in Neurobiology, 40, 31–37. https://doi.org/10.1016/j.conb.2016.06.003.
Boltz, M. G. (2011). Illusory tempo changes due to musical characteristics. Music Perception, 28, 367–386. https://doi.org/10.1525/mp.2011.28.4.367.
Bonetti, L., & Costa, M. (2017). Pitch-verticality and pitch-size cross-modal interactions. Psychology of Music. https://doi.org/10.1177/0305735617710734.
Bregman, A. S., & Steiger, H. (1980). Auditory streaming and vertical localization: Interdependence of “what” and “where” decisions in audition. Perception and Psychophysics, 28, 539–546. https://doi.org/10.3758/BF03198822.
Burr, D., Banks, M. S., & Morrone, M. C. (2009). Auditory dominance over vision in the perception of interval duration. Experimental Brain Research, 198, 49–57. https://doi.org/10.1007/s00221-009-1933-z.
Calvert, G. A., & Thesen, T. (2004). Multisensory integration: Methodological approaches and emerging principles in the human brain. Journal of Physiology—Paris, 98, 191–205. https://doi.org/10.1016/j.physparis.2004.03.018.
Campbell, J. I., & Scheepers, F. (2015). Effects of pitch on auditory number comparisons. Psychological Research, 79, 389–400. https://doi.org/10.1007/s00426-014-0571-7.
Cho, Y. S., Bae, G. Y., & Proctor, R. W. (2012). Referential coding contributes to the horizontal SMARC effect. Journal of Experimental Psychology: Human Perception and Performance, 38, 726–734. https://doi.org/10.1037/a0026157.
Cho, Y. S., & Proctor, R. W. (2002). Influences of hand posture and hand position on compatibility effects for up-down stimuli mapped to left-right responses: Evidence for a hand referent hypothesis. Perception & Psychophysics, 64, 1301–1315. https://doi.org/10.3758/bf03194773.
Cho, Y. S., & Proctor, R. W. (2003). Stimulus and response representations underlying orthogonal stimulus-response compatibility effects. Psychonomic Bulletin & Review, 10, 45–73. https://doi.org/10.3758/bf03196468.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd Edn.). Lawrence Erlbaum Associates.
Dehaene, S., Bossini, S., & Giraux, P. (1993). The mental representation of parity and number magnitude. Journal of Experimental Psychology: General, 122, 371–396. https://doi.org/10.1037//0096-3445.122.3.371.
Dolscheid, S., Hunnius, S., Casasanto, D., & Majid, A. (2014). Prelinguistic infants are sensitive to space-pitch associations found across cultures. Psychological Science, 25, 1256–1261. https://doi.org/10.1177/0956797614528521.
Eitan, Z., & Granot, R. Y. (2006). How music moves. Music Perception, 23, 221–248. https://doi.org/10.1525/mp.2006.23.3.221.
Eitan, Z., & Timmers, R. (2010). Beethoven’s last piano sonata and those who follow crocodiles: Cross-domain mappings of auditory pitch in a musical context. Cognition, 114, 405–422. https://doi.org/10.1016/j.cognition.2009.10.013.
Evans, K. K., & Treisman, A. (2010). Natural cross-modal mappings between visual and auditory features. Journal of Vision, 10(1), 1–12. https://doi.org/10.1167/10.1.6.
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.
Fernández-Prieto, I., & Navarra, J. (2017). The higher the pitch the larger its crossmodal influence on visuospatial processing. Psychology of Music, 45, 713–724. https://doi.org/10.1177/0305735616684205.
Fischer, M. H., & Brugger, P. (2011). When digits help digits: spatial? Numerical associations point to finger counting as prime example of embodied cognition. Frontiers in Psychology, 2, 1–7. https://doi.org/10.3389/fpsyg.2011.00260.
Fischer, M. H., Riello, M., Giordano, B. L., & Rusconi, E. (2013). Singing numbers…in cognitive space—a dual-task study of the link between pitch, space, and numbers. Topics in Cognitive Science, 5(2), 354–366. https://doi.org/10.1111/tops.12017.
Fitts, P. M., & Seeger, C. M. (1953). S-R compatibility: Spatial characteristics of stimulus and response codes. Journal of Experimental Psychology, 46, 199–210. https://doi.org/10.1037/e441262004-001.
Grassi, M. (2005). Do we hear size or sound? Balls dropped on plates. Perception and Psychophysics, 67, 274–284. https://doi.org/10.3758/BF03206491.
Guilbert, A. (2019). About the existence of a horizontal mental pitch line in non-musicians. Laterality: Asymmetries of Body, Brain and Cognition. https://doi.org/10.1080/1357650X.2019.1646756.
Hartmann, M. (2015). Numbers in the eye of the beholder: What do eye movements reveal about numerical cognition? Cognitive Processing, 16, 245–248. https://doi.org/10.1007/s10339-015-0716-7.
Hartmann, M. (2017). Non-musicians also have a piano in the head: Evidence for spatial–musical associations from line bisection tracking. Cognitive Processing, 18, 75–80. https://doi.org/10.1007/s10339-016-0779-0.
Holmes, K. J., & Lourenco, S. F. (2011). Common spatial organization of number and emotional expression: A mental magnitude line. Brain and Cognition, 77(2), 315–323. https://doi.org/10.1016/j.bandc.2011.07.002.
Ito, Y., & Hatta, T. (2004). Spatial structure of quantitative representation of numbers: Evidence from the SNARC effect. Memory and Cognition, 32(4), 662–673. https://doi.org/10.3758/BF03195857.
Keller, P. E., & Koch, I. (2008). Action planning in sequential skills: Relations to music performance. The Quarterly Journal of Experimental Psychology, 61, 275–291. https://doi.org/10.1080/17470210601160864.
Lachmair, M., Cress, U., Fissler, T., Kurek, S., Leininger, J., & Nuerk, H.-C. (2019). Music-space associations are grounded, embodied, and situated: Examination of cello experts and non-musicians in a standard tone discrimination task. Psychological Research, Advanced Online Publication.
Leboe, L. C., & Mondor, T. A. (2007). Item-specific congruency effects in nonverbal auditory Stroop. Psychological Research, 71, 568–575. https://doi.org/10.1007/s00426-006-0049-3.
Lega, C., Cattaneo, Z., Merabet, L. B., Vecchi, T., & Cucchi, S. (2014). The effect of musical expertise on the representation of space. Frontiers in Human Neuroscience. https://doi.org/10.3389/fnhum.2014.00250.
Lewkowicz, D. J., & Minar, N. J. (2014). Infants are not sensitive to synesthetic cross-modality correspondences: a comment on Walker et al. (2010). Psychological Science, 25, 832–834. https://doi.org/10.1177/0956797613516011.
Lidji, P., Kolinsky, R., Lochy, A., & Morais, J. (2007). Spatial associations for musical stimuli: A piano in the head? Journal of Experimental Psychology: Human Perception and Performance, 33, 1189–1207. https://doi.org/10.1037/0096-1523.33.5.1189.
Miller, J. E., Carlson, L. A., & McAuley, J. D. (2013). When what you hear influences when you see. Psychological Science, 24, 11–18. https://doi.org/10.1177/0956797612446707.
Mossbridge, J. A., Grabowecky, M., & Suzuki, S. (2011). Changes in auditory frequency guide visual–spatial attention. Cognition, 121, 133–139. https://doi.org/10.1016/j.cognition.2011.06.003.
Nava, E., Grassi, M., & Turati, C. (2016). Audio-visual, visuo-tactile and audio-tactile correspondences in preschoolers. Multisensory Research, 29(1–3), 93–111. https://doi.org/10.1163/22134808-00002493.
Parise, C. V., Knorre, K., & Ernst, M. O. (2014). Natural auditory scene statistics shapes human spatial hearing. Proceedings of the National Academy of Sciences, 111, 6104–6108. https://doi.org/10.1073/pnas.1322705111.
Parkinson, C., Kohler, P. J., Sievers, B., & Wheatley, T. (2012). Associations between auditory pitch and visual elevation do not depend on language: Evidence from a remote population. Perception, 41, 854–861. https://doi.org/10.1068/p7225.
Pitteri, M., Marchetti, M., Priftis, K., & Grassi, M. (2017). Naturally together: Pitch-height and brightness as coupled factors for eliciting the SMARC effect in non-musicians. Psychological Research, 81, 243–254. https://doi.org/10.1007/s00426-015-0713-6.
Pratt, C. C. (1930). The spatial character of high and low tones. Journal of Experimental Psychology, 13, 278–285. https://doi.org/10.1037/h0072651.
Roach, N. W., Heron, J., & Mcgraw, P. V. (2006). Resolving multisensory conflict: A strategy for balancing the costs and benefits of audio-visual integration. Proceedings of the Royal Society B: Biological Sciences, 273(1598), 2159–2168. https://doi.org/10.1098/rspb.2006.3578.
Rusconi, E., Kwan, B., Giordano, B., Umiltà, C., & Butterworth, B. (2006). Spatial representation of pitch height: The SMARC effect. Cognition, 99, 113–129. https://doi.org/10.1016/j.cognition.2005.01.004.
Simon, J. R. (1990). Effects of an irrelevant directional CUE on human information processing. Advances in Psychology, 65, 31–86. https://doi.org/10.1016/S0166-4115(08)61218-2.
Simon, J. R., & Craft, J. L. (1970). Effects of an irrelevant auditory stimulus on visual choice reaction time. Journal of Experimental Psychology, 86, 272–274. https://doi.org/10.1037/h0029961.
Sonnadara, R. R., Gonzalez, D. A., Hansen, S., Elliott, D., & Lyons, J. L. (2009). Spatial properties of perceived pitch. Annals of the New York Academy of Sciences, 1169, 503–507. https://doi.org/10.1111/j.1749-6632.2009.04858.x.
Spence, C. (2011). Crossmodal correspondences: A tutorial review. Attention, Perception, and Psychophysics, 73, 971–995. https://doi.org/10.3758/s13414-010-0073-7.
Spence, C., Baddeley, R., Zampini, M., James, R., & Shore, D. (2003). Multisensory temporal order judgments: When two locations are better than one. Perception and Psychophysics, 65, 318–328. https://doi.org/10.3758/BF03194803.
Stewart, L., Walsh, V., & Frith, U. (2004). Reading music modifies spatial mapping in pianists. Perception & Psychophysics, 66(2), 183–195. https://doi.org/10.3758/BF03194871.
Stewart, L., Verdonschot, R. G., Nasralla, P., & Lanipekun, J. (2013). Action-perception coupling in pianists: learned mappings or spatial musical association of response codes (SMARC) effect? Quarterly Journal of Experimental Psychology, 66(1), 37–50. https://doi.org/10.1080/17470218.2012.687385.
Timmers, R., & Li, S. (2016). Representation of pitch in horizontal space and its dependence on musical and instrumental experience. Psychomusicology: Music, Mind, and Brain, 26, 139–148. https://doi.org/10.1037/pmu0000146.
Townsend, J. T., & Ashby, F. G. (1978). Methods of modeling capacity in simple processing systems. In J. Castellan & F. Restle (Eds.), Cognitive theory (Vol. 3, pp. 200–239). Hillsdale, NJ: Erlbaum.
Van der Burg, E., Olivers, C. N. L., Bronkhorst, A. W., & Theeuwes, J. (2008). Audiovisual events capture attention: Evidence from temporal order judgments. Journal of Vision, 8, 1–10. https://doi.org/10.1167/8.5.2.
Vu, K. P. L., Proctor, R. W., & Pick, D. F. (2000). Vertical versus horizontal spatial compatibility: Right-left prevalence with bimanual responses. Psychological Research, 64(1), 25–40. https://doi.org/10.1007/s004260000035.
Walker, P. (2012). Cross-sensory correspondences and cross talk between dimensions of connotative meaning: Visual angularity is hard, high-pitched, and bright. Attention, Perception, and Psychophysics, 74(8), 1792–1809. https://doi.org/10.3758/s13414-012-0341-9.
Walker, P., Bremner, J. G., Mason, U., Spring, J., Mattock, K., Slater, A., & Johnson, S. P. (2010). Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences. Psychological Science, 21(1), 21–25. https://doi.org/10.1177/0956797609354734.
Walker, P., Bremner, J. G., Mason, U., Spring, J., Mattock, K., Slater, A., & Johnson, S. P. (2014). Preverbal infants are sensitive to cross-sensory correspondences: Much ado about the null results of Lewkowicz and Minar (2014). Psychological Science, 25, 835–836. https://doi.org/10.1177/0956797613520170.
Walsh, V. (2003). A theory of magnitude: Common cortical metrics of time, space and quantity. Trends in Cognitive Sciences, 7(11), 483–488. https://doi.org/10.1016/j.tics.2003.09.002.
Weis, T., Estner, B., & Lachmann, T. (2016). When speech enhances spatial musical association of response codes: Joint spatial associations of pitch and timbre in nonmusicians. The Quarterly Journal of Experimental Psychology, 69, 1687–1700. https://doi.org/10.1080/17470218.2015.1091850.
Welch, R. B., & Warren, D. H. (1980). Immediate perceptual response to intersensory discrepancy. Psychological Bulletin, 88, 638–667. https://doi.org/10.1037//0033-2909.88.3.638.
Wilbiks, J. (2018). Effects of temporal distribution on utility of temporal factors in competitive audio-visual perceived synchrony. Multisensory Research, 31, 579–593. https://doi.org/10.1163/22134808-00002613.
Wilbiks, J. M. P., & Dyson, B. J. (2013). Effects of temporal asynchrony and stimulus magnitude on competitive audio–visual binding. Attention, Perception, & Psychophysics, 75, 1883–1891. https://doi.org/10.3758/s13414-013-0527-9.
Wolter, S., Dudschig, C., & Kaup, B. (2016). Reading sentences describing high- or low-pitched auditory events: Only pianists show evidence for a horizontal space-pitch association. Psychological Research, 81, 1213–1223. https://doi.org/10.1007/s00426-016-0812-z.
Zarate, J. M., Ritson, C. R., & Poeppel, D. (2012). Pitch-interval discrimination and musical expertise: Is the semitone a perceptual boundary? The Journal of the Acoustical Society of America z, 132, 984–993. https://doi.org/10.1121/1.4733535.
Funding
This study was funded by a Margaret and Wallace McCain Postdoctoral Fellowship.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
All authors declare that they have no conflict of interest.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Klapman, S.F., Munn, J.T. & Wilbiks, J.M.P. Response orientation modulates pitch–space relationships: the ROMPR effect. Psychological Research 85, 2197–2212 (2021). https://doi.org/10.1007/s00426-020-01388-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00426-020-01388-z