Advertisement

Experimental Brain Research

, Volume 220, Issue 3–4, pp 319–333 | Cite as

Audiovisual crossmodal correspondences and sound symbolism: a study using the implicit association test

  • Cesare V. Parise
  • Charles Spence
Research Article

Abstract

A growing body of empirical research on the topic of multisensory perception now shows that even non-synaesthetic individuals experience crossmodal correspondences, that is, apparently arbitrary compatibility effects between stimuli in different sensory modalities. In the present study, we replicated a number of classic results from the literature on crossmodal correspondences and highlight the existence of two new crossmodal correspondences using a modified version of the implicit association test (IAT). Given that only a single stimulus was presented on each trial, these results rule out selective attention and multisensory integration as possible mechanisms underlying the reported compatibility effects on speeded performance. The crossmodal correspondences examined in the present study all gave rise to very similar effect sizes, and the compatibility effect had a very rapid onset, thus speaking to the automatic detection of crossmodal correspondences. These results are further discussed in terms of the advantages of the IAT over traditional techniques for assessing the strength and symmetry of various crossmodal correspondences.

Keywords

Multisensory perception Audition Vision Crossmodal correspondences Sound symbolism Implicit association test 

Notes

Acknowledgments

Cesare Parise was supported by the Bernstein Center for Computational Neuroscience, Tübingen, funded by the German Federal Ministry of Education and Research (BMBF; FKZ: 01GQ1002).

Supplementary material

Supplementary material 1 (MPG 161 kb)

References

  1. Bernstein IH, Edelstein BA (1971) Effects of some variations in auditory input upon visual choice reaction time. J Exp Psychol 87(2):241–247PubMedCrossRefGoogle Scholar
  2. Bien N, ten Oever S, Goebel R, Sack AT (2012) The sound of size Crossmodal binding in pitch-size synesthesia: A combined TMS, EEG and psychophysics study. Neuroimage 59:663–672Google Scholar
  3. Blair IV (2002) The malleability of automatic stereotypes and prejudice. Personal Soc Psychol Rev 6:242–261CrossRefGoogle Scholar
  4. Bozzi P, Flores D’Arcais G (1967) Experimental research on the intermodal relationships between expressive qualities. Arch Psicol Neurol Psichiatr 28(5):377–420PubMedGoogle Scholar
  5. Brang D, Rouw R, Ramachandran VS, Coulson S (2011) Similarly shaped letters evoke similar colors in grapheme–color synesthesia. Neuropsychologia 49:1355–1358PubMedCrossRefGoogle Scholar
  6. Bremner A, Caparos S, Davidoff J, de Fockert J, Linnell K, Spence C (submitted) Bouba and Kiki in Namibia? Western shape-symbolism does not extend to taste in a remote population. CognitionGoogle Scholar
  7. Chen Y-C, Spence C (2011) Crossmodal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity. J Exp Psychol Hum Percept Perform 37:1554–1568PubMedCrossRefGoogle Scholar
  8. Chiou R, Rich AN (2012) Cross-modality correspondence between pitch and spatial location modulates attentional orienting. Perception 41:339–353CrossRefGoogle Scholar
  9. Cohen J (1988) Statistical power analysis for the behavioral sciences. Lawrence Erlbaum Associates, Hillsdale, NJGoogle Scholar
  10. Cohen Kadosh R, Henik A (2007) Can synaesthesia research inform cognitive science? Trends Cogn Sci 11(4):177–184PubMedCrossRefGoogle Scholar
  11. Cohen Kadosh R, Cohen Kadosh K, Henik A (2007a) The neuronal correlate of bidirectional synesthesia: a combined event-related potential and functional magnetic resonance imaging study. J Cogn Neurosci 19(12):2050–2059PubMedCrossRefGoogle Scholar
  12. Cohen Kadosh R, Henik A, Walsh V (2007b) Small is bright and big is dark in synaesthesia. Curr Biol 17(19):R834–R835PubMedCrossRefGoogle Scholar
  13. Cowey A, Weiskrantz L (1975) Demonstration of cross-modal matching in rhesus monkeys, Macaca mulatta. Neuropsychologia 13(1):117–120PubMedCrossRefGoogle Scholar
  14. Crisinel AS, Spence C (2009) Implicit association between basic tastes and pitch. Neurosci Lett 464(1):39–42PubMedCrossRefGoogle Scholar
  15. Crisinel AS, Spence C (2010) A sweet sound? Food names reveal implicit associations between taste and pitch. Perception 39(3):417–425PubMedCrossRefGoogle Scholar
  16. Davis R (1961) The fitness of names to drawings: a cross-cultural study in Tanganyika. Br J Psychol 52:259–268PubMedCrossRefGoogle Scholar
  17. De Jong R, Liang CC, Lauber E (1994) Conditional andunconditional automaticity: a dual-process model of effects of spatial stimulus-response correspondence. J Exp Psychol Hum Percept Perform 20(4):731–750PubMedCrossRefGoogle Scholar
  18. Demattè M, Sanabria D, Spence C (2006) Cross-modal associations between odors and colors. Chem Senses 31(6):531–538CrossRefGoogle Scholar
  19. Demattè M, Sanabria D, Spence C (2007) Olfactory-tactile compatibility effects demonstrated using a variation of the implicit association test. Acta Psychol 124(3):332–343CrossRefGoogle Scholar
  20. Dixon MJ, Smilek D, Cudahy C, Merikle PM (2000) Five plus two equals yellow. Nature 406(6794):365PubMedCrossRefGoogle Scholar
  21. Ernst MO (2007) Learning to integrate arbitrary signals from vision and touch. J Vis 7(5):1–14PubMedCrossRefGoogle Scholar
  22. Evans KK, Treisman A (2010) Natural cross-modal mappings between visual and auditory features. J Vis 10(1):1–12PubMedGoogle Scholar
  23. Fiedler K, Messner C, Bluemke M (2006) Unresolved problems with the “I”, the “A”, and the “T”: a logical and psychometric critique of the implicit association test (IAT). Eur Rev Soc Psychol 17:74–147CrossRefGoogle Scholar
  24. Freed DJ (1990) Auditory correlates of perceived mallet hardness for a set of recorded percussive sound events. J Acoust Soc Am 87(1):311–322PubMedCrossRefGoogle Scholar
  25. Gallace A, Spence C (2006) Multisensory synesthetic interactions in the speeded classification of visual size. Percept Psychophys 68(7):1191–1203PubMedCrossRefGoogle Scholar
  26. Greenwald AG, McGhee DE, Schwartz JLK (1998) Measuring individual differences in implicit cognition: the implicit association test. J Pers Soc Psychol 74(6):1464–1480PubMedCrossRefGoogle Scholar
  27. Grossenbacher PG, Lovelace CT (2001) Mechanisms of synesthesia: cognitive and physiological constraints. Trends Cogn Sci 5(1):36–41PubMedCrossRefGoogle Scholar
  28. Hinton L, Nichols J, Ohala JJ (eds) (2006) Sound symbolism. Cambridge University Press, CambridgeGoogle Scholar
  29. Johnson A, Jepma M, De Jong R (2007) Colours sometimes count: awareness and bidirectionality in grapheme-colour synaesthesia. Q J Exp Psychol 60(10):1406–1422CrossRefGoogle Scholar
  30. Klapetek A, Ngo MK, Spence C (in press) Do crossmodal correspondences enhance the facilitatory effect of auditory cues on visual search? Atten Percept PsychophysGoogle Scholar
  31. Klatzky RL, Pai DK, Krotkov EP (2000) Perception of material from contact sounds. Presence Teleoper Virtual Environ 9(4):399–410CrossRefGoogle Scholar
  32. Klein R, Brennan M, Gilani A (1987) Covert cross-modality orienting of attention in space. Paper presented at the Annual meeting of the Psychonomic Society, Seattle, WAGoogle Scholar
  33. Köhler W (1929) Gestalt psychology. Liveright, New YorkGoogle Scholar
  34. Köhler W (1947) Gestalt psychology: an introduction to new concepts in modern psychology. Liveright Publ. Corporation, New York, NYGoogle Scholar
  35. Kovic V, Plunkett K, Westermann G (2010) The shape of words in the brain. Cognition 114(1):19–28PubMedCrossRefGoogle Scholar
  36. Ludwig VU, Adachi I, Matzuzawa T (2011) Visuoauditory mappings between high luminance and high pitch are shared by chimpanzees (Pan troglodytes) and humans. Proc Natl Acad Sci USA 108:20661–20665PubMedCrossRefGoogle Scholar
  37. Marks LE (1987) On cross-modal similarity: auditory–visual interactions in speeded discrimination. J Exp Psychol Hum Percept Perform 13(3):384–394PubMedCrossRefGoogle Scholar
  38. Marks LE (1989) On cross-modal similarity: the perceptual structure of pitch, loudness, and brightness. J Exp Psychol Hum Percept Perform 15(3):586–602PubMedCrossRefGoogle Scholar
  39. Marks LE (2004) Cross-modal interactions in speeded classification. In: Calvert GA, Spence C, Stein BE (eds) The handbook of mutisensory processes. MIT Press, Cambridge, MA, pp 85–106Google Scholar
  40. Martino G, Marks LE (2001) Synesthesia: strong and weak. Curr Dir Psychol Sci 10(2):61–65CrossRefGoogle Scholar
  41. Meier B, Rothen N (2007) When conditioned responses “fire back”: bidirectional cross-activation creates learning opportunities in synesthesia. Neuroscience 147(3):569–572PubMedCrossRefGoogle Scholar
  42. Melara RD, O’Brien TP (1987) Interaction between synesthetically corresponding dimensions. J Exp Psychol Gen 116(4):323–336CrossRefGoogle Scholar
  43. Melara RD, O’Brien TP (1990) Effects of cuing on cross-modal congruity. J Mem Lang 29(6):655–686CrossRefGoogle Scholar
  44. Mills CB (1999) Digit synaesthesia: a case study using a Stroop-type test. Cogn Neuropsychol 16(2):181–191CrossRefGoogle Scholar
  45. Mulvenna CM, Walsh V (2006) Synaesthesia: supernormal integration? Trends Cogn Sci 10(8):350–352PubMedCrossRefGoogle Scholar
  46. Newman S (1933) Further experiments in phonetic symbolism. Am J Psychol 45(1):53–75CrossRefGoogle Scholar
  47. Oberman LM, Ramachandran VS (2008) Preliminary evidence for deficits in multisensory integration in autism spectrum disorders: the mirror neuron hypothesis. Soc Neurosci 3(3–4):348–355PubMedCrossRefGoogle Scholar
  48. Osgood CE (1960) The cross-cultural generality of visual–verbal synesthetic tendencies. Behav Sci 5(2):146–169CrossRefGoogle Scholar
  49. Osgood CE, Suci G, Tannenbaum P (1957) The measurement of meaning. University of Illinois Press, UrbanaGoogle Scholar
  50. Oyama T, Yamada H, Iwasawa H (1998) Synesthetic tendencies as the basis of sensory symbolism: a review of a series of experiments by means of semantic differential. Psychologia 41:203–215Google Scholar
  51. Parise CV, Pavani F (2011) Evidence of sound symbolism in simple vocalizations. Exp Brain Res 214(3):373–380PubMedCrossRefGoogle Scholar
  52. Parise CV, Spence C (2008) Synesthetic congruency modulates the temporal ventriloquism effect. Neurosci Lett 442(3):257–261PubMedCrossRefGoogle Scholar
  53. Parise CV, Spence C (2009) When birds of a feather flock together: synesthetic correspondences modulate audiovisual integration in non-synesthetes. PLoS One 4(5):e5664PubMedCrossRefGoogle Scholar
  54. Parise CV, Spence C (2012) Assessing the associations between brand packaging and brand attributes using an indirect performance measure. Food Qual Prefer 24:17–23CrossRefGoogle Scholar
  55. Parise CV, Spence C (in press) Audiovisual crossmodal correspondences. In Simner J, Hubbard EM (eds) Oxford handbook of synaesthesia. Oxford University Press, OxfordGoogle Scholar
  56. Parise CV, Spence C, Ernst MO (2012) When correlation implies causation in multisensory integration. Curr Biol 22:46–49PubMedCrossRefGoogle Scholar
  57. Parker A, Easton A (2004) Cross-modal memory in primates: the neural basis of learning about the multisensory properties of objects and events. In: Calvert GA, Spence C, Stein BE (eds) The handbook of multisensory processes. MIT Press, Cambridge, MA, pp 333–342Google Scholar
  58. Poffenberger A, Barrows B (1924) The feeling value of lines. J Appl Psychol 8(2):187–205CrossRefGoogle Scholar
  59. Premack D, Premack AJ (2003) Original intelligence: unlocking the mystery of who we are. McGraw-Hill, New YorkGoogle Scholar
  60. Rader C, Tellegen A (1987) An investigation of synesthesia. J Pers Soc Psychol 52(5):981–987CrossRefGoogle Scholar
  61. Ramachandran VS, Oberman LM (2007) Broken mirrors: a theory of autism. Sci Am Spec Ed 17(2):20–29Google Scholar
  62. Robson D (2011) Language’s missing link. New Sci 211(2821):30–33CrossRefGoogle Scholar
  63. Rogers SK, Ross AS (1968) A cross-cultural test of the Maluma-Takete phenomenon. Perception 4(1):105–106CrossRefGoogle Scholar
  64. Rousseeuw PJ, Ruts I, Tukey JW (1999) The bagplot: a bivariate boxplot. Am Stat 53(4):382–387Google Scholar
  65. Rudmin F, Cappelli M (1983) Tone-taste synesthesia: a replication. Percept Mot Skills 56:118PubMedCrossRefGoogle Scholar
  66. Sapir E (1929) A study in phonetic symbolism. J Exp Psychol 12(3):225–239CrossRefGoogle Scholar
  67. Seo H-S, Arshamian A, Schemmer K, Scheer I, Sander T, Ritter G, Hummel T (2010) Cross-modal integration between odors and abstract symbols. Neurosci Lett 478:175–178PubMedCrossRefGoogle Scholar
  68. Shepherd GM (2012) Neurogastronomy: how the brain creates flavor and why it matters. Columbia University Press, New YorkGoogle Scholar
  69. Simpson RH, Quinn M, Ausubel DP (1956) Synesthesia in children: association of colors with pure tone frequencies. J Genet Psychol Res Theory Hum Dev 89(1):95–103Google Scholar
  70. Soto-Faraco S, Lyons J, Gazzaniga M, Spence C, Kingstone A (2002) The ventriloquist in motion: illusory capture of dynamic information across sensory modalities. Cogn Brain Res 14(1):139–146CrossRefGoogle Scholar
  71. Spence C (2011) Crossmodal correspondences: a tutorial review. Atten Percept Psychophys 73(4):1–25CrossRefGoogle Scholar
  72. Spence C, Deroy O (2012) Are chimpanzees really synaesthetic? i-Perception 3:316–318Google Scholar
  73. Spence C, Squire S (2003) Multisensory integration: maintaining the perception of synchrony. Curr Biol 13(13):R519–R521PubMedCrossRefGoogle Scholar
  74. Stevens JC, Marks LE (1965) Cross-modality matching of brightness and loudness. Proc Natl Acad Sci USA 54(2):407–411PubMedCrossRefGoogle Scholar
  75. Stumpf K (1883) Tonpsychologie. S. Hirzel, LeipzigGoogle Scholar
  76. Svartdal F, Iversen T (1989) Consistency in synesthetic experience to vowels and consonants: five case studies. Scand J Psychol 30:220–227PubMedCrossRefGoogle Scholar
  77. Vallesi A, Mapelli D, Schiff S, Amodio P, Umiltà C (2005) Horizontal and vertical Simon effect: different underlying mechanisms? Cognition 96(1):B33–B43PubMedCrossRefGoogle Scholar
  78. Van den Doel K, Pai DK (1998) The sounds of physical shapes. Presence 7(4):382–395CrossRefGoogle Scholar
  79. Walker P, Smith S (1985) Stroop interference based on the multimodal correlates of haptic size and auditory pitch. Perception 14(6):729–736PubMedCrossRefGoogle Scholar
  80. Walker P, Bremner J, Mason U, Spring J, Mattock K, Slater A, Johnson S (2010) Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences. Psychol Sci 21(1):21–25PubMedCrossRefGoogle Scholar
  81. Ward J, Huckstep B, Tsakanikos E (2006) Sound-colour synaesthesia: to what extent does it use cross-modal mechanisms common to us all? Cortex 42(2):264–280PubMedCrossRefGoogle Scholar
  82. Watson AB, Pelli DG (1983) QUEST-a Bayesian adaptive psychometric method. Percept Psychophys 33(2):113–120PubMedCrossRefGoogle Scholar
  83. Weiskrantz L, Cowey A (1975) Cross-modal matching in the rhesus monkey using a single pair of stimuli. Neuropsychologia 13(3):257–261PubMedCrossRefGoogle Scholar
  84. Westbury C (2005) Implicit sound symbolism in lexical access: evidence from an interference task. Brain Lang 93(1):10–19PubMedCrossRefGoogle Scholar
  85. Xu J, Yu L, Rowland BA, Stanford TR, Stein BE (2012) Incorporating cross-modal statistics in the development and maintenance of multisensory integration. J Neurosci 32:2287–2298PubMedCrossRefGoogle Scholar
  86. Zigler MJ (1930) Tone shapes: a novel type of synaesthesia. J Gen Psychol 3:276–287CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2012

Authors and Affiliations

  1. 1.Department of Experimental PsychologyUniversity of OxfordOxfordUK
  2. 2.Max Planck Institute for Biological CyberneticsTübingenGermany
  3. 3.Bernstein Centre for Computational NeuroscienceTübingenGermany
  4. 4.Department of Cognitive Neuroscience and Center of Excellence Cognitive Interaction Technology (CITEC)University of BielefeldBielefeldGermany

Personalised recommendations