Advertisement

Cross-modal correspondences in sine wave: Speech versus non-speech modes

  • Daniel Márcio Rodrigues Silva
  • Samuel C. Bellini-LeiteEmail author
SHORT REPORT
  • 17 Downloads

Abstract

The present study aimed to investigate whether or not the so-called “bouba-kiki” effect is mediated by speech-specific representations. Sine-wave versions of naturally produced pseudowords were used as auditory stimuli in an implicit association task (IAT) and an explicit cross-modal matching (CMM) task to examine cross-modal shape-sound correspondences. A group of participants trained to hear the sine-wave stimuli as speech was compared to a group that heard them as non-speech sounds. Sound-shape correspondence effects were observed in both groups and tasks, indicating that speech-specific processing is not fundamental to the “bouba-kiki” phenomenon. Effects were similar across groups in the IAT, while in the CMM task the speech-mode group showed a stronger effect compared with the non-speech group. This indicates that, while both tasks reflect auditory-visual associations, only the CMM task is additionally sensitive to associations involving speech-specific representations.

Keywords

Cross-modal correspondences Sound symbolism Bouba-kiki effect Sine-wave speech 

Notes

Acknowledgements

Both authors were supported by Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) - Brazil. The authors also gratefully acknowledge Professors Barry C. Smith, Sarah Garfinkel, Vincent Hayward, and André J. Abath, as well as anonymous reviewers for thoughtful comments and suggestions.

Compliance with ethical standards

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Open practices statement

The data used in this research were made available to reviewers and to journal editors.

References

  1. Adeli, M., Rouat, J., & Molotchnikoff, S. (2014). Audiovisual correspondence between musical timbre and visual shapes. Frontiers in Human Neuroscience, 8.  https://doi.org/10.3389/fnhum.2014.00352
  2. Adler, D. (2019). vioplot: Violin Plot (Version 0.3.0). Retrieved from https://CRAN.R-project.org/package=vioplot
  3. Anikin, A., & Johansson, N. (2019). Implicit associations between individual properties of color and sound. Attention, Perception, & Psychophysics, 81(3), 764–777.  https://doi.org/10.3758/s13414-018-01639-7 Google Scholar
  4. Baart, M., Stekelenburg, J. J., & Vroomen, J. (2014). Electrophysiological evidence for speech-specific audiovisual integration. Neuropsychologia, 53, 115-121.  https://doi.org/10.1016/j.neuropsychologia.2013.11.011 Google Scholar
  5. Blasi, D. E., Wichmann, S., Hammarström, H., Stadler, P. F., & Christiansen, M. H. (2016). Sound–meaning association biases evidenced across thousands of languages. Proceedings of the National Academy of Sciences, 113(39), 10818–10823.  https://doi.org/10.1073/pnas.1605782113 Google Scholar
  6. Boersma, P., & Weenink, D. (2013). Praat: Doing phonetics by computer (Version 5.3. 41)[Software].Google Scholar
  7. Bremner, A., Caparos, S., Davidoff, J., de Fockert, J., Linnell, K. and Spence, C. (2013). Bouba and Kiki in Namibia? A remote culture makes similar shape–sound matches, but different shape–taste matches to Westerners. Cognition 126, 165–172.  https://doi.org/10.1016/j.cognition.2012.09.007 Google Scholar
  8. Chen, Y.-C., Huang, P.-C., Woods, A., & Spence, C. (2016). When “Bouba” equals “Kiki”: Cultural commonalities and cultural differences in sound-shape correspondences. Scientific Reports, 6, 26681.  https://doi.org/10.1038/srep26681 Google Scholar
  9. Chow, H. M., & Ciaramitaro, V. (2019). What makes a shape “baba”? The shape features prioritized in sound–shape correspondence change with development. Journal of Experimental Child Psychology, 179, 73–89.  https://doi.org/10.1016/j.jecp.2018.10.005 Google Scholar
  10. Cuskley, C., Simner, J., & Kirby, S. (2017). Phonological and orthographic influences in the bouba-kiki effect. Psychological Research, 81(1), 119-130.  https://doi.org/10.1007/s00426-015-0709-2 Google Scholar
  11. D’Onofrio, A. (2014). Phonetic detail and dimensionality in sound-shape correspondences: Refining the bouba-kiki paradigm. Language and Speech, 57(3), 367-393.  https://doi.org/10.1177/0023830913507694 Google Scholar
  12. Darwin, C. (2003). Sine-wave speech produced automatically using a script for the PRAAT program. http://www.lifesci.sussex.ac.uk/home/Chris_Darwin/SWS/ (Last viewed July, 2017)
  13. Davidson, D. J., & Martin, A. E. (2013). Modeling accuracy as a function of response time with the generalized linear mixed effects model. Acta Psychologica, 144(1), 83–96.  https://doi.org/10.1016/j.actpsy.2013.04.016 Google Scholar
  14. Dehaene-Lambertz, G., Pallier, C., Serniclaes, W., Sprenger-Charolles, L., Jobert, A., & Dehaene, S. (2005). Neural correlates of switching from auditory to speech perception. Neuroimage, 24(1), 21-33.  https://doi.org/10.1016/j.neuroimage.2004.09.039 Google Scholar
  15. Deroy, O., & Spence, C. (2013). Why we are not all synesthetes (not even weakly so). Psychonomic Bulletin & Review, 20(4), 643-664.  https://doi.org/10.3758/s13423-013-0387-2 Google Scholar
  16. Eskelund, K., Tuomainen, J., & Andersen, T. S. (2011). Multistage audiovisual integration of speech: Dissociating identification and detection. Experimental Brain Research, 208(3), 447–457.  https://doi.org/10.1007/s00221-010-2495-9 Google Scholar
  17. Evans, K. K., & Treisman, A. (2010). Natural cross-modal mappings between visual and auditory features. Journal of Vision, 10(1), 1–12.  https://doi.org/10.1167/10.1.6 Google Scholar
  18. Fort, M., Martin, A., & Peperkamp, S. (2015). Consonants are more important than vowels in the Bouba-kiki effect. Language and Speech, 58(2), 247–266.  https://doi.org/10.1177/0023830914534951 Google Scholar
  19. Fort, M., Weiß, A., Matin, A., & Peperkamp, S. (2013). Looking for the bouba-kiki effect in prelexical infants. In S. Ouni, F. Berthommier, & A. Jesse (Eds.), Proceedings of the 12th international conference on auditory-visual speech processing (pp. 71–76). Annecy France: Inria.Google Scholar
  20. Godoy, M., Duarte, A. C. V., Silva, F. L. F., Albano, G. F., Souza, R. J. P., & Silva, Y. U. A. P. M. (2018). The replication of takete-maluma effect in Brazilian Portuguese. Revista do GELNE, 20(1), 87–100.  https://doi.org/10.21680/1517-7874.2018v20n1ID13331 Google Scholar
  21. Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009). Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity. Journal of Personality and Social Psychology, 97(1), 17–41.  https://doi.org/10.1037/a0015575 Google Scholar
  22. Hickok, G., & Poeppel, D. (2007). The cortical organization of speech processing. Nature Reviews Neuroscience, 8(5), 393–402.  https://doi.org/10.1038/nrn2113 Google Scholar
  23. Jaeger, T. F. (2008). Categorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed models. Journal of Memory and Language, 59(4), 434-446.  https://doi.org/10.1016/j.jml.2007.11.007 Google Scholar
  24. Khoshkhoo, S., Leonard, M. K., Mesgarani, N., & Chang, E. F. (2018). Neural correlates of sine-wave speech intelligibility in human frontal and temporal cortex. Brain and Language, 187, 83–91.  https://doi.org/10.1016/j.bandl.2018.01.007 Google Scholar
  25. Knoeferle, K., Li, J., Maggioni, E., & Spence, C. (2017). What drives sound symbolism? Different acoustic cues underlie sound-size and sound-shape mappings. Scientific Reports, 7(1), 5562.  https://doi.org/10.1038/s41598-017-05965-y Google Scholar
  26. Kohler, W. (1947). Gestalt psychology (1929). New York, NY: Liveright.Google Scholar
  27. Liew, K., Lindborg, P., Rodrigues, R., & Styles, S. J. (2018). Cross-modal perception of noise-in-music: Audiences generate spiky shapes in response to auditory roughness in a novel electroacoustic concert setting. Frontiers in Psychology, 9.  https://doi.org/10.3389/fpsyg.2018.00178
  28. Lockwood, G., & Dingemanse, M. (2015). Iconicity in the lab: A review of behavioral, developmental, and neuroimaging research into sound-symbolism. Frontiers in Psychology, 6.  https://doi.org/10.3389/fpsyg.2015.01246
  29. Marks, L. E. (1987). On cross-modal similarity: Auditory–visual interactions in speeded discrimination. Journal of Experimental Psychology: Human Perception and Performance, 13(3), 384.  https://doi.org/10.1037/0096-1523.13.3.384 Google Scholar
  30. Maurer, D., Pathman, T., & Mondloch, C. J. (2006). The shape of boubas: Sound–shape correspondences in toddlers and adults. Developmental Science, 9(3), 316-322.  https://doi.org/10.1111/j.1467-7687.2006.00495.x Google Scholar
  31. McCormick, K., Kim, J., List, S., & Nygaard, L. C. (2015). Sound to Meaning Mappings in the Bouba-Kiki Effect. In Noelle, D. C., Dale, R., Warlaumont, A. S., Yoshimi, J., Matlock, T., Jennings, C. D., & Maglio, P. P. (Eds.). Proceedings of the 37th annual meeting of the cognitive science society. Austin, TX: Cognitive Science Society.Google Scholar
  32. Monahan, P. J. (2018). Phonological knowledge and speech comprehension. Annual Review of Linguistics, 4(1), 21–47.  https://doi.org/10.1146/annurev-linguistics-011817-045537 Google Scholar
  33. Morey, R. D. (2008) Confidence intervals from normalized data: A correction to Cousineau (2005). Tutorial in Quantitative Methods for Psychology, 4(2), 61-64.  https://doi.org/10.20982/tqmp.04.2.p061 Google Scholar
  34. Nielsen, A., & Rendall, D. (2011). The sound of round: Evaluating the sound-symbolic role of consonants in the classic Takete-Maluma phenomenon. Canadian Journal of Experimental Psychology/Revue Canadienne de Psychologie Experimentale, 65(2), 115–124.  https://doi.org/10.1037/a0022268 Google Scholar
  35. Nielsen, A. K., & Rendall, D. (2013). Parsing the role of consonants versus vowels in the classic Takete-Maluma phenomenon. Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale, 67(2), 153-163.  https://doi.org/10.1037/a0030553 Google Scholar
  36. O'Boyle, M. W., & Tarte, R. D. (1980). Implications for phonetic symbolism: The relationship between pure tones and geometric figures. Journal of Psycholinguistic Research, 9(6), 535-544.  https://doi.org/10.1007/BF01068115 Google Scholar
  37. Ozturk, O., Krehm, M., & Vouloumanos, A. (2013). Sound symbolism in infancy: Evidence for sound–shape cross-modal correspondences in 4-month-olds. Journal of Experimental Child Psychology, 114(2), 173-186.  https://doi.org/10.1016/j.jecp.2012.05.004 Google Scholar
  38. Parise, C., Spence, C. (2012). Audiovisual cross-modal correspondences and sound symbolism: A study using the implicit association task. Experimental Brain Research, 220 (3-4), 319-33.  https://doi.org/10.1007/s00221-012-3140-6 Google Scholar
  39. Ramachandran, V. S., & Hubbard, E. M. (2001). Synaesthesia--A window into perception, thought and language. Journal of Consciousness Studies, 8(12), 3-34.Google Scholar
  40. Remez, R. E., Rubin, P. E., & Pisoni, D. B. (1983). Coding of the speech spectrum in three time-varying sinusoids. Annals of the New York Academy of Sciences, 405(1), 485–489.  https://doi.org/10.1111/j.1749-6632.1983.tb31663.x Google Scholar
  41. Remez, R. E., Rubin, P. E., Pisoni, D. B., & Carrell, T. D. (1981). Speech perception without traditional speech cues. Science, 212(4497), 947–949.  https://doi.org/10.1126/science.7233191 Google Scholar
  42. Remez, R. E., & Thomas, E. F. (2013). Early recognition of speech. Wiley Interdisciplinary Reviews: Cognitive Science, 4(2), 213–223.  https://doi.org/10.1002/wcs.1213 Google Scholar
  43. Schellinger, S. K., Munson, B., & Edwards, J. (2017). Gradient perception of children’s productions of /s/ and /θ/: A comparative study of rating methods. Clinical Linguistics & Phonetics, 31(1), 80–103.  https://doi.org/10.1080/02699206.2016.1205665 Google Scholar
  44. Shang, N., & Styles, S. J. (2017). Is a high tone pointy? Speakers of different languages match Mandarin Chinese tones to visual shapes differently. Frontiers in Psychology, 8.  https://doi.org/10.3389/fpsyg.2017.02139
  45. Sidhu, D. M., & Pexman, P. M. (2018). Five mechanisms of sound symbolic association. Psychonomic Bulletin & Review, 25(5), 1619–1643.  https://doi.org/10.3758/s13423-017-1361-1 Google Scholar
  46. Spence, C. (2011). Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics, 73(4), 971-995.  https://doi.org/10.3758/s13414-010-0073-7 Google Scholar
  47. Styles, S. J., & Gawne, L. (2017). When does Maluma/Takete fail? Two key failures and a meta-analysis suggest that phonology and phonotactics matter. I-Perception, 8(4), 2041669517724807.  https://doi.org/10.1177/2041669517724807 Google Scholar
  48. Vroomen, J., & Stekelenburg, J. J. (2011). Perception of intersensory synchrony in audiovisual speech: Not that special. Cognition, 118(1), 75–83.  https://doi.org/10.1016/j.cognition.2010.10.002 Google Scholar
  49. Walker, P., Bremner, J. G., Mason, U., Spring, J., Mattock, K., Slater, A., & Johnson, S. P. (2010). Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences. Psychological Science, 21(1), 21-25.  https://doi.org/10.1177/0956797609354734 Google Scholar
  50. Wickham, H. (2009). ggplot2: Elegant graphics for data analysis. New York, NY: Springer-Verlag.Google Scholar
  51. Wobbrock, J. O., Findlater, L., Gergle, D., & Higgins, J. J. (2011). The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 143-146). ACM.Google Scholar

Copyright information

© The Psychonomic Society, Inc. 2019

Authors and Affiliations

  1. 1.Federal University of Minas Gerais (UFMG)Belo HorizonteBrazil
  2. 2.State University of Minas Gerais (UEMG) and Federal University of Minas Gerais (UFMG)Belo HorizonteBrazil

Personalised recommendations