Experimental Brain Research

, 198:339 | Cite as

Multisensory integration of drumming actions: musical expertise affects perceived audiovisual asynchrony

  • Karin Petrini
  • Sofia Dahl
  • Davide Rocchesso
  • Carl Haakon Waadeland
  • Federico Avanzini
  • Aina Puce
  • Frank E. Pollick
Research Article

Abstract

We investigated the effect of musical expertise on sensitivity to asynchrony for drumming point-light displays, which varied in their physical characteristics (Experiment 1) or in their degree of audiovisual congruency (Experiment 2). In Experiment 1, 21 repetitions of three tempos × three accents × nine audiovisual delays were presented to four jazz drummers and four novices. In Experiment 2, ten repetitions of two audiovisual incongruency conditions × nine audiovisual delays were presented to 13 drummers and 13 novices. Participants gave forced-choice judgments of audiovisual synchrony. The results of Experiment 1 show an enhancement in experts’ ability to detect asynchrony, especially for slower drumming tempos. In Experiment 2 an increase in sensitivity to asynchrony was found for incongruent stimuli; this increase, however, is attributable only to the novice group. Altogether the results indicated that through musical practice we learn to ignore variations in stimulus characteristics that otherwise would affect our multisensory integration processes.

Keywords

Synchrony perception Audiovisual integration Audiovisual congruency Drumming actions Musical expertise 

Notes

Acknowledgments

This work was supported by a grant from the British Academy (LRG-42455). We would like to thank Melanie Russell for her contribution during the pilot phase of this study and Gerry Rossi of Strathclyde University for his help in recruiting the jazz drummers. We also would like to thank Jim Kay and Michael Kubovy for their valuable suggestions on data analysis and statistical methods.

References

  1. Arai T, Greenberg S (1997). The temporal properties of spoken Japanese are similar to those of English. In: Proceedings of Eurospeech, Rhodes, Greece, pp 1011–1014Google Scholar
  2. Arrighi R, Alais D, Burr D (2006) Perceptual synchrony of audiovisual streams for natural and artificial motion sequences. J Vision 6:260–268CrossRefGoogle Scholar
  3. Bengtsson SL, Nagy Z, Skare S, Forsman L, Forssberg H, Ullén F (2005) Extensive piano practicing has regionally specific effects on white matter development. Nature Neurosci 8:1148–1150PubMedCrossRefGoogle Scholar
  4. Bermudez P, Zatorre RJ (2005) Differences in gray matter between musicians and nonmusicians. Ann N Y Acad Sci 1060:395–399PubMedCrossRefGoogle Scholar
  5. Brainard DH (1997) The psychophysics toolbox. Spatial Vision 10:433–436PubMedCrossRefGoogle Scholar
  6. Brooks A, van der Zwan R, Billard A, Petreska B, Clarke S, Blanke O (2007) Auditory motion affects visual biological motion processing. Neuropsychologia 45:523–530PubMedCrossRefGoogle Scholar
  7. Clifford CWG, Arnold DH, Pearson J (2003) A paradox of temporal perception revealed by a stimulus oscillating in colour and orientation. Vision Res 43:2245–2253PubMedCrossRefGoogle Scholar
  8. Collyer CE, Broadbent HA, Church RM (1994) Preferred rates of repetitive tapping and categorical time production. Percept Psychophys 55:443–453PubMedGoogle Scholar
  9. Dahl S (2004) Playing the accent—comparing striking velocity and timing in an ostinato rhythm performed by four drummers. Acta Acustica United Acustica 90:762–776Google Scholar
  10. Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9:719–721PubMedCrossRefGoogle Scholar
  11. Drake C, Botte MC (1993) Tempo sensitivity in auditory sequences: evidence for a multiple-look model. Percept Psychophys 54:277–286PubMedGoogle Scholar
  12. Drake C, Jones MR, Baruch C (2000) The development of rhythmic attending in auditory sequences: attunement, referent period, focal attending. Cognition 77:251–288PubMedCrossRefGoogle Scholar
  13. Fain GL (2003) Sensory transduction. Sinauer Associates, Sunderland, MAGoogle Scholar
  14. Fontana F, Rocchesso D, Avanzini F (2004) Computation of nonlinear filter networks containing delay-free paths. In Proceedings of the 7th international conference on digital audio effects (DAFX-04), Naples, Italy, pp 113–118Google Scholar
  15. Friberg A, Sundberg J (1995) Time discrimination in a monotonic, isochronous sequence. J Acoust Soc Am 98:2524–2531CrossRefGoogle Scholar
  16. Fujisaki W, Nishida S (2005) Temporal frequency characteristics of synchrony–asynchrony discrimination of audio–visual signals. Exp Brain Res 166:455–464PubMedCrossRefGoogle Scholar
  17. Gaser C, Schlaug G (2003) Brain structures differ between musicians and non-musicians. J Neurosci 23:9240–9245PubMedGoogle Scholar
  18. Grant KW, Greenberg S (2001) Speech intelligibility derived from asynchronous processing of auditory–visual speech information. In: Proceedings of AVSP 2001 international conference of auditory–visual speech processing, Scheelsminde, Denmark, pp 132–137Google Scholar
  19. Grant KW, van Wassenhove V, Poeppel D (2004) Detection of auditory (cross-spectral) and auditory–visual (cross-modal) synchrony. J Acoust Soc Am 108:1197–1208CrossRefGoogle Scholar
  20. Hodges DA, Hairston WD, Burdette JH (2005) Aspects of multisensory perception: the integration of visual and auditory information in musical experiences. Ann N Y Acad Sci 1060:175–185PubMedCrossRefGoogle Scholar
  21. Hollier MP, Rimell AN (1998) An experimental investigation into multi-modal synchronisation sensitivity for perceptual model development. 105th AES Convention Preprint No. 4790Google Scholar
  22. Johansson G (1973) Visual perception of biological motion and model for its analysis. Percept Psychophys 14:201–211Google Scholar
  23. Jones MR (1987) Dynamic pattern structure in music: recent theory and research. Percept Psychophys 41:621–634PubMedGoogle Scholar
  24. Kay BA, Kelso JAS, Saltzman EL, Schöner G (1987) Space–time behavior of single and bimanual rhythmical movements: data and limit cycle model. J Exp Psychol: Human Percept Perform 13:178–192CrossRefGoogle Scholar
  25. King AJ (2005) Multisensory integration: strategies for synchronization. Curr Biol 15:R339–R341PubMedCrossRefGoogle Scholar
  26. King AJ, Palmer AR (1985) Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp Brain Res 60:492–500PubMedCrossRefGoogle Scholar
  27. Luck G, Nte S (2008) An investigation of conductors’ temporal gestures and conductor— musician synchronization, and a first experiment. Psychol Music 36:81–99CrossRefGoogle Scholar
  28. Luck G, Sloboda JA (2007) An investigation of musicians’ synchronization with traditional conducting beat patterns. Music Perform Res 1:26–46Google Scholar
  29. MacDonald J, McGurk H (1978) Visual influences on speech perception processes. Percept Psychophys 24:253–257PubMedGoogle Scholar
  30. MacDougall HG, Moore ST (2005) Marching to the beat of the same drummer: the spontaneous tempo of human locomotion. J Appl Physiol 99:1164–1173PubMedCrossRefGoogle Scholar
  31. Massaro DW, Cohen MM, Smeele PM (1996) Perception of asynchronous and conflicting visual and auditory speech. J Acoust Soc Am 100:1777–1786PubMedCrossRefGoogle Scholar
  32. McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748PubMedCrossRefGoogle Scholar
  33. Miner N, Caudell T (1998) Computational requirements and synchronization issues for virtual acoustic displays. Presence-Teleoperat Virtual Environ 7(4):396–409CrossRefGoogle Scholar
  34. Moelants D (2002) Preferred tempo reconsidered. In: Proceedings of the 7th international conference on music perception and cognition, Sydney, 2002, Stevens C, Burnham D, McPherson G, Schubert E, Renwick J (eds). Causal productions, Adelaide, pp 580–583Google Scholar
  35. Munhall KG, Gribble P, Sacco L, Ward M (1996) Temporal constraints on the McGurk effect. Percept Psychophys 58:351–362PubMedGoogle Scholar
  36. Murray M, Drought AB, Kory RC (1964) Walking patterns of normal men. J Bone Joint Surgery 46:335–360Google Scholar
  37. Navarra J, Vatakis A, Zampini M, Soto-Faraco S, Humphreys W, Spence C (2005) Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration. Cognitive Brain Res 25(2):499–507CrossRefGoogle Scholar
  38. Palmer C, Krumhansl CL (1990) Mental representations for musical meter. J Exp Psychol Hum Percept Perform 16:728–741PubMedCrossRefGoogle Scholar
  39. Pelli DG (1997) The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spatial Vision 10:437–442PubMedCrossRefGoogle Scholar
  40. Petrini K, Russell M, Pollick F (2009) When knowing can replace seeing in audiovisual integration of actions. Cognition 110:432–439PubMedCrossRefGoogle Scholar
  41. Saygin AP, Driver J, de Sa VR (2008) In the footsteps of biological motion and multisensory perception: judgements of audio-visual temporal relations are enhanced for upright walkers. Psychol Sci 19:469–475PubMedCrossRefGoogle Scholar
  42. Schutz M, Kubovy M (2008) The effect of tone envelope on sensory integration: support for the ‘unity assumption’. J Acoust Soc Am 123:3412CrossRefGoogle Scholar
  43. Schutz M, Lipscomb S (2007) Hearing gestures, seeing music: vision influences perceived tone duration. Perception 36:888–897PubMedCrossRefGoogle Scholar
  44. Spence C, Squire S (2003) Multisensory integration: maintaining the perception of synchrony. Curr Biol 13:519–521CrossRefGoogle Scholar
  45. Spence C, Walton M (2005) On the inability to ignore touch when responding to vision in the crossmodal congruency task. Acta Psychol 118:47–70CrossRefGoogle Scholar
  46. Spence C, Shore DI, Klein RM (2001) Multisensory prior entry. J Exp Psychol: Gen 130:799–832CrossRefGoogle Scholar
  47. Steinmetz R (1996) Human perception of jitter and media synchronization. IEEE 14:61–72Google Scholar
  48. Stone JV, Hunkin NM, Porrill J, Wood R, Keeler V, Beanland M, Port M, Porter NR (2001) When is now? Perception of simultaneity. Proc R Soc, B 268:31–38CrossRefGoogle Scholar
  49. Sugita Y, Suzuki Y (2003) Audiovisual perception: implicit estimation of sound-arrival time. Nature 421:911PubMedCrossRefGoogle Scholar
  50. van Wassenhove V, Grant KW, Poeppel D (2007) Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45:598–607PubMedCrossRefGoogle Scholar
  51. Vatakis A, Spence C (2006a) Audiovisual synchrony perception for speech and music using a temporal order judgment task. Neurosci Lett 393:40–44PubMedCrossRefGoogle Scholar
  52. Vatakis A, Spence C (2006b) Audiovisual synchrony perception for music, speech, and object actions. Brain Res 1111:134–142PubMedCrossRefGoogle Scholar
  53. Vatakis A, Spence C (2007) Crossmodal binding: evaluating the “unity assumption” using audiovisual speech stimuli. Percept Psychophys 69:744–756PubMedGoogle Scholar
  54. Vatakis A, Spence C (2008) Evaluating the influence of the ‘unity assumption’ on the temporal perception of realistic audiovisual stimuli. Acta Psychol 127:12–23CrossRefGoogle Scholar
  55. Vatakis A, Navarra J, Soto-Faraco S, Spence C (2007a) Audiovisual temporal adaptation of speech: temporal order versus simultaneity judgments. Exp Brain Res 185:521–529PubMedCrossRefGoogle Scholar
  56. Vatakis A, Navarra J, Soto-Faraco S, Spence C (2007b) Temporal recalibration during asynchronous audiovisual speech perception. Exp Brain Res 181:173–181PubMedCrossRefGoogle Scholar
  57. Vatakis A, Ghazanfar AA, Spence C (2008) Facilitation of multisensory integration by the “unity effect” reveals that speech is special. J Vision 8:1–11CrossRefGoogle Scholar
  58. Vroomen J, Keetels M, de Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cognitive Brain Res 22:32–35CrossRefGoogle Scholar
  59. Waadeland CH (2003) Analysis of jazz drummers’ movements in performance of swing grooves-a preliminary report. In: Bresin R (ed) Proceedings of Stockholm music acoustic conference, volume II, Stockholm, pp 573–576Google Scholar
  60. Waadeland CH (2006) Strategies in empirical studies of swing groove. Studia Musicol Norvegica 32:169–191Google Scholar
  61. Wagner A (2006) Analysis of drumbeats—interaction between Drummer, Drumstick and Instrument. Master’s Thesis at the Department of Speech, Music and Hearing (TMH)Google Scholar
  62. Watson AB, Hu J (1999) ShowTime: a QuickTime-based infrastructure for vision research displays. Perception 28 ECVP Abstract Supplement, 45bGoogle Scholar
  63. Zampini M, Shore DI, Spence C (2003) Audiovisual temporal order judgments. Exp Brain Res 152:198–210PubMedCrossRefGoogle Scholar
  64. Zhou F, Wong V, Sekuler R (2007) Multi-sensory integration of spatio-temporal segmentation cues: one plus one does not always equal two. Exp Brain Res 180:641–654PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2009

Authors and Affiliations

  • Karin Petrini
    • 1
  • Sofia Dahl
    • 2
  • Davide Rocchesso
    • 3
  • Carl Haakon Waadeland
    • 4
  • Federico Avanzini
    • 5
  • Aina Puce
    • 6
  • Frank E. Pollick
    • 1
  1. 1.Department of PsychologyUniversity of GlasgowGlasgowScotland, UK
  2. 2.Department of Media TechnologyAalborg University CopenhagenCopenhagenDenmark
  3. 3.Department of Art and Industrial DesignIUAV University of VeniceVeniceItaly
  4. 4.Department of MusicNorwegian University of Science and TechnologyTrondheimNorway
  5. 5.Department of Information EngineeringUniversity of PadovaPadovaItaly
  6. 6.Department of Psychological and Brain SciencesIndiana UniversityBloomingtonUSA

Personalised recommendations