Advertisement

Experimental Brain Research

, Volume 198, Issue 2–3, pp 245–259 | Cite as

Audio–tactile superiority over visuo–tactile and audio–visual combinations in the temporal resolution of synchrony perception

  • Waka Fujisaki
  • Shin’ya Nishida
Research Article

Abstract

To see whether there is a difference in temporal resolution of synchrony perception between audio–visual (AV), visuo–tactile (VT), and audio–tactile (AT) combinations, we compared synchrony–asynchrony discrimination thresholds of human participants. Visual and auditory stimuli were, respectively, a luminance-modulated Gaussian blob and an amplitude-modulated white noise. Tactile stimuli were mechanical vibrations presented to the index finger. All the stimuli were temporally modulated by either single pulses or repetitive-pulse trains. The results show that the temporal resolution of synchrony perception was similar for AV and VT (e.g., ~4 Hz for repetitive-pulse stimuli), but significantly higher for AT (~10 Hz). Apart from having a higher temporal resolution, however, AT synchrony perception was similar to AV synchrony perception in that participants could select matching features through attention, and a change in the matching-feature attribute had little effect on temporal resolution. The AT superiority in temporal resolution was indicated not only by synchrony–asynchrony discrimination but also by simultaneity judgments. Temporal order judgments were less affected by modality combination than the other two tasks.

Keywords

Psychophysics Temporal synchrony Audio–visual–tactile Synchrony–asynchrony discrimination Simultaneity judgment Temporal order judgment 

Notes

Acknowledgment

Most of this work was carried out when WF was a JSPS Research Fellow at NTT Communication Science Laboratories.

References

  1. Aghdaee SM, Cavanagh P (2007) Temporal limits of long-range phase discrimination across the visual field. Vis Res 47(16):2156–2163PubMedCrossRefGoogle Scholar
  2. Amano K, Johnston A, Nishida S (2007) Two mechanisms underlying the effect of angle of motion direction change on colour-motion asynchrony. Vis Res 47(5):687–705PubMedCrossRefGoogle Scholar
  3. Arnold DH (2005) Perceptual pairing of colour and motion. Vis Res 45(24):3015–3026PubMedCrossRefGoogle Scholar
  4. Arrighi R, Alais D, Burr D (2006) Perceptual synchrony of audiovisual streams for natural and artificial motion sequences. J Vis 6(3):260–268PubMedCrossRefGoogle Scholar
  5. Axelrod S, Thompson LW, Cohen LD (1968) Effects of senescence on the temporal resolution of somesthetic stimuli presented to one hand or both. J Gerontol 23(2):191–195PubMedGoogle Scholar
  6. Caetano G, Jousmaki V (2006) Evidence of vibrotactile input to human auditory cortex. Neuroimage 29(1):15–28PubMedCrossRefGoogle Scholar
  7. Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9(6):719–721PubMedCrossRefGoogle Scholar
  8. Duncan J, Martens S, Ward R (1997) Restricted attentional capacity within but not between sensory modalities. Nature 387(6635):808–810PubMedCrossRefGoogle Scholar
  9. Foxe JJ, Wylie GR, Martinez A, Schroeder CE, Javitt DC, Guilfoyle D et al (2002) Auditory-somatosensory multisensory processing in auditory association cortex: an fmri study. J Neurophysiol 88(1):540–543PubMedGoogle Scholar
  10. Fujisaki W, Nishida S (2005) Temporal frequency characteristics of synchrony–asynchrony discrimination of audio–visual signals. Exp Brain Res 166(3–4):455–464PubMedCrossRefGoogle Scholar
  11. Fujisaki W, Nishida S (2007) Feature-based processing of audio–visual synchrony perception revealed by random pulse trains. Vis Res 47(8):1075–1093PubMedCrossRefGoogle Scholar
  12. Fujisaki W, Nishida S (2008a) Temporal limits of within- and cross-modal cross-attribute bindings. Paper presented at the 9th international multisensory research forum, Hamburg, GermanyGoogle Scholar
  13. Fujisaki W, Nishida S (2008b) Top-down feature-based selection of matching features for audio–visual synchrony discrimination. Neurosci Lett 433(3):225–230PubMedCrossRefGoogle Scholar
  14. Fujisaki W, Shimojo S, Kashino M, Nishida S (2004) Recalibration of audiovisual simultaneity. Nat Neurosci 7(7):773–778PubMedCrossRefGoogle Scholar
  15. Fujisaki W, Koene A, Arnold D, Johnston A, Nishida S (2006) Visual search for a target changing in synchrony with an auditory signal. Proc Biol Sci 273(1588):865–874PubMedCrossRefGoogle Scholar
  16. Hanson JV, Heron J, Whitaker D (2008) Recalibration of perceived time across sensory modalities. Exp Brain Res 185(2):347–352PubMedCrossRefGoogle Scholar
  17. Harrar V, Harris LR (2008) The effect of exposure to asynchronous audio, visual, and tactile stimulus combinations on the perception of simultaneity. Exp Brain Res 186(4):517–524PubMedCrossRefGoogle Scholar
  18. Hirsh IJ, Sherrick C E Jr (1961) Perceived order in different sense modalities. J Exp Psychol 62:423–432PubMedCrossRefGoogle Scholar
  19. Holcombe AO, Cavanagh P (2001) Early binding of feature pairs for visual perception. Nat Neurosci 4(2):127–128PubMedCrossRefGoogle Scholar
  20. Hoshiyama M, Kakigi R, Tamura Y (2004) Temporal discrimination threshold on various parts of the body. Muscle Nerve 29(2):243–247PubMedCrossRefGoogle Scholar
  21. Kandil FI, Fahle M (2001) Purely temporal figure-ground segregation. Eur J Neurosci 13(10):2004–2008PubMedCrossRefGoogle Scholar
  22. Kayser C, Petkov CI, Augath M, Logothetis NK (2005) Integration of touch and sound in auditory cortex. Neuron 48(2):373–384PubMedCrossRefGoogle Scholar
  23. Kelly DH (1979) Motion and vision. Ii. Stabilized spatio-temporal threshold surface. J Opt Soc Am 69(10):1340–1349PubMedCrossRefGoogle Scholar
  24. Lavie N, Tsal Y (1994) Perceptual load as a major determinant of the locus of selection in visual attention. Percept Psychophys 56(2):183–197PubMedGoogle Scholar
  25. Lee BB, Martin PR, Valberg A, Kremers J (1993) Physiological mechanisms underlying psychophysical sensitivity to combined luminance and chromatic modulation. J Opt Soc Am A 10(6):1403–1412PubMedCrossRefGoogle Scholar
  26. Levanen S, Jousmaki V, Hari R (1998) Vibration-induced auditory-cortex activation in a congenitally deaf adult. Curr Biol 8(15):869–872PubMedCrossRefGoogle Scholar
  27. Lewkowicz DJ (1996) Perception of auditory-visual temporal synchrony in human infants. J Exp Psychol Hum Percept Perform 22(5):1094–1106PubMedCrossRefGoogle Scholar
  28. Lu ZL, Sperling G (2001) Three-systems theory of human visual motion perception: Review and update. J Opt Soc Am A Opt Image Sci Vis 18(9):2331–2370PubMedCrossRefGoogle Scholar
  29. Lu T, Liang L, Wang X (2001) Temporal and rate representations of time-varying signals in the auditory cortex of awake primates. Nat Neurosci 4(11):1131–1138PubMedCrossRefGoogle Scholar
  30. Martens W, Woszczyk W (2004) Perceived synchrony in a bimodal display: optimal intermodal delay for coordinated auditory and haptic reproduction. Paper presented at the ICAD 04-tenth meeting of the international conference on auditory display, Sydney, AustraliaGoogle Scholar
  31. Moore B (2003) An introduction to the psychology of hearing, 5th edn. Academic Press, San DiegoGoogle Scholar
  32. Navarra J, Soto-Faraco S, Spence C (2007) Adaptation to audiotactile asynchrony. Neurosci Lett 413(1):72–76PubMedCrossRefGoogle Scholar
  33. Nishida S, Johnston A (2002) Marker correspondence, not processing latency, determines temporal binding of visual attributes. Curr Biol 12(5):359–368PubMedCrossRefGoogle Scholar
  34. Occelli V, Spence C, Zampini M (2008) Audiotactile temporal order judgments in sighted and blind individuals. Neuropsychologia 46(11):2845–2850PubMedCrossRefGoogle Scholar
  35. Pöppel E (1988) Mindworks: time and conscious experience. New York, Harcourt Brace JovanovichGoogle Scholar
  36. Recanzone GH (2003) Auditory influences on visual temporal rate perception. J Neurophysiol 89(2):1078–1093PubMedCrossRefGoogle Scholar
  37. Schneider KA, Bavelier D (2003) Components of visual prior entry. Cogn Psychol 47(4):333–366PubMedCrossRefGoogle Scholar
  38. Schroeder CE, Lindsley RW, Specht C, Marcovici A, Smiley JF, Javitt DC (2001) Somatosensory input to auditory association cortex in the macaque monkey. J Neurophysiol 85(3):1322–1327PubMedGoogle Scholar
  39. Shapiro KL, Arnell KM, Raymond JE (1997) The attentional blink. Trends Cogn Sci 1:291–296CrossRefGoogle Scholar
  40. Shipley T (1964) Auditory flutter-driving of visual flicker. Science 145:1328–1330PubMedCrossRefGoogle Scholar
  41. Shore DI, Spry E, Spence C (2002) Confusing the mind by crossing the hands. Brain Res Cogn Brain Res 14(1):153–163PubMedCrossRefGoogle Scholar
  42. Soto-Faraco S, Deco G (2009) Multisensory contributions to the perception of vibrotactile events. Behav Brain Res 196(2):145–154PubMedCrossRefGoogle Scholar
  43. Spence C, Baddeley R, Zampini M, James R, Shore DI (2003) Multisensory temporal order judgments: When two locations are better than one. Percept Psychophys 65(2):318–328PubMedGoogle Scholar
  44. Van de Burg E, Olivers CN, Bronkhorst AW, Theeuves J (2008) Pip and pop: nonspatial auditory signals improve spatial visual search. J Exp Psychol Hum Percept Perform 34(5):1053–1065PubMedCrossRefGoogle Scholar
  45. van de Par S, Kohlrausch A (2004) Visual and auditory object selection based on temporal correlations between auditory and visual cues. Paper presented at the 18th int. congress on acoustics, Kyoto, Japan, 4–9 April 2004Google Scholar
  46. van de Par S, Kohlrausch A, Juola F (2002) Some methodological aspects for measuring asynchrony detection in audio–visual stimuli. Paper presented at the Forum Acousticum Sevilla, Sevilla, SpainGoogle Scholar
  47. van Eijk RL, Kohlrausch A, Juola JF, van de Par S (2008) Audiovisual synchrony and temporal order judgments: effects of experimental method and stimulus type. Percept Psychophys 70(6):955–968PubMedCrossRefGoogle Scholar
  48. Vatakis A, Spence C (2006a) Audiovisual synchrony perception for music, speech, and object actions. Brain Res 1111(1):134–142PubMedCrossRefGoogle Scholar
  49. Vatakis A, Spence C (2006b) Audiovisual synchrony perception for speech and music assessed using a temporal order judgment task. Neurosci Lett 393(1):40–44PubMedCrossRefGoogle Scholar
  50. Vatakis A, Spence C (2006c) Temporal order judgments for audiovisual targets embedded in unimodal and bimodal distractor streams. Neurosci Lett 408(1):5–9PubMedCrossRefGoogle Scholar
  51. Vatakis A, Spence C (2007) How ‘special’ is the human face? Evidence from an audiovisual temporal order judgment task. Neuroreport 18(17):1807–1811PubMedCrossRefGoogle Scholar
  52. Vatakis A, Spence C (2008) Investigating the effects of inversion on configural processing with an audiovisual temporal-order judgment task. Perception 37(1):143–160PubMedCrossRefGoogle Scholar
  53. Vatakis A, Navarra J, Soto-Faraco S, Spence C (2008) Audiovisual temporal adaptation of speech: temporal order versus simultaneity judgments. Exp Brain Res 185(3):521–529PubMedCrossRefGoogle Scholar
  54. Verstraten FA, Cavanagh P, Labianca AT (2000) Limits of attentive tracking reveal temporal properties of attention. Vision Res 40(26):3651–3664CrossRefGoogle Scholar
  55. Von Békésy G (1959) Similarities between hearing and skin sensations. Psychol Rev 66(1):1–22CrossRefGoogle Scholar
  56. Von Békésy G (1963) Interaction of paired sensory stimuli and conduction in peripheral nerves. J Appl Physiol 18:1276–1284Google Scholar
  57. Vroomen J, Keetels M, de Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio–visual asynchrony. Brain Res Cogn Brain Res 22(1):32–35PubMedCrossRefGoogle Scholar
  58. Yamamoto S, Kitazawa S (2001) Reversal of subjective temporal order due to arm crossing. Nat Neurosci 4(7):759–765PubMedCrossRefGoogle Scholar
  59. Yau JM, Olenczak JB, Dammann JF, Bensmaia SJ (2009) Temporal frequency channels are linked across audition and touch. Curr Biol 19(7):561–566PubMedCrossRefGoogle Scholar
  60. Zampini M, Shore DI, Spence C (2003) Audiovisual temporal order judgments. Exp Brain Res 152(2):198–210PubMedCrossRefGoogle Scholar
  61. Zampini M, Brown T, Shore DI, Maravita A, Roder B, Spence C (2005a) Audiotactile temporal order judgments. Acta Psychol (Amst) 118(3):277–291CrossRefGoogle Scholar
  62. Zampini M, Guest S, Shore DI, Spence C (2005b) Audio–visual simultaneity judgments. Percept Psychophys 67(3):531–544PubMedGoogle Scholar

Copyright information

© Springer-Verlag 2009

Authors and Affiliations

  1. 1.National Institute of Advanced Industrial Science and Technology (AIST)TsukubaJapan
  2. 2.NTT Communication Science LaboratoriesNTT CorporationAtsugi-shiJapan

Personalised recommendations