Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Audio-visual synchrony and feature-selective attention co-amplify early visual processing

Abstract

Our brain relies on neural mechanisms of selective attention and converging sensory processing to efficiently cope with rich and unceasing multisensory inputs. One prominent assumption holds that audio-visual synchrony can act as a strong attractor for spatial attention. Here, we tested for a similar effect of audio-visual synchrony on feature-selective attention. We presented two superimposed Gabor patches that differed in colour and orientation. On each trial, participants were cued to selectively attend to one of the two patches. Over time, spatial frequencies of both patches varied sinusoidally at distinct rates (3.14 and 3.63 Hz), giving rise to pulse-like percepts. A simultaneously presented pure tone carried a frequency modulation at the pulse rate of one of the two visual stimuli to introduce audio-visual synchrony. Pulsed stimulation elicited distinct time-locked oscillatory electrophysiological brain responses. These steady-state responses were quantified in the spectral domain to examine individual stimulus processing under conditions of synchronous versus asynchronous tone presentation and when respective stimuli were attended versus unattended. We found that both, attending to the colour of a stimulus and its synchrony with the tone, enhanced its processing. Moreover, both gain effects combined linearly for attended in-sync stimuli. Our results suggest that audio-visual synchrony can attract attention to specific stimulus features when stimuli overlap in space.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2

References

  1. An X, Hohne J, Ming D, Blankertz B (2014) Exploring combinations of auditory and visual stimuli for gaze-independent brain–computer interfaces. PLoS One 9:e111070. doi:10.1371/journal.pone.0111070

  2. Andersen SK, Hillyard SA, Müller MM (2008) Attention facilitates multiple stimulus features in parallel in human visual cortex. Curr Biol 18:1006–1009

  3. Atmaca S, Stadler W, Keitel A, Ott DV, Lepsien J, Prinz W (2013) Prediction processes during multiple object tracking (MOT): involvement of dorsal and ventral premotor cortices. Brain Behav 3:683–700. doi:10.1002/brb3.180

  4. Bidet-Caulet A, Fischer C, Besle J, Aguera PE, Giard MH, Bertrand O (2007) Effects of selective attention on the electrophysiological representation of concurrent sounds in the human auditory cortex. J Neurosci 27:9252–9261

  5. Busse L, Roberts KC, Crist RE, Weissman DH, Woldorff MG (2005) The spread of attention across modalities and space in a multisensory object. Proc Natl Acad Sci USA 102:18751–18756. doi:10.1073/pnas.0507704102

  6. Delorme A, Makeig S (2004) EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods 134:9–21

  7. Donohue SE, Green JJ, Woldorff MG (2015) The effects of attention on the temporal integration of multisensory stimuli. Front Integr Neurosci 9:32. doi:10.3389/fnint.2015.00032

  8. Eimer M, Grubert A (2014) The gradual emergence of spatially selective target processing in visual search: from feature-specific to object-based attentional control. J Exp Psychol Hum Percept Perform 40:1819–1831. doi:10.1037/a0037387

  9. Franconeri SL, Pylyshyn ZW, Scholl BJ (2012) A simple proximity heuristic allows tracking of multiple objects through occlusion. Atten Percept Psychophys 74:691–702. doi:10.3758/s13414-011-0265-9

  10. Fujisaki W, Nishida S (2005) Temporal frequency characteristics of synchrony-asynchrony discrimination of audio-visual signals. Exp Brain Res 166:455–464. doi:10.1007/s00221-005-2385-8

  11. Fujisaki W, Nishida S (2008) Top-down feature-based selection of matching features for audio-visual synchrony discrimination. Neurosci Lett 433:225–230. doi:10.1016/j.neulet.2008.01.031

  12. Fujisaki W, Nishida S (2009) Audio-tactile superiority over visuo-tactile and audio-visual combinations in the temporal resolution of synchrony perception. Exp Brain Res 198:245–259. doi:10.1007/s00221-009-1870-x

  13. Giani AS, Ortiz E, Belardinelli P, Kleiner M, Preissl H, Noppeney U (2012) Steady-state responses in MEG demonstrate information integration within but not across the auditory and visual senses. Neuroimage 60:1478–1489. doi:10.1016/j.neuroimage.2012.01.114

  14. Herrmann B, Schlichting N, Obleser J (2014) Dynamic range adaptation to spectral stimulus statistics in human auditory cortex. J Neurosci 34:327–331. doi:10.1523/JNEUROSCI.3974-13.2014

  15. Jarmasz J, Hollands JG (2009) Confidence intervals in repeated-measures designs: the number of observations principle. Can J Exp Psychol 63:124–138. doi:10.1037/a0014164

  16. Jenkins J 3rd, Rhone AE, Idsardi WJ, Simon JZ, Poeppel D (2011) The elicitation of audiovisual steady-state responses: multi-sensory signal congruity and phase effects. Brain Topogr 24:134–148. doi:10.1007/s10548-011-0174-1

  17. Keitel C, Schröger E, Saupe K, Müller MM (2011) Sustained selective intermodal attention modulates processing of language-like stimuli. Exp Brain Res 213(2–3):321–327

  18. Keitel C, Maess B, Schroger E, Muller MM (2013) Early visual and auditory processing rely on modality-specific attentional resources. Neuroimage 70:240–249. doi:10.1016/j.neuroimage.2012.12.046

  19. Keitel C, Quigley C, Ruhnau P (2014) Stimulus-driven brain oscillations in the alpha range: Entrainment of intrinsic rhythms or frequency-following response? J Neurosci 34:10137–10140. doi:10.1523/Jneurosci.1904-14.2014

  20. Kim YJ, Grabowecky M, Paller KA, Muthu K, Suzuki S (2007) Attention induces synchronization-based response gain in steady-state visual evoked potentials. Nat Neurosci 10:117–125

  21. Kim YJ, Grabowecky M, Paller KA, Suzuki S (2011) Differential roles of frequency-following and frequency-doubling visual responses revealed by evoked neural harmonics. J Cogn Neurosci 23:1875–1886. doi:10.1162/jocn.2010.21536

  22. Lakatos P, O’Connell MN, Barczak A, Mills A, Javitt DC, Schroeder CE (2009) The leading sense: supramodal control of neurophysiological context by attention. Neuron 64:419–430. doi:10.1016/j.neuron.2009.10.014

  23. Leek MR (2001) Adaptive procedures in psychophysical research. Percept Psychophys 63:1279–1292

  24. Macaluso E, Driver J (2003) Multimodal spatial representations in the human parietal cortex: evidence from functional imaging. Adv Neurol 93:219–233

  25. Maunsell JH, Treue S (2006) Feature-based attention in visual cortex. Trends Neurosci 29:317–322. doi:10.1016/j.tins.2006.04.001

  26. Müller MM (2014) Neural mechanisms of feature-based attention. In: Mangun GR (ed) Cognitive electrophysiology of attention: signals of the mind. Elsevier, Amsterdam, pp 123–135

  27. Müller MM, Picton TW, Valdes-Sosa P, Riera J, Teder-Salejarvi WA, Hillyard SA (1998) Effects of spatial selective attention on the steady-state visual evoked potential in the 20–28 Hz range. Brain Res Cogn Brain Res 6:249–261

  28. Müller MM, Malinowski P, Gruber T, Hillyard SA (2003) Sustained division of the attentional spotlight. Nature 424:309–312

  29. Müller MM, Andersen S, Trujillo NJ, Valdes-Sosa P, Malinowski P, Hillyard SA (2006) Feature-selective attention enhances color signals in early visual areas of the human brain. Proc Natl Acad Sci USA 103:14250–14254

  30. Nolan H, Whelan R, Reilly RB (2010) FASTER: fully automated statistical thresholding for EEG artifact rejection. J Neurosci Methods 192:152–162. doi:10.1016/j.jneumeth.2010.07.015

  31. Nozaradan S, Peretz I, Mouraux A (2012) Steady-state evoked potentials as an index of multisensory temporal binding. Neuroimage 60:21–28. doi:10.1016/j.neuroimage.2011.11.065

  32. Pastor MA, Valencia M, Artieda J, Alegre M, Masdeu JC (2007) Topography of cortical activation differs for fundamental and harmonic frequencies of the steady-state visual-evoked responses. An EEG and PET H215O study. Cereb Cortex 17:1899–1905. doi:10.1093/cercor/bhl098

  33. Porcu E, Keitel C, Muller MM (2013) Concurrent visual and tactile steady-state evoked potentials index allocation of inter-modal attention: a frequency-tagging study. Neurosci Lett 556:113–117. doi:10.1016/j.neulet.2013.09.068

  34. Porcu E, Keitel C, Muller MM (2014) Visual, auditory and tactile stimuli compete for early sensory processing capacities within but not between senses. Neuroimage 97:224–235. doi:10.1016/j.neuroimage.2014.04.024

  35. Regan D (1989) Human brain electrophysiology: evoked potentials and evoked magnetic fields in science and medicine. Elsevier, New York

  36. Ross B, Picton TW, Herdman AT, Pantev C (2004) The effect of attention on the auditory steady-state response. Neurol Clin Neurophysiol 2004:22

  37. Saenz M, Buracas GT, Boynton GM (2002) Global effects of feature-based attention in human visual cortex. Nat Neurosci 5:631–632. doi:10.1038/nn876

  38. Saupe K, Schröger E, Andersen SK, Müller MM (2009a) Neural mechanisms of intermodal sustained selective attention with concurrently presented auditory and visual stimuli. Front Hum Neurosci 3:58

  39. Saupe K, Widmann A, Bendixen A, Müller MM, Schröger E (2009b) Effects of intermodal attention on the auditory steady-state response and the event-related potential. Psychophysiology 46:321–327

  40. Schall S, Quigley C, Onat S, König P (2009) Visual stimulus locking of EEG is modulated by temporal congruency of auditory stimuli. Exp Brain Res 198:137–151

  41. Snyder AC, Foxe JJ (2010) Anticipatory attentional suppression of visual features indexed by oscillatory alpha-band power increases: a high-density electrical mapping study. J Neurosci 30:4024–4032. doi:10.1523/jneurosci.5684-09.2010

  42. Talsma D, Senkowski D, Soto-Faraco S, Woldorff MG (2010) The multifaceted interplay between attention and multisensory integration. Trends Cogn Sci 14:400–410

  43. Treisman AM, Gelade G (1980) A feature-integration theory of attention. Cogn Psychol 12:97–136

  44. Treue S, Martinez Trujillo JC (1999) Feature-based attention influences motion processing gain in macaque visual cortex. Nature 399:575–579

  45. Van der Burg E, Olivers CN, Bronkhorst AW, Theeuwes J (2008) Pip and pop: nonspatial auditory signals improve spatial visual search. J Exp Psychol Hum Percept Perform 34:1053–1065

  46. Van der Burg E, Cass J, Olivers CN, Theeuwes J, Alais D (2010) Efficient visual search from synchronized auditory signals requires transient audiovisual events. PLoS One 5:e10664. doi:10.1371/journal.pone.0010664

  47. Van der Burg E, Talsma D, Olivers CN, Hickey C, Theeuwes J (2011) Early multisensory interactions affect the competition among multiple visual objects. Neuroimage 55:1208–1218. doi:10.1016/j.neuroimage.2010.12.068

  48. Van der Burg E, Cass J, Alais D (2014) Window of audio-visual simultaneity is unaffected by spatio-temporal visual clutter. Sci Rep 4:5098. doi:10.1038/srep05098

  49. Vroomen J, Keetels M (2010) Perception of intersensory synchrony: a tutorial review. Atten Percept Psychophys 72:871–884. doi:10.3758/APP.72.4.871

  50. Wagner G, Boynton RM (1972) Comparison of four methods of heterochromatic photometry. J Opt Soc Am 62:1508–1515

  51. Werner S, Noppeney U (2011) The contributions of transient and sustained response codes to audiovisual integration. Cereb Cortex 21:920–931

  52. Winkler I, Denham S, Mill R, Bohm TM, Bendixen A (2012) Multistability in auditory stream segregation: a predictive coding view. Philos Trans R Soc Lond B Biol Sci 367:1001–1012. doi:10.1098/rstb.2011.0359

  53. Wolfe JM (1994) Guided search 2.0—a revised model of visual-search. Psychon Bull Rev 1:202–238. doi:10.3758/Bf03200774

  54. Yi DJ, Turk-Browne NB, Flombaum JI, Kim MS, Scholl BJ, Chun MM (2008) Spatiotemporal object continuity in human ventral visual cortex. Proc Natl Acad Sci USA 105:8840–8845. doi:10.1073/pnas.0802525105

Download references

Acknowledgments

Work was supported by the Deutsche Forschungsgemeinschaft (Grant No. MU972/21-1). Data presented here were recorded at the Institut für Psychologie, Universität Leipzig. The authors appreciate the assistance of Renate Zahn and Emanuele Porcu in data collection. The experimental stimulation was realized using Cogent 2000 developed by the Cogent 2000 team at the Functional Imaging Laboratory and the Institute of Cognitive Neuroscience and Cogent Graphics developed by John Romaya at the Laboratory of Neurobiology at the Wellcome Department of Imaging Neuroscience.

Author information

Correspondence to Christian Keitel.

Ethics declarations

Conflict of interest

None declared.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Keitel, C., Müller, M.M. Audio-visual synchrony and feature-selective attention co-amplify early visual processing. Exp Brain Res 234, 1221–1231 (2016). https://doi.org/10.1007/s00221-015-4392-8

Download citation

Keywords

  • Feature-based attention
  • Selective attention
  • Multisensory integration
  • Audio-visual synchrony
  • Brain oscillations
  • Neural rhythms
  • Steady-state visual evoked potentials (SSVEP)
  • EEG
  • Brain–computer interface (BCI)