Abstract
Listening and speaking constitute two most fundamental aspects of human communication. While speaking is largely intentional and calls for voluntary control and planning, listening could be largely unintentional. One listens and understands speech even when one is not being deliberate. Not much is known about control, attentional mechanisms and intention during listening. Most psycholinguistic studies ask participants to comprehend speech under different task requirements and measure comprehension. However, recently it has been observed that listening without any intention to comprehend can lead to attention capture and change in action plan. This nexus between speech and action control can be very useful in understanding goal planning and cognition in humans. The abundance of speech around us and its evolutionary importance makes it a very important biological stimulus in understanding different facets of control. In this chapter, we will review studies where unintentional comprehension of speech has been shown to cause changes in goal-directed action. We particularly look at studies where eye movements and other such motor actions have been shown to deviate during such multi-modal processing. It appears that speech just cannot be ignored in many situations and this penetrates into processing modules that require no language. We look at visual world studies with bilinguals where listening to words causes slowness in a visual search task.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Allopenna, P. D., Magnuson, J. S., & Tanenhaus, M. K. (1998). Tracking the time course ofspoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory and Language, 38(4), 419–439.
Altmann, G. T. M., & Kamide, Y. (2004). Now you see it, now you don’t: Mediating the mapping between language and the visual world. In J. Henderson & F. Ferreira (Eds.), The integration of language, vision and action (pp. 347–386). Psychology Press.
Bargh, J. A. (1994). The four horsemen of automaticity. In R. S. Wyer & T. K. Srull (Eds.), Handbook of social cognition (pp. 1–40). Hillsdale, NJ: Erlbaum.
Bertels, J., Kolinsky, R., Bernaerts, A., & Morais, J. (2011). Effects of emotional spoken words on exogenous attentional orienting. Journal of Cognitive Psychology, 23(4), 435–452.
Blumenfeld, H. K., & Marian, V. (2007). Constraints on parallel activation in bilingual spoken language processing: Examining proficiency and lexical status using eye-tracking. Language and Cognitive Processes, 22, 633–660.
Blumenfeld, H. K., & Marian, V. (2013). Parallel language activation and cognitive control during spoken word recognition in bilinguals. Journal of Cognitive Psychology, 25(5), 547–567.
Buchner, A., & Erdfelder, E. (2005). Word frequency of irrelevant speech distractors affects serial recall. Memory and Cognition, 33, 86–97.
Buchner, A., Rothermund, K., Wentura, D., & Mehl, B. (2004). Valence of distractor words increases the effects of irrelevant speech on serial recall. Memory and Cognition, 32, 722–731.
Buchner, A. (1996). On the irrelevance of semantic information for the llIrrelevant speech effect. The Quarterly Journal of Experimental Psychology: Section A, 49(3), 765–779.
Chua, F. K. (2013). Attentional capture by onsets and offsets. Visual Cognition, 21(5), 569–598.
Cleland, A. A., Tamminen, J., Quinlan, P. T., & Gaskell, M. G. (2012). Spoken word processing creates a lexical bottleneck. Language and Cognitive Processes, 27(4), 572–593.
Cooper, R. M. (1974). The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6(1), 84–107.
Dahan, D., Magnuson, J. S., & Tanenhaus, M. K. (2001). Time course of frequency effects in spoken-word recognition: evidence from eye movements. Cognitive Psychology, 42, 317–367.
Dijkstra, T., Miwa, K., Brummelhuis, B., Sappelli, M., & Baayen, H. (2010). How crosslanguage similarity and task demands affect cognate recognition. Journal of Memory and Language, 62, 284–301.
Dijkstra, A. F. J., & Van Heuven, W. J. B. (2002). The architecture of the bilingual word recognition system: From identification to decision. Bilingualism: Language and Cognition, 5(3), 175–197.
Downing, P. E. (2000). Interactions between visual working memory and selective attention. Psychological Science, 11, 467–473.
Driver, J., & Spence, C. (1998). Cross-modal links in spatial attention. Philosophical Transactions of the Royal Society B: Biological Sciences, 353(1373), 1319.
Elliott, E. M., & Briganti, A. M. (2012). Investigating the role of attentional processes in the irrelevant speech effect. Acta Psychologica, 140, 64–74.
Guo, T., Misra, M., Tam, J. W., & Kroll, J. F. (2012). On the time course of accessing meaning in a second language: An electrophysiological investigation of translation recognition. Journal of Experimental Psychology. Learning, Memory, and Cognition, 38, 1165–1186. doi:10.1037/a0028076.
Huettig, F., & Altmann, G. T. M. (2004). Language-mediated eye movements and the resolution of lexical ambiguity. In M. Carreiras & C. Clifton (Eds.), The on-line study of sentence comprehension: Eye-tracking, ERP, and beyond (pp. 187–207). New York, NY: Psychology Press.
Huettig, F., & Altmann, G. T. M. (2005). Word meaning and the control of eye fixation: semantic competitor effects and the visual world paradigm. Cognition, 96(1), 23–32.
Huettig, F., & Altmann, G. T. M. (2007). Visual-shape competition and the control of eye fixation during the processing of unambiguous and ambiguous words. Visual Cognition, 15(8), 985–1018.
Huettig, F., & Altmann, G. T. M. (2011). Looking at anything that is green when hearing ‘frog’—How object surface colour and stored object colour knowledge influence language-mediated overt attention. Quarterly Journal of Experimental Psychology, 64(1), 122–145.
Huettig, F., & Hartsuiker, R. J. (2008). When you name the pizza you look at the coin and the bread: Eye movements reveal semantic activation during word production. Memory and Cognition, 36(2), 341–360. doi:10.3758/MC.36.2.341.
Huettig, F., Mishra, R. K., & Olivers, C. N. (2012). On the mechanisms and representations of language-mediated visual attention. Frontiers in Cognition, 2, 394.
Huettig, F., & McQueen, J. M. (2007). The tug of war between phonological, semantic and shape information in language-mediated visual search. Journal of Memory and Language, 57(4), 460–482. doi:10.1016/j.jml.2007.02.001.
Irwin, D. E., Colcombe, A. M., Kramer, A. F., & Hahn, S. (2000). Attention and oculomotor capture by onset luminance and color singletons. Vision Research, 40, 1443–1458.
Jones, D. M., Miles, C., & Page, J. (1990). Disruption of proofreading by irrelevant speech: Effects of attention, arousal or memory? Applied Cognitive Psychology, 4(2), 89–108.
Jones, D. M., Madden, C., & Miles, C. (1992). Privileged access by irrelevant speech to short-term memory: The role of changing state. Quarterly Journal of Experimental Psychology: Human Experimental Psychology, 44, 645–669.
Jonides, J., Naveh-Benjamin, M., & Palmer, J. (1985). Assessing automaticity. Acta Psychologica, 60(2), 157–171.
Ju, M., & Luce, P. A. (2004). Falling on sensitive ears. Psychological Science, 15, 314–318.
Kamide, Y., & Altmann, G. T. M. (2004). The time-course of constraint-application during sentence processing in visual contexts: Anticipatory eye-movements in English and Japanese. In M. Tanenhaus & J. Trueswell (Eds.), World situated language use: Psycholinguistic, linguistic and computational perspectives on bridging the product and action traditions. Cambridge: MIT Press.
Kamide, Y., Altmann, G. T. M., & Haywood, S. (2003). The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye-movements. Journal of Memory and Language, 49, 133–159.
Klatte, M., Lachmann, T., Schlittmeier, S., & Hellbrück, J. (2010). The irrelevant sound effect in short-term memory: Is there developmental change?European. Journal of Cognitive Psychology, 22(8), 1168–1191.
Lavie, N. (2005). Distracted and confused? Selective attention under load. Trends in Cognitive Sciences, 9, 75–82.
Lavie, N., & Tsal, Y. (1994). Perceptual load as a major determinant of the locus of selection in visual attention. Perception and Psychophysics, 56, 183–197.
LeCompte, D. C., Neely, C. B., & Wilson, J. R. (1997). Irrelevant speech and irrelevant tones: The relative importance of speech to the irrelevant speech effect. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23, 472–483.
Logan, G. D. (1998). What is learned during automatization? Obligatory encoding of location information. Journal of Experimental Psychology: Human Perception and Performance, 24, 1720–1736.
Lupyan, G., & Swingley, D. (2011). Self-directed speech affects visual processing. Quarterly Journal of Experimental Psychology, 65(6), 1068–1085.
Ljungberg, J. K., Parmentier, F. B., Hughes, R. W., Macken, W. J., & Jones, D. M. (2012). Listen out! Behavioural and subjective responses to verbal warnings. Applied Cognitive Psychology, 26(3), 451–461.
Mani, N., & Huettig, F. (2012). Prediction is a piece of cake—but only for skilled producers. Journal of Experimental Psychology: Human Perception and Performance, 38, 843–847.
Marsh, J. E., Hughes, R. W., & Jones, D. M. (2008). Auditory distraction in semantic memory: A process-based approach. Journal of Memory and Language, 58(3), 682–700.
Marian, V., & Spivey, M. (2003a). Bilingual and monolingual processing of competing lexical items. Applied Psycholinguistics, 24, 173–193.
Marian, V., & Spivey, M. (2003b). Competing activation in bilingual language processing: Within and between-language competition. Bilingualism: Language and Cognition, 6, 97–115.
Martin, R. C., Wogalter, M. S., & Forlano, J. G. (1988). Reading comprehension in the presence of unattended speech and music. Journal of Memory and Language, 27, 382–398.
Mishra, R. (2009). Interface of language and visual attention: Evidence from production and comprehension. Progress in Brain Research, 176, 277–292.
Mishra, R. K., Huettig, F., & Olivers, C. N. (2012). Automaticity and conscious decisions during language-mediated eye gaze in the visual world. In N. Srinivasan & V. S. C. Pammi (Eds.), Progress in Brain Research: Decision Making: Neural and Behavioral Approaches. Amsterdam: Elsevier.
Moors, A.., & De Houwer, J. (2006). Automaticity: a theoretical and conceptual analysis. Psychological bulletin, 132(2), 297.
Moores, E., Laiti, L., & Chelazzi, L. (2003). Associative knowledge controls deployment of visual selective attention. Nature Neuroscience, 6, 182–189.
Nation, K., Marshall, C., & Altmann, G. T. M. (2003). Investigating individual differences in children’s real-time sentence comprehension using language-mediated eye movements. Journal of Experimental Child Psychology, 86, 314–329.
Neely, C. B., & LeCompte, D. C. (1999). The importance of semantic similarity to the irrelevant speech effect. Memory and Cognition, 27, 37–44.
Oswald, C. J. P., Tremblay, S., & Jones, D. M. (2000). Disruption of comprehension by the meaning of irrelevant sound. Memory, 8, 345–350. doi:10.1080/09658210050117762.
Parmentier, F. B. R., Turner, J., & Perez, L. (2013). A dual contribution to the involuntary semantic processing of unexpected spoken words. Journal of Experimental Psychology: General. doi:10.1037/a0031550.
Pickering, M. J., & Garrod, S. (2004). Toward a mechanistic psychology of dialogue. Behavioral and Brain Sciences, 27(2), 169–189.
Pulvermüller, F., & Shtyrov, Y. (2006). Language outside the focus of attention: The mismatch negativity as a tool for studying higher cognitive processes. Progress in Neurobiology, 79(1), 49.
Salamé, P., & Baddeley, A. (1982). Disruption of short term memory by unattended speech: Implications for thestructure of working memory. Journal of Verbal Learning and Verbal Behavior, 21, 150–164.
Salverda, A. P., & Altmann, G. T. M. (2011). Attentional capture of objects referred to by spoken language. Journal of Experimental Psychology: Human Perception and Performance, 37, 1122–1133.
Shtyrov. Y., Kimppa, L., Pulvermüller, F., Kujala, T. (2010). Event-related potentials reflecting the frequency of unattended spoken words: A neuronal index of connection strength in lexical memory circuits? Neuroimage, 15;55(2), 658–668.
Singh, N., & Mishra, R. K. (2013). Second language proficiency modulates conflict-monitoring in an oculomotor stroop task: Evidence from Hindi-English Bilinguals. Frontiers in Psychology,. doi:10.3389/fpsyg.2013.00322.
Singh, N., & Mishra, R. K. (2012). Does language proficiency modulate oculomotor control? Evidence from Hindi-English bilinguals. Bilingualism: Language and Cognition, 15, 771–781.
Soto, D., & Humphreys, G. W. (2007). Automatic guidance of visual attention from verbal working memory. The Journal of Experimental Psychology: Human Perception and Performance, 33(3), 730–737.
Spivey, M. J., & Marian, V. (1999). Cross talk between native and second languages: Partial activation of an irrelevant lexicon. Psychological Science, 10(3), 281–284.
Sunderman, G., & Priya, K. (2012). Translation recognition in highly proficient Hindi-Englis bilinguals the influence of different scripts but connectable phonologies. Language and Cognitive Processes, 27(9), 1265–1285.
Telling, A. L., Kumar, S., Meyer, A. S., & Humphreys, G. W. (2010). Electrophysiological evidence of semantic interference in visual search. Journal of Cognitive Neuroscience, 22(10), 2212–2225.
Tellinghuisen, D. J., & Nowak, E. J. (2003). The inability to ignore auditory distractors as a function of visual task perceptual load. Perception and Psychophysics, 65, 817–828.
Thierry, G., & Wu, Y. J. (2010). Chinese–English bilinguals reading english hear Chinese. The Journal of Neuroscience, 30, 7646–7651.
Tun, P. A., O’Kane, G., & Wingfield, A. (2002). Distraction by competing speech in younger and older listeners. Psychology and Aging, 17, 453–467.
Weaver, M. D., Lauwereyns, J., & Theeuwes, J. (2011). The effect of semantic information on saccade trajectory deviations. Vision Research, 51, 1124–1128.
Weber, A., & Cutler, A. (2004). Lexical competition in non-native spoken-word recognition. Journal of Memory and Language, 50, 1–25.
Yantis, S., & Jonides, J. (1984). Abrupt visual onsets and selective attention: Evidence from visual search. Journal of Experimental Psychology: Human Perception and Performance, 5, 625–638.
Yee, E., Overton, E., & Thompson-Schill, S. L. (2009). Looking for meaning: Eye movements are sensitive to overlapping semantic features, not association. Psychonomic Bulletin and Review, 16(5), 869–874.
Yee, E., & Sedivy, J. C. (2006). Eye movements to pictures reveal transient semantic activation during spoken word recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32, 1–14.
Zeelenberg, R., & Bocanegra, B. R. (2010). Auditory emotional cues enhance visual perception. Cognition, 115, 202–206.
Acknowledgments
Support for this chapter came from a DST grant awarded to RKM under the Cognitive Science Initiative Scheme.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer India
About this chapter
Cite this chapter
Mishra, R.K., Singh, N. (2015). Spoken Word Mediated Interference in a Visual Task: Eye Tracking Evidence from Bilinguals. In: Mishra, R., Srinivasan, N., Huettig, F. (eds) Attention and Vision in Language Processing. Springer, New Delhi. https://doi.org/10.1007/978-81-322-2443-3_10
Download citation
DOI: https://doi.org/10.1007/978-81-322-2443-3_10
Published:
Publisher Name: Springer, New Delhi
Print ISBN: 978-81-322-2442-6
Online ISBN: 978-81-322-2443-3
eBook Packages: Humanities, Social Sciences and LawSocial Sciences (R0)