Advertisement

Journal of Cultural Cognitive Science

, Volume 3, Issue 2, pp 113–139 | Cite as

Fixations in the visual world paradigm: where, when, why?

  • James S. MagnusonEmail author
Review

Abstract

Over the last 25 years, the visual world paradigm has enabled discoveries and theoretical advances in spoken language processing. However, the intuitive interpretation of fixations in the visual world paradigm—that fixations directly reflect over-time processes of activation and competition governing cognitive and language processing—deserves scrutiny. This paper provides a selective review of studies that suggest that the relations between fixations and ongoing processing are more complex than suggested by the intuitive interpretation. A particular challenge is explaining why context sometimes appears to have deep effects on language processing, while other times fixations appear to violate strong contextual constraints. I discuss implications of these seemingly contradictory patterns for theories of real-world language processing, and practical implications for using the visual world paradigm. Along the way, I review four possible linking hypotheses for connecting measures in the paradigm to theories of language and cognition. This review leads to the conclusion that implemented computational models will be needed to assess to what degree different linking hypotheses generate distinguishable predictions.

Keywords

Visual world paradigm Eye tracking Psycholinguistics 

Notes

Acknowledgements

This paper is based on a talk presented at the Attentive Listener in the Visual World meeting in Trondheim, Norway, in August, 2018. I am grateful to Falk Huettig, Mila Vulchanova, Valentin Vulchanov, Inge-Marie Eigsti, and Kenny Coventry for stimulating discussions that reshaped this paper. Preparation of this paper was supported in part by U.S. National Science Foundation Grants 1754284, Computational approaches to human spoken word recognition, and 1735225, Science of learning, from neurobiology to real-world application.

Compliance with ethical standards

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

References

  1. Allopenna, P. D., Magnuson, J. S., & Tanenhaus, M. K. (1998). Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory and Language,38, 419–439.Google Scholar
  2. Altmann, G. T. M., & Kamide, Y. (1999). Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition,73, 247–264.Google Scholar
  3. Altmann, G. T. M., & Kamide, Y. (2007). The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing. Journal of Memory and Language,57, 502–518.Google Scholar
  4. Anderson, J. R. (1991). Is human cognition adaptive? Behavioral and Brain Sciences,14(3), 471–517.Google Scholar
  5. Ballard, D. H. (1991). Animate vision. Artificial Intelligence,48, 57–86.Google Scholar
  6. Ballard, D. H., Hayhoe, M. M., Pook, P. K., & Rao, R. P. (1997). Deictic codes for the embodiment of cognition. Behavioural and Brain Sciences,20(4), 723–742.Google Scholar
  7. Balota, D. A. (1990). The role of meaning in word recognition. In D. A. Balota, G. Flores D’Arcais, & K. Rayner (Eds.), Comprehension processes in reading (pp. 9–32). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  8. Brooks, R. A. (1991). Intelligence without representation. Artificial Intelligence,47, 139–159.Google Scholar
  9. Brown-Schmidt, S., & Tanenhaus, M. K. (2008). Real-time interpretation of referential domains in unscripted conversation: A targeted language game approach. Cognitive Science,32, 643–684.  https://doi.org/10.1080/03640210802066816.CrossRefPubMedPubMedCentralGoogle Scholar
  10. Chambers, C., & San Juan, V. (2008). Perception and presupposition in real-time language comprehension: Insights from anticipatory processing. Cognition,108, 26–50.PubMedGoogle Scholar
  11. Chambers, C. G., Tanenhaus, M. K., & Magnuson, J. S. (2004). Actions and affordances in syntactic ambiguity resolution. Journal of Experimental Psychology. Learning, Memory, and Cognition,30, 687–696.PubMedGoogle Scholar
  12. Chiu, E. M., & Spivey, M. J. (2014). Timing of speech and display affects the linguistic mediation of visual search. Perception,43, 527–548.PubMedGoogle Scholar
  13. Contini, E. W., Wardle, S. G., & Carlson, T. A. (2017). Decoding the time-course of object recognition in the human brain: From visual features to categorical decisions. Neuropsychologia,105, 165–176.PubMedGoogle Scholar
  14. Cooper, R. M. (1974). The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology,6, 84–107.Google Scholar
  15. Cree, G. S., McRae, K., & McNorgan, C. (1999). An attractor model of lexical conceptual processing: Simulating semantic priming. Cognitive Science,23, 371–414.Google Scholar
  16. Dahan, D., Magnuson, J. S., & Tanenhaus, M. K. (2001a). Time course of frequency effects in spoken-word recognition: Evidence from eye movements. Cognitive Psychology,42, 317–367.PubMedGoogle Scholar
  17. Dahan, D., Magnuson, J. S., Tanenhaus, M. K., & Hogan, E. M. (2001b). Tracking the time course of subcategorical mismatches: Evidence for lexical competition. Language and Cognitive Processes,16(5/6), 507–534.Google Scholar
  18. Dahan, D., & Tanenhaus, M. K. (2005). Looking at the rope when looking for the snake: Conceptually mediated eye movements during spoken-word recognition. Psychonomic Bulletin and Review,12, 453–459.PubMedGoogle Scholar
  19. De Groot, F., Huettig, F., & Olivers, C. N. L. (2016). When meaning matters: The temporal dynamics of semantic influences on visual attention. Journal of Experimental Psychology: Human Perception and Performance,42(2), 180–196.  https://doi.org/10.1037/xhp0000102.CrossRefPubMedGoogle Scholar
  20. Dell’Acqua, R., & Grainger, J. (1999). Unconscious semantic priming from pictures. Cognition,73(1), B1–B15.PubMedGoogle Scholar
  21. Elman, J. L. (1990). Finding structure in time. Cognitive Science,14, 179–211.Google Scholar
  22. Elman, J. L. (2009). On the meaning of words and dinosaur bones: Lexical knowledge without a lexicon. Cognitive Science,33, 1–36.Google Scholar
  23. Elman, J. L. (2011). Lexical knowledge without a lexicon? The Mental Lexicon,6(1), 1–33.PubMedPubMedCentralGoogle Scholar
  24. Frauenfelder, U. H., & Peeters, G. (1998). Simulating the time course of spoken word recognition: An analysis of lexical competition in TRACE. In J. Grainger & A. M. Jacobs (Eds.), Localist connectionist approaches to human cognition (pp. 101–146). Mahwah, NJ: Erlbaum.Google Scholar
  25. Gibson, J. J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin Company.Google Scholar
  26. Gibson, E. J., & Pick, A. D. (2000). An ecological approach to perceptual learning and development. New York: Oxford University Press.Google Scholar
  27. Greeno, J. G. (1994). Gibson’s affordances. Psychological Review,101(2), 336–342.PubMedGoogle Scholar
  28. Hanna, J. E., Tanenhaus, M. K., & Trueswell, J. C. (2003). Journal of Memory and Language,49, 43–61.Google Scholar
  29. Hannagan, T., Magnuson, J. S., & Grainger, J. (2013). Spoken word recognition without a TRACE. Frontiers in Psychology,4, 563.PubMedPubMedCentralGoogle Scholar
  30. Harm, M. W., & Seidenberg, M. S. (1999). Phonology, reading acquisition, and dyslexia: Insights from connectionist models. Psychological Review,106(3), 491–528.PubMedGoogle Scholar
  31. Harm, M. W., & Seidenberg, M. S. (2004). Computing the meanings of words in reading: cooperative division of labor between visual and phonological processes. Psychological Review,111(3), 662–720.PubMedGoogle Scholar
  32. Hollingworth, A., & Henderson, J. M. (2002). Accurate visual memory for previously attended objects in natural scenes. Journal of Experimental Psychology: Human Perception and Performance,28(1), 113–136.  https://doi.org/10.1037/0096-1523.28.1.113.CrossRefGoogle Scholar
  33. Huettig, F. (2015). Four central questions about prediction in language processing. Brain Research,1626, 118–135.  https://doi.org/10.1016/j.brainres.2015.02.014.CrossRefGoogle Scholar
  34. Huettig, F., & Altmann, G. T. M. (2005). Word meaning and the control of eye fixation: semantic competitor effects and the visual world paradigm. Cognition,96(1), 23–32.Google Scholar
  35. Huettig, F., & McQueen, J. M. (2007). The tug of war between phonological, semantic and shape information in language-mediated visual search. Journal of Memory and Language,57(4), 460–482.  https://doi.org/10.1016/j.jml.2007.02.001.CrossRefGoogle Scholar
  36. Huettig, F., Olivers, C. N. L., & Hartsuiker, R. J. (2011a). Looking, language, and memory: Bridging research from the visual world and visual search paradigms. Acta Psychologica,137, 138–150.  https://doi.org/10.1016/j.actpsy.2010.07.013.CrossRefPubMedPubMedCentralGoogle Scholar
  37. Huettig, F., Rommers, J., & Meyer, A. S. (2011b). Using the visual world paradigm to study language processing: A review and critical evaluation. Acta Psychologica,137, 151–171.  https://doi.org/10.1016/j.actpsy.2010.11.003.CrossRefGoogle Scholar
  38. Kahneman, D. (2011). Thinking. Fast and Slow: Macmillan.Google Scholar
  39. Kamide, Y., Altmann, G. T. M., & Haywood, S. (2003). The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye movements. Journal of Memory and Language,49, 133–156.Google Scholar
  40. Keysar, B., Barr, D. J., & Horton, W. S. (1998). The egocentric basis of language use: Insights from a processing approach. Current Directions in Psychological Science,7, 46–50.Google Scholar
  41. Knoeferle, P., & Crocker, M. W. (2006). The coordinated interplay of scene, utterance, and world knowledge: Evidence from eye tracking. Cognitive Science,30, 481–529.PubMedPubMedCentralGoogle Scholar
  42. Knoeferle, P., & Crocker, M. W. (2007). The influence of recent scene events on spoken comprehension: Evidence from eye-movements. Journal of Memory and Language,57, 519–543.Google Scholar
  43. Kukona, A., Braze, D., Johns, C. L., Mencl, W. E., Van Dyke, J. A., Magnuson, J. S., et al. (2016). The real-time prediction and inhibition of linguistic outcomes: Effects of language and literacy skill. Acta Psychologica,171, 72–84.PubMedPubMedCentralGoogle Scholar
  44. Kukona, A., Cho, P. W., Magnuson, J. S., & Tabor, W. (2014). Lexical interference effects in sentence processing: Evidence from the visual world paradigm and self-organizing models. Journal of Experimental Psychology. Learning, Memory, and Cognition,40(2), 326–347.PubMedGoogle Scholar
  45. Kukona, A., Fang, S., Aicher, K. A., Chen, H., & Magnuson, J. S. (2011). The time course of anticipatory constraint integration. Cognition,119, 23–42.PubMedPubMedCentralGoogle Scholar
  46. Li, M. Y. C., Braze, D., Kukona, A., Johns, C. L., Tabor, W., Van Dyke, J. A., et al. (2019). Individual differences in subphonemic sensitivity and phonological skills. Journal of Memory and Language,105, 195–215.Google Scholar
  47. Luce, R. D. (1959). Individual choice behavior: A theoretical analysis. New York: Wiley.Google Scholar
  48. MacDonald, M. C., Pearlmutter, N. J., & Seidenberg, M. S. (1994). The lexical nature of syntactic ambiguity resolution. Psychological Review,101(4), 676–703.PubMedPubMedCentralGoogle Scholar
  49. Magnuson, J. S. (2008). Nondeterminism, pleiotropy, and single word reading: Theoretical and practical concerns. In E. Grigorenko & A. Naples (Eds.), single word reading (pp. 377–404). Mahweh, NJ: Erlbaum.Google Scholar
  50. Magnuson, J. S. (2017). Mapping spoken words to meaning. In G. Gaskell & J. Mirkovic (Eds.), Speech Perception and spoken word recognition (pp. 76–96). New York: Routledge.Google Scholar
  51. Magnuson, J. S. (2019a). Schematic of the time course of priming. figshare. Figure.  https://doi.org/10.6084/m9.figshare.9465416.v1
  52. Magnuson, J. (2019). Working memory visual world linking hypothesis (Version 2). figshare. Figure.  https://doi.org/10.6084/m9.figshare.8019518.v2
  53. Magnuson, J. (2019c). Deep interaction visual world paradigm linking hypothesis (Version 1). figshare. Figure.  https://doi.org/10.6084/m9.figshare.8020184.v1
  54. Magnuson, J. S. (in preparation). Comparative modeling of spoken word recognition.Google Scholar
  55. Magnuson, J. S. (in preparation). Similar microstructure of spoken word recognition across computational architectures.Google Scholar
  56. Magnuson, J. S., Dixon, J., Tanenhaus, M. K., & Aslin, R. N. (2007). The dynamics of lexical competition during spoken word recognition. Cognitive Science,31, 133–156.PubMedGoogle Scholar
  57. Magnuson, J. S., Mirman, D., & Harris, H. D. (2012). Computational models of spoken word recognition. In M. Spivey, K. McRae, & M. Joanisse (Eds.), The Cambridge handbook of psycholinguistics (pp. 76–103). Cambridge: Cambridge University Press.Google Scholar
  58. Magnuson, J. S., & Nusbaum, H. C. (2007). Acoustic differences, listener expectations, and the perceptual accommodation of talker variability. Journal of Experimental Psychology: Human Perception and Performance,33, 391–409.PubMedGoogle Scholar
  59. Magnuson, J. S., Tanenhaus, M. K., & Aslin, R. N. (2008). Immediate effects of form-class constraints on spoken word recognition. Cognition,108(3), 866–873.PubMedPubMedCentralGoogle Scholar
  60. Magnuson, J. S., Tanenhaus, M. K., Aslin, R. N., & Dahan, D. (2003). The time course of spoken word recognition and learning: Studies with artificial lexicons. Journal of Experimental Psychology: General,132(2), 202–227.Google Scholar
  61. Mani, N., & Plunkett, K. (2010). In the infant's mind's ear: Evidence for implicit naming in 18-month-olds. Psychological Science,21, 908–913.Google Scholar
  62. Marslen-Wilson, W., & Zwitserlood, P. (1989). Accessing spoken words: The importance of word onsets. Journal of Experimental Psychology: Human Perception and Performance,15, 576–585.Google Scholar
  63. McClelland, J. L. (1979). On the time relations of mental processes: An examination of systems of processes in cascade. Psychological Review,86, 287–330.Google Scholar
  64. McClelland, J. L., & Elman, J. L. (1986). The TRACE model of speech perception. Cognitive Psychology,18, 1–86.PubMedGoogle Scholar
  65. McClelland, J. L., & Rumelhart, D. E. (1981). An interactive activation model of context effects in letter perception: Part 1. An account of basic findings. Psychological Review,88, 375–407.Google Scholar
  66. McMurray, B., Tanenhaus, M., & Aslin, R. (2002). Gradient effects of within-category phonetic variation on lexical access. Cognition,86(2), B33–B42.PubMedGoogle Scholar
  67. McQueen, J. M., & Huettig, F. (2014). Interference of spoken word recognition through phonological priming from visual objects and printed words. Attention, Perception and Psychophysics,76, 190–200.  https://doi.org/10.3758/s13414-013-0560-8.CrossRefPubMedGoogle Scholar
  68. McRae, K., Cree, G. S., Seidenberg, M. S., & McNorgan, C. (2005). Semantic feature production norms for a large set of living and nonliving things. Behavior Research Methods,37, 547–559.PubMedGoogle Scholar
  69. McRae, K., Spivey-Knowlton, M. J., & Tanenhaus, M. K. (1998). Modeling the influence of thematic fit (and other constraints) in on-line sentence processing. Journal of Memory and Language, 38, 283–312.Google Scholar
  70. Mirman, D., & Magnuson, J. S. (2008). Attractor dynamics and semantic neighborhood density: Processing is slowed by near neighbors and speeded by distant neighbors. Journal of Experimental Psychology. Learning, Memory, and Cognition,34, 65–79.PubMedPubMedCentralGoogle Scholar
  71. Mirman, D., & Magnuson, J. S. (2009a). The effect of frequency of shared features on judgments of semantic similarity. Psychonomic Bulletin & Review,16(4), 671–677.Google Scholar
  72. Mirman, D., & Magnuson, J. S. (2009b). Dynamics of activation of semantically similar concepts during spoken word recognition. Memory and Cognition,37, 1026–1039.PubMedGoogle Scholar
  73. Mishra, R. K., Olivers, C. N. L., & Huettig, F. (2013). Spoken language and the decision to move the eyes: To what extent are language-mediated eye movements automatic? In V. S. C. Pammi & N. Srinivasan (Eds.), Progress in brain research: Decision making: Neural and behavioural approaches (pp. 135–149). New York: Elsevier.Google Scholar
  74. Nisbett, R. E., Peng, K., Choi, I., & Norenzayan, A. (2001). Culture and systems of thought: Holistic vs. analytic cognition. Psychological Review,108, 291–310.PubMedGoogle Scholar
  75. Noizet, G., & Pynte, J. (1976). Implict labeling and rediness for pronunciation during the perceptual process. Perception,5, 217–223.PubMedGoogle Scholar
  76. Pirog Revill, K., Tanenhaus, M. K., & Aslin, R. N. (2008). Context and spoken word recognition in a novel lexicon. Journal of Experimental Psychology. Learning, Memory, and Cognition,34(5), 1207–1223.Google Scholar
  77. Plaut, D. C., McClelland, J. L., Seidenberg, M. S., & Patterson, K. (1996). Understanding normal and impaired word reading: Computational principles in quasi-regular domains. Psychological Review,103, 56–115.PubMedGoogle Scholar
  78. Reali, F., Spivey, M. J., Tyler, M. J., & Terranova, J. (2006). Inefficient conjunction search made efficient by concurrent spoken delivery of target identity. Perception and Psychophysics,68, 959–974.PubMedGoogle Scholar
  79. Rensink, R. A., O’Regan, J. K., & Clark, J. J. (1997). To see or not to see: The need for attention to perceive changes in scenes. Psychological Science,8(5), 368–373.Google Scholar
  80. Richardson, D. C., & Spivey, M. J. (2000). Representation, space, and Hollywood Squares: Looking at things that aren’t there anymore. Cognition,76(3), 269–295.PubMedGoogle Scholar
  81. Salverda, A. P., Dahan, D., & McQueen, J. M. (2003). The role of prosodic boundaries in the resolution of lexical embedding in speech comprehension. Cognition,90, 51–89.PubMedGoogle Scholar
  82. Sedivy, J. C., Tanenhaus, M. K., Chambers, C. G., & Carlson, G. N. (1999). Achieving incremental semantic interpretation through contextual representation. Cognition,71, 109–147.PubMedPubMedCentralGoogle Scholar
  83. Seidenberg, M. S., & McClelland, J. L. (1989). A distributed, developmental model of word recognition and naming. Psychological Review,96, 523–568.PubMedGoogle Scholar
  84. Spivey, M. J. (2007). The continuity of mind. New York: Oxford University Press.Google Scholar
  85. Spivey, J. J., & Marian, V. (1999). Cross talk between native and second languages: Partial activation of an irrelevant lexicon. Psychological Science,10(3), 281–284.Google Scholar
  86. Spivey, M. J., Richardson, D. C., & Fitneva, S. A. (2004). Thinking outside the brain: Spatial indices to visual and linguistic information. In J. M. Henderson & F. Ferreira (Eds.), The interface of language, vision, and action: Eye movements and the visual world (pp. 161–189). New York, NY, US: Psychology Press.Google Scholar
  87. Spivey, M. J., & Spevack, S. C. (2017). An inclusive account of mind across spatiotemporal scales of cognition. Journal of Cultural Cognition,1, 25–38.  https://doi.org/10.1007/s41809-017-0002-6.CrossRefGoogle Scholar
  88. Strand, J. F., Brown, V. A., Brown, H. E., & Berg, J. J. (2017). Keep listening: Grammatical context reduces but does not eliminate activation of unexpected words. Journal of Experimental Psychology. Learning, Memory, and Cognition,44(6), 962–973.PubMedGoogle Scholar
  89. Tabor, W., Galantucci, B., & Richardson, D. (2004). Effects of merely local syntactic coherence on sentence processing. Journal of Memory and Language,50, 355–370.  https://doi.org/10.1016/j.jml.2004.01.001.CrossRefGoogle Scholar
  90. Tabor, W., & Hutchins, S. (2004). Evidence for self-organized sentence processing: Digging in effects. Journal of Experimental Psychology. Learning, Memory, and Cognition,30, 431–450.  https://doi.org/10.1037/0278-7393.30.2.431.CrossRefPubMedGoogle Scholar
  91. Tanenhaus, M. K., Magnuson, J. S., Dahan, D., & Chambers, C. (2000). Eye movements and lexical access in spoken-language comprehension: Evaluating a linking hypothesis between fixations and linguistic processing. Journal of Psycholinguistic Research,29, 557–580.PubMedGoogle Scholar
  92. Tanenhaus, M. K., & Spivey-Knowlton, M. J. (1996). Eye-tracking. Language and Cognitive Processes,11, 583–588.Google Scholar
  93. Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science,268, 1632–1634.Google Scholar
  94. Trueswell, J. C., & Tanenhaus, M. K. (1994). Toward a lexicalist framework for constraint-based syntactic ambiguity resolution. In C. Clifton, L. Frazier, & K. Rayner (Eds.), Perspectives in sentence processing (pp. 155–179). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  95. Turvey, M. (1973). On peripheral and central processes in vision: Inferences from an information-processing analysis of masking with patterned stimuli. Psychological Review,80, 1–52.PubMedGoogle Scholar
  96. Venhuizen, N. J., Crocker, M. W., & Brouwer, H. (2019). Expectation-based comprehension: Modeling the interaction of world knowledge and linguistic experience. Discourse Processes,56(3), 229–255.Google Scholar
  97. Vivianni, P. (1990). Eye movements in visual search: Cognitive, perceptual, and motor control aspects. In E. Kowler (Ed.), Eye movements and their role in visual and cognitive processes. Reviews of oculomotor research V4 (pp. 353–383). Amsterdam: Elsevier.Google Scholar
  98. Simmons, E. S., & Magnuson, J. S. (accepted with minor revisions). Word length, proportion of overlap, and the time course of phonological competition in spoken word recognition: An empirical and computational investigation. Cognitive Science. Google Scholar
  99. Yee, E., & Sedivy, J. (2001). Using eye movements to track the spread of semantic activation during spoken word recognition. Paper presented to the 13th annual CUNY sentence processing conference, Philadelphia.Google Scholar
  100. Yee, E., & Sedivy, J. C. (2006). Eye movements to pictures reveal transient semantic activation during spoken word recognition. Journal of Experimental Psychology. Learning, Memory, and Cognition,32, 1–14.PubMedGoogle Scholar
  101. You, H., & Magnuson, J. S. (2018). TISK 1.0: An easy-to-use Python implementation of the time-invariant string kernel model of spoken word recognition. Behavior Research Methods.  https://doi.org/10.3758/s13428-017-1012-5.CrossRefPubMedGoogle Scholar
  102. Zelinsky, G. J., & Murphy, G. L. (2000). Synchronizing visual and language processing: An effect of object name length on oculomotor behavior. Psychological Science, 11, 125–131.Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.Psychological SciencesUniversity of ConnecticutStorrsUSA

Personalised recommendations