The right look for the job: decoding cognitive processes involved in the task from spatial eye-movement patterns

Abstract

The aim of the study was not only to demonstrate whether eye-movement-based task decoding was possible but also to investigate whether eye-movement patterns can be used to identify cognitive processes behind the tasks. We compared eye-movement patterns elicited under different task conditions, with tasks differing systematically with regard to the types of cognitive processes involved in solving them. We used four tasks, differing along two dimensions: spatial (global vs. local) processing (Navon, Cognit Psychol, 9(3):353–383 1977) and semantic (deep vs. shallow) processing (Craik and Lockhart, J Verbal Learn Verbal Behav, 11(6):671–684 1972). We used eye-movement patterns obtained from two time periods: fixation cross preceding the target stimulus and the target stimulus. We found significant effects of both spatial and semantic processing, but in case of the latter, the effect might be an artefact of insufficient task control. We found above chance task classification accuracy for both time periods: 51.4% for the period of stimulus presentation and 34.8% for the period of fixation cross presentation. Therefore, we show that task can be to some extent decoded from the preparatory eye-movements before the stimulus is displayed. This suggests that anticipatory eye-movements reflect the visual scanning strategy employed for the task at hand. Finally, this study also demonstrates that decoding is possible even from very scant eye-movement data similar to Coco and Keller, J Vis 14(3):11–11 (2014). This means that task decoding is not limited to tasks that naturally take longer to perform and yield multi-second eye-movement recordings.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. Altmann, G.T.M. (2004). Language-mediated eye movements in the absence of a visual world: The ‘blank screen paradigm’. Cognition, 93(2), B79–B87. https://doi.org/10.1016/j.cognition.2004.02.005.

    Article  PubMed  Google Scholar 

  2. Bernstein, L.J., Beig, S., Siegenthaler, A.L., & Grady, C.L. (2002). The effect of encoding strategy on the neural correlates of memory for faces. Neuropsychologia, 40(1), 86–98. https://doi.org/10.1016/S0028-3932(01)00070-7.

    Article  PubMed  Google Scholar 

  3. Betz, T., Kietzmann, T.C., Wilming, N., & König, P. (2010). Investigating task-dependent top-down effects on overt visual attention. Journal of Vision, 10(3), 1–14. https://doi.org/10.1167/10.3.15.

    Article  PubMed  Google Scholar 

  4. Boisvert, J.F.G., & Bruce, N.D.B. (2016). Predicting task from eye movements: On the importance of spatial distribution, dynamics, and image features. Neurocomputing, 207, 653–668. https://doi.org/10.1016/j.neucom.2016.05.047.

    Article  Google Scholar 

  5. Borji, A., & Itti, L. (2014). Defending Yarbus: Eye movements reveal observers’ task. Journal of Vision, 14(3:29), 1–22. https://doi.org/10.1167/14.3.29.

    Article  Google Scholar 

  6. Bower, G.H., & Karlin, M.B. (1974). Depth of processing pictures of faces and recognition memory. Journal of Experimental Psychology, 103(4), 751–757. https://doi.org/10.1037/h0037190.

    Article  Google Scholar 

  7. Castelhano, M.S., Mack, M.L., & Henderson, J.M. (2009). Viewing task influences eye movement control during active scene perception. Journal of Vision, 9(3), 6–6. https://doi.org/10.1167/9.3.6.

    Article  PubMed  Google Scholar 

  8. Coco, M.I., & Keller, F. (2014). Classification of visual and linguistic tasks using eye-movement features. Journal of Vision, 14(3), 11–11. https://doi.org/10.1167/14.3.11.

    Article  PubMed  Google Scholar 

  9. Cowen, L., Ball, L.J., & Delin, J. (2002). An eye movement analysis of web page usability. In People and computers XVI—memorable yet invisible (pp. 317–335). London: Springer. https://doi.org/10.1007/978-1-4471-0105-5_19.

    Chapter  Google Scholar 

  10. Craik, F.I.M., & Lockhart, R.S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11(6), 671–684. https://doi.org/10.1016/S0022-5371(72)80001-X.

    Article  Google Scholar 

  11. DeAngelus, M., & Pelz, J.B. (2009). Top-down control of eye movements: Yarbus revisited. Visual Cognition, 17(6–7), 790–811. https://doi.org/10.1080/13506280902793843.

    Article  Google Scholar 

  12. Ehinger, K.A., Hidalgo-Sotelo, B., Torralba, A., & Oliva, A. (2009). Modelling search for people in 900 scenes: A combined source model of eye guidance. Visual Cognition, 17(6–7), 945–978. https://doi.org/10.1080/13506280902834720.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Friedman, A. (1979). Framing pictures: The role of knowledge in automatized encoding and memory for gist. Journal of Experimental Psychology General, 108, 316–355.

    Article  Google Scholar 

  14. Grady, C.L., McIntosh, A.R., Rajah, M.N., & Craik, F.I.M. (1998). Neural correlates of the episodic encoding of pictures and words. Proceedings of the National Academy of Science USA, 95, 2703–8.

    Article  Google Scholar 

  15. Greene, M.R., Liu, T., & Wolfe, J.M. (2012). Reconsidering Yarbus: A failure to predict observers’ task from eye movement patterns. Vision Research, 62, 1–8. https://doi.org/10.1016/j.visres.2012.03.019.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Haji-Abolhassani, A., & Clark, J.J. (2014). An inverse Yarbus process: Predicting observers’ task from eye movement patterns. Vision Research, 103, 127–142. https://doi.org/10.1016/j.visres.2014.08.014.

    Article  PubMed  Google Scholar 

  17. Henderson, J.M., & Hollingworth, A. (1999). High-level scene perception. Annual Review of Psychology, 50, 243–271. https://doi.org/10.1146/annurev.psych.50.1.243.

    Article  PubMed  Google Scholar 

  18. Henderson, J.M., Shinkareva, S.V., Wang, J., Luke, S.G., & Olejarczyk, J. (2013). Predicting cognitive state from eye movements. PLoS ONE, 8(5), e64937. https://doi.org/10.1371/journal.pone.0064937.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., & van de Weijer, J. (2011). Eye Tracking: A comprehensive guide to methods and measures. Oxford: Oxford University Press.

    Google Scholar 

  20. Intraub, H., & Nicklos, S. (1985). Levels of processing and picture memory: The physical superiority effect. Journal of Experimental Psychology Learning, Memory, and Cognition, 11(2), 284–298. https://doi.org/10.1037/0278-7393.11.2.284.

    Article  PubMed  Google Scholar 

  21. Kanan, C., Ray, N.A., Bseiso, D.N.F., Hsiao, J.H., & Cottrell, G.W. (2014). Predicting an observer’s task using multi-fixation pattern analysis. In proceedings of the symposium on eye tracking research and applications—ETRA’14 (pp. 287–290). New York, USA: ACM Press. https://doi.org/10.1145/2578153.2578208.

  22. Kardan, O., Berman, M.G., Yourganov, G., Schmidt, J., & Henderson, J.M. (2015). Classifying mental states from eye movements during scene viewing. Journal of Experimental Psychology Human Perception and Performance, 41(6), 1502–1514. https://doi.org/10.1037/a0039673.

    Article  PubMed  Google Scholar 

  23. Kardan, O., Henderson, J.M., Yourganov, G., & Berman, M.G. (2016). Observers’ cognitive states modulate how visual inputs relate to gaze control. Journal of Experimental Psychology Human Perception and Performance, 42(9), 1429–1442. https://doi.org/10.1037/xhp0000224.

    Article  PubMed  Google Scholar 

  24. Kollmorgen, S., Nortmann, N., Schröder, S., & König, P. (2010). Influence of low-level stimulus features, task dependent factors, and spatial biases on overt visual attention. PLoS Computational Biology, 6(5), e1000791. https://doi.org/10.1371/journal.pcbi.1000791.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Laeng, B., Bloem, I.M., D’Ascenzo, S., & Tommasi, L. (2014). Scrutinizing visual images: The role of gaze in mental imagery and memory. Cognition, 131(2), 263–283. https://doi.org/10.1016/j.cognition.2014.01.003.

    Article  PubMed  Google Scholar 

  26. Lockhart, R.S., & Craik, F.I. (1990). Levels of processing: A retrospective commentary on a framework for memory research. Canadian Journal of Psychology/Revue Canadienne de Psychologie, 44(1), 87–112. https://doi.org/10.1037/h0084237.

    Article  Google Scholar 

  27. Loftus, G.R., & Mackworth, N.H. (1978). Cognitive determinants of fixation location during picture viewing. Journal of Experimental Psychology Human Perception and Performance, 4, 565–572.

    Article  Google Scholar 

  28. Mills, M., Hollingworth, A., Van der Stigchel, S., Hoffman, L., & Dodd, M.D. (2011). Examining the influence of task set on eye movements and fixations. Journal of Vision, 11(8), 17–17. https://doi.org/10.1167/11.8.17.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Navon, D. (1977). Forest before trees: The precedence of global features in visual perception. Cognitive Psychology, 9(3), 353–383. https://doi.org/10.1016/0010-0285(77)90012-3.

    Article  Google Scholar 

  30. Tatler, B.W., Wade, N.J., Kwan, H., Findlay, J.M., & Velichkovsky, B.M. (2010). Yarbus, eye movements, and vision. I Perception, 1(1), 7–27. https://doi.org/10.1068/i0382.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Yarbus, A. (1967). Eye movements and vision. New York: Plenum Press.

    Book  Google Scholar 

Download references

Funding

This work was supported by the National Science Centre in Poland under Grant 2013/11/D/HS6/04683.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Magdalena Ewa Król.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (XLSX 32 KB)

Supplementary material 2 (DOCX 17 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Król, M.E., Król, M. The right look for the job: decoding cognitive processes involved in the task from spatial eye-movement patterns. Psychological Research 84, 245–258 (2020). https://doi.org/10.1007/s00426-018-0996-5

Download citation