The aim of the study was not only to demonstrate whether eye-movement-based task decoding was possible but also to investigate whether eye-movement patterns can be used to identify cognitive processes behind the tasks. We compared eye-movement patterns elicited under different task conditions, with tasks differing systematically with regard to the types of cognitive processes involved in solving them. We used four tasks, differing along two dimensions: spatial (global vs. local) processing (Navon, Cognit Psychol, 9(3):353–383 1977) and semantic (deep vs. shallow) processing (Craik and Lockhart, J Verbal Learn Verbal Behav, 11(6):671–684 1972). We used eye-movement patterns obtained from two time periods: fixation cross preceding the target stimulus and the target stimulus. We found significant effects of both spatial and semantic processing, but in case of the latter, the effect might be an artefact of insufficient task control. We found above chance task classification accuracy for both time periods: 51.4% for the period of stimulus presentation and 34.8% for the period of fixation cross presentation. Therefore, we show that task can be to some extent decoded from the preparatory eye-movements before the stimulus is displayed. This suggests that anticipatory eye-movements reflect the visual scanning strategy employed for the task at hand. Finally, this study also demonstrates that decoding is possible even from very scant eye-movement data similar to Coco and Keller, J Vis 14(3):11–11 (2014). This means that task decoding is not limited to tasks that naturally take longer to perform and yield multi-second eye-movement recordings.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Altmann, G.T.M. (2004). Language-mediated eye movements in the absence of a visual world: The ‘blank screen paradigm’. Cognition, 93(2), B79–B87. https://doi.org/10.1016/j.cognition.2004.02.005.
Bernstein, L.J., Beig, S., Siegenthaler, A.L., & Grady, C.L. (2002). The effect of encoding strategy on the neural correlates of memory for faces. Neuropsychologia, 40(1), 86–98. https://doi.org/10.1016/S0028-3932(01)00070-7.
Betz, T., Kietzmann, T.C., Wilming, N., & König, P. (2010). Investigating task-dependent top-down effects on overt visual attention. Journal of Vision, 10(3), 1–14. https://doi.org/10.1167/10.3.15.
Boisvert, J.F.G., & Bruce, N.D.B. (2016). Predicting task from eye movements: On the importance of spatial distribution, dynamics, and image features. Neurocomputing, 207, 653–668. https://doi.org/10.1016/j.neucom.2016.05.047.
Borji, A., & Itti, L. (2014). Defending Yarbus: Eye movements reveal observers’ task. Journal of Vision, 14(3:29), 1–22. https://doi.org/10.1167/14.3.29.
Bower, G.H., & Karlin, M.B. (1974). Depth of processing pictures of faces and recognition memory. Journal of Experimental Psychology, 103(4), 751–757. https://doi.org/10.1037/h0037190.
Castelhano, M.S., Mack, M.L., & Henderson, J.M. (2009). Viewing task influences eye movement control during active scene perception. Journal of Vision, 9(3), 6–6. https://doi.org/10.1167/9.3.6.
Coco, M.I., & Keller, F. (2014). Classification of visual and linguistic tasks using eye-movement features. Journal of Vision, 14(3), 11–11. https://doi.org/10.1167/14.3.11.
Cowen, L., Ball, L.J., & Delin, J. (2002). An eye movement analysis of web page usability. In People and computers XVI—memorable yet invisible (pp. 317–335). London: Springer. https://doi.org/10.1007/978-1-4471-0105-5_19.
Craik, F.I.M., & Lockhart, R.S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11(6), 671–684. https://doi.org/10.1016/S0022-5371(72)80001-X.
DeAngelus, M., & Pelz, J.B. (2009). Top-down control of eye movements: Yarbus revisited. Visual Cognition, 17(6–7), 790–811. https://doi.org/10.1080/13506280902793843.
Ehinger, K.A., Hidalgo-Sotelo, B., Torralba, A., & Oliva, A. (2009). Modelling search for people in 900 scenes: A combined source model of eye guidance. Visual Cognition, 17(6–7), 945–978. https://doi.org/10.1080/13506280902834720.
Friedman, A. (1979). Framing pictures: The role of knowledge in automatized encoding and memory for gist. Journal of Experimental Psychology General, 108, 316–355.
Grady, C.L., McIntosh, A.R., Rajah, M.N., & Craik, F.I.M. (1998). Neural correlates of the episodic encoding of pictures and words. Proceedings of the National Academy of Science USA, 95, 2703–8.
Greene, M.R., Liu, T., & Wolfe, J.M. (2012). Reconsidering Yarbus: A failure to predict observers’ task from eye movement patterns. Vision Research, 62, 1–8. https://doi.org/10.1016/j.visres.2012.03.019.
Haji-Abolhassani, A., & Clark, J.J. (2014). An inverse Yarbus process: Predicting observers’ task from eye movement patterns. Vision Research, 103, 127–142. https://doi.org/10.1016/j.visres.2014.08.014.
Henderson, J.M., & Hollingworth, A. (1999). High-level scene perception. Annual Review of Psychology, 50, 243–271. https://doi.org/10.1146/annurev.psych.50.1.243.
Henderson, J.M., Shinkareva, S.V., Wang, J., Luke, S.G., & Olejarczyk, J. (2013). Predicting cognitive state from eye movements. PLoS ONE, 8(5), e64937. https://doi.org/10.1371/journal.pone.0064937.
Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., & van de Weijer, J. (2011). Eye Tracking: A comprehensive guide to methods and measures. Oxford: Oxford University Press.
Intraub, H., & Nicklos, S. (1985). Levels of processing and picture memory: The physical superiority effect. Journal of Experimental Psychology Learning, Memory, and Cognition, 11(2), 284–298. https://doi.org/10.1037/0278-73184.108.40.2064.
Kanan, C., Ray, N.A., Bseiso, D.N.F., Hsiao, J.H., & Cottrell, G.W. (2014). Predicting an observer’s task using multi-fixation pattern analysis. In proceedings of the symposium on eye tracking research and applications—ETRA’14 (pp. 287–290). New York, USA: ACM Press. https://doi.org/10.1145/2578153.2578208.
Kardan, O., Berman, M.G., Yourganov, G., Schmidt, J., & Henderson, J.M. (2015). Classifying mental states from eye movements during scene viewing. Journal of Experimental Psychology Human Perception and Performance, 41(6), 1502–1514. https://doi.org/10.1037/a0039673.
Kardan, O., Henderson, J.M., Yourganov, G., & Berman, M.G. (2016). Observers’ cognitive states modulate how visual inputs relate to gaze control. Journal of Experimental Psychology Human Perception and Performance, 42(9), 1429–1442. https://doi.org/10.1037/xhp0000224.
Kollmorgen, S., Nortmann, N., Schröder, S., & König, P. (2010). Influence of low-level stimulus features, task dependent factors, and spatial biases on overt visual attention. PLoS Computational Biology, 6(5), e1000791. https://doi.org/10.1371/journal.pcbi.1000791.
Laeng, B., Bloem, I.M., D’Ascenzo, S., & Tommasi, L. (2014). Scrutinizing visual images: The role of gaze in mental imagery and memory. Cognition, 131(2), 263–283. https://doi.org/10.1016/j.cognition.2014.01.003.
Lockhart, R.S., & Craik, F.I. (1990). Levels of processing: A retrospective commentary on a framework for memory research. Canadian Journal of Psychology/Revue Canadienne de Psychologie, 44(1), 87–112. https://doi.org/10.1037/h0084237.
Loftus, G.R., & Mackworth, N.H. (1978). Cognitive determinants of fixation location during picture viewing. Journal of Experimental Psychology Human Perception and Performance, 4, 565–572.
Mills, M., Hollingworth, A., Van der Stigchel, S., Hoffman, L., & Dodd, M.D. (2011). Examining the influence of task set on eye movements and fixations. Journal of Vision, 11(8), 17–17. https://doi.org/10.1167/11.8.17.
Navon, D. (1977). Forest before trees: The precedence of global features in visual perception. Cognitive Psychology, 9(3), 353–383. https://doi.org/10.1016/0010-0285(77)90012-3.
Tatler, B.W., Wade, N.J., Kwan, H., Findlay, J.M., & Velichkovsky, B.M. (2010). Yarbus, eye movements, and vision. I Perception, 1(1), 7–27. https://doi.org/10.1068/i0382.
Yarbus, A. (1967). Eye movements and vision. New York: Plenum Press.
This work was supported by the National Science Centre in Poland under Grant 2013/11/D/HS6/04683.
About this article
Cite this article
Król, M.E., Król, M. The right look for the job: decoding cognitive processes involved in the task from spatial eye-movement patterns. Psychological Research 84, 245–258 (2020). https://doi.org/10.1007/s00426-018-0996-5