Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Eye movement analysis with switching hidden Markov models


Here we propose the eye movement analysis with switching hidden Markov model (EMSHMM) approach to analyzing eye movement data in cognitive tasks involving cognitive state changes. We used a switching hidden Markov model (SHMM) to capture a participant’s cognitive state transitions during the task, with eye movement patterns during each cognitive state being summarized using a regular HMM. We applied EMSHMM to a face preference decision-making task with two pre-assumed cognitive states—exploration and preference-biased periods—and we discovered two common eye movement patterns through clustering the cognitive state transitions. One pattern showed both a later transition from the exploration to the preference-biased cognitive state and a stronger tendency to look at the preferred stimulus at the end, and was associated with higher decision inference accuracy at the end; the other pattern entered the preference-biased cognitive state earlier, leading to earlier above-chance inference accuracy in a trial but lower inference accuracy at the end. This finding was not revealed by any other method. As compared with our previous HMM method, which assumes no cognitive state change (i.e., EMHMM), EMSHMM captured eye movement behavior in the task better, resulting in higher decision inference accuracy. Thus, EMSHMM reveals and provides quantitative measures of individual differences in cognitive behavior/style, making a significant impact on the use of eyetracking to study cognitive behavior across disciplines.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7


  1. 1.

    Note that there are other hierarchical HMMs for other purposes. For example, please see Camci and Chinnam (2006) and Hariri, Shirmohammadi, and Pakravan (2008) for more details.

  2. 2.

    Since we were interested in the differences in the transitions between the two sides, we forced all HMMs to use the same set of ROIs that covered each face.

  3. 3.

    Greenhouse–Geisser correction was applied whenever the assumption of sphericity was not met.


  1. Ahlstrom, U., & Friedman-Berg, F. J. (2006). Using eye movement activity as a correlate of cognitive workload. International Journal of Industrial Ergonomics, 36, 623–636.

  2. Altman, R. (2007). Mixed hidden Markov models: An extension of the hidden Markov model to the longitudinal data setting. Journal of the American Statistical Association, 102, 201–210.

  3. Andrews, T. J., & Coppola, D. M. (1999). Idiosyncratic characteristics of saccadic eye movements when viewing different visual environments. Vision Research, 39, 2947–2953. https://doi.org/10.1016/S0042-6989(99)00019-X

  4. Bee, N., Prendinger, H., Nakasone, A., André, E., & Ishizuka, M. (2006). Auto select: What you want is what you get: Real-time processing of visual attention and affect. In International Tutorial and Research Workshop on Perception and Interactive Technologies for Speech-Based Systems (pp. 40–52). Heidelberg, Germany: Springer.

  5. Camci, F., & Chinnam, R. B. (2006). Hierarchical HMMs for autonomous diagnostics and prognostics. In Proceedings of the 2006 IEEE International Joint Conference on Neural Networks (pp. 2445–2452). Piscataway, NJ: IEEE Press.

  6. Castelhano, M. S., & Henderson, J. M. (2008). Stable individual differences across images in human saccadic eye movements. Canadian Journal of Experimental Psychology, 62, 1–14. https://doi.org/10.1037/1196-1961.62.1.1

  7. Chan, C. Y. H., Chan, A. B., Lee, T. M. C., & Hsiao, J. H. (2018). Eye movement patterns in face recognition are associated with cognitive decline in older adults. Psychonomic Bulletin & Review, 25, 2200–2207.

  8. Chandon, P., Hutchinson, J. W., Bradlow, E. T., & Young, S. H. (2009). Does in-store marketing work? Effects of the number and position of shelf facings on brand attention and evaluation at the point of purchase. Journal of Marketing, 73(6), 1–17. https://doi.org/10.1509/jmkg.73.6.1

  9. Chuk, T., Chan, A. B., & Hsiao, J. H. (2014). Understanding eye movements in face recognition using hidden Markov models. Journal of Vision, 14(11), 8:1–14. https://doi.org/10.1167/14.11.8

  10. Chuk, T., Chan, A. B., & Hsiao, J. H. (2017). Is having similar eye movement patterns during face learning and recognition beneficial for recognition performance? Evidence from hidden Markov modeling. Vision Research, 141, 204–216.

  11. Chuk, T., Crookes, K., Hayward, W. G., Chan, A. B., & Hsiao, J. H. (2017). Hidden Markov model analysis reveals the advantage of analytic eye movement patterns in face recognition across cultures. Cognition, 169, 102–117.

  12. Coutrot, A., Hsiao, J. H., & Chan, A. B. (2018). Scanpath modeling and classification with hidden Markov models. Behavior Research Methods, 50, 362–379. https://doi.org/10.3758/s13428-017-0876-8

  13. Coviello, E., Chan, A. B., & Lanckriet, G. R. (2014). Clustering hidden Markov models with variational HEM. Journal of Machine Learning Research, 15, 697–747.

  14. Ferrari, J. R., & Dovidio, J. F. (2000). Examining behavioral proceses in indecision: Decisional procrastination and decision-making style. Journal of Research in Personality, 34, 127–137.

  15. Germeijs, V., & De Boeck, P. (2002). A measurement scale for indecisiveness and its relationship to career indecision and other types of indecision. European Journal of Psychological Assessment, 18, 113–122.

  16. Glaholt, M. G., Wu, M. C., & Reingold, E. M. (2009). Predicting preference from fixations. PsychNology Journal, 7, 141–158.

  17. Haji-Abolhassani, A., & Clark, J. J. (2014). An inverse Yarbus process: Predicting observers’ task from eye movement patterns. Vision Research, 103, 127–142.

  18. Hariri, B., Shirmohammadi, S., & Pakravan, M. R. (2008). A hierarchical HMM model for online gaming traffic patterns. In Proceedings of 2008 IEEE Instrumentation and Measurement Technology Conference (pp. 2195–2200). Piscataway, NJ: IEEE Press.

  19. Hayhoe, M., & Ballard, D. (2014). Modeling task control of eye movements. Current Biology, 24, R622–R628.

  20. Henderson, J. M., Shinkareva, S. V., Wang, J., Luke, S. G., & Olejarczyk, J. (2013). Predicting cognitive state from eye movements. PLoS ONE, 8, e64937. https://doi.org/10.1371/journal.pone.0064937

  21. Kanan, C., Bseiso, D. N., Ray, N. A., Hsiao, J. H., & Cottrell, G. W. (2015). Humans have idiosyncratic and task-specific scanpaths for judging faces. Vision Research, 108, 67–76.

  22. Lemonnier, S., Brémond, R., & Baccino, T. (2014). Discriminating cognitive processes with eye movements in a decision-making driving task. Journal of Eye Movement Research, 7(4), 3:1–14.

  23. Liechty, J., Pieters, R., & Wedel, M. (2003). Global and local covert visual attention: Evidence from a Bayesian hidden Markov model. Psychometrika, 68, 519–541.

  24. MacQueen, J. (1967). Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability (pp. 281–297). Berkeley, CA: University of California Press.

  25. Poynter, W., Barber, M., Inman, J., & Wiggins, C. (2013). Individuals exhibit idiosyncratic eye-movement behavior profiles across tasks. Vision Research, 89, 32–38.

  26. Rassin, E., Muris, P, Booster, E., & Kolsloot, I. (2008). Indecisiveness and informational tunnel vision. Personality and Individual Differences, 45, 96–102.

  27. Risko, E. F., Anderson, N. C., Lanthier, S., & Kingstone, A. (2012). Curious eyes: Individual differences in personality predict eye movement behavior in scene-viewing. Cognition, 122, 86–90.

  28. Sekiguchi, T. (2011). Individual differences in face memory and eye fixation patterns during face learning. Acta Psychologica, 137, 1–9. https://doi.org/10.1016/j.actpsy.2011.01.014

  29. Shimojo, S., Simion, C., & Changizi, M. A. (2011). Gaze and preference-orienting behavior as a somatic precursor of preference decision. In R. B. Adams (Ed.), The science of social vision (pp. 151–163). Oxford, UK: Oxford University Press.

  30. Shimojo, S., Simion, C., Shimojo, E., & Scheier, C. (2003). Gaze bias both reflects and influences preference. Nature Neuroscience, 6, 1317–1322.

  31. Simion, C., & Shimojo, S. (2007). Interrupting the cascade: Orienting contributes to decision making even in the absence of visual stimulation. Perception & Psychophysics, 69, 591–595. https://doi.org/10.3758/BF03193916

  32. Simola, J., Salojarvi, J., & Kojo, I. (2008). Using hidden Markov model to uncover processing states from eye movements in information search tasks. Cognitive Systems Research, 9, 237–251.

  33. Sutterlin, B., Brunner, T. A., & Opwis, K. (2008). Eye-tracking the cancellation and focus model for preference judgments. Journal of Experimental Social Psychology, 44, 904–911.

  34. Van der Lans, R., Pieters, R., & Wedel, M. (2008). Eye-movement analysis of search effectiveness. Journal of the American Statistical Association, 103, 452–461.

  35. Wu, D. W. L., Bischof, W. F., Anderson, N. C., Jakobsen, T., & Kingstone, A. (2014). The influence of personality on social attention. Personality and Individual Differences, 60, 25–29.

  36. Yeung, P. Y., Wong, L. L., Chan, C. C., Leung, J. L., & Yung, C. Y. (2014). A validation study of the Hong Kong version of Montreal Cognitive Assessment (HK-MoCA) in Chinese older adults in Hong Kong. Hong Kong Medical Journal, 20, 504–510.

  37. Yi, W., & Ballard, D. H. (2009). Recognizing behavior in hand–eye coordination patterns. International Journal of Humanoid Robotics, 6, 337–359.

  38. Zhang, J., Chan, A. B., Lau, E. Y. Y., & Hsiao, J. H. (2019). Individuals with insomnia misrecognize angry faces as fearful faces while missing the eyes: An eye-tracking study. Sleep, 42, zsy220. https://doi.org/10.1093/sleep/zsy220

Download references


We are grateful to the Research Grant Council of Hong Kong (project 17609117 to J.H.H. and CityU 110513 to A.B.C.) and to JST.CREST (to S.S.). A.B.C. and J.H.H. contributed equally to this article. We thank the editor and two anonymous reviewers for the helpful comments.

Open Practices Statement

The code (Matlab Toolbox EMSHMM) and data of the study are available to the research community for noncommercial use at http://visal.cs.cityu.edu.hk/research/emshmm/. The experiment reported here was not preregistered.

Author information

Correspondence to Antoni B. Chan or Janet H. Hsiao.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chuk, T., Chan, A.B., Shimojo, S. et al. Eye movement analysis with switching hidden Markov models. Behav Res (2019). https://doi.org/10.3758/s13428-019-01298-y

Download citation


  • Hidden Markov model
  • Eye movement
  • Preference decision making