Skip to main content
SpringerLink
Log in

Identification of human implicit visual search intention based on eye movement and pupillary analysis

  • Original Paper
  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

We propose a novel approach for the identification of human implicit visual search intention based on eye movement patterns and pupillary analysis, in general, as well as pupil size, gradient of pupil size variation, fixation length and fixation count corresponding to areas of interest, and fixation count corresponding to non-areas of interest, in particular. The proposed model identifies human implicit visual search intention as task-free visual browsing or task-oriented visual search. Task-oriented visual search is further identified as task-oriented visual search intent generation, task-oriented visual search intent maintenance, or task-oriented visual search intent disappearance. During a visual search, measurement of the pupillary response is greatly influenced by external factors such the intensity and size of the visual stimulus. To alleviate the effects of external factors, we propose a robust baseline model that can accurately measure the pupillary response. Graphical representation of the measured parameter values shows significant differences among the different intent conditions, which can then be used as features for identification. By using the eye movement patterns and pupillary analysis, we can detect the transitions between different implicit intentions—task-free visual browsing intent to task-oriented visual search intent and task-oriented visual search intent maintenance to task-oriented visual search intent disappearance—using a hierarchical support vector machine. In the proposed model, the hierarchical support vector machine is able to identify the transitions between different intent conditions with greater than 90 % accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  • Ahsan, M.R., Ibrahimy, M.I., O-Khalifa, O.: EMG signal classification for human computer interaction: a review. Eur. J. Sci. Res. 33, 480–501 (2009)

    Google Scholar 

  • Beatty, J.: Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychol. Bull. 91, 276–292 (1982)

    Article  Google Scholar 

  • Bradshaw, J.L.: Pupil size and problem solving. Quaterly J. Exp. Psychol. 20, 116–122 (1968)

    Article  Google Scholar 

  • Carnel, E., Crawford, S., Chen, H.: Browsing in hypertext: a cognitive study. IEEE Transact. Syst. Man Cybern. 22, 865–884 (1992)

    Article  Google Scholar 

  • Chen, Z., Lin, F., Liu, H., Ma, W.Y., Wenyin, L.: User intention modelling in web applications using data mining. Internet Web Inf. Syst. J, Kluwer Academic Publishers 5, 181–192 (2002)

    Google Scholar 

  • Claudio, M. P., Laura, W. R., Thom, C., Stanley, K., Mario, A.: The Pupil Dilation Response to Visual Detection, Human Vision and Electronic Imaging XIII / SPIE-IS &T, vol. 6806, pp. 68060T–68060T-11 (2008)

  • Duchowski, A.: Eye Tracking Methodology: Theory and Practice. Springer, Heidelberg (2007)

  • Dumais, S., Chen, H.: Hierarchical classification of Web content, presented at the Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 256–263. Greece, Athens (2000)

  • Erol, A., Bebis, G., Nicolescu, M., Boyle, R. D., Twombly, X.: Vision-Based Hand Pose Estimation: A Review, Computer Vision and Image Understanding, vol. 108, pp. 52–73. (2007)

  • Eye tracking system of Tobii technology, http://www.tobii.com/ (2007)

  • Ferreira, A., Celeste, W. C., Cheein, F. A., Bastos-Filho, T. F., Sarcinelli-Filho, M., Carelli, R.: Human-machine interfaces based on EMG and EEG applied to robotic systems, J. NeuroEng. Rehabil. pp. 5–10 (2008)

  • Fleetwood, M.D., Byrne, M.D.: Modeling the visual search of displays: a revised ACT-R model of icon search based on eye-tracking data. Hum. Comput. Interact. 21, 153–197 (2006)

    Article  Google Scholar 

  • Gareze, L., Harris, J.M., Barenghi, C.F., Tadmor, Y.: Characterising patterns of eye movements in natural images and visual scanning. J. Mod. Opt. 55, 533–555 (2008)

    Article  Google Scholar 

  • Goldwater, B.: Psychological significance of pupillary movements. Psychol. Bull. 77, 340–355 (1972)

    Article  Google Scholar 

  • Goodrich, M.A., Schultz, A.C.: Human-robot interaction: a survey. Found. Trends Hum.-Comput. Interact 1(3), 203–275 (2007)

    Article  MATH  Google Scholar 

  • Granholm, E., Asarnow, R.F., Sarkin, A.J., Dykes, K.L.: Pupillary responses index cognitive resource limitations. Psychophysiology 33, 457–461 (1996)

    Article  Google Scholar 

  • Granka, L.A., T. Joachims, et al.: Eye-tracking analysis of user behavior in WWW search. Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval. Sheffield, United Kingdom, ACM, pp. 478–479 (2004)

  • Hakerem, G., Sutton, S.: Pupillary response at visual threshold. Nature 212, 485–486 (1966)

    Article  Google Scholar 

  • Hakerem, G., Sutton, S., Zubin, J.: Parameters of the Pupillary Response, in Meetings of the Psychonomic Society, pp. 1–10 (1960)

  • Halverson, T., Hornof, A.J.: A computational model of “Active Vision” for visual search in human-computer interaction. Hum.-Comput. Interact. 26, 285–314 (2011)

    Article  Google Scholar 

  • Hess, E.H.: Attitude and pupil size. Sci. Am. 212, 46–54 (1965)

    Article  Google Scholar 

  • Hess, E.H., Polt, J.M.: Pupil size in relation to mental activity during problem-solving. Science 143, 1190–1192 (1964)

    Article  Google Scholar 

  • Hoecks, B., Levelt, W.: Pupillary dilation as a measure of attention: a quatitative system analysis. Behav. Res. Methods Instrum. Comput. 25, 16–26 (1993)

    Article  Google Scholar 

  • Holmqvist, K., Nyström, M., et al. Eye Tracking: A Comprehensive Guide to Methods and Measures, Oxford University Press, Oxford (2011)

  • Iqbal, S. T., Zheng, X. S., Bailey, B. P.: Task-evoked pupillary response to mental worload in human-computer interaction, Human factors in, computing systems, pp. 1477–1480 (2004)

  • Jaimes, A., Sebe, N.: Multimodal human-computer interaction: a survey. Comput. Vis. Image Underst. 108, 116–134 (2007)

    Article  Google Scholar 

  • Jang, Y.-M., Lee, S., Mallipeddi, R., Kwak, H.-W., Lee, M.: Recognition of human’s implicit intention based on an eyeball movement pattern analysis. Lect. Notes Comput. Sci. 7062, 138–145 (2011)

    Article  Google Scholar 

  • Jang, Y.-M., Mallipeddi, R., Lee, S., Kwak, H.-W., Lee, M.: Human implicit intent transition detection based on pupillary analysis. IEEE International Conference on Neural Networks (IJCNN), pp. 1–7, Brisbane, Australia (2012)

  • Janisse, M.P.: Pupillometry: The Psychology of the Pupillary Response. Hemisphere Publishing Co., Washington, D. C. (1977)

    Google Scholar 

  • Jansen, B.J., Booth, D.L., Spink, A.: Determining the informational, navigational, and transactional intent of Web queries. Inf. Process. Manag. 44, 1251–1266 (2008)

    Article  Google Scholar 

  • Just, M.A., Carpenter, P.A.: Eye fixations and cognitive processes. Cogn. Psychol. 8, 441–480 (1976)

    Article  Google Scholar 

  • Kahneman, D., Peavler, W.S.: Incentive effects and pupillary changes in association learning. J. Exp. Psychol. 79, 312–318 (1969)

    Article  Google Scholar 

  • Kahneman, D., Beatty, J.: Pupil diameter and load on memory. Science 154, 1583–1585 (1966)

    Article  Google Scholar 

  • Kecman, V.: Learning and Soft Computing. MIT Press, Cambridge (2001)

    MATH  Google Scholar 

  • Kendall, E., A.: An Introduction to Numerical Analysis, 2nd edn. Wiley, New York (1989)

  • Klingner, J., Kumar, R., Hanrahan, P.: Measuring the task-evoked pupillary response with a remote eye tracker. Proceedings of the: symposium on Eye tracking research & applications, pp. 69–72. Savannah, Georgia, ACM (2008)

  • Land, M.F., Tatler, B.W.: Looking and Acting: Vision and Eye Movements in Natural Behaviour. Oxford University Press, Oxford (2009)

    Google Scholar 

  • Libby, W.L., Lacey, B.C., Lacey, J.I.: Pupillary and cardiac activity during visual attention. Psychophysiology 10, 270–294 (1973)

    Article  Google Scholar 

  • Liu, J., Cole, M. J., Liu, C., Bierig, R., Gwizdka, J. Belkin, N., J., Zhang, J., Zhang, X.: Search behaviors in different task types. Proceedings of the 10th annual joint conference on digital libraries. pp. 69–78. Gold Coast, Queensland, Australia, ACM (2010)

  • Lorigo, L., Haridasan, M.: Eye tracking and online search: lessons learned and challenges ahead. J. Am. Soc. Inf. Sci. Technol. 59(7), 1041–1052 (2008)

    Article  Google Scholar 

  • Lowenstein, O., Lowenfeld, I. E. : The Pupil, vol. 3. Academic Press, New York (1962)

  • Majaranta, P., Räihä K.-J. (2002). Twenty years of eye typing: systems and design issues. Proceedings of the: symposium on Eye tracking research & applications, pp. 15–22. New Orleans, Louisiana, ACM (2002)

  • Majaranta, P., Aoki, H. et al.: Gaze Interaction and Applications of Eye Tracking, Advances in Assistive Technologies (2011)

  • Marchionini, G.: Information Seeking in Electronic Environments. Cambridge University Press, Cambridge (1995)

    Book  Google Scholar 

  • Marshall, S. P., Pleydell-Pearce, C., Dickson, B. T.: Integrating psychophysiological measures of cognitive workload and eye movements to detect strategy shifts, International Conference on System Sciences, pp. 1–6, Hawaii (2003)

  • Oliveira, F.T.P., Aula, A., Russell, D.M.: Discriminating the Relevance of Web Search Results with Measures of Pupil Size, in Human Factors in Computing Systems, pp. 2209–2212. Boston, USA (2009)

  • Oyekoya, O., Stentiford, F.: Eye tracking: a new interface for visual exploration. BT Technol. J. 24(3), 57–66 (2006)

    Article  Google Scholar 

  • Paivio, A., Simpson, H.M.: The effect of word abstractness and pleasantness on pupil size during an imagery task. Psychon. Sci. 5, 55–56 (1966)

    Article  Google Scholar 

  • Park, S.-M., Ko, K.-E., Park, J., Sim, K.-B.: A study on hybrid model of HMMs and GMMs for mirror neuron system modeling using EEG signals, IEEE International Conference on Fuzzy Systems, pp. 2752–2755 (2011)

  • Pomplun, M., Sunkara, S.: Pupil Dilation as an Indicator of Cognitive Workload in Human-Computer Interaction. International Conference on Human-Computer Interaction, pp. 1–5, Crete, Greece (2003)

  • Poole, A., Ball, L.J.: Eye Tracking in Human-Computer Interaction and Usability Research: Current Status and Future Prospects. Encyclopedia of Human-Computer Interaction. C. Ghaoui. Pennsylvania, Idea Group, Inc. (2005)

  • Premack, D., Woodruff, G.: Does the chimpanzee have a theory of mind? Behav. Brain Sci. 1, 515–526 (1978)

    Article  Google Scholar 

  • Rani, P., Sarkar, N.: Anxiety-based affective communication for implicit human-machine interaction. Adv. Eng. Inform. 21(3), 323–334 (2007)

    Article  Google Scholar 

  • Ratwani, R.M., Trafton, J.G.: A real-time eye tracking system for predicting and preventing postcompletion errors. Hum.-Comput. Interact. 26, 205–245 (2011)

    Google Scholar 

  • Rayner, K., Pollatsek, A.: The Psychology of Reading. Prentice Hall, Englewood Cliffs (1989)

    Google Scholar 

  • Rose, D. E., Levinson, D.: Understanding User Goals in Web Search, in Proceedings of the 13th International Conference on, World Wide Web, pp. 13–19 (2004)

  • Salvucci, D.D.: Inferring intent in eye-based interfaces: tracing eye movements with process models, pp. 254–261. Proceedings of the SIGCHI conference on Human Factors in Computing Systems. Pittsburgh, Pennsylvania, United States, ACM (1999)

  • Salvucci, D. D., J. H. Goldberg (2000). Identifying fixations and saccades in eye-tracking protocols. Proceedings of the: symposium on Eye tracking research & applications, pp. 71–78. Palm Beach Gardens, Florida, United States, ACM (2000)

  • Salvucci, D.D., Anderson, J.R.: Automated eye-movement protocol analysis. Hum.-Comput. Interact. 16, 39–86 (2001)

    Article  Google Scholar 

  • Schwarz, U., Schmuckle, T.: Cognitive eyes. Schweiz Arch. Neurol. Psychiatr. 153, 175–179 (2002)

    Google Scholar 

  • Schütz, A.C., Braun, D.I., Gegenfurtner, K.R.: Eye movements and perception: a selective review. J. Vis. 11, 1–30 (2011)

    Google Scholar 

  • Selker, T.: Visual attentive interfaces. BT Technol. J. 22(4), 146–150 (2004)

    Article  Google Scholar 

  • Simpson, H.M., Paivio, A.: Changes in pupil size during an imagery task without motor response involvement. Psychon. Sci. 5, 405–406 (1966)

    Article  Google Scholar 

  • Sodhi, M., Reimer, B., et al.: On-road driver eye movement tracking using head-mounted devices. Proceedings of the: symposium on eye tracking research \(\backslash \) & applications, pp. 61–68. New Orleans, Louisiana, ACM (2002)

  • Spyrou, T., Darzentas, J.: Intention Modelling: Approximating Computer User Intentions for Detection and Prediction of Intrusions. Chapman & Hall, London (1996)

    Google Scholar 

  • Steinhauer, S.R., Boller, F., Zubin, J., Pearlman, S.: Pupillary dilation to emotional visual stimuli revisited. Psychophysiology 20, 472 (1983)

    Google Scholar 

  • Wegner, D.M.: The Illusion of Conscious Will. MIT Press, Cambridge (2002)

    Google Scholar 

  • Wong, F., Park, K.-H., Kim, D.-J., Jung, J.-W., Bien, Z.: Intention reading towards engineering applications for the elderly and people with disabilities. Int. J. ARM 7, 3–15 (2006)

    Google Scholar 

  • Wright, P., Kahneman, D.: Evidence for alternative strategies of sentence retention. Quaterly J. Exp. Psychol. 23, 197–213 (1971)

    Article  Google Scholar 

  • Yarbus, A.L.: Eye Movements and Vision. Plenum Press, New York (1967)

    Book  Google Scholar 

  • Youn, S.-J., Oh, K.-W.: Intention recognition using graph representation. World Acad. Sci. Eng. Technol. 25, 13–18 (2007)

    Google Scholar 

  • Young, F.A., Biersdorf, W.R.: Pupillary contraction and dilation in light and darkness. J. Comp. Physiol. Psychol. 47, 264–268 (1954)

    Article  Google Scholar 

  • Zhai, S., Morimoto, C., Ihde, S.: Manual and gaze input cascaded (MAGIC) pointing, Proceedings ACM, pp. 246–253 (1999)

Download references

Acknowledgments

This research was supported by the Converging Research Center Program funded by the Ministry of Education, Science and Technology (2012K001342) (50 %) and also by the Original Technology Research Program for Brain Science through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2012-0005794) (50 %).

Author information

Authors and Affiliations

  1. School of Electronics Engineering, Kyungpook National University, 1370 Sankyuk-Dong, Puk-Gu, Taegu, 702-701, South Korea

    Young-Min Jang, Rammohan Mallipeddi & Minho Lee

Authors
  1. Young-Min Jang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rammohan Mallipeddi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Minho Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Minho Lee.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jang, YM., Mallipeddi, R. & Lee, M. Identification of human implicit visual search intention based on eye movement and pupillary analysis. User Model User-Adap Inter 24, 315–344 (2014). https://doi.org/10.1007/s11257-013-9142-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-013-9142-7

Keywords

Search

Navigation