What Users Do: The Eyes Have It

  • Paul Thomas
  • Falk Scholer
  • Alistair Moffat
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8281)


Search engine result pages – the ten blue links – are a staple of document retrieval services. The usual presumption is that users read these one-by-one from the top, making judgments about the usefulness of documents based on the snippets presented, accessing the underlying document when a snippet seems attractive, and then moving on to the next snippet. In this paper we re-examine this assumption, and present the results of a user experiment in which gaze-tracking is combined with click analysis. We conclude that in very general terms, users do indeed read from the top, but that at a detailed level there are complex behaviors evident, suggesting that a more sophisticated model of user interaction might be appropriate. In particular, we argue that users retain a number of snippets in an “active band” that shifts down the result page, and that reading and clicking activity tends to takes place within the band in a manner that is not strictly sequential.


Retrieval evaluation user behavior user model 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aula, A., Majaranta, P., Räihä, K.-J.: Eye-tracking reveals the personal styles for search result evaluation. In: Costabile, M.F., Paternó, F. (eds.) INTERACT 2005. LNCS, vol. 3585, pp. 1058–1061. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  2. 2.
    Borlund, P.: Experimental components for the evaluation of interactive information retrieval systems. J. Documentation 56(1), 71–90 (2000)CrossRefGoogle Scholar
  3. 3.
    Chapelle, O., Zhang, Y.: A dynamic Bayesian network click model for web search ranking. In: Proc. WWW, Madrid, Spain, pp. 1–10 (2009)Google Scholar
  4. 4.
    Dumais, S., Buscher, G., Cutrell, E.: Individual differences in gaze patterns for web search. In: Proc. IIiX, London, England, pp. 185–194 (2010)Google Scholar
  5. 5.
    Granka, L.A., Joachims, T., Gay, G.: Eye-tracking analysis of user behavior in WWW search. In: Proc. SIGIR, Sheffield, England, pp. 478–479 (2004)Google Scholar
  6. 6.
    Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Information Systems 20(4), 422–446 (2002)CrossRefGoogle Scholar
  7. 7.
    Joachims, T., Granka, L., Pan, B., Hembrooke, H., Gay, G.: Accurately interpreting clickthrough data as implicit feedback. In: Proc. SIGIR, Salvador, Brazil, pp. 154–161 (2005)Google Scholar
  8. 8.
    Jones, T., Hawking, D., Thomas, P., Sankaranarayana, R.: Relative effect of spam and irrelevant documents on user interaction with search engines. In: Proc. CIKM, Glasgow, Scotland, pp. 2113–2116 (2011)Google Scholar
  9. 9.
    Kelly, D., Harper, D.J., Landau, B.: Questionnaire mode effects in interactive information retrieval experiments. Information Processing & Management 44(1), 122–141 (2008)CrossRefGoogle Scholar
  10. 10.
    Moffat, A., Thomas, P., Scholer, F.: Users versus models: What observation tells us about effectiveness metrics. In: Proc. CIKM, San Francisco, California (to appear, 2013)Google Scholar
  11. 11.
    Moffat, A., Zobel, J.: Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Information Systems 27(1), 2:1–2:27 (2008)Google Scholar
  12. 12.
    Robertson, S.: A new interpretation of average precision. In: Proc. SIGIR, Singapore, pp. 689–690 (2008)Google Scholar
  13. 13.
    Smucker, M.D., Clarke, C.L.A.: Time-based calibration of effectiveness measures. In: Proc. SIGIR, Portland, Oregon, pp. 95–104 (2012)Google Scholar
  14. 14.
    Thomas, P., Scholer, F., Moffat, A.: Fading away: Dilution and user behaviour. In: Proc. 3rd Europ. Wrkshp. HCI and IR, Dublin, Ireland, pp. 3–6 (2013)Google Scholar
  15. 15.
    Voorhees, E.M.: Variations in relevance judgements and the measurement of retrieval effectiveness. Information Processing & Management 36(5), 697–716 (2000)CrossRefGoogle Scholar
  16. 16.
    Wu, W.C., Kelly, D., Edwards, A., Arguello, J.: Grannies, tanning beds, tattoos and NASCAR: Evaluation of search tasks with varying levels of cognitive complexity. In: Proc. IIiX, Nijmegen, The Netherlands, pp. 254–257 (2012)Google Scholar
  17. 17.
    Yilmaz, E., Shokouhi, M., Craswell, N., Robertson, S.: Expected browsing utility for web search evaluation. In: Proc. CIKM, Toronto, Canada, pp. 1561–1564 (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Paul Thomas
    • 1
  • Falk Scholer
    • 2
  • Alistair Moffat
    • 3
  1. 1.CSIRO and The Australian National UniversityAustralia
  2. 2.RMIT UniversityAustralia
  3. 3.The University of MelbourneAustralia

Personalised recommendations