Skip to main content
SpringerLink
Log in
Book cover

European Conference on Information Retrieval

ECIR 2018: Advances in Information Retrieval pp 210–222Cite as

Information Scent, Searching and Stopping

Modelling SERP Level Stopping Behaviour

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 10772))

Abstract

Current models and measures of the Interactive Information Retrieval (IIR) process typically assume that a searcher will always examine the first snippet in a given Search Engine Results Page (SERP), and then with some probability or cutoff, he or she will stop examining snippets and/or documents in the ranked list (snippet level stopping). Prior work has however shown that searchers will form an initial impression of the SERP, and will often abandon a page without clicking on or inspecting in detail any snippets or documents. That is, the information scent affects their decision to continue. In this work, we examine whether considering the information scent of a page leads to better predictions of stopping behaviour. In a simulated analysis, grounded with data from a prior user study, we show that introducing a SERP level stopping strategy can improve the performance attained by simulated users, resulting in an increase in gain across most snippet level stopping strategies. When compared to actual search and stopping behaviour, incorporating SERP level stopping offers a closer approximation than without. These findings show that models and measures that naïvely assume snippets and documents in a ranked list are actually examined in detail are less accurate, and that modelling SERP level stopping is required to create more realistic models of the search process.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    For a more detailed description of the flow of interaction and various processes represented within the CSM, refer to Maxwell et al. [18].

  2. 2.

    Whoosh is available on PyPi at https://pypi.python.org/pypi/Whoosh/.

  3. 3.

    SimIIR is available at https://github.com/leifos/simiir.

  4. 4.

    Despite the allocation of 10 min per topic, only the first six minutes (360 s) of interaction data were considered in the results of Maxwell et al. [21]. As such, we use this as our simulated search session time limit. Refer to Maxwell et al. [21] for the rationale behind this decision.

  5. 5.

    Human subjects issued queries of 3.31 terms on average. This means that the three term queries generated by QS3 can be considered as a reasonable approximation.

  6. 6.

    For example, a robust snippet level stopping strategy would ideally stop early in the ranked list for poor performing queries, and later for good queries – good queries will return more relevant documents in the ranked list of results.

  7. 7.

    This also meant that for our comparison runs, only the four topics selected by Maxwell et al. [21] were trialled, rather than the full set of 50 topics from the TREC 2005 Robust Track as used in our performance runs.

References

  1. Bailey, P., Moffat, A., Scholer, F., Thomas, P.: User variability and IR system evaluation. In: Proceedings of 38th ACM SIGIR, pp. 625–634 (2015)

    Google Scholar 

  2. Baskaya, F., Keskustalo, H., Järvelin, K.: Modeling behavioral factors in interactive information retrieval. In: Proceedings of 22nd ACM CIKM, pp. 2297–2302 (2013)

    Google Scholar 

  3. Browne, G., Pitts, M., Wetherbe, J.: Stopping rule use during web-based search. In: Proceedings of HICSS-38, p. 271b (2005)

    Google Scholar 

  4. Card, S., Pirolli, P., Van Der Wege, M., Morrison, J., Reeder, R., Schraedley, P., Boshart, J.: Information scent as a driver of web behavior graphs: results of a protocol analysis method for web usability. In: Proceedings of 19th ACM CHI, pp. 498–505 (2001)

    Google Scholar 

  5. Chi, E., Pirolli, P., Chen, K., Pitkow, J.: Using information scent to model user information needs and actions and the web. In: Proceedings of 19th ACM CHI, pp. 490–497 (2001)

    Google Scholar 

  6. Cooper, W.: On selecting a measure of retrieval effectiveness part II. Implementation of the philosophy. J. Am. Soc. Info. Sci. 24(6), 413–424 (1973)

    Article  Google Scholar 

  7. Diriye, A., White, R., Buscher, G., Dumais, S.: Leaving so soon? Understanding and predicting web search abandonment rationales. In: Proceedings of 21st ACM CIKM, pp. 1025–1034 (2012)

    Google Scholar 

  8. Hagen, M., Michel, M., Stein, B.: Simulating ideal and average users. In: Proceedings of 12th AIRS, pp. 138–154 (2016)

    Google Scholar 

  9. Hassan, A., White, R.: Personalized models of search satisfaction. In: Proceedings of 22nd ACM CIKM, pp. 2009–2018 (2013)

    Google Scholar 

  10. Hassan, A., Shi, X., Craswell, N., Ramsey, B.: Beyond clicks: query reformulation as a predictor of search satisfaction. In: Proceedings of 22nd CIKM, pp. 2019–2028 (2013)

    Google Scholar 

  11. Hearst, M.: Search user interfaces. Cambridge University Press, Cambridge (2009)

    Google Scholar 

  12. Ingwersen, P., Järvelin, K.: The Turn: Integration of Information Seeking and Retrieval in Context. Springer, Dordrecht (2005). https://doi.org/10.1007/1-4020-3851-8

    MATH  Google Scholar 

  13. Keskustalo, H., Järvelin, K., Pirkola, A., Sharma, T., Lykke, M.: Test collection-based IR evaluation needs extension toward sessions – a case of extremely short queries. In: Proceedings of 5th AIRS, pp. 63–74 (2009)

    Google Scholar 

  14. Kiseleva, J., Kamps, J., Nikulin, V., Makarov, N.: Behavioral dynamics from the SERP’s perspective: what are failed SERPs and how to fix them? In: Proceedings of 24th ACM CIKM, pp. 1561–1570 (2015)

    Google Scholar 

  15. Kraft, D., Lee, T.: Stopping rules and their effect on expected search length. IPM 15(1), 47–58 (1979)

    MATH  Google Scholar 

  16. Loumakis, F., Stumpf, S., Grayson, D.: This image smells good: effects of image information scent in search engine results pages. In: Proceedings of 20th ACM CIKM, pp. 475–484 (2011)

    Google Scholar 

  17. Marshall, C.C., Shipman III, F.M.: Spatial hypertext and the practice of information triage. In: Proceedings of 8th ACM HYPERTEXT, pp. 124–133 (1997)

    Google Scholar 

  18. Maxwell, D., Azzopardi, L.: Agents, simulated users and humans: an analysis of performance and behaviour. In: Proceedings of 25th ACM CIKM, pp. 731–740 (2016)

    Google Scholar 

  19. Maxwell, D., Azzopardi, L., Järvelin, K., Keskustalo, H.: An initial investigation into fixed and adaptive stopping strategies. In: Proceedings of 38th ACM SIGIR, pp. 903–906 (2015)

    Google Scholar 

  20. Maxwell, D., Azzopardi, L., Järvelin, K., Keskustalo, H.: Searching and stopping: an analysis of stopping rules and strategies. In: Proceedings of 24th ACM CIKM, pp. 313–322 (2015)

    Google Scholar 

  21. Maxwell, D., Azzopardi, L., Moshfeghi, Y.: A study of snippet length and informativeness: behaviour, performance and UX. In: Proceedings of 40th ACM SIGIR (2017)

    Google Scholar 

  22. McNair, J.N.: Optimal giving-up times and the marginal value theorem. Am. Nat. 119(4), 511–529 (1982)

    Article  Google Scholar 

  23. Moffat, A., Bailey, P., Scholer, F., Thomas, P.: INST: An adaptive metric for IR evaluation. In: Proceedings of 20th ADCS, pp. 5:1–5:4 (2015)

    Google Scholar 

  24. Moffat, A., Zobel, J.: Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Info. Syst. 27(1), 2:1–2:27 (2008)

    Article  Google Scholar 

  25. Nickles, K.: Judgment-based and reasoning-based stopping rules in decision making under uncertainty. Ph.D. thesis, University of Minnesota (1995)

    Google Scholar 

  26. Olston, C., Chi, E.: ScentTrails: integrating browsing and searching on the web. ACM Trans. Comput. Hum. Interact. 10(3), 177–197 (2003)

    Article  Google Scholar 

  27. Ong, K., Järvelin, K., Sanderson, M., Scholer, F.: Using information scent to understand mobile and desktop web search behavior. In: Proceedings of 40th ACM SIGIR (2017)

    Google Scholar 

  28. Pirolli, P.: Information Foraging Theory: Adaptive Interaction with Information, 1st edn. Oxford University Press, New York (2007)

    Book  Google Scholar 

  29. Pirolli, P., Card, S.: Informayion foraging in information access environments. In: Proceedings of 13th ACM SIGCHI, pp. 51–58 (1995)

    Google Scholar 

  30. Pirolli, P., Card, S.: Information foraging. Psychol. Rev. 106, 643–675 (1999)

    Article  Google Scholar 

  31. Smucker, M., Clarke, C.: Modeling optimal switching behavior. In: Proceedings of 1st ACM CHIIR, pp. 317–320 (2016)

    Google Scholar 

  32. Stephens, D., Krebs, J.: Foraging Theory. Princeton University Press, Princeton (1986)

    Google Scholar 

  33. Sundar, S., Knobloch-Westerwick, S., Hastall, M.: News cues: information scent and cognitive heuristics. J. Am. Soc. Inf. Sci. Technol. 58(3), 366–378 (2007)

    Article  Google Scholar 

  34. Thomas, P., Moffat, A., Bailey, P., Scholer, F.: Modeling decision points in user search behavior. In: Proceedings of 5th IIiX, pp. 239–242 (2014)

    Google Scholar 

  35. Tombros, A., Sanderson, M.: Advantages of query biased summaries in information retrieval. In: Proceedings of 21st ACM SIGIR, pp. 2–10 (1998)

    Google Scholar 

  36. Wu, W.: How far will you go? Using need for closure and information scent to model search stopping behavior. In: Proceedings of 4th IIiX, pp. 328–328 (2012)

    Google Scholar 

  37. Wu, W., Kelly, D., Sud, A.: Using information scent and need for cognition to understand online search behavior. In: Proceedings of 37th ACM SIGIR, pp. 557–566 (2014)

    Google Scholar 

  38. Zach, L.: When is “enough” enough? Modeling the information-seeking and stopping behavior of senior arts administrators. J. Am. Soc. Info. Sci. Tech. 56(1), 23–35 (2005)

    Article  Google Scholar 

Download references

Acknowledgements

Our thanks to Horaţiu Bota and Alastair Maxwell for their feedback – including Horaţiu’s helpful comments on our results. We would also like to thank the anonymous reviewers for their comments and feedback. Finally, the lead author is funded by the UK Government though the EPSRC, grant number 1367507.

Author information

Authors and Affiliations

  1. School of Computing Science, University of Glasgow, Glasgow, Scotland

    David Maxwell

  2. Computer and Information Sciences, University of Strathclyde, Glasgow, Scotland

    Leif Azzopardi

Authors
  1. David Maxwell
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Leif Azzopardi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David Maxwell .

Editor information

Editors and Affiliations

  1. Department of Informatics, Systems, and Communication, University of Milano-Bicocca, Milan, Italy

    Gabriella Pasi

  2. LIP6 – UPMC/CNRS, University Pierre et Marie Curie, Paris, France

    Benjamin Piwowarski

  3. University of Glasgow, Glasgow, United Kingdom

    Leif Azzopardi

  4. Technical University of Vienna, Vienna, Austria

    Allan Hanbury

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Maxwell, D., Azzopardi, L. (2018). Information Scent, Searching and Stopping. In: Pasi, G., Piwowarski, B., Azzopardi, L., Hanbury, A. (eds) Advances in Information Retrieval. ECIR 2018. Lecture Notes in Computer Science(), vol 10772. Springer, Cham. https://doi.org/10.1007/978-3-319-76941-7_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-76941-7_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-76940-0

  • Online ISBN: 978-3-319-76941-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation