Skip to main content
SpringerLink
Log in
Book cover

Asia Information Retrieval Symposium

AIRS 2016: Information Retrieval Technology pp 138–154Cite as

Simulating Ideal and Average Users

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 9994))

Abstract

We propose a framework for deterministic simulation of user behavior that allows to analyze the cost-gain-based performance on single result lists or whole search sessions. The ideal user representing optimal behavior (i.e., most gain with lowest effort) is contrasted with more “average” users that employ the spreading activation model from cognitive theory. On TREC Session Track data, the ideal user achieves about double the gain of real users at the same costs while the average gain of our different simulated users correlates well with the session-DCG metric—another argument for that metric in session-based evaluation.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Anderson, J.R., Matessa, M., Lebiere, C.: ACT-R: a theory of higher level cognition and its relation to visual attention. Hum. Comput. Interact. 12(4), 439–462 (1997)

    Article  Google Scholar 

  2. Arif, A., Stuerzlinger, W.: Analysis of text entry performance metrics. In: IEEE TIC-STH 2009, pp. 100–105 (2009)

    Google Scholar 

  3. Azzopardi, L.: Modelling interaction with economic models of search. In: SIGIR 2014, pp. 3–12 (2014)

    Google Scholar 

  4. Azzopardi, L.: The economics in interactive information retrieval. In: SIGIR 2011, pp. 15–24 (2011)

    Google Scholar 

  5. Azzopardi, L., de Rijke, M., Balog, K.: Building simulated queries for known-item topics: an analysis using six European languages. In: SIGIR 2007, pp. 455–462 (2007)

    Google Scholar 

  6. Azzopardi, L., Kelly, D., Brennan, K.: How query cost affects search behavior. In: SIGIR 2013, pp. 23–32 (2013)

    Google Scholar 

  7. Azzopardi, L., Zuccon, G.: An analysis of theories of search and search behavior. In: ICTIR 2015, pp. 81–90 (2015)

    Google Scholar 

  8. Azzopardi, L., Zuccon, G.: Two scrolls or one click: a cost model for browsing search results. In: Ferro, N., et al. (eds.) ECIR 2016. LNCS, vol. 9626, pp. 696–702. Springer, Heidelberg (2016). doi:10.1007/978-3-319-30671-1_55

    Chapter  Google Scholar 

  9. Barker, K., Cornacchia, N.: Using noun phrase heads to extract document keyphrases. In: AI 2000, pp. 40–52 (2000)

    Google Scholar 

  10. Baskaya, F., Keskustalo, H., Järvelin, K.: Time drives interaction: simulating sessions in diverse searching environments. In: SIGIR 2012, pp. 105–114 (2012)

    Google Scholar 

  11. Baskaya, F., Keskustalo, H., Järvelin, K.: Modeling behavioral factors in interactive information retrieval. In: CIKM 2013, pp. 2297–2302 (2013)

    Google Scholar 

  12. Budiu, R., Royer, C., Pirolli, P.: Modeling information scent: a comparison of LSA, PMI and GLSA similarity measures on common tests and corpora. In: RIAO 2007, pp. 314–332 (2007)

    Google Scholar 

  13. Busemeyer, J.R., Diederich, A.: Cognitive Modeling. Sage Publications, Thousand Oaks (2009)

    Google Scholar 

  14. Carterette, B., Bah, A., Zengin, M.: Dynamic test collections for retrieval evaluation. In: ICTIR 2015, pp. 91–100 (2015)

    Google Scholar 

  15. Carterette, B., Kanoulas, E., Yilmaz, E.: Simulating simple user behavior for system effectiveness evaluation. In: CIKM 2011, pp. 611–620 (2011)

    Google Scholar 

  16. Chapelle, O., Metlzer, D., Zhang, Y., Grinspan, P.: Expected reciprocal rank for graded relevance. In: CIKM 2009, pp. 621–630 (2009)

    Google Scholar 

  17. Chapelle, O., Zhang, Y.: A dynamic Bayesian network click model for web search ranking. In: WWW 2009, pp. 1–10 (2009)

    Google Scholar 

  18. Church, H.: Word association norms, mutual information, and lexicography. Comput. Linguist. 16(1), 22–29 (1990)

    Google Scholar 

  19. Dang, V., Croft, B.W.: Query reformulation using anchor text. In: WSDM 2010, pp. 41–50 (2010)

    Google Scholar 

  20. Dyer, M.E., Kayal, N., Walker, J.: A branch and bound algorithm for solving the multiple-choice knapsack problem. J. Comp. Appl. Math. 11(2), 231–249 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  21. Fu, W.-T., Pirolli, P.: SNIF-ACT: a cognitive model of user navigation on the world wide web. Hum. Comput. Interact. 22(4), 355–412 (2007)

    Google Scholar 

  22. Hagen, M., Gomoll, J., Beyer, A., Stein, B.: From search session detection to search mission detection. In: OAIR 2013, pp. 85–92 (2013)

    Google Scholar 

  23. He, Y., Wang, K.: Inferring search behaviors using partially observable Markov model with duration (POMD). In: WSDM 2011, pp. 415–424 (2011)

    Google Scholar 

  24. Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM TOIS 20(4), 422–446 (2002)

    Article  Google Scholar 

  25. Järvelin, K., Price, S.L., Delcambre, L.M.L., Nielsen, M.L.: Discounted cumulated gain based evaluation of multiple-query IR sessions. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) ECIR 2008. LNCS, vol. 4956, pp. 4–15. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  26. Keane, M.T., O’Brien, M., Smyth, B.: Are people biased in their use of search engines? CACM 51(2), 49–52 (2008)

    Article  Google Scholar 

  27. Kellerer, H., Pferschy, U., Pisinger, D.: Knapsack Problems. Springer, Heidelberg (2004)

    Book  MATH  Google Scholar 

  28. Klöckner, K., Wirschum, N., Jameson, A.: Depth- and breadth-first processing of search result lists. In: CHI 2004, p. 1539 (2004)

    Google Scholar 

  29. Manktelow, K.I.: Reasoning and Thinking. Psychology Press, Hove (1999)

    Google Scholar 

  30. Maxwell, D., Azzopardi, L., Järvelin, K., Keskustalo, H.: Searching and stopping: an analysis of stopping rules and strategies. In: CIKM 2015, pp. 313–322 (2015)

    Google Scholar 

  31. O’Brien, M., Keane, M.T.: Modeling user behavior using a search-engine. In: IUI 2007, pp. 357–360 (2007)

    Google Scholar 

  32. Pirolli, P., Card, S.: Information foraging in information access environments. In: CHI 1995, pp. 51–58 (1995)

    Google Scholar 

  33. Pisinger, D.: A minimal algorithm for the multiple-choice knapsack problem. Eur. J. Oper. Res. 83(2), 394–410 (1995)

    Article  MATH  Google Scholar 

  34. Sanderson, M., Paramita, M.L., Clough, P., Kanoulas, E.: Do user preferences and evaluation measures line up? In: SIGIR 2010, pp. 555–562 (2010)

    Google Scholar 

  35. Smucker, M.D., Clarke, C.L.A.: Modeling optimal switching behavior. In: CHIIR 2016, pp. 317–320 (2016)

    Google Scholar 

  36. Smucker, M.D., Clarke, C.L.A.: Modeling user variance in time-biased gain. In: HCIR 2012, paper 3 (2012)

    Google Scholar 

  37. Smucker, M.D., Jethani, C.P.: Human performance and retrieval precision revisited. In: SIGIR 2010, pp. 595–602 (2010)

    Google Scholar 

  38. Thomas, P., Moffat, A., Bailey, P., Scholer, F.: Modeling decision points in user search behavior. In: IIiX 2014, pp. 239–242 (2014)

    Google Scholar 

  39. Tran, V.T., Fuhr, N.: Using eye-tracking with dynamic areas of interest for analyzing interactive information retrieval. In: SIGIR 2012, pp. 1165–1166 (2012)

    Google Scholar 

  40. Tran, V.T., Fuhr, N.: Markov modeling for user interaction in retrieval. In: MUBE 2013, pp. 13–14 (2013)

    Google Scholar 

  41. Turpin, A., Scholer, F.: User performance versus precision measures for simple search tasks. In: SIGIR 2006, pp. 11–18 (2006)

    Google Scholar 

  42. Zhang, Y., Chen, W., Wang, D., Yang, Q.: User-click modeling for understanding and predicting search-behavior. In: KDD 2011, pp. 1388–1396 (2011)

    Google Scholar 

Download references

Acknowledgment

Working on simulated ideal and average users was very much inspired by many discussions the first author had with Leif Azzopardi, Charlie Clarke, Gianmaria Silvello, and Robert Villa in the “User simulation” working group of the Dagstuhl seminar 13441, organized by Maristella Agosti, Norbert Fuhr, Elaine Toms, and Pertti Vakkari.

Author information

Authors and Affiliations

  1. Bauhaus-Universität Weimar, Weimar, Germany

    Matthias Hagen, Maximilian Michel & Benno Stein

Authors
  1. Matthias Hagen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maximilian Michel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Benno Stein
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matthias Hagen .

Editor information

Editors and Affiliations

  1. Tsinghua University , Beijing, China

    Shaoping Ma

  2. Renmin University of China , Beijing, China

    Ji-Rong Wen

  3. Tsinghua University , Beijing, China

    Yiqun Liu

  4. Renmin University of China , Beijing, China

    Zhicheng Dou

  5. Tsinghua University , Beijing, China

    Min Zhang

  6. Yahoo Labs , Sunnyvale, California, USA

    Yi Chang

  7. Renmin University of China , Beijing, China

    Xin Zhao

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing AG

About this paper

Cite this paper

Hagen, M., Michel, M., Stein, B. (2016). Simulating Ideal and Average Users. In: Ma, S., et al. Information Retrieval Technology. AIRS 2016. Lecture Notes in Computer Science(), vol 9994. Springer, Cham. https://doi.org/10.1007/978-3-319-48051-0_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-48051-0_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-48050-3

  • Online ISBN: 978-3-319-48051-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation