Skip to main content
SpringerLink
Log in

The User in Interactive Information Retrieval Evaluation

  • Chapter
Advanced Topics in Information Retrieval

Part of the book series: The Information Retrieval Series ((INRE,volume 33))

Abstract

This chapter initially defines what characterizes and distinguishes research frameworks from research models. The Laboratory Research Framework for IR illustrates the case. We define briefly what is meant by the concept of research design, including research questions, and what this chapter regards as central IIR evaluation research settings and variables. This is followed by a description of IIR components, pointing to the elements of the Integrated Cognitive Research Framework for IR that incorporates the Laboratory Framework in a contextual manner. The following sections describe and exemplify: (1) Request types, test persons, task-based simulations of search situations and relevance or performance measures in IIR; (2) Ultra-Light Interactive IR experiments; (3) Interactive-Light IR studies; and (4) Naturalistic field investigations of IIR. The chapter concludes with a summary section, a reference list and a thematically classified bibliography.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Anick P (2003) Using terminological feedback for web search refinement: A log based study. In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 88–95

    Google Scholar 

  • Baeza-Yates R, Ribeiro-Neto B (1999) Modern Information Retrieval. ACM Press/Addison-Wesley, New York, NY

    Google Scholar 

  • Barry CL (1994) User-defined relevance criteria: An exploratory study. Journal of the American Society for Information Science 45:149–159

    Article  Google Scholar 

  • Barry CL, Schamber L (1998) User’s criteria for relevance evaluation: A cross situational comparison. Information Processing and Management 34(2/3):219–236

    Article  Google Scholar 

  • Beaulieu M (1997) Experiments with interfaces to support query expansion. Journal of Documentation 53:8–19

    Article  Google Scholar 

  • Beaulieu M (2000) Interaction in information searching and retrieval. Journal of Documentation 431–439

    Google Scholar 

  • Beaulieu M, Jones S (1998) Interactive searching and interface issues in the okapi best match probabilistic retrieval system. Interacting with Computers 10:237–248

    Article  Google Scholar 

  • Belew R (2000) Finding out About: A Cognitive Perspective on Search Engine Technology and the WWW. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  • Belkin N, Vickery A (1985) Interaction in information systems. British Library, London

    Google Scholar 

  • Belkin N, Oddy R, Brooks H (1982) Ask for information retrieval: Part 2. Journal of Documentation 38:145–164

    Article  Google Scholar 

  • Belkin N, Seeger T, Wersig G (1983) Distributed expert problem treatment as a model for information systems analysis and design. Journal of Information Science: Principles and Practice 5:153–167

    Google Scholar 

  • Belkin N, Cool C, Croft W, Callan J (1993) The effect of multiple query representations on information retrieval performance. In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 339–346

    Google Scholar 

  • Belkin N, Cool C, Koenemann J, Park S, Ng W (1996a) Information seeking behaviour in new searching environments. In: Ingwersen P, Pors N (eds) Proceedings of the Conference on Conceptions of Library and Information Science, pp 403–416

    Google Scholar 

  • Belkin N, Cool C, Koenemann J, Ng W, Park S (1996b) Using relevance feedback and ranking in interactive searching. In: Harman D (ed) Proceedings of the Text Retrieval Conference, pp 181–210

    Google Scholar 

  • Belkin N, Cool C, Kelly D, Lee HJ, Muresan G, Tang MC, Yuan XJ (2003) Query length in interactive information retrieval. In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 205–212

    Google Scholar 

  • Bellotti V, Ducheneaut N, Howard M, Smith I (2003) Taking email to task: The design and evaluation of a task management centered email tool. In: Proceedings of the ACM Conference on Human Factors in Computing Systems, pp 345–352

    Chapter  Google Scholar 

  • Bilal D (2000) Children’s use of the Yahooligans! web search engine: 1. Cognitive, physical, and affective behaviors on fact-based search tasks. Journal of the American Society for Information Science, pp 1170–1183

    Google Scholar 

  • Bilal D (2002) Children’s use of the Yahooligans! web search engine. III. Cognitive and physical behaviors on fully self-generated search tasks. Journal of the American Society for Information Science and Technology 53:1170–1183. http://portal.acm.org/citation.cfm?id=772458.772468, visited on December, 2010

    Article  Google Scholar 

  • Borlund P (2000) Experimental components for the evaluation of interactive information retrieval systems. Journal of Documentation 56:71–90

    Article  Google Scholar 

  • Borlund P (2003a) The IIR evaluation model: A framework for evaluation of interactive information retrieval systems. http://informationr.net/ir/8-3/paper152.html, visited on February, 2011

  • Borlund P (2003b) The concept of relevance in IR. Journal of the American Society for Information Science and Technology 54:913–925. http://portal.acm.org/citation.cfm?id=941240.941242, visited on December, 2010

    Article  Google Scholar 

  • Borlund P, Ingwersen P (1998) Measures of relative relevance and ranked half-life: performance indicators for interactive IR. In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 324–331. http://doi.acm.org/10.1145/290941.291019, visited on December, 2010

    Google Scholar 

  • Buckland L, Voorhees E (eds) (2005) Proceedings of the Text Retrieval Conference. Government Printing Office, Washington, DC

    Google Scholar 

  • Byström K, Järvelin K (1995) Task complexity affects information seeking and use. Information Processing and Management 31:191–213. http://portal.acm.org/citation.cfm?id=208327.208360, visited on December, 2010

    Article  Google Scholar 

  • Campbell I (2000) Interactive evaluation of the ostensive model using a new test collection of images with multiple relevance assessments. Journal of Information Retrieval 2:89–114. http://portal.acm.org/citation.cfm?id=593954.593979, visited on December, 2010

    Article  Google Scholar 

  • Cosijn E, Ingwersen P (2000) Dimensions of relevance. Information Processing and Management 36:533–550. http://portal.acm.org/citation.cfm?id=348834.348836, visited on December, 2010

    Article  Google Scholar 

  • Cothey V (2002) A longitudinal study of world wide web users’ information-searching behavior. Journal of the American Society for Information Science and Technology 53:67–78. http://portal.acm.org/citation.cfm?id=506072.506073, visited on December, 2010

    Article  Google Scholar 

  • Croft W, Thompson R (1987) I3r: a new approach to the design of document retrieval systems. Journal of the American Society for Information Science 38(6):389–404

    Article  Google Scholar 

  • Czerwinski M, Horvitz E, Cutrell E (2001) Subjective duration assessment: An implicit probe for software usability. In: Proceedings of the Conference on Interaction Homme-Machine and Human-Computer Interaction, pp 167–170

    Google Scholar 

  • Dennis S, Bruza P, McArthur R (2002) Web searching: a process-oriented experimental study of three interactive search paradigms. Journal of the American Society for Information Science and Technology 53:120–133. http://portal.acm.org/citation.cfm?id=506072.506077, visited on December, 2010

    Article  Google Scholar 

  • Dumais S, Cutrell E, Cadiz J, Jancke G, Sarin R, Robbins D (2003) Stuff i’ve seen: A system for personal information retrieval and re-use. In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 72–79

    Google Scholar 

  • Efron M (2009) Using multiple query aspects to build test collections without human relevance judgments. In: Proceedings of the European Conference on Information Retrieval. Springer, Heidelberg, pp 276–287

    Google Scholar 

  • Efthimiadis E (1993) A user-centered evaluation of ranking algorithms for interactive query expansion. In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 146–156

    Google Scholar 

  • Ericsson K, Simon H (1996) Protocol Analysis: Verbal Reports as Data. MIT Press, Cambridge, MA

    Google Scholar 

  • Fidel R (1993) Qualitative methods in information retrieval research. Library and Information Science Research 15:219–247

    Google Scholar 

  • Fidel R, Soergel D (1983) Factors affecting online bibliographic retrieval: A conceptual framework for research. Journal of the American Society for Information Science 34:163–180

    Article  Google Scholar 

  • Fidel R, Davies RK, Douglass MH, Holder JK, Hopkins CJ, Kushner EJ, Miyagishima BK, Toney CD (1999) A visit to the information mall: Web searching behavior of high school students. Journal of the American Society for Information Science 50:24–37

    Article  Google Scholar 

  • Ford N, Miller D, Moss N (2001) The role of individual differences in internet searching: An empirical study. Journal of the American Society for Information Science and Technology 52:1049–1066

    Article  Google Scholar 

  • Frankfort-Nachmias C, Nachmias D (2000) Research Methods in the Social Sciences. Worth Publishers, New York

    Google Scholar 

  • Frokjaer E, Hertzum M, Hormbaek K (2000) Measuring usability: Are effectiveness and efficiency and satisfaction really correlated? In: Proceedings of the ACM Conference on Human Factors in Computing Systems, pp 345–352

    Chapter  Google Scholar 

  • Harman D (ed) (1996) Proceedings of the Text Retrieval Conference. Government Printing Office, Washington, DC

    Google Scholar 

  • Harper D, Koychev I, Sun Y, Pirie I (2004) Within-document retrieval: A user-centred evaluation of relevance profiling. Journal of Information Retrieval 7:265–290

    Article  Google Scholar 

  • Hawking D, Voorhees E, Craswell N, Bailey P (2000) Overview of the TREC web track. In: Voorhees E, Harman D (eds) Proceedings of the Text Retrieval Conference, pp 131–150

    Google Scholar 

  • Hersh W, Pentecost J, Hickam D (1996) A task oriented approach to information retrieval evaluation. Journal of the American Society for Information Science 47:50–56

    Article  Google Scholar 

  • Hirsh S (1999) Children’s relevance criteria and information-seeking on electronic resources. Journal of the American Society for Information Science 50:1265–1283

    Article  Google Scholar 

  • Hornbaek K (2006) Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies 64:79–102

    Article  Google Scholar 

  • Hsieh-Yee I (1998) Search tactics of web users in searching for texts, graphics, known items and subjects: A search simulation study. Reference Librarian 60:61–85

    Article  Google Scholar 

  • Ingwersen P (1992) Information Retrieval Interaction. Taylor Graham, London

    Google Scholar 

  • Ingwersen P, Järvelin K (2005) The Turn: Integration of Information Seeking and Retrieval in Context. Springer, Berlin

    MATH  Google Scholar 

  • Ingwersen P, Järvelin K (2007) On the holistic cognitive theory for information retrieval: Drifting outside the cave of the laboratory framework. In: Dominich S, Kisss F (eds) Studies in Theory of Information Retrieval. Foundation for Information Society, pp 135–147

    Google Scholar 

  • Ingwersen P, Willett P (1995) An introduction to algorithmic and cognitive approaches for information retrieval. Libri 45:160–177

    Article  Google Scholar 

  • Järvelin K (2007) An analysis of two approaches in information retrieval: From frameworks to study designs. Journal of the American Society for Information Science and Technology 58:971–986

    Article  Google Scholar 

  • Järvelin K, Kekäläinen J (2000) IR evaluation methods for highly relevant documents. In: Belkin N, Ingwersen P, Leong MK (eds) Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 41–48

    Google Scholar 

  • Järvelin K, Kekäläinen J (2002) Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 20:422–446

    Article  Google Scholar 

  • Joachims T, Granka L, Pan B, Hembrooke H, Gay G (2005a) Accurately interpreting click-through data as implicit feedback. In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 154–161

    Google Scholar 

  • Käki M (2004) Proportional search interface usability measures. In: Proceedings of the Nordic Forum for Human-Computer Interaction Research, pp 365–372

    Google Scholar 

  • Kekäläinen J (2005) Binary and graded relevance in IR evaluations—comparison of the effects on ranking of IR systems. Information Processing and Management 41:1019–1033

    Article  Google Scholar 

  • Kekäläinen J, Järvelin K (2002b) Using graded relevance assessments in IR evaluation. Journal of the American Society for Information Science and Technology 53:1120–1129

    Article  Google Scholar 

  • Kekäläinen J, Järvelin K (2002a) User-oriented evaluation methods for information retrieval: A case study based on conceptual models for query expansion. In: Lakemeyer G, Nebel B (eds) Exploring Artificial Intelligence in the New Millennium. Morgan Kaufmann, San Diego, CA, pp 355–379

    Google Scholar 

  • Kelly D, Belkin N (2004) Display time as implicit feedback: Understanding task effects. In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 377–384

    Google Scholar 

  • Kelly D, Fu X (2007) Eliciting better information need descriptions from users of information search systems. Information Processing and Management 43:30–46

    Article  Google Scholar 

  • Kelly D, Dollu V, Fu X (2005) The loquacious user: A document-independent source of terms for query expansion. In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 457–464

    Google Scholar 

  • Koenemann J, Belkin N (1996) A case for interaction: A study of interactive information retrieval behavior and effectiveness. In: Proceedings of the ACM Conference on Human Factors in Computing Systems, pp 205–212

    Google Scholar 

  • Larsen SM, Tombros A (2006) The interactive track at INEX. In: Fuhr N, Lalmas M, Malik S, Kazai G (eds) Proceedings of the Workshop of Initiative for the Evaluation of XML Retrieval. Springer, Berlin, pp 398–410

    Chapter  Google Scholar 

  • Lun A (2001) Measuring usability with the use questionnaire. http://www.stcsig.org/usability/newsletter/0110_measuring_with_use.html, visited on February, 2011

  • Lykke M (2001) A framework for work task based thesaurus design. Journal of Documentation 57:774–797

    Article  Google Scholar 

  • Lykke M (2004) Task-based evaluation of associative thesaurus in real-life environment. In: Proceedings of the Meeting of the American Society for Information Science and Technology, pp 437–447

    Google Scholar 

  • Magennis M, van Rijsbergen K (1997) The potential and actual effectiveness of interactive query expansion. In: Belkin N, Narasimhalu A, Willett P (eds) Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 324–332

    Google Scholar 

  • Maglaughlin K, Sonnenwald D (2002) User perspectives on relevance criteria: A comparison among relevant, partially relevant and not-relevant judgments. Journal of the American Society for Information Science and Technology 53:327–342

    Article  Google Scholar 

  • Nielsen J (2003) Usability 101: Introduction to usability. http://www.useit.com/alertbox/20030825.html, visited on February, 2011

  • Palmquist R, Kim K (2000) Cognitive style and on line database search experience as predictors of web search performance. Journal of the American Society for Information Science 51:558–566

    Article  Google Scholar 

  • Papaeconomou C, Zijlema AF, Ingwersen P (2008) Searchers’ relevance judgments and criteria in evaluating web pages in a learning style perspective. In: Proceedings of the International Symposium on Information Interaction in Context. IIiX ’08. ACM Press, New York, NY, pp 123–132. http://doi.acm.org/10.1145/1414694.1414722, visited on December, 2010

    Chapter  Google Scholar 

  • Petrelli D, Beaulieu M, Sanderson G, Demetriou M, Herring P, Hansen P (2004) Observing users and designing clarity: A case study on the user-centered design of a cross-language information retrieval system. Journal of the American Society for Information Science and Technology 55:923–934

    Article  Google Scholar 

  • Puolamäki K, Salojärvi J, Savia E (2005) Combining eye movements and collaborative filtering for proactive information retrieval. In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 146–153

    Google Scholar 

  • Ruthven I, Lalmas M, van Rijsbergen K (2002) Ranking expansion terms with partial and ostensive evidence. In: Proceedings of the Conference on Conceptions of Library and Information Science, pp 199–220

    Google Scholar 

  • Ruthven I, Lalmas M, van Rijsbergen K (2003) Incorporating user search behaviour into relevance feedback. Journal of the American Society for Information Science and Technology 54:529–549

    Article  Google Scholar 

  • Salojärvi J, Kojo I, Simola J, Kaski S (2003) Can relevance be inferred from eye movements in information retrieval? In: Proceedings of the Workshop on Self-Organizing Maps, Kitakyushu, Japan, pp 261–266

    Google Scholar 

  • Sanderson M, Zobel J (2005) Information retrieval system evaluation: Effort, sensitivity and reliability. In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 162–169

    Google Scholar 

  • Saracevic T, Kantor P (1988a) A study of information seeking and retrieving: II. Users, questions and effectiveness. Journal of the American Society for Information Science 39:177–196

    Article  Google Scholar 

  • Saracevic T, Kantor P (1988b) A study of information seeking and retrieving: III. Searchers, searches, and overlaps. Journal of the American Society for Information Science 39:197–216

    Article  Google Scholar 

  • Saracevic T, Kantor P, Chamis A, Trivison D (1988) A study of information seeking and retrieving: I. Background and methodology. Journal of the American Society for Information Science 39:161–176

    Article  Google Scholar 

  • Sormunen E (2002) Liberal relevance criteria of TREC—counting on negligible documents? In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 320–330

    Google Scholar 

  • Spink A, Greisdorf H, Bateman J (1998) From highly relevant to not relevant: Examining different regions of relevance. Information Processing and Management 34:599–621

    Article  Google Scholar 

  • Tague J, Schultz R (1988) Some measures and procedures for evaluation of the userinterface in an information retrieval system. In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 371–385

    Google Scholar 

  • Teevan J, Alvarado C, Ackerman M, Karger D (2004) The perfect search engine is not enough: A study of orienteering behavior in directed search. In: Proceedings of the ACM Conference on Human Factors in Computing Systems, pp 415–422

    Chapter  Google Scholar 

  • Tombros A, Ruthven I, Jose J (2003) Searchers’ criteria for assessing web pages. In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 96–104

    Google Scholar 

  • Vakkari P (2001) Changes in search tactics and relevance judgments in preparing a research proposal: A summary of findings of a longitudinal study. Journal of Information Retrieval 4(3/4):295–310

    MATH  Google Scholar 

  • Vakkari P, Hakala N (2000) Changes in relevance criteria and problem stages in task performance. Journal of Documentation 56:540–562

    Article  Google Scholar 

  • Vakkari P, Sormunen E (2004) The influence of relevance levels on the effectiveness of interactive information retrieval. Journal of the American Society for Information Science and Technology 55:963–969

    Article  Google Scholar 

  • Voorhees E (1998) Variations in relevance judgments and the measurement of retrieval effectiveness. In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 315–323

    Google Scholar 

  • Wang P (1997) Users’ information needs at different stages of a research project: a cognitive view. In: Proceedings of the International Conference on Information Seeking in Context. Taylor Graham, London, pp 307–318. http://portal.acm.org/citation.cfm?id=267190.267209, visited on December, 2010

    Google Scholar 

  • Wang P, Hawk W, Tenopir C (2000) Users’ interaction with World Wide Web resources: An exploratory study using a holistic approach. Information Processing and Management 36:229–251

    Article  Google Scholar 

  • White R (2006) Using searcher simulations to redesign a polyrepresentative implicit feedback interface. Information Processing and Management 42:1185–1202

    Article  Google Scholar 

  • White R, Jose JM, Ruthven I (2003) A task-oriented study of the influencing effects of query-biased summarization in web searching. Information Processing and Management 39:707–733

    Article  MATH  Google Scholar 

  • White R, Ruthven I, Jose J (2005a) A study of factors affecting the utility of implicit relevance feedback. In: Proceedings of the ACM Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp 35–42

    Google Scholar 

  • White R, Ruthven I, Jose J, van Rijsbergen K (2005b) Evaluating implicit feedback models using searcher simulations. ACM Transactions on Information Systems 325–361

    Google Scholar 

  • White R, Jose J, Ruthven I (2006) An implicit feedback approach for interactive information retrieval. Information Processing and Management 42:166–190

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Royal School of Library and Information Science, Birketinget 6, 2300, Copenhagen S, Denmark

    Peter Ingwersen

  2. Oslo University College, Oslo, Norway

    Peter Ingwersen

Authors
  1. Peter Ingwersen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peter Ingwersen .

Editor information

Editors and Affiliations

  1. , Department of Information Engineering, University of Padua, Via G. Gradenigo, 6, Padua, 35131, Italy

    Massimo Melucci

  2. Yahoo! Research Barcelona, Ocata 1, Barcelona, 08003, Spain

    Ricardo Baeza-Yates

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Ingwersen, P. (2011). The User in Interactive Information Retrieval Evaluation. In: Melucci, M., Baeza-Yates, R. (eds) Advanced Topics in Information Retrieval. The Information Retrieval Series, vol 33. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-20946-8_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-20946-8_4

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-20945-1

  • Online ISBN: 978-3-642-20946-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation