Selecting Success Criteria: Experiences with an Academic Library Catalogue
Multiple methods exist for evaluating search systems, ranging from more user-oriented approaches to those more focused on evaluating system performance. When preparing an evaluation, key questions include: (i) why conduct the evaluation, (ii) what should be evaluated, and (iii) how the evaluation should be conducted. Over recent years there has been more focus on the end users of search systems and understanding what they view as ‘success’. In this paper we consider what to evaluate; in particular what criteria users of search systems consider most important and whether this varies by user characteristic. Using our experience with evaluating an academic library catalogue, input was gathered from end users relating to the perceived importance of different evaluation criteria prior to conducting an evaluation. We analyse results to show which criteria users most value, together with the inter-relationships between them. Our results highlight the necessity of conducting multiple forms of evaluation to ensure that search systems are deemed successful by their users.
KeywordsEvaluation success criteria digital libraries
Unable to display preview. Download preview PDF.
- 1.Saracevic, T.: Evaluation of evaluation in information retrieval. In: Fox, E., Ingwersen, P., Fidel, R. (eds.) Proc. 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, Washington, USA, July 9-13, pp. 138–146. ACM Press, New York (1995)Google Scholar
- 2.Harman, D.: Information retrieval evaluation. Synthesis Lectures on Information Concepts, Retrieval, and Services, vol. 3(2). Morgan & Claypool Publishers, San Raphael (2011)Google Scholar
- 5.Voorhees, E.M., Harman, D.K.: TREC: experiments and evaluation in information retrieval. MIT Press, Cambridge (2005)Google Scholar
- 6.Ingwersen, P., Järvelin, K.: The turn: integration of information seeking and retrieval in context. Springer, New York (2005)Google Scholar
- 7.Borlund, P.: User-Centred Evaluation of Information Retrieval Systems. In: Göker, A., Davies, J. (eds.) Information Retrieval: Searching in the 21st Century. John Wiley & Sons, Chichester (2009)Google Scholar
- 8.Kelly, D.: Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval 3(1-2), 1–224 (2009)Google Scholar
- 9.van Rijsbergen, C.J.: Information retrieval, 2nd edn. Butterworths, London (1979)Google Scholar
- 10.Saracevic, T.: Digital Library Evaluation: Toward Evolution of Concepts. Library Trends 49(2), 350–369 (2000)Google Scholar
- 14.Nielsen, J.: Enhancing the Explanatory Power of Usability Heuristics. In: Proc. SIGCHI Conference on Human Factors in Computing Systems, pp. 152–158. ACM Press, New York (1994)Google Scholar