Relevance in Technicolor
In this article we propose the concept of relevance criteria profiles, which provide a global view of user behaviour in judging the relevance of retrieved information. We further propose a plotting technique which provides a session based overview of the relevance judgement processes interlaced with interactions that allow the researcher to visualise and quickly detect emerging patterns in both interactions and relevance criteria usage. We discuss by example, using data from a user study conducted between the months of January and August of 2008, how these tools support the better understanding of task based user valuation of documents that is likely to lead to recommendations for improving end-user services in digital libraries.
KeywordsRelevance Criterion Relevance Judgement Search Session Metadata Element Interactive Information Retrieval
Unable to display preview. Download preview PDF.
- 4.Borlund, P.: The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Information Research 8(3), 8–13 (2003)Google Scholar
- 5.Cleverdon, C.W., Mills, J., Keen, E.M.: Factors Determining the Performance of Indexing Systems. Design, vol. 1., Test Results, vol. 2. Aslib Cranfield Research Project, Cranfield, Lagland (1966)Google Scholar
- 6.Ericsson, K.A., Simon, H.A.: Protocol analysis: verbal reports as data. MIT Press, Cambridge (1993)Google Scholar
- 11.Schamber, L.: Users’criteria for evaluation in a multimedia environment. Proceedings of the 54 Annual Meeting of the American Society for Information Science 28, 126–133 (1991)Google Scholar