Information Retrieval

, Volume 11, Issue 3, pp 209–228

Evaluating the effectiveness of relevance feedback based on a user simulation model: effects of a user scenario on cumulated gain value


    • Department of Information StudiesUniversity of Tampere
  • Kalervo Järvelin
    • Department of Information StudiesUniversity of Tampere
  • Ari Pirkola
    • Department of Information StudiesUniversity of Tampere

DOI: 10.1007/s10791-007-9043-7

Cite this article as:
Keskustalo, H., Järvelin, K. & Pirkola, A. Inf Retrieval (2008) 11: 209. doi:10.1007/s10791-007-9043-7


We propose a method for performing evaluation of relevance feedback based on simulating real users. The user simulation applies a model defining the user’s relevance threshold to accept individual documents as feedback in a graded relevance environment; user’s patience to browse the initial list of retrieved documents; and his/her effort in providing the feedback. We evaluate the result by using cumulated gain-based evaluation together with freezing all documents seen by the user in order to simulate the point of view of a user who is browsing the documents during the retrieval process. We demonstrate the method by performing a simulation in the laboratory setting and present the “branching” curve sets characteristic for the presented evaluation method. Both the average and topic-by-topic results indicate that if the freezing approach is adopted, giving feedback of mixed quality makes sense for various usage scenarios even though the modeled users prefer finding especially the most relevant documents.


EvaluationRelevance feedbackSimulationUser modeling

Copyright information

© Springer Science+Business Media, LLC 2008