Information Retrieval

, Volume 11, Issue 3, pp 209-228

First online:

Evaluating the effectiveness of relevance feedback based on a user simulation model: effects of a user scenario on cumulated gain value

  • Heikki KeskustaloAffiliated withDepartment of Information Studies, University of Tampere Email author 
  • , Kalervo JärvelinAffiliated withDepartment of Information Studies, University of Tampere
  • , Ari PirkolaAffiliated withDepartment of Information Studies, University of Tampere

Rent the article at a discount

Rent now

* Final gross prices may vary according to local VAT.

Get Access


We propose a method for performing evaluation of relevance feedback based on simulating real users. The user simulation applies a model defining the user’s relevance threshold to accept individual documents as feedback in a graded relevance environment; user’s patience to browse the initial list of retrieved documents; and his/her effort in providing the feedback. We evaluate the result by using cumulated gain-based evaluation together with freezing all documents seen by the user in order to simulate the point of view of a user who is browsing the documents during the retrieval process. We demonstrate the method by performing a simulation in the laboratory setting and present the “branching” curve sets characteristic for the presented evaluation method. Both the average and topic-by-topic results indicate that if the freezing approach is adopted, giving feedback of mixed quality makes sense for various usage scenarios even though the modeled users prefer finding especially the most relevant documents.


Evaluation Relevance feedback Simulation User modeling