Information Retrieval Journal

, Volume 18, Issue 5, pp 445–472

Pooling-based continuous evaluation of information retrieval systems

  • Alberto Tonon
  • Gianluca Demartini
  • Philippe Cudré-Mauroux
Article

DOI: 10.1007/s10791-015-9266-y

Cite this article as:
Tonon, A., Demartini, G. & Cudré-Mauroux, P. Inf Retrieval J (2015) 18: 445. doi:10.1007/s10791-015-9266-y

Abstract

The dominant approach to evaluate the effectiveness of information retrieval (IR) systems is by means of reusable test collections built following the Cranfield paradigm. In this paper, we propose a new IR evaluation methodology based on pooled test-collections and on the continuous use of either crowdsourcing or professional editors to obtain relevance judgements. Instead of building a static collection for a finite set of systems known a priori, we propose an IR evaluation paradigm where retrieval approaches are evaluated iteratively on the same collection. Each new retrieval technique takes care of obtaining its missing relevance judgements and hence contributes to augmenting the overall set of relevance judgements of the collection. We also propose two metrics: Fairness Score, and opportunistic number of relevant documents, which we then use to define new pooling strategies. The goal of this work is to study the behavior of standard IR metrics, IR system ranking, and of several pooling techniques in a continuous evaluation context by comparing continuous and non-continuous evaluation results on classic test collections. We both use standard and crowdsourced relevance judgements, and we actually run a continuous evaluation campaign over several existing IR systems.

Keywords

Information retrieval evaluation Crowdsourcing Continuous evaluation Pooling techniques 

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Alberto Tonon
    • 1
  • Gianluca Demartini
    • 2
  • Philippe Cudré-Mauroux
    • 1
  1. 1.University of FribourgFribourgSwitzerland
  2. 2.University of SheffieldSouth YorkshireUK

Personalised recommendations