A Case for Automatic System Evaluation

  • Claudia Hauff
  • Djoerd Hiemstra
  • Leif Azzopardi
  • Franciska de Jong
Conference paper

DOI: 10.1007/978-3-642-12275-0_16

Part of the Lecture Notes in Computer Science book series (LNCS, volume 5993)
Cite this paper as:
Hauff C., Hiemstra D., Azzopardi L., de Jong F. (2010) A Case for Automatic System Evaluation. In: Gurrin C. et al. (eds) Advances in Information Retrieval. ECIR 2010. Lecture Notes in Computer Science, vol 5993. Springer, Berlin, Heidelberg

Abstract

Ranking a set retrieval systems according to their retrieval effectiveness without relying on relevance judgments was first explored by Soboroff et al. [13]. Over the years, a number of alternative approaches have been proposed, all of which have been evaluated on early TREC test collections. In this work, we perform a wider analysis of system ranking estimation methods on sixteen TREC data sets which cover more tasks and corpora than previously. Our analysis reveals that the performance of system ranking estimation approaches varies across topics. This observation motivates the hypothesis that the performance of such methods can be improved by selecting the “right” subset of topics from a topic set. We show that using topic subsets improves the performance of automatic system ranking methods by 26% on average, with a maximum of 60%. We also observe that the commonly experienced problem of underestimating the performance of the best systems is data set dependent and not inherent to system ranking estimation. These findings support the case for automatic system evaluation and motivate further research.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Claudia Hauff
    • 1
  • Djoerd Hiemstra
    • 1
  • Leif Azzopardi
    • 2
  • Franciska de Jong
    • 1
  1. 1.University of TwenteEnschedeThe Netherlands
  2. 2.University of GlasgowGlasgowUK

Personalised recommendations