The Combination and Evaluation of Query Performance Prediction Methods

  • Claudia Hauff
  • Leif Azzopardi
  • Djoerd Hiemstra
Conference paper

DOI: 10.1007/978-3-642-00958-7_28

Part of the Lecture Notes in Computer Science book series (LNCS, volume 5478)
Cite this paper as:
Hauff C., Azzopardi L., Hiemstra D. (2009) The Combination and Evaluation of Query Performance Prediction Methods. In: Boughanem M., Berrut C., Mothe J., Soule-Dupuy C. (eds) Advances in Information Retrieval. ECIR 2009. Lecture Notes in Computer Science, vol 5478. Springer, Berlin, Heidelberg

Abstract

In this paper, we examine a number of newly applied methods for combining pre-retrieval query performance predictors in order to obtain a better prediction of the query’s performance. However, in order to adequately and appropriately compare such techniques, we critically examine the current evaluation methodology and show how using linear correlation coefficients (i) do not provide an intuitive measure indicative of a method’s quality, (ii) can provide a misleading indication of performance, and (iii) overstate the performance of combined methods. To address this, we extend the current evaluation methodology to include cross validation, report a more intuitive and descriptive statistic, and apply statistical testing to determine significant differences. During the course of a comprehensive empirical study over several TREC collections, we evaluate nineteen pre-retrieval predictors and three combination methods.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Claudia Hauff
    • 1
  • Leif Azzopardi
    • 2
  • Djoerd Hiemstra
    • 1
  1. 1.University of TwenteThe Netherlands
  2. 2.University of GlasgowUnited Kingdom

Personalised recommendations