A Case for Automatic System Evaluation

  • Claudia Hauff
  • Djoerd Hiemstra
  • Leif Azzopardi
  • Franciska de Jong
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5993)


Ranking a set retrieval systems according to their retrieval effectiveness without relying on relevance judgments was first explored by Soboroff et al. [13]. Over the years, a number of alternative approaches have been proposed, all of which have been evaluated on early TREC test collections. In this work, we perform a wider analysis of system ranking estimation methods on sixteen TREC data sets which cover more tasks and corpora than previously. Our analysis reveals that the performance of system ranking estimation approaches varies across topics. This observation motivates the hypothesis that the performance of such methods can be improved by selecting the “right” subset of topics from a topic set. We show that using topic subsets improves the performance of automatic system ranking methods by 26% on average, with a maximum of 60%. We also observe that the commonly experienced problem of underestimating the performance of the best systems is data set dependent and not inherent to system ranking estimation. These findings support the case for automatic system evaluation and motivate further research.


Retrieval System Data Fusion Good System Mean Average Precision Relevance Judgment 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Rank Correlation Methods. Hafner Publishing Co., New York (1955)Google Scholar
  2. 2.
    Amitay, E., Carmel, D., Lempel, R., Soffer, A.: Scaling ir-system evaluation using term relevance sets. In: SIGIR 2004, pp. 10–17 (2004)Google Scholar
  3. 3.
    Aslam, J.A., Pavlu, V.: Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions. In: Amati, G., Carpineto, C., Romano, G. (eds.) ECiR 2007. LNCS, vol. 4425, pp. 198–209. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  4. 4.
    Aslam, J.A., Pavlu, V., Yilmaz, E.: A statistical method for system evaluation using incomplete judgments. In: SIGIR 2006, pp. 541–548 (2006)Google Scholar
  5. 5.
    Aslam, J.A., Savell, R.: On the effectiveness of evaluating retrieval systems in the absence of relevance judgments. In: SIGIR 2003, pp. 361–362 (2003)Google Scholar
  6. 6.
    Carterette, B., Allan, J.: Incremental test collections. In: CIKM 2005, pp. 680–687 (2005)Google Scholar
  7. 7.
    Diaz, F.: Performance prediction using spatial autocorrelation. In: SIGIR 2007, pp. 583–590 (2007)Google Scholar
  8. 8.
    Efron, M.: Using multiple query aspects to build test collections without human relevance judgments. In: ECIR 2009, pp. 276–287 (2009)Google Scholar
  9. 9.
    Guiver, J., Mizzaro, S., Robertson, S.: A few good topics: Experiments in topic set reduction for retrieval evaluation. To appear in TOISGoogle Scholar
  10. 10.
    Krovetz, R.: Viewing morphology as an inference process. In: SIGIR 1993, pp. 191–202 (1993)Google Scholar
  11. 11.
    Mizzaro, S., Robertson, S.: Hits hits trec: exploring ir evaluation results with network analysis. In: SIGIR 2007, pp. 479–486 (2007)Google Scholar
  12. 12.
    Nuray, R., Can, F.: Automatic ranking of information retrieval systems using data fusion. Information Processing and Management 42(3), 595–614 (2006)zbMATHCrossRefGoogle Scholar
  13. 13.
    Soboroff, I., Nicholas, C., Cahan, P.: Ranking retrieval systems without relevance judgments. In: SIGIR 2001, pp. 66–73 (2001)Google Scholar
  14. 14.
    Spoerri, A.: Using the structure of overlap between search results to rank retrieval systems without relevance judgments. Information Processing and Management 43(4), 1059–1070 (2007)CrossRefGoogle Scholar
  15. 15.
    Voorhees, E.M.: Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing and Management 36, 697–716 (2000)CrossRefGoogle Scholar
  16. 16.
    Wu, S., Crestani, F.: Methods for ranking information retrieval systems without relevance judgments. In: Matsui, M., Zuccherato, R.J. (eds.) SAC 2003. LNCS, vol. 3006, pp. 811–816. Springer, Heidelberg (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Claudia Hauff
    • 1
  • Djoerd Hiemstra
    • 1
  • Leif Azzopardi
    • 2
  • Franciska de Jong
    • 1
  1. 1.University of TwenteEnschedeThe Netherlands
  2. 2.University of GlasgowGlasgowUK

Personalised recommendations