Exploring the Impact of Inter-query Variability on the Performance of Retrieval Systems

  • Francesco Brughi
  • Debora Gil
  • Llorenç Badiella
  • Eva Jove Casabella
  • Oriol Ramos Terrades
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8814)


This paper introduces a framework for evaluating the performance of information retrieval systems. Current evaluation metrics provide an average score that does not consider performance variability across the query set. In this manner, conclusions lack of any statistical significance, yielding poor inference to cases outside the query set and possibly unfair comparisons. We propose to apply statistical methods in order to obtain a more informative measure for problems in which different query classes can be identified. In this context, we assess the performance variability on two levels: overall variability across the whole query set and specific query class-related variability. To this end, we estimate confidence bands for precision-recall curves, and we apply ANOVA in order to assess the significance of the performance across different query classes.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Everingham, M., Ali Eslami, S.M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: Assessing the significance of performance differences on the pascal voc challenges via bootstrapping. Tech. rep. (2013)Google Scholar
  2. 2.
    Bertail, P., Clemencon, S., Vayatis, N.: On bootstrapping the roc curve. In: NIPS, pp. 137–144. Curran Associates Inc. (2008)Google Scholar
  3. 3.
    Clémençon, S., Vayatis, N.: Nonparametric estimation of the precision-recall curve. In: ICML, pp. 185–192 (2009)Google Scholar
  4. 4.
    Macskassy, S.A., Provost, F.J.: Confidence bands for roc curves: Methods and an empirical study. In: ROCAI, pp. 61–70 (2004)Google Scholar
  5. 5.
    Brughi, F., Gil, D., Ramos Terrades, O.: Artistic heritage motive retrieval: an explorative study. Tech. rep. (2013)Google Scholar
  6. 6.
    Crowley, E.J., Zisserman, A.: Of gods and goats: Weakly supervised learning of figurative art. In: BMVC (2013)Google Scholar
  7. 7.
    Bamber, D.: The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J. Math. Psy. 12, 387–415 (1975)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    DeLong, E.R., DeLong, D.M., Clarke-Pearson, D.L.: Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics 44, 837–845 (1988)CrossRefzbMATHGoogle Scholar
  9. 9.
    Wieand, S., Gail, M.H., James, B.R., James, K.L.: A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika 76(3), 585–592 (1989)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. Cambridge University Press (2008)Google Scholar
  11. 11.
    Davis, J., Goadrich, M.: The relationship between precision-recall and roc curves. In: ICML, pp. 233–240 (2006)Google Scholar
  12. 12.
    Casella, G., Berger, R.: Statistical inference. Duxbury Press (1990)Google Scholar
  13. 13.
    Hochberg, Y., Tamhane, A.C.: Multiple Comparison Procedures. John Wiley & Sons Inc. (1987)Google Scholar
  14. 14.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. IJCV 60(2), 91–110 (2004)CrossRefGoogle Scholar
  15. 15.
    Sivic, J., Zisserman, A.: Video google: A text retrieval approach to object matching in videos. In: ICCV, pp. 1470–1477 (2003)Google Scholar
  16. 16.
    Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval with large vocabularies and fast spatial matching. In: CVPR (2007)Google Scholar
  17. 17.
    Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Lost in quantization: Improving particular object retrieval in large scale image databases. In: CVPR, pp. 1–8 (2008)Google Scholar
  18. 18.
    Lebeda, K., Matas, J., Chum, O.: Fixing the locally optimized ransac. In: BMVC, pp. 1–11 (2012)Google Scholar
  19. 19.
    Badiella, L., Puig, P., Leton, E.: Evaluacion diagnostica mediante curvas roc. Tech. rep. (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Francesco Brughi
    • 1
  • Debora Gil
    • 1
  • Llorenç Badiella
    • 2
  • Eva Jove Casabella
    • 3
  • Oriol Ramos Terrades
    • 1
  1. 1.Department Ciències de la Computació, Computer Vision CenterUniv. Autónoma de BarcelonaBarcelonaSpain
  2. 2.Servei de Estadística AplicadaUniv. Autónoma de BarcelonaBarcelonaSpain
  3. 3.Department História i História de l’ArtUniv. de GironaGironaSpain

Personalised recommendations