Tie-Breaking Bias: Effect of an Uncontrolled Parameter on Information Retrieval Evaluation

  • Guillaume Cabanac
  • Gilles Hubert
  • Mohand Boughanem
  • Claude Chrisment
Conference paper

DOI: 10.1007/978-3-642-15998-5_13

Part of the Lecture Notes in Computer Science book series (LNCS, volume 6360)
Cite this paper as:
Cabanac G., Hubert G., Boughanem M., Chrisment C. (2010) Tie-Breaking Bias: Effect of an Uncontrolled Parameter on Information Retrieval Evaluation. In: Agosti M., Ferro N., Peters C., de Rijke M., Smeaton A. (eds) Multilingual and Multimodal Information Access Evaluation. CLEF 2010. Lecture Notes in Computer Science, vol 6360. Springer, Berlin, Heidelberg

Abstract

We consider Information Retrieval evaluation, especially at Trec with the trec_eval program. It appears that systems obtain scores regarding not only the relevance of retrieved documents, but also according to document names in case of ties (i.e., when they are retrieved with the same score). We consider this tie-breaking strategy as an uncontrolled parameter influencing measure scores, and argue the case for fairer tie-breaking strategies. A study of 22 Trec editions reveals significant differences between the Conventional unfair Trec’s strategy and the fairer strategies we propose. This experimental result advocates using these fairer strategies when conducting evaluations.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Guillaume Cabanac
    • 1
  • Gilles Hubert
    • 1
  • Mohand Boughanem
    • 1
  • Claude Chrisment
    • 1
  1. 1.Université de Toulouse — IRIT UMR 5505 CNRSToulouse cedex 9

Personalised recommendations