Tie-Breaking Bias: Effect of an Uncontrolled Parameter on Information Retrieval Evaluation
- Cite this paper as:
- Cabanac G., Hubert G., Boughanem M., Chrisment C. (2010) Tie-Breaking Bias: Effect of an Uncontrolled Parameter on Information Retrieval Evaluation. In: Agosti M., Ferro N., Peters C., de Rijke M., Smeaton A. (eds) Multilingual and Multimodal Information Access Evaluation. CLEF 2010. Lecture Notes in Computer Science, vol 6360. Springer, Berlin, Heidelberg
We consider Information Retrieval evaluation, especially at Trec with the trec_eval program. It appears that systems obtain scores regarding not only the relevance of retrieved documents, but also according to document names in case of ties (i.e., when they are retrieved with the same score). We consider this tie-breaking strategy as an uncontrolled parameter influencing measure scores, and argue the case for fairer tie-breaking strategies. A study of 22 Trec editions reveals significant differences between the Conventional unfair Trec’s strategy and the fairer strategies we propose. This experimental result advocates using these fairer strategies when conducting evaluations.
Unable to display preview. Download preview PDF.