Best and Fairest: An Empirical Analysis of Retrieval System Bias

  • Colin Wilkie
  • Leif Azzopardi
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8416)


In this paper, we explore the bias of term weighting schemes used by retrieval models. Here, we consider bias as the extent to which a retrieval model unduly favours certain documents over others because of characteristics within and about the document. We set out to find the least biased retrieval model/weighting. This is largely motivated by the recent proposal of a new suite of retrieval models based on the Divergence From Independence (DFI) framework. The claim is that such models provide the fairest term weighting because they do not make assumptions about the term distribution (unlike most other retrieval models). In this paper, we empirically examine whether fairness is linked to performance and answer the question; is fairer better?


Language Model Retrieval System Term Frequency Retrieval Model Vector Space Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Amati, G., Van Rijsbergen, C.J.: Probabilistic models of ir based on measuring the divergence from randomness. ACM Trans. on Info. Sys., 357–389 (2002)Google Scholar
  2. 2.
    Azzopardi, L., Bache, R.: On the relationship between effectiveness and accessibility. In: Proc. of the 33rd international ACM SIGIR, pp. 889–890 (2010)Google Scholar
  3. 3.
    Azzopardi, L., Vinay, V.: Retrievability: An evaluation measure for higher order information access tasks. In: Proc. of the 17th ACM CIKM, pp. 561–570 (2008)Google Scholar
  4. 4.
    Bashir, S., Rauber, A.: On the relationship bw query characteristics and ir functions retrieval bias. J. Am. Soc. Inf. Sci. Technol. 62(8), 1515–1532 (2011)CrossRefGoogle Scholar
  5. 5.
    Clinchant, S., Gaussier, E.: Bridging language modeling and divergence from randomness models: A log-logistic model for ir. In: Azzopardi, L., Kazai, G., Robertson, S., Rüger, S., Shokouhi, M., Song, D., Yilmaz, E. (eds.) ICTIR 2009. LNCS, vol. 5766, pp. 54–65. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  6. 6.
    Crestani, F., Lalmas, M., Van Rijsbergen, C.J., Campbell, I.: Is this document relevant? probably: a survey of probabilistic models in information retrieval. ACM Computing Survey 30(4), 528–552 (1998)CrossRefGoogle Scholar
  7. 7.
    Dinçer, B.T., Kocabas, I., Karaoglan, B.: Irra at trec 2010: Index term weighting by divergence from independence model. In: TREC (2010)Google Scholar
  8. 8.
    Fang, H., Tao, T., Zhai, C.: A formal study of information retrieval heuristics. In: Proc. of the 27th ACM SIGIR Conference, SIGIR 2004, pp. 49–56 (2004)Google Scholar
  9. 9.
    Fuhr, N.: Probabilistic models in ir. Computer Journal 35(3), 243–255 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Gastwirth, J.: The estimation of the lorenz curve and gini index. The Review of Economics and Statistics 54, 306–316 (1972)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Harter, S.P.: A probabilistic approach to automatic keyword indexing. part i. on the distribution of specialty words in a technical literature. Journal of the American Society for Information Science 26(4), 197–206 (1975)CrossRefGoogle Scholar
  12. 12.
    Hiemstra, D.: A probabilistic justification for using tf.idf term weighting in information retrieval. International Journal on Digital Libraries 3(2), 131–139 (2000)CrossRefGoogle Scholar
  13. 13.
    Jones, K.S., Walker, S., Robertson, S.E.: A probabilistic model of information retrieval: development and comparative experiments (parts 1 and 2). Information Processing and Management 36(6), 779–808 (2000)CrossRefGoogle Scholar
  14. 14.
    Kocabas, I., Dinçer, B.T., Karaoglan, B.: A nonparametric term weighting method for information retrieval based on measuring the divergence from independence. Information Retrieval, 1–24 (2013)Google Scholar
  15. 15.
    Losada, D.E., Azzopardi, L., Baillie, M.: Revisiting the relationship between doc. length and relevance. In: Proc. of the 17th ACM CIKM 2008, pp. 419–428 (2008)Google Scholar
  16. 16.
    Maron, M.E., Kuhns, J.L.: On relevance, probabilistic indexing and information retrieval. Journal of the ACM 7(3), 216–244 (1960)CrossRefGoogle Scholar
  17. 17.
    Ponte, J.M., Croft, W.B.: A language modeling approach to information retrieval. In: Proc. of the 21st ACM SIGIR Conference, SIGIR 1998, pp. 275–281 (1998)Google Scholar
  18. 18.
    Robertson, S.E., Walker, S.: Some simple effective approx. to the 2-poisson model for probabilistic weighted retrieval. In: Proc. of ACM SIGIR 1994, pp. 232–241 (1994)Google Scholar
  19. 19.
    Salton, G., Wong, A., Yang, C.S.: A vector space model for automatic indexing. Communications of the ACM 18(11), 613–620 (1975)CrossRefzbMATHGoogle Scholar
  20. 20.
    Salton, G.: Automatic Information Organization and Retrieval (1968)Google Scholar
  21. 21.
    Singhal, A., Buckley, C., Mitra, M.: Pivoted document length normalization. In: Proce. of the 19th ACM SIGIR Conference, SIGIR 1996, pp. 21–29 (1996)Google Scholar
  22. 22.
    Wilkie, C., Azzopardi, L.: Relating retrievability, performance and length. In: Proc. of the 36th ACM SIGIR Conference, SIGIR 2013, pp. 937–940 (2013)Google Scholar
  23. 23.
    Zhai, C., Lafferty, J.: A study of smoothing methods for language models applied to ad hoc ir. In: Proc. of the 24th ACM SIGIR, pp. 334–342 (2001)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Colin Wilkie
    • 1
  • Leif Azzopardi
    • 1
  1. 1.School of Computing ScienceUniversity of GlasgowScotland, UK

Personalised recommendations