Advertisement

International Journal on Digital Libraries

, Volume 3, Issue 2, pp 131–139 | Cite as

A probabilistic justification for using tf×idf term weighting in information retrieval

  • Djoerd Hiemstra
Natural language processing for digital libraries

Abstract.

This paper presents a new probabilistic model of information retrieval. The most important modeling assumption made is that documents and queries are defined by an ordered sequence of single terms. This assumption is not made in well-known existing models of information retrieval, but is essential in the field of statistical natural language processing. Advances already made in statistical natural language processing will be used in this paper to formulate a probabilistic justification for using tf×idf term weighting. The paper shows that the new probabilistic interpretation of tf×idf term weighting might lead to better understanding of statistical ranking mechanisms, for example by explaining how they relate to coordination level ranking. A pilot experiment on the TREC collection shows that the linguistically motivated weighting algorithm outperforms the popular BM25 weighting algorithm.

Key words: Information retrieval theory – Statistical information retrieval – Statistical natural language processing 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag 2000

Authors and Affiliations

  • Djoerd Hiemstra
    • 1
  1. 1.Centre for Telematics and Information Technology, University of Twente, The Netherlands; E-mail: hiemstra@ctit.utwente.nlNL

Personalised recommendations