Document Length Normalization Using Effective Level of Term Frequency in Large Collections

  • Soheila Karbasi
  • Mohand Boughanem
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3936)


The effectiveness of the information retrieval systems is largely dependent on term-weighting. Most current term-weighting approaches involve the use of term frequency normalization. We develop here a method to assess the potential role of the term frequency-inverse document frequency measures that are commonly used in text retrieval systems. Since automatic information retrieval systems have to deal with documents of varying sizes and terms of varying frequencies, we carried out preliminary tests to evaluate the effect of term-weighing items on the retrieval performance. With regard to the preliminary tests, we identify a novel factor (effective level of term frequency) that represents the document content based on its length and maximum term-frequency. This factor is used to find the maximum main terms within the documents and an appropriate subset of documents containing the query terms. We show that, all document terms need not be considered for ranking a document with respect to a query. Regarding the result of the experiments, the effective level of term frequency (EL) is a significant factor in retrieving relevant documents, especially in large collections. Experiments were under-taken on TREC collections to evaluate the effectiveness of our proposal.


Average Precision Term Frequency Query Term Information Retrieval System Test Collection 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw-Hill, New York (1983)zbMATHGoogle Scholar
  2. 2.
    van Rijsbergen, C.J.: Information retrieval. Butterworths (1979)Google Scholar
  3. 3.
    Salton, G., Buckley, C.: Term-Weighting Approaches in Automatic Text Retrieval. Information Processing & Management 24(5), 513–523 (1988)CrossRefGoogle Scholar
  4. 4.
    Luhn, H.P.: The Automatic Creation of Literature Abstracts. IBM Journal of Research and Development 2(2), 159–165, 317 (1958)MathSciNetCrossRefGoogle Scholar
  5. 5.
    He, B., Ounis, I.: Term frequency normalisation tuning for BM25 and DFR models. In: Losada, D.E., Fernández-Luna, J.M. (eds.) ECIR 2005. LNCS, vol. 3408, pp. 200–214. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  6. 6.
    Robertson, S., Walker, S., Beaulieu, M.M., Gatford, M., Payne, A.: Okapi at trec-4. In: The Fourth Text Retrieval Conference (TREC-4), pp. 73–96. NIST Special Publication 500-236 (1995)Google Scholar
  7. 7.
    Buckley, C., Singhal, A., Mitra, M., Salton, G.: New retrieval approaches using SMART. In: Proceedings of TREC-4, pp. 25–48. NIST Publication #500-236, Gaithersburg (1996)Google Scholar
  8. 8.
    Singhal, A., Choi, J., Hindle, D., Lewis, D.D., Pereira, F.: AT&T at TREC-7. In: Proceedings of TREC-7, pp. 239–251. NIST Publication #500-242, Gaithersburg (1999)Google Scholar
  9. 9.
    Singhal, A., Buckley, C., Mitra, M.: Pivoted document length normalization. In: Proceedings of the 19th Annual International ACM SIGIR Conference on Researchand Development in Information Retrieval, pp. 21–29 (1996)Google Scholar
  10. 10.
    He, B., Ounis, I.: A study of parameter tuning for term frequency normalization. In: Proceedings of the twelfth international conference on Information and knowledge management, New Orleans, LA, USA (2003)Google Scholar
  11. 11.
    Fang, H., Tao, T., Zhai, C.: A formal study of information retrieval heuristics. In: SIGIR 2004, pp. 49–56 (2004)Google Scholar
  12. 12.
    Boughanem, M., Chrisment, C., Soulé-Dupuy, C.: Query Modification based on relevance back-propagation in adhoc environnement. Information Processing & Management 35, 121–139 (1999)CrossRefGoogle Scholar
  13. 13.
    Robertson, S.E., Walker, S.: Okapi/Keenbow at TREC- 8. In: Voorhees, E.M., Harman, D.K. (eds.) The Eighth Text Retrieval Conference (TREC-8), pp. 151–162. NIST Special Publication 500-246, Gaithersburg (2000)Google Scholar
  14. 14.
    Craswell, N., Hawking, D.: Overview of the TREC-2002 Web Track. In: NIST Special Publication SP 500-251: The 11 th Text Retrieval Conference (TREC 2002), Gaithersburg, Maryland, USA (2002)Google Scholar
  15. 15.
    Bailey, P., Craswell, N., Hawking, D.: Engineering a multipurpose test collection for web retrieval experiments draft. In: Proceedings of the 24th annual international ACM SIGIR conference (2001)Google Scholar
  16. 16.
    Singhal, A., Salton, G., Mitra, M., Buckley, C.: Document length normalization. Information Processing & Management 32, 619–633 (1996)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Soheila Karbasi
    • 1
  • Mohand Boughanem
    • 1
  1. 1.IRIT-SIGToulouseFrance

Personalised recommendations