Abstract
The effectiveness of the information retrieval systems is largely dependent on term-weighting. Most current term-weighting approaches involve the use of term frequency normalization. We develop here a method to assess the potential role of the term frequency-inverse document frequency measures that are commonly used in text retrieval systems. Since automatic information retrieval systems have to deal with documents of varying sizes and terms of varying frequencies, we carried out preliminary tests to evaluate the effect of term-weighing items on the retrieval performance. With regard to the preliminary tests, we identify a novel factor (effective level of term frequency) that represents the document content based on its length and maximum term-frequency. This factor is used to find the maximum main terms within the documents and an appropriate subset of documents containing the query terms. We show that, all document terms need not be considered for ranking a document with respect to a query. Regarding the result of the experiments, the effective level of term frequency (EL) is a significant factor in retrieving relevant documents, especially in large collections. Experiments were under-taken on TREC collections to evaluate the effectiveness of our proposal.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw-Hill, New York (1983)
van Rijsbergen, C.J.: Information retrieval. Butterworths (1979)
Salton, G., Buckley, C.: Term-Weighting Approaches in Automatic Text Retrieval. Information Processing & Management 24(5), 513–523 (1988)
Luhn, H.P.: The Automatic Creation of Literature Abstracts. IBM Journal of Research and Development 2(2), 159–165, 317 (1958)
He, B., Ounis, I.: Term frequency normalisation tuning for BM25 and DFR models. In: Losada, D.E., Fernández-Luna, J.M. (eds.) ECIR 2005. LNCS, vol. 3408, pp. 200–214. Springer, Heidelberg (2005)
Robertson, S., Walker, S., Beaulieu, M.M., Gatford, M., Payne, A.: Okapi at trec-4. In: The Fourth Text Retrieval Conference (TREC-4), pp. 73–96. NIST Special Publication 500-236 (1995)
Buckley, C., Singhal, A., Mitra, M., Salton, G.: New retrieval approaches using SMART. In: Proceedings of TREC-4, pp. 25–48. NIST Publication #500-236, Gaithersburg (1996)
Singhal, A., Choi, J., Hindle, D., Lewis, D.D., Pereira, F.: AT&T at TREC-7. In: Proceedings of TREC-7, pp. 239–251. NIST Publication #500-242, Gaithersburg (1999)
Singhal, A., Buckley, C., Mitra, M.: Pivoted document length normalization. In: Proceedings of the 19th Annual International ACM SIGIR Conference on Researchand Development in Information Retrieval, pp. 21–29 (1996)
He, B., Ounis, I.: A study of parameter tuning for term frequency normalization. In: Proceedings of the twelfth international conference on Information and knowledge management, New Orleans, LA, USA (2003)
Fang, H., Tao, T., Zhai, C.: A formal study of information retrieval heuristics. In: SIGIR 2004, pp. 49–56 (2004)
Boughanem, M., Chrisment, C., Soulé-Dupuy, C.: Query Modification based on relevance back-propagation in adhoc environnement. Information Processing & Management 35, 121–139 (1999)
Robertson, S.E., Walker, S.: Okapi/Keenbow at TREC- 8. In: Voorhees, E.M., Harman, D.K. (eds.) The Eighth Text Retrieval Conference (TREC-8), pp. 151–162. NIST Special Publication 500-246, Gaithersburg (2000)
Craswell, N., Hawking, D.: Overview of the TREC-2002 Web Track. In: NIST Special Publication SP 500-251: The 11 th Text Retrieval Conference (TREC 2002), Gaithersburg, Maryland, USA (2002)
Bailey, P., Craswell, N., Hawking, D.: Engineering a multipurpose test collection for web retrieval experiments draft. In: Proceedings of the 24th annual international ACM SIGIR conference (2001)
Singhal, A., Salton, G., Mitra, M., Buckley, C.: Document length normalization. Information Processing & Management 32, 619–633 (1996)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Karbasi, S., Boughanem, M. (2006). Document Length Normalization Using Effective Level of Term Frequency in Large Collections. In: Lalmas, M., MacFarlane, A., Rüger, S., Tombros, A., Tsikrika, T., Yavlinsky, A. (eds) Advances in Information Retrieval. ECIR 2006. Lecture Notes in Computer Science, vol 3936. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11735106_8
Download citation
DOI: https://doi.org/10.1007/11735106_8
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-33347-0
Online ISBN: 978-3-540-33348-7
eBook Packages: Computer ScienceComputer Science (R0)