The Effiectiveness of a Graph-Based Algorithm for Stemming
In Information Retrieval (IR), stemming enables a matching of query and document terms which are related to a same meaning but which can appear in different morphological variants. In this paper we will propose and evaluate a statistical graph-based algorithm for stemming. Considering that a word is formed by a stem (prefix) and a derivation (suffix), the key idea is that strongly interlinked prefixes and suffixes form a community of sub-strings. Discovering these communities means searching for the best word splits which give the best word stems. We conducted some experiments on CLEF 2001 test subcollections for Italian language. The results show that stemming improve the IR effectiveness. They also show that effectiveness level of our algorithm is comparable to that of an algorithm based on a-priori linguistic knowledge. This is an encouraging result, particularly in a multi-lingual context.
- 1.M. Agosti, M. Bacchin, N. Ferro and M. Melucci. University of Padua at CLEF 2002: Experiments to evaluate a statistical stemming algorithm. In Cross-Language Information Retrieval and Evaluation: Proceedings of the CLEF 2002 workshop, Lecture Notes in Computer Science series, Springer Verlag (forthcoming).Google Scholar
- 4.C. Cleverdon. The Cranfield Tests on Index Language Devices. In K. Sparck Jones and P. Willett (Eds.). Readings in Information Retrieval, pages 47–59, Morgan Kaufmann, 1997.Google Scholar
- 5.W.B. Frakes and R. Baeza-Yates. Information Retrieval: data structures and algorithms. Prentice Hall, 1992.Google Scholar
- 10.R. Krovetz. Viewing Morphology as an Inference Process,. In Proceedings of the ACM International Conference on Research and Development in Information Retrieval (SIGIR), 1993.Google Scholar
- 11.J. Lovins. Development of a stemming algorithm. Mechanical Translation and Computational Linguistics, 11:22–31, 1968.Google Scholar
- 13.C.D. Manning and H. Schütze. Foundations of statistical natural language processing. The MIT Press, 1999.Google Scholar
- 17.M. Porter. Snowball: A language for stemming algorithms. http://snowball.sourceforge.net, 2001.
- 18.M.F. Porter. An algorithm for suffix stripping. Program, 14(3):130–137, 1980.Google Scholar
- 19.G. Salton and M. McGill. Introduction to modern Information Retrieval. McGraw-Hill, New York, NY, 1983.Google Scholar
- 21.Institut interfacultaire d’informatique. CLEF and Multilingual information retrieval. University of Neuchatel. http://www.unine.ch/info/clef/, 2002.
- 22.C. Buckley. Trec eval. ftp://ftp.cs.cornell.edu/pub/smart/, 2002.