SPIRE 2003: String Processing and Information Retrieval pp 266-276 | Cite as
The Implementation and Evaluation of a Lexicon-Based Stemmer
Conference paper
Abstract
This paper describes a stemming technique that depends principally on a target language’s lexicon, organised as an automaton of word strings. The clear distinction between the lexicon and the procedure itself allows the stemmer to be customised for any language with little or even no changes to the program’s source code. An implementation of the stemmer, with a medium sized Portuguese lexicon is evaluated using Paice’s [16] evaluation method.
Keywords
Retrieval Performance Lexical Entry Input Word Stem Group Deterministic Finite Automaton
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
Preview
Unable to display preview. Download preview PDF.
References
- 1.Adamson, G., Boreham, J.: The Use of an Association Measure Based on Character Structure to Identify Semantically Related Pairs of Words and Document Titles. Information Storage and Retrieval 10 (1974)Google Scholar
- 2.Aronoff, M., Anshen, F.: Morphology and the Lexicon: Lexicalization and Productivity. In: Spencer, A., Zwicky, A. (eds.) The Handbook of Morphology, Blackwell Publishers, Malden (1998)Google Scholar
- 3.Couto, M.: Representação de Léxicos através de Autômatos Finitos. MsC Dissertation. ICMC Universidade de São Paulo, São Carlos (1999)Google Scholar
- 4.Crystal, D.: An Encyclopedic Dictionary of Language and Languages. Penguin, London (1992)Google Scholar
- 5.Dawson, J.: Suffix Removal and Word Conflation. ALLC Bulletin, Michelmas (1974)Google Scholar
- 6.Frakes, W.B.: Stemming Algorithms. In: Frakes, W.B., Baeza-Yates, R. (eds.) Information Retrieval: Data Structures & Algorithms, Prentice-Hall, Englewood Cliffs (1992)Google Scholar
- 7.Fuller, M., Zobel, J.: Conflation-based Comparison of Stemming Algorithms. In: Proc. of the Third Australian Document Computing Symposium, Sydney, Australia (1998)Google Scholar
- 8.Hafer, M., Weiss, S.: Word Segmentation by Letter Succession Varieties. Information Storage and Retrieval 10 (1974)Google Scholar
- 9.Harman, D.: How Effective is Suffixing? Journal of the American Society for Information Science 42(1) (1991)Google Scholar
- 10.Hull, D.A.: Stemming Algorithms: A Case Study for Detailed Evaluation. Journal of the American Society for Information Science 47(1), 70–84 (1996)CrossRefGoogle Scholar
- 11.Kowaltolwski, T., Lucchesi, C., Stolfi, J.: Finite Automata and Efficient Lexicon Implementation. Technical Report, Instituto de Computação, Universidade de Campinas (1998)Google Scholar
- 12.Krovetz, R.: Viewing morphology as an inference process. In: ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 191-202 (1993)Google Scholar
- 13.Lovins, J. B.: Development of a Stemming Algorithm. Mechanical Translation and Computational Linguistics 11(1&2) (1968) Google Scholar
- 14.Pacheco, H.: Uma Ferramenta de Auxílio à Redação. MsC Dissertation. Universidade Federal de Minas Gerais (1996) Google Scholar
- 15.Paice, C.: Another Stemmer. ACM Sigir Forum 24(3) (1990)Google Scholar
- 16.Paice, C.: An Evaluation Method for Stemming Algorithms. In: ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 42–50 (1994)Google Scholar
- 17.Porter, M.: An Algorithm for Suffix Stripping. Program 14(3) (1980) Google Scholar
- 18.Pequeno Vocabulário Ortográfico da Língua Portuguesa. Academia Brasileira de Letras, Rio de Janeiro (1999)Google Scholar
- 19.Revuz, D.: Minimisation of acyclic deterministic automata in linear time. Theoretical Computer Science 92 (1992)Google Scholar
- 20.Salton, G.: Automatic Information Organization and Retrieval. McGraw Hill, New York (1968)Google Scholar
- 21.Snowball, http://snowball.tartarus.org
Copyright information
© Springer-Verlag Berlin Heidelberg 2003