Putting Successor Variety Stemming to Work
Stemming algorithms find canonical forms for inflected words, e. g. for declined nouns or conjugated verbs. Since such a unification of words with respect to gender, number, time, and case is a language-specific issue, stemming algorithms operationalize a set of linguistically motivated rules for the language in question. The most well-known rule-based algorithm for the English language is from [Porter (1980)].
The paper presents a statistical stemming approach which is based on the analysis of the distribution of word prefixes in a document collection, and which thus is widely language-independent. In particular, our approach tackles the problem of index construction for multi-lingual documents. Related work for statistical stemming either focuses on stemming quality (such as [Bachin et al. (2002) or Bordag (2005)]) or investigates runtime performance ([Mayfield and McNamee (2003)] for example), but neither provides a reasonable tradeoff between both. For selected retrieval tasks under vector-based document models we report on new results related to stemming quality and collection size dependency.
Interestingly, successor variety stemming has neither been investigated under similarity concerns for index construction nor is it applied as a technology in current retrieval applications. The results show that this disregard is not justified.
KeywordsCategorization Task Vector Space Model Successor Variety Compound Word Document Model
Unable to display preview. Download preview PDF.
- ABDOU, S., RUCH, P. and SAVOY, J. (2005): Evaluation of Stemming, Query Expansion and Manual Indexing Approaches for the Genomic Task. In: Proc.TREC’05.Google Scholar
- BACCHIN, M., FERRO, N. and MELUCCI, M. (2002): Experiments to Evaluate a Statistical Stemming Algorithm. In: Proc. of CLEF’02.Google Scholar
- BORDAG, S. (2005): Unsupervised Knowledge-free Morpheme Boundary Detection. In: Proc. of RANLP’05.Google Scholar
- FRAKES, W.B. (1984): Term Conflation for Information Retrieval. In: Proc. SIGIR’84.Google Scholar
- FRAKES, W.B. and BAEZA-YATES, R. (1992): Information Retrieval: Data Structures and Algorithms. Prentice-Hall, Upper Saddle River.Google Scholar
- GUSFIELD, D. (1997): Algorithms on Strings, Trees, and Sequences: Computer Science and Computational Biology. Cambridge University Press.Google Scholar
- HARMAN, D. (1991): How Effective is Suffixing? J. of the ASIS&T., 42,1, 7–15.Google Scholar
- KROVETZ, R. (1993): Viewing Morphology as an Inference Process. In: Proc. SIGIR’93.Google Scholar
- LOVINS, J.B. (1968): Development of a Stemming Algorithm. Mechanical Translation and Computation Linguistics, 11,1, 23–31.Google Scholar
- MAYFIELD, J. and MCNAMEE, P. (2003): Single n-gram Stemming. In: Proc. SIGIR’03.Google Scholar
- PORTER, M. (2001): Snowball. http://snowball.tartarus.org/.Google Scholar
- ROSE, T.G., STEVENSON, M. and WHITEHEAD, M. (2002): The Reuters Corpus Volume 1-From Yesterday’s News to Tomorrow’s Language Resources. In: Proc. of LREC’02.Google Scholar
- STEIN, B., MEYER ZU EISSEN, S. and WISSBROCK, F. (2003): On Cluster Validity and the Information Need of Users. In: Proc. of AIA’03.Google Scholar
- WEINER, P. (1973): Linear Pattern Matching Algorithm. In: Proc. of the 14th IEEE Symp. on Switching and Automata Theory.Google Scholar