A Probabilistic Model for Guessing Base Forms of New Words by Analogy

  • Krister Lindén
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4919)


Language software applications encounter new words, e.g., acronyms, technical terminology, loan words, names or compounds of such words. Looking at English, one might assume that they appear in base form, i.e., the lexical look-up form. However, in more highly inflecting languages like Finnish or Swahili only 40-50 % of new words appear in base form. In order to index documents or discover translations for these languages, it would be useful to reduce new words to their base forms as well. We often have access to analyzes for more frequent words which shape our intuition for how new words will inflect. We formalize this into a probabilistic model for lemmatization of new words using analogy, i.e., guessing base forms, and test the model on English, Finnish, Swedish and Swahili demonstrating that we get a recall of 89-99 % with an average precision of 76-94 % depending on language and the amount of training material.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Creutz, M., Lagus, K., Lindén, K., Virpioja, S.: Morfessor and Hutmegs: Unsupervised Morpheme Segmentation for Highly-Inflecting and Compounding Languages. In: Proceedings of the Second Baltic Conference on Human Language Technologies, Tallinn, Estonia, April 4–8 (2005)Google Scholar
  2. 2.
    Goldsmith, J.: Morphological Analogy: Only a Beginning (2007), http://hum.uchicago.edu/~jagoldsm/Papers/analogy.pdf
  3. 3.
    Kuenning, G.: Dictionaries for International Ispell (2007), http://www.lasr.cs.ucla.edu/geoff/ispell-dictionaries.html
  4. 4.
    Kurimo, M., Creutz, M., Turunen, V.: Overview of Morpho Challenge in CLEF 200. In: Nardi, A., Peters, C. (eds.) Working Notes of the CLEF 2007 Workshop, udapest, Hungary, September 19–21 (2007)Google Scholar
  5. 5.
    Lindén, K.: Multilingual Modeling of Cross-lingual Spelling Variants. Journal of Information Retrieval 9, 295–310 (2006)CrossRefGoogle Scholar
  6. 6.
    Allauzen, C., Riley, M., Schalkwyk, J., Skut, W., Mohri, M.: OpenFst: A General and Efficient Weighted Finite-State Transducer Library. LNCS (to appear)Google Scholar
  7. 7.
    Lombardy, S., Régis-Gianas, Y., Sakarovitch, J.: Introducing Vaucanson. Theoretical Computer Science 328, 77–96 (2004)MATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Wicentowski, R.: Modeling and Learning Multilingual Inflectional Morphology in a Minimally Supervised Framework. PhD Thesis. Baltimore, Maryland (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Krister Lindén
    • 1
  1. 1.Department of General LinguisticsUniversity of Helsinki 

Personalised recommendations