Division of Spanish Words into Morphemes with a Genetic Algorithm

  • Alexander Gelbukh
  • Grigori Sidorov
  • Diego Lara-Reyes
  • Liliana Chanona-Hernandez
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5039)

Abstract

We discuss an unsupervised technique for determining morpheme structure of words in an inflective language, with Spanish as a case study. For this, we use a global optimization (implemented with a genetic algorithm), while most of the previous works are based on heuristics calculated using conditional probabilities of word parts. Thus, we deal with complete space of solutions and do not reduce it with the risk to eliminate some correct solutions beforehand. Also, we are working at the derivative level as contrasted with the more traditional grammatical level interested only in flexions. The algorithm works as follows. The input data is a wordlist built on the base of a large dictionary or corpus in the given language and the output data is the same wordlist with each word divided into morphemes. First, we build a redundant list of all strings that might possibly be prefixes, suffixes, and stems of the words in the wordlist. Then, we detect possible paradigms in this set and filter out all items from the lists of possible prefixes and suffixes (though not stems) that do not participate in such paradigms. Finally, a subset of those lists of possible prefixes, stems, and suffixes is chosen using the genetic algorithm. The fitness function is based on the ideas of minimum length description, i.e. we choose the minimum number of elements that are necessary for covering all the words. The obtained subset is used for dividing the words from the wordlist. Algorithm parameters are presented. Preliminary evaluation of the experimental results for a dictionary of Spanish is given.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Baroni, M., Matiasek, J., Trost, H.: Unsupervised discovery of morphologically related words based on orthographic and semantic similarity. In: ACL Workshop on Morphological and Phonological Learning (2002)Google Scholar
  2. 2.
    Gelbukh, A., Alexandrov, M., Han, S.: Detecting Inflection Patterns in Natural Language by Minimization of Morphological Model. In: Sanfeliu, A., Martínez Trinidad, J.F., Carrasco Ochoa, J.A. (eds.) CIARP 2004. LNCS, vol. 3287, pp. 432–438. Springer, Heidelberg (2004)Google Scholar
  3. 3.
    Goldsmith, J.: Unsupervised Learning of the Morphology of a Natural Language. Computational Linguistics 27(2), 153–198 (2001)CrossRefMathSciNetGoogle Scholar
  4. 4.
    Creutz, M.: Unsupervised Segmentation of Words Using Prior Distributions of Morph Length and Frequency. In: Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL 2003), Sapporo, Japan, pp. 280–287 (2003)Google Scholar
  5. 5.
    Haahr, P., Baker, S.: Making search better in Catalonia, Estonia, and everywhere else. Google blog (2007), http://googleblog.blogspot.com/2008/03/making-search-better-in-catalonia.htm
  6. 6.
    Kazakov, D.: Unsupervised learning of naıve morphology with genetic algorithms. In: Workshop Notes of the ECML/MLnet Workshop on Empirical Learning of Natural Language Processing Tasks. Prague, Czech Republic, pp. 105–112 (1997)Google Scholar
  7. 7.
    Rehman, K., Hussain, I.: Unsupervised Morphemes Segmentation. In: Pascal Morphochallenge, 5p. (2005)Google Scholar
  8. 8.
    Pascal Morphochallenge (2007), http://www.cis.hut.fi/morphochallenge2007/
  9. 9.
    Pascal Morphochallenge (2005), http://www.cis.hut.fi/morphochallenge2005/

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Alexander Gelbukh
    • 1
  • Grigori Sidorov
    • 1
  • Diego Lara-Reyes
    • 1
  • Liliana Chanona-Hernandez
    • 2
  1. 1.Natural Language and Text Processing Laboratory, Center for Research in Computer ScienceNational Polytechnic InstituteMexico CityMexico
  2. 2.ESIME ZacatencoNational Polytechnic InstituteMexico CityMexico

Personalised recommendations