HFST Tools for Morphology – An Efficient Open-Source Package for Construction of Morphological Analyzers

  • Krister Lindén
  • Miikka Silfverberg
  • Tommi Pirinen
Part of the Communications in Computer and Information Science book series (CCIS, volume 41)

Abstract

Morphological analysis of a wide range of languages can be implemented efficiently using finite-state transducer technologies. Over the last 30 years, a number of attempts have been made to create tools for computational morphologies. The two main competing approaches have been parallel vs. cascaded rule application. The parallel rule application was originally introduced by Koskenniemi [1] and implemented in tools like TwolC and LexC. Currently many applications of morphologies could use dictionaries encoding the a priori likelihoods of words and expressions as well as the likelihood of relations to other representations or languages. We have made the choice to create open-source tools and language descriptions in order to let as many as possible participate in the effort. The current article presents some of the main tools that we have created such as HFST-LexC, HFST-TwolC and HFST-Compose-Intersect. We evaluate their efficiency in comparison to some similar tools and libraries. In particular, we evaluate them using several full-fledged morphological descriptions. Our tools compare well with similar open source tools, even if we still have some challenges ahead before we can catch up with the commercial tools. We demonstrate that for various reasons a parallel rule approach still seems to be more efficient than a cascaded rule approach when developing finite-state morphologies.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Koskenniemi, K.: Two-Level Morphology: A General Computational Model for Word-Form Recognition and Production. University of Helsinki, Department of General Linguistics (1983)Google Scholar
  2. 2.
    Karttunen, L., Koskenniemi, K., Kaplan, R.: A Compiler for Two-Level Phonological Rules. CSLI Publications (1987), http://www2.parc.com/istl/members/karttune/publications/archive/twolcomp.pdf
  3. 3.
    Karttunen, L.: Two-Level Rule Compiler, Technical Report ISTL-92-2, Xerox Palo Alto Research Center (1992), http://www.xrce.xerox.com/competencies/content-analysis/fssoft/docs/twolc-92/twolc92.html
  4. 4.
    Karttunen, L.: Finite-State Lexicon Compiler. Technical Report, ISTL-NLTT2993-04-02, Xerox Palo Alto Research Center, Palo Alto, California (1993)Google Scholar
  5. 5.
    Karttunen, L.: Constructing Lexical Transducers. In: The Proceedings of the 15th International Conference on Computational Linguistics COLING 1994, I, pp. 406–411 (1994)Google Scholar
  6. 6.
    Beesley, K., Karttunen, L.: Finite State Morphology. CSLI Publications, Stanford (2003), http://www.fsmbook.com Google Scholar
  7. 7.
    Schmid, H.: A programming language for finite state transducers. In: Yli-Jyrä, A., Karttunen, L., Karhumäki, J. (eds.) FSMNLP 2005. LNCS (LNAI), vol. 4002, pp. 308–309. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  8. 8.
    Lombardy, S., Régis-Gianas, Y., Sakharovitch, J.: Introducing Vaucanson. Theoretical Computer Science 328, 77–96 (2004)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Allauzen, C., Riley, M., Schalkwyk, J., Skut, W., Mohri, M.: OpenFst: A general and efficient weighted finite-state transducer library. In: Holub, J., Žďárek, J. (eds.) CIAA 2007. LNCS, vol. 4783, pp. 11–23. Springer, Heidelberg (2007), http://www.openfst.org CrossRefGoogle Scholar
  10. 10.
    Yli-Jyrä, A., Koskenniemi, K.: Compiling Generalized Two-Level Rules and Grammars. In: Salakoski, T., Ginter, F., Pyysalo, S., Pahikkala, T. (eds.) FinTAL 2006. LNCS (LNAI), vol. 4139, pp. 174–185. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  11. 11.
    Pirinen, T.: Suomen kielen äärellistilainen automaattinen morfologia avoimen lähdekoodin menetelmin. Master’s thesis, Helsingin yliopisto (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Krister Lindén
    • 1
  • Miikka Silfverberg
    • 1
  • Tommi Pirinen
    • 1
  1. 1.Department of General LinguisticsUniversity of HelsinkiFinland

Personalised recommendations