HFST — A System for Creating NLP Tools

  • Krister Lindén
  • Erik Axelson
  • Senka Drobac
  • Sam Hardwick
  • Juha Kuokkala
  • Jyrki Niemi
  • Tommi A Pirinen
  • Miikka Silfverberg
Part of the Communications in Computer and Information Science book series (CCIS, volume 380)

Abstract

The paper presents and evaluates various NLP tools that have been created using the open source library HFST – Helsinki Finite-State Technology and outlines the minimal extensions that this has required to a pure finite-state system. In particular, the paper describes an implementation and application of Pmatch presented by Karttunen at SFCM 2011.

Keywords

finite-state technology language identification morphological guessers spell-checking named-entity recognition language generation parsing HFST XFST Pmatch 

References

  1. 1.
    Bouma, G., Noord, G.V., Malouf, R.: Alpino: Wide-coverage computational analysis of Dutch. In: CLIN 2000, vol. 8, pp. 45–59. Rodopi (2000)Google Scholar
  2. 2.
    Brants, S., Dipper, S., Hansen, S., Lezius, W., Smith, G.: The TIGER treebank. In: Proceedings of the Workshop on Treebanks and Linguistic Theories. Sozopol (2002)Google Scholar
  3. 3.
    Cavnar, W.B., Trenkle, J.M.: N-gram-based text categorization. In: Proceedings of SDAIR 1994, 3rd Annual Symposium on Document Analysis and Information Retrieval, pp. 161–175 (1994)Google Scholar
  4. 4.
    Drobac, S., Silfverberg, M., Yli-Jyrä, A.: Implementation of replace rules using preference operator. In: Proceedings of the 10th International Workshop on Finite State Methods and Natural Language Processing, pp. 55–59. Association for Computational Linguistics, Donostia–San Sebastián (2012), http://www.aclweb.org/anthology/W12-6210 Google Scholar
  5. 5.
    Einarsson, J.: Talbankens skriftspråkskonkordans. Lund University (1976)Google Scholar
  6. 6.
    Karlsson, F.: Constraint grammar as a framework for parsing running text. In: Karlgren, H. (ed.) Proceedings of the 13th Conference on Computational linguistics, COLING 1990, vol. 3, pp. 168–173. Association for Computational Linguistics, Stroudsburg (1990), http://dx.doi.org/10.3115/991146.991176 CrossRefGoogle Scholar
  7. 7.
    Karttunen, L.: Beyond morphology: Pattern matching with FST. In: Mahlow, C., Piotrowski, M. (eds.) SFCM 2011. CCIS, vol. 100, pp. 1–13. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  8. 8.
    Kokkinakis, D.: Swedish NER in the Nomen Nescio project. In: Holmboe, H. (ed.) Nordisk Sprogteknologi – Nordic Language Technology 2002, pp. 379–398. Museum Tusculanums Forlag, Copenhagen (2003)Google Scholar
  9. 9.
    Lindén, K., Axelson, E., Hardwick, S., Pirinen, T.A., Silfverberg, M.: HFST—framework for compiling and applying morphologies. In: Mahlow, C., Piotrowski, M. (eds.) SFCM 2011. CCIS, vol. 100, pp. 67–85. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  10. 10.
    Lindén, K., Pirinen, T.: Weighted finite-state morphological analysis of Finnish compounds. In: Jokinen, K., Bick, E. (eds.) Nodalida 2009. NEALT Proceedings, vol. 4 (2009), http://www.ling.helsinki.fi/~klinden/pubs/linden09dnodalida.pdf
  11. 11.
    Manning, C.D.: Part-of-speech tagging from 97% to 100%: Is it time for some linguistics? In: Gelbukh, A.F. (ed.) CICLing 2011, Part I. LNCS, vol. 6608, pp. 171–189. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  12. 12.
    Marcus, M.P., Santorini, B., Marcinkiewicz, M.A.: Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics 19(2), 313–330 (1993)Google Scholar
  13. 13.
    McDonald, D.D.: Internal and external evidence in the identification and semantic categorization of proper names. In: Boguraev, B., Pustejovsky, J. (eds.) Corpus Processing for Lexical Acquisition, pp. 21–39. MIT Press, Cambridge (1996)Google Scholar
  14. 14.
    Nadeau, D., Sekine, S.: A survey of named entity recognition and classification. Lingvisticae Investigationes 30(1), 3–26 (2007)CrossRefGoogle Scholar
  15. 15.
    Pirinen, T.: Suomen kielen äärellistilainen automaattinen morfologinen analyysi avoimen lähdekoodin menetelmin. Master’s thesis, Helsingin yliopisto (2008), http://www.helsinki.fi/~tapirine/gradu/
  16. 16.
    Pirinen, T., Silfverberg, M., Lindén, K.: Improving finite-state spell-checker suggestions with part of speech n-grams. In: IJCLA (2012)Google Scholar
  17. 17.
    Pirinen, T.A., Lindén, K.: Finite-state spell-checking with weighted language and error models. In: Proceedings of the Seventh SaLTMiL Workshop on Creation and Use of Basic Lexical Resources for Less-Resourced Languagages, Valletta, Malta, pp. 13–18 (2010), http://siuc01.si.ehu.es/
  18. 18.
    Silfverberg, M., Lindén, K.: HFST runtime format—a compacted transducer format allowing for fast lookup. In: Watson, B., Courie, D., Cleophas, L., Rautenbach, P. (eds.) FSMNLP 2009 (July 13, 2009), http://www.ling.helsinki.fi/~klinden/pubs/fsmnlp2009runtime.pdf
  19. 19.
    Silfverberg, M., Lindén, K.: Combining statistical models for POS tagging using finite-state calculus. In: Nodalida 2011, Riga, Latvia (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Krister Lindén
    • 1
  • Erik Axelson
    • 1
  • Senka Drobac
    • 1
  • Sam Hardwick
    • 1
  • Juha Kuokkala
    • 1
  • Jyrki Niemi
    • 1
  • Tommi A Pirinen
    • 1
  • Miikka Silfverberg
    • 1
  1. 1.Department of Modern LanguagesUniversity of HelsinkiFinland

Personalised recommendations