Ship-LemmaTagger: Building an NLP Toolkit for a Peruvian Native Language

  • José Pereira-Noriega
  • Rodolfo Mercado-Gonzales
  • Andrés Melgar
  • Marco Sobrevilla-Cabezudo
  • Arturo Oncevay-MarcosEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10415)


Natural Language Processing deals with the understanding and generation of texts through computer programs. There are many different functionalities used in this area, but among them there are some functions that are the support of the remaining ones. These methods are related to the core processing of the morphology of the language (such as lemmatization) and automatic identification of the part-of-speech tag. Thereby, this paper describes the implementation of a basic NLP toolkit for a new language, focusing in the features mentioned before, and testing them in an own corpus built for the occasion. The obtained results exceeded the expected results and could be used for more complex tasks such as machine translation.


Part-of-speech tagging Lemmatization Low resource language Shipibo-konibo 



For this study, the authors appreciate the linguistic team effort that made possible the corpus annotation, and also acknowledge the support of the “Consejo Nacional de Ciencia, Tecnología e Innovación Tecnológica” (CONCYTEC Perú) under the contract 225-2015-FONDECYT.


  1. 1.
    Acosta, S., Natalia, K., Huamancayo Curi, E., Mori Clement, M., Carbajal Solis, V.: Documento nacional de lenguas originarias del Perú (2013)Google Scholar
  2. 2.
    Daelemans, W., Groenewald, H.J., Van Huyssteen, G.B.: Prototype-Based Active Learning for Lemmatization (2009)Google Scholar
  3. 3.
    Ekbal, A., Bandyopadhyay, S.: Part of speech tagging in Bengali using support vector machine. In: International Conference on Information Technology 2008, ICIT 2008, pp. 106–111. IEEE (2008)Google Scholar
  4. 4.
    Gupta, V., Joshi, N., Mathur, I.: Design and development of a rule-based Urdu lemmatizer. In: Proceedings of International Conference on ICT for Sustainable Development, pp. 161–169. Springer (2016)Google Scholar
  5. 5.
    Jurafsky, D., Martin, J.H.: Speech and Language Processing, vol. 3. Pearson, London (2014)Google Scholar
  6. 6.
    Juršic, M., Mozetic, I., Erjavec, T., Lavrac, N.: Lemmagen: multilingual lemmatisation with induced ripple-down rules. J. Univers. Comput. Sci. 16(9), 1190–1214 (2010)Google Scholar
  7. 7.
    Khaltar, B.O., Fujii, A.: A lemmatization method for mongolian and its application to indexing for information retrieval. Inf. Process. Manag. 45(4), 438–451 (2009)CrossRefGoogle Scholar
  8. 8.
    Ministerio de Educación del Perú: Minedu oficializa alfabetos de 24 lenguas originarias a ser utilizados por todas las entidades públicas. Accessed 31 Mar 2016
  9. 9.
    Nivre, J., de Marneffe, M.C., Ginter, F., Goldberg, Y., Hajic, J., Manning, C.D., McDonald, R., Petrov, S., Pyysalo, S., Silveira, N., et al.: Universal dependencies v1: A multilingual treebank collection. In: Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), pp. 1659–1666 (2016)Google Scholar
  10. 10.
    Paul, A., Purkayastha, B.S., Sarkar, S.: Hidden Markov model based part of speech tagging for Nepali language. In: 2015 International Symposium on Advanced Computing and Communication (ISACC), pp. 149–156. IEEE (2015)Google Scholar
  11. 11.
    Singh, S., Jha, G.N.: Statistical tagger for Bhojpuri (employing support vector machine). In: 2015 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pp. 1524–1529. IEEE (2015)Google Scholar
  12. 12.
    Valenzuela, P.: Transitivity in Shipibo-Konibo grammar. Ph.D. thesis, University of Oregon (2003)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • José Pereira-Noriega
    • 1
  • Rodolfo Mercado-Gonzales
    • 2
  • Andrés Melgar
    • 2
  • Marco Sobrevilla-Cabezudo
    • 2
  • Arturo Oncevay-Marcos
    • 2
    Email author
  1. 1.Facultad de Ciencias e IngenieríaPontificia Universidad Católica del PerúLimaPeru
  2. 2.Research Group on Pattern Recognition and Applied Artificial Intelligence, Departamento de IngenieríaPontificia Universidad Católica del PerúLimaPeru

Personalised recommendations