Resource Creation for Training and Testing of Normalisation Systems for Konkani-English Code-Mixed Social Media Text

  • Akshata Phadte
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10859)


Code-Mixing is the mixing of two or more languages or language varieties in speech. Apart from the inherent linguistic complexity, the analysis of code-mixed content poses complex challenges owing to the presence of spelling variations and non-adherence to a formal grammar. However, for any downstream Natural Language Processing task, tools that are able to process and analyze code-mixed social media data are required. Currently there is a lack of publicly available resources for code-mixed Konkani-English social media data, while the amount of such text is increasing everyday. The lack of a standard dataset to evaluate these systems makes it difficult to make any meaningful comparisons of their relative accuracies.

In this paper, we describe the methodology for the creation of a normalisation dataset for Konkani-English Code-Mixed Social Media Text (CMST). We believe that this dataset will prove useful not only for the evaluation and training of normalisation systems but also help in the linguistic analysis of the process of normalisation Indian languages from native scripts to Roman. Normalisation refers to the process of writing the text of one language using the script of another language whereby the sound of the text is preserved as far as possible [3].


Code-mixing Social media text Normalisation Natural Language Processing 


  1. 1.
    Phadte, A., Thakkar, G.: Towards normalising Konkani-English code-mixed social media text. In: Proceedings of the 14th International Conference on Natural Language Processing (ICON-2017), pp. 85–94 (2017)Google Scholar
  2. 2.
    Gumperz, J.J.: ZIB Discourse Strategies. Cambridge University Press, Cambridge (1982)CrossRefGoogle Scholar
  3. 3.
    Knight, K., Graehl, J.: Machine Transliteration. Comput. Linguist. 24(4), 599–612 (1998)Google Scholar
  4. 4.
    Kunchukuttan, A. Puduppully, R. Bhattacharyya, P.: Brahmi-Net: a transliteration and script conversion system for languages of the Indian subcontinent. In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pp. 81–85 (2015)Google Scholar
  5. 5.
    Hidayat, T.: An analysis of code switching used by facebookers (a case study in a social network site). Sekolah Tinggi Keguruan dan Ilmu Pendidikan (STKIP) Siliwangi Bandung (2012)Google Scholar
  6. 6.
    Barman, U., Das, A., Wagner, J., Foster, J.: Code mixing: a challenge for language identification in the language of social media. In: EMNLP 2014, p. 13 (2014)Google Scholar
  7. 7.
    Han, B., Baldwin, T.: Lexical normalisation of short text messages: Makn sens a# twitter. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 368–378. Association for Computational Linguistics (2011)Google Scholar
  8. 8.
    Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., et al.: Moses: open source toolkit for statistical machine translation. In: Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, pp. 177–180. Association for Computational Linguistics (2007)Google Scholar
  9. 9.
    Federico, M., Bertoldi, N., Cettolo, M.: IRSTLM: an open source toolkit for handling large scale language models. In: Interspeech, pp. 1618–1621 (2008)Google Scholar
  10. 10.
    Kneser, R., Ney, H.: Improved backing-off for m-gram language modeling. In: 1995 International Conference on Acoustics, Speech, and Signal Processing, ICASSP-1995, vol. 1, pp. 181–184. IEEE (1995)Google Scholar
  11. 11.
    Sharma, A., Gupta, S., Motlani, R., Bansal, P., Srivastava, M., Mamidi, R., Sharma, D.M.: Shallow parsing pipeline for Hindi-English code-mixed social media text (2016)Google Scholar
  12. 12.
    Vyas, Y., Gella, S., Sharma, J., Bali, K., Choudhury, M.: Pos tagging of English-Hindi code-mixed social media content. In: EMNLP, vol. 14, PP. 974–979 (2014)Google Scholar
  13. 13.
    Wong, K.-F., Xia, Y.: Normalization of Chinese chat language. Lang. Resour. Eval. 42(2), 219–242 (2008)CrossRefGoogle Scholar
  14. 14.
    Gupta, K., Choudhury, M., Bali, K.: Mining Hindi-English transliteration pairs from online Hindi lyrics. In: LREC, pp. 2459–2465 (2012)Google Scholar
  15. 15.
    Choudhury, M., Bali, K., Dasgupta, T., Basu, A.: Resource creation for training and testing of transliteration systems for Indian languages. In: LREC (2010)Google Scholar
  16. 16.
    Paul Cook and Suzanne Stevenson.: An unsupervised model for text message normalization. In: Proceedings of the Workshop on Computational Approaches to Linguistic Creativity, pp. 71–78. Association for Computational Linguistics (2009)Google Scholar
  17. 17.
    Xue, Z., Yin, D., Davison, B.D.: Normalizing microtext. In: Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence (2011)Google Scholar
  18. 18.
    Gella, S., Sharma, J., Bali, K.: Query word labeling and back transliteration for Indian languages: shared task system description. In: FIRE Working Notes (2013)Google Scholar
  19. 19.
    Owoputi, O., O’Connor, B., Dyer, C., Gimpel, K., Schneider, N., Smith, N.A.: Improved part-of-speech tagging for online conversational text with word clusters. In: Association for Computational Linguistics (2013)Google Scholar
  20. 20.
    Mayzner, M.S., Tresselt, M.E.: Tables of single-letter and digram frequency counts for various word-length and letter-position combinations. Psychonomic monograph supplements (1965)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer Science and TechnologyGoa UniversityTaleigao PlateauIndia

Personalised recommendations