Abstract
The translation task of social media comments has attracted researchers in recent times because of the challenges to understand the nature of the comments and its representation and the need of its translation into other target languages. In the present work, we attempt two approaches of translating the Facebook comments – one using a language identifier and other without using the language identifier. We also attempt to handle some form of spelling variation of these comments towards improving the translation quality with the help of state-of-the-art statistical machine translation techniques. Our approach employs n-best-list generation of the source language of the training dataset to address the spelling variation in the comments and also enrich the resource for translation. A small in-domain dataset could further boost the performance of the translation system. Our translation task focuses on Hindi-English mixed comments collected from Facebook and our systems show improvement of translation quality over the baseline system in terms of automatic evaluation scores.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
References
Stolcke, A.: SRILM - an extensible language modeling toolkit. In: Proceedings of the International Conference Spoken Language Processing, Denver, Colorado (2002)
Dey, A., Fung, P.: A Hindi-English code-switching corpus. In: The 9th International Conference on Language Resources and Evaluation (LREC) (2014)
Kunchukuttan, A., Puduppully, R., Bhattacharyya, P.: Brahmi-Net: a transliteration and script conversion system for languages of the Indian subcontinent. In: Conference of the North American Chapter of the Association for Computational Linguistics—Human Language Technologies: System Demonstrations (2015)
King, B., Abney, S.: Labelling the languages of words in mixed-language documents using weakly supervised methods. In: Proceedings of NAACL-HLT 2013, Atlanta, Georgia, 9–14 June, pp. 1110–1119 (2013)
Cherry, C., Foster, G.: Batch tuning strategies for statistical machine translation. In: Proceedings of the NAACL-HLT, 2012, Montreal, Canada, June 3–8, pp. 427–436 (2012)
Čermák, F., Rosen, A.: The case of InterCorp, a multilingual parallel corpus. Int. J. Corpus Linguis. 13(3), 411–427 (2012)
Och, F.J.: Minimum error rate training in statistical machine translation. In: Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, ACL 2003, Stroudsburg, PA, USA, vol. 1, pp. 160–167. Association for Computational Linguistics (2003)
Franz Josef Och and Herman Ney: A systematic comparison of various statistical alignment models. Comput. Linguis. 29(1), 19–51 (2003)
Bali, K., Sharma, J., Choudhury, M., Vyas, Y.: “I am borrowing ya mixing ?” An analysis of English-Hindi code mixing in Facebook. In: Proceedings of the First Workshop on Computational Approaches to Code Switching, Oct 25, 2014, Doha, Qatar, pp. 116–126 (2014)
Gupta, K., Choudhury, M., Bali, K.: Mining Hindi-English transliteration pairs from online Hindi lyrics. In: Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC 2012), 23–25 May, Istanbul, Turkey, pp. 2459–2465 (2012)
Heafield, K.: KenLM: faster and smaller language model queries. In: Proceedings of the EMNLP 2011 Sixth Workshop on Statistical Machine Translation, Edinburgh, Scotland, United Kingdom, pp. 187–197 (2011)
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of 40th ACL, Philadelphia, PA (2002)
Birch, L., Callison-Burch, C., Osborne, M., Post, M.: The Indic multi-parallel corpus (2011). http://homepages.inf.ed.uk/miles/babel.html
Post, M., Callison-Burch, C., Osborne, M.: Constructing parallel corpora for six Indian languages via crowdsourcing. In: Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, pp. 401–409. Association of Computational Linguistics (2012)
Bertoldi, N., Cettolo, M., Federico, M.: Statistical machine translation of texts with misspelled words. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, June 02–04, Los Angeles, California, pp. 412–419 (2010)
Bojar, O., Diatka, V., Rychlý, P., Stranak, P., Suchomel, V., Tamchyna, A., Zeman, D.: HindEnCorp—Hindi-English and Hindi-only corpus for machine translation. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014), Reykjavik, Iceland (2014)
Gupta, P., Bali, K., Banchs, R.E., Choudhury, M., Rosso, P.: Query expansion for mixed-script information retrieval. In: SIGIR 2014, Gold Coast, Queensland, Australia, July 6–11, 2014 (2014)
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E.: Moses: open source toolkit for statistical machine translation. In: Annual Meeting of the Association for Computational Linguistics (ACL), demonstration session, Prague, Czech Republic (2007)
Gao, Q., Vogel, S.: Parallel implementations of word alignment tool. In: Software Engineering, Testing, and Quality Assurance for Natural Language Processing, pp. 49–57 (2008)
Sinha, R.M.K.: An engineering perspective of machine translation: AnglaBharti-II and AnuBharti-II architectures. In: Invited Paper, Proceedings of International Symposium on Machine Translation, NLP and Translation Support System (iSTRANS-2004), November 17–19, Tata Mc Graw Hill, New Delhi (2004)
Sinha, R.M.K., Thakur, A.: Machine translation of bi-lingual Hindi-English (Hinglish) text. In: Proceedings of the 10th Conference on Machine Translation (2005)
Sowmya, V.B., Choudhury, M., Bali, K., Dasgupta, T., Basu, A.: Resource creation for training and testing of transliteration systems for Indian languages. In: LREC 2010 (2010)
Stanley, F.C., Goodman, J.: An empirical study of smoothing techniques for language modeling. Technical Report TR-10–98, Harvard, Cambridge, MA, USA (1998)
Solorio, T., Liu, Y.: Learning to predict code-switching points. In: Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, Honolulu, Hawaii, pp. 973–981 (2008)
Ling, W., Xiang, G., Dyer, C., Black, A., Trancoso, I.: Microblogs as parallel corpora. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria. Long Papers, vol. 1, pp. 176–186 (2013)
Acknowledgements
We acknowledge the HPCC of UH for providing facility to carry out experiments. We thank Kunal Parmar for his help in the data preparation. Thanks to Paolo Rosso, UPV, Valencia, Spain for his valuable feedback and Heba Elfardy of George Washington University for helping during the data collection process.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Singh, T.D., Solorio, T. (2018). Towards Translating Mixed-Code Comments from Social Media. In: Gelbukh, A. (eds) Computational Linguistics and Intelligent Text Processing. CICLing 2017. Lecture Notes in Computer Science(), vol 10762. Springer, Cham. https://doi.org/10.1007/978-3-319-77116-8_34
Download citation
DOI: https://doi.org/10.1007/978-3-319-77116-8_34
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-77115-1
Online ISBN: 978-3-319-77116-8
eBook Packages: Computer ScienceComputer Science (R0)