Skip to main content
SpringerLink
Log in

Enhancing sentiment and emotion translation of review text through MLM knowledge integration in NMT

  • Research
  • Published:
Journal of Intelligent Information Systems Aims and scope Submit manuscript

Abstract

Producing a high-quality review translation is a multifaceted process. It goes beyond successful semantic transfer and requires conveying the original message’s tone and style in a way that resonates with the target audience, whether they are human readers or Natural Language Processing (NLP) applications. Capturing these subtle nuances of the review text demands a deeper understanding and better encoding of the source message. In order to achieve this goal, we explore the use of self-supervised masked language modeling (MLM) and a variant called polarity masked language modeling (p-MLM) as auxiliary tasks in a multi-learning setup. MLM is widely recognized for its ability to capture rich linguistic representations of the input and has been shown to achieve state-of-the-art accuracy in various language understanding tasks. Motivated by its effectiveness, in this paper we adopt joint learning, combining the neural machine translation (NMT) task with source polarity-masked language modeling within a shared embedding space to induce a deeper understanding of the emotional nuances of the text. We analyze the results and observe that our multi-task model indeed exhibits a better understanding of linguistic concepts like sentiment and emotion. Intriguingly, this is achieved even without explicit training on sentiment-annotated or domain-specific sentiment corpora. Our multi-task NMT model consistently improves the translation quality of affect sentences from diverse domains in three language pairs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2

Similar content being viewed by others

Availability of supporting data

The datasets used are publicly available. See the Data section for details.

Notes

  1. see Section 1 for definition of conventional NMT.

  2. We discuss Fig. 1 and all subsequent discussion considering p-MLM as the selected auxiliary task. In the case of MLM task too, we use the same architecture.

  3. The sub-component of encoder used to calculate contextual sentence embedding. It is this component that is specifically dedicated to creating contextual word embeddings of words w.r.t their context of appearance.

  4. The encoder-side component used to obtain word embedding representation of tokens. This encoder-side component is used to obtain word embedding representation of tokens, without considering their context of appearance between the aforementioned two encoder(s)

  5. We subsequently referred it as S and T in the given task-specific context.

  6. from this we filter out sentences larger than 80 tokens length.

  7. See Section 1 for definition.

References

  • Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In: Y. Bengio, & Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference track proceedings. arXiv:1409.0473.

  • Berard, A., Calapodescu, I., Dymetman, M., & et al. (2019). Machine translation of restaurant reviews: New corpus for domain adaptation and robustness. In: A. Birch, A. Finch, H. Hayashi, & et al. (Eds.), Proceedings of the 3rd workshop on neural generation and translation (pp. 168–176). Association for Computational Linguistics, Hong Kong. https://doi.org/10.18653/v1/D19-5617.

  • Devlin, J., Chang, M.-W., Lee, K., & et al. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: Human language technologies, volume 1 (Long and Short Papers) (pp. 4171–4186). Association for Computational Linguistics, Minneapolis, Minnesota. https://doi.org/10.18653/v1/N19-1423.

  • Ekman, P., & et al. (1999). Basic emotions. p. 16. https://www.paulekman.com/wp-content/uploads/2013/07/Basic-Emotions.pdf.

  • Feng, J., Li, H., & Yu, Z. (2023). Enhancing aspect-based sentiment analysis with dependency-attention gcn and mutual assistance mechanism. Journal of Intelligent Information Systems. https://doi.org/10.1007/s10844-023-00811-2

    Article  Google Scholar 

  • Freitag, M., & Al-Onaizan, Y. (2016). Fast domain adaptation for neural machine translation. arXiv:1612.06897.

  • Graham, Y., Baldwin, T., Moffat, A., & et al. (2013). Continuous measurement scales in human evaluation of machine translation. In: A. Pareja-Lora, M. Liakata, & S. Dipper (Eds.), Proceedings of the 7th linguistic annotation workshop and interoperability with discourse (pp. 33–41). Association for Computational Linguistics, Sofia, Bulgaria. https://aclanthology.org/W13-2305.

  • Jones, A., & Wijaya, D. (2021). Sentiment-based candidate selection for NMT. In: K. Duh, & F. Guzmán (Eds.), Proceedings of machine translation summit XVIII: Research track (pp. 188–201). Association for Machine Translation in the Americas, Virtual. https://aclanthology.org/2021.mtsummit-research.16.

  • Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In: Y. Bengio, & Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference track proceedings. arXiv:1412.6980.

  • Klein, G., Kim, Y., Deng, Y., & et al. (2017). OpenNMT: Open-source toolkit for neural machine translation. In: M. Bansal, & H. Ji (Eds.), Proceedings of ACL 2017, system demonstrations (pp. 67–72). Association for Computational Linguistics, Vancouver, Canada. https://aclanthology.org/P17-4012.

  • Koehn, P. (2004). Statistical significance tests for machine translation evaluation. In: D. Lin, & D. Wu (Eds.), Proceedings of the 2004 conference on empirical methods in natural language processing (pp. 388–395). Association for Computational Linguistics, Barcelona, Spain. https://aclanthology.org/W04-3250.

  • Kumari, D., Chennabasavaraj, S., Garera, N., & et al. (2021a). Sentiment preservation in review translation using curriculum-based re-inforcement framework. In: K. Duh, & F. Guzmán (Eds.), Proceedings of machine translation summit XVIII: Research track (pp. 150–162). Association for Machine Translation in the Americas, Virtual. https://aclanthology.org/2021.mtsummit-research.13.

  • Kumari, D., Ekbal, A., Haque, R., amp, et al. (2021). Reinforced nmt for sentiment and content preservation in low-resource scenario, vol 20. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3450970

  • Kunchukuttan, A., Mehta, P., & Bhattacharyya, P. (2018). The IIT Bombay English-Hindi parallel corpus. In: N. Calzolari, K. Choukri, C. Cieri, & et al (Eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, Japan. https://aclanthology.org/L18-1548.

  • Lohar, P., Afli, H., & Way, A. (2017). Maintaining sentiment polarity in translation of user-generated content. (pp. 73–84). https://doi.org/10.1515/pralin-2017-0010.

  • Lohar, P., Afli, H., & Way, A. (2018). Balancing translation quality and sentiment preservation. In: Proceedings of the 13th conference of the association for machine translation in the Americas (Volume 1: Research Papers, pp. 81–88). Boston, MA. https://www.aclweb.org/anthology/W18-1808.

  • Luong, T., Pham, H., & Manning, C. D. (2015). Effective approaches to attention-based neural machine translation. In: Proceedings of the 2015 conference on empirical methods in natural language processing (pp. 1412–1421). Lisbon, Portugal. https://doi.org/10.18653/v1/D15-1166.

  • Mariana, V. R. (2014). The multidimensional quality metric (mqm) framework: A new framework for translation quality assessment. Brigham Young University. https://scholarsarchive.byu.edu/etd/4312/.

  • Miao, M., Meng, F., Liu, Y., & et al. (2021). Prevent the language model from being overconfident in neural machine translation. In: C. Zong, F. Xia, W. Li, & et al (Eds.), Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (Volume 1: Long Papers, pp. 3456–3468). Association for Computational Linguistics, Online. https://doi.org/10.18653/v1/2021.acl-long.268.

  • Michel, P., & Neubig, G. (2018). Extreme adaptation for personalized neural machine translation. In: I. Gurevych, & Y. Miyao (Eds.), Proceedings of the 56th annual meeting of the association for computational linguistics (Volume 2: Short Papers, pp. 312–318). Association for Computational Linguistics, Melbourne, Australia. https://doi.org/10.18653/v1/P18-2050.

  • Mima, H., Furuse, O., & Iida, H. (1997). Improving performance of transfer-driven machine translation with extra-linguistic informatioon from context, situation and environment. In: Proceedings of the fifteenth International Joint Conference on Artificial Intelligence, IJCAI 97, Nagoya, Japan, August 23-29, 1997, 2 volumes (pp. 983–989). Morgan Kaufmann. http://ijcai.org/Proceedings/97-2/Papers/027.pdf.

  • Mohammad, S. (2012). #emotional tweets. In: E. Agirre, J. Bos, M. Diab, & et al. (Eds.), *SEM 2012: The first joint conference on lexical and computational semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the sixth international workshop on semantic evaluation (SemEval 2012, pp. 246–255). Association for Computational Linguistics, Montréal, Canada. https://aclanthology.org/S12-1033.

  • Mohammad, S. M., & Turney, P. D. (2013). Nrc emotion lexicon. p. 234. https://doi.org/10.4224/21270984.

  • Mohammad, S. M., Salameh, M., & Kiritchenko, S. (2016). How translation alters sentiment (vol 55, pp. 95-130). AI Access Foundation, El Segundo, CA, USA. https://doi.org/10.5555/3013558.3013562.

  • Nelatoori, Babu, K., Kommanti, & et al. (2023). Multi-task learning for toxic comment classification and rationale extraction (vol 60, pp. 495–519). Springer. https://doi.org/10.1007/s10844-022-00726-4.

  • Papineni, K., Roukos, S., Ward, T., & et al. (2002). Bleu: A method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting on sssociation for computational linguistics (ACL ’02, pp. 311-318). Association for Computational Linguistics, USA. https://doi.org/10.3115/1073083.1073135.

  • Poncelas, A., Lohar, P., Hadley, J., & et al. (2020). The impact of indirect machine translation on sentiment classification. In: Proceedings of the 14th conference of the association for machine translation in the Americas (Volume 1: Research Track, pp 78–88). Association for Machine Translation in the Americas, Virtual. https://aclanthology.org/2020.amta-research.7.pdf.

  • Pontiki, M., Galanis, D., Papageorgiou, H., & et al. (2016). SemEval-2016 task 5: Aspect based sentiment analysis. In: S. Bethard, M. Carpuat, D. Cer, & et al. (Eds.), Proceedings of the 10th international workshop on Semantic Evaluation (SemEval-2016) (pp. 19–30). Association for Computational Linguistics, San Diego, California. https://doi.org/10.18653/v1/S16-1002.

  • Rabinovich, E., Patel, R. N., Mirkin, S., & et al. (2017). Personalized machine translation: Preserving original author traits. In: M. Lapata, P. Blunsom, & A. Koller (Eds.), Proceedings of the 15th conference of the European chapter of the association for computational linguistics: Volume 1, Long papers (pp. 1074–1084). Association for Computational Linguistics, Valencia, Spain. https://aclanthology.org/E17-1101.

  • Rei, R., Stewart, C., Farinha, A. C., & et al. (2020). COMET: A neural framework for MT evaluation. In: B. Webber, T. Cohn, Y. He, & et al. (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 2685–2702). Association for Computational Linguistics, Online. https://doi.org/10.18653/v1/2020.emnlp-main.213.

  • Salameh, M., Mohammad, S., & Kiritchenko, S. (2015). Sentiment after translation: A case-study on Arabic social media posts. In: R. Mihalcea, J. Chai, & A. Sarkar (Eds.), Proceedings of the 2015 conference of the North American chapter of the association for computational linguistics: Human language technologies (pp. 767–777). Association for Computational Linguistics, Denver, Colorado. https://doi.org/10.3115/v1/N15-1078.

  • Shalunts, G., Backfried, G., & Commeignes, N. (2016). The impact of machine translation on sentiment analysis (pp. 51–56). https://aclanthology.org/2020.amta-research.7.pdf.

  • Si, C., Wu, K., Aw, A. T., & et al. (2019). Sentiment aware neural machine translation. In: T. Nakazawa, C. Ding, R. Dabre & et al. (Eds.), Proceedings of the 6th workshop on asian translation (pp. 200–206). Association for Computational Linguistics, Hong Kong, China. https://doi.org/10.18653/v1/D19-5227.

  • Snover, M., Dorr, B., Schwartz, R., & et al. (2006). A study of translation edit rate with targeted human annotation. In: Proceedings of the 7th conference of the association for machine translation in the Americas: Technical papers (pp 223–231). Association for Machine Translation in the Americas, Cambridge, Massachusetts, USA. https://aclanthology.org/2006.amta-papers.25.

  • Song, K., Tan, X., Qin, T., & et al. (2019). Mass: Masked sequence to sequence pre-training for language generation. In: International conference on machine learning (pp. 5926–5936). arXiv:1905.02450.

  • Stanovsky, G., Smith, N. A., & Zettlemoyer, L. (2019). Evaluating gender bias in machine translation. In: A. Korhonen, D. Traum, & L. Màrquez (Eds.), Proceedings of the 57th annual meeting of the association for computational linguistics. (pp. 1679–1684). Association for Computational Linguistics, Florence, Italy. https://doi.org/10.18653/v1/P19-1164.

  • Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. In: Proceedings of the 27th international conference on neural information processing systems. (Volume 2, pp. 3104-3112). MIT Press, Cambridge, MA, USA. NIPS’14. https://dl.acm.org/doi/10.5555/2969033.2969173.

  • Troiano, E., Klinger, R., & Padó, S. (2020). Lost in back-translation: Emotion preservation in neural machine translation. In: D. Scott, N. Bel, & C. Zong (Eds.), Proceedings of the 28th tnternational conference on computational linguistics (pp. 4340–4354). International Committee on Computational Linguistics, Barcelona, Spain (Online). https://doi.org/10.18653/v1/2020.coling-main.384.

  • Vanmassenhove, E., Hardmeier, C., & Way, A. (2018). Getting gender right in neural machine translation. In: E. Riloff, D. Chiang, J. Hockenmaier & et al. (Eds.), Proceedings of the 2018 conference on empirical methods in natural language processing (pp. 3003–3008). Association for Computational Linguistics, Brussels, Belgium. https://doi.org/10.18653/v1/D18-1334.

  • Vaswani, A., Shazeer, N., Parmar, N., & et al. (2017). Attention is all you need. In: Proceedings of the 31st international conference on neural information processing systems (pp. 6000-6010). Curran Associates Inc., Red Hook, NY, USA. NIPS’17. https://dl.acm.org/doi/10.5555/3295222.3295349.

  • Wan, Y., Chen, Y., Shi, L., amp, et al. (2023). A knowledge-enhanced interactive graph convolutional network for aspect-based sentiment analysis. Journal of Intelligent Information Systems, 61, 343–365. https://doi.org/10.1007/s10844-022-00761-1

  • Yu, J., & Jiang, J. (2016). Learning sentence embeddings with auxiliary tasks for cross-domain sentiment classification. In: J. Su, K. Duh, & X. Carreras (Eds.), Proceedings of the 2016 conference on empirical methods in natural language processing (pp. 236–246). Association for Computational Linguistics, Austin, Texas. https://doi.org/10.18653/v1/D16-1023.

  • Žagar, A., & Robnik-Šikonja, M. (2022). Cross-lingual transfer of abstractive summarizer to less-resource language (pp. 1–21). Springer. https://doi.org/10.1007/s10844-021-00663-8

  • Zhu, J., Xia, Y., Wu, L., & et al. (2020). Incorporating bert into neural machine translation. In: International conference on learning representations. arXiv:2002.06823.

Download references

Acknowledgements

Authors gratefully acknowledge the unrestricted research grant received from the Flipkart Internet Private Limited to carry out the research. Authors thank Muthusamy Chelliah for his continuous feedback.

Funding

We receive research grant from Flipkart Internet Private Limited to carry out this research.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Indian Institute of Technology Patna, Patna, 801106, Bihar, India

    Divya Kumari & Asif Ekbal

Authors
  1. Divya Kumari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Asif Ekbal
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Asif Ekbal provide supervision and helped with the acquisition of funding and resources. Divya Kumari performed experiments, data collection, and analysis. The first draft of the manuscript was written by Divya Kumari and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Divya Kumari.

Ethics declarations

Competing interests

No competing interest.

Ethical Approval

Not Applicable

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumari, D., Ekbal, A. Enhancing sentiment and emotion translation of review text through MLM knowledge integration in NMT. J Intell Inf Syst (2024). https://doi.org/10.1007/s10844-024-00843-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10844-024-00843-2

Keywords

Search

Navigation