Improving Performance of NMT Using Semantic Concept of WordNet Synset
Neural machine translation (NMT) has shown promising progress in recent years. However, for reducing the computational complexity, NMT typically needs to limit its vocabulary scale to a fixed or relatively acceptable size, which leads to the problem of rare word and out-of-vocabulary (OOV). In this paper, we present that the semantic concept information of word can help NMT learn better semantic representation of word and improve the translation accuracy. The key idea is to utilize the external semantic knowledge base WordNet to replace rare words and OOVs with their semantic concepts of WordNet synsets. More specifically, we propose two semantic similarity models to obtain the most similar concepts of rare words and OOVs. Experimental results on 4 translation tasks (We verify the effectiveness of our method on four translation tasks, including English-to- German, German-to-English, English-to-Chinese and Chinese-to-English.) show that our method outperforms the baseline RNNSearch by 2.38–2.88 BLEU points. Furthermore, the proposed hybrid method by combining BPE and our proposed method can also gain 0.39–0.97 BLEU points improvement over BPE. Experiments and analysis presented in this study also demonstrate that the proposed method can significantly improve translation quality of OOVs in NMT.
KeywordsNMT Semantic concept of synset Rare words Unknown words
The research work has been supported by the National Nature Science Foundation of China (Contract 61370130, 61473294 and 61502149), and Beijing Natural Science Foundation under Grant No. 4172047, and the Fundamental Research Funds for the Central Universities (2015JBM033), and the International Science and Technology Cooperation Program of China under grant No. 2014DFA11350.
- 1.Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: ICLR 2015 (2015)Google Scholar
- 2.Vaswani, A., et al.: Attention is all you need. CoRR abs/1706.03762 (2017)Google Scholar
- 3.Wu, Y., Schuster, M., Chen, Z., et al.: Google’s neural machine translation system: bridging the gap between human and machine translation (2016)Google Scholar
- 4.Luong, M.T., Sutskever, I., Le, Q.V., et al.: Addressing the rare word problem in neural machine translation. Bull. Univ. Agric. Sci. Vet. Med. Cluj-Napoca. Vet. Med. 27(2), 82–86 (2014)Google Scholar
- 5.Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. Computer Science (2015)Google Scholar
- 8.Papineni, K., Roukos, S., et al.: BLEU: a method for automatic valuation of machine translation. In: Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, July 2002, pp. 311–318 (2002)Google Scholar
- 9.Ziemski, M., Junczys-Dowmunt, M., Pouliquen, B.: The United Nations Parallel Corpus, Language Resources and Evaluation (LREC 2016), Portorož, Slovenia, May 2016 (2016)Google Scholar
- 10.Stolcke, A.: SRILM—an extensible language modeling toolkit. In: International Conference on Spoken Language Processing, pp. 901–904 (2002)Google Scholar
- 11.Li, X., Zhang, J., Zong, C.: Towards zero unknown word in neural machine translation. In: International Joint Conference on Artificial Intelligence, pp. 2852–2858. AAAI Press (2016)Google Scholar
- 12.Li, S., Xu, J., Miao, G., Zhang, Y., Chen, Y.: A semantic concept based unknown words processing method in neural machine translation. In: Huang, X., Jiang, J., Zhao, D., Feng, Y., Hong, Yu. (eds.) NLPCC 2017. LNCS (LNAI), vol. 10619, pp. 233–242. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73618-1_20CrossRefGoogle Scholar
- 13.Sutskever, I., et al.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)Google Scholar