Abstract
Machine Translation (MT) evaluation is very important for a MT system. In this paper, we investigate an improved Cilin-based smoothed BLEU (BLEUS-syn). As the possible cases that the short translation or English abbreviations in candidate may cause unigram have no matches, this evaluation metric smoothed the traditional BLEUS n-gram. It applied synonym substitution in unigram matching, and calculated the other 2–4-gram. It performed experiments in Russian and Chinese bilingual sentence data set and evaluated the output translations of online translation systems such as Google, Baidu, Bing and Youdao. The experimental results show that the effectiveness of our BLEUS-syn and traditional BLEUS are consistent. The performance of Baidu is the best, that of Youdao is the second, and that of Bing is the worst. Using BLEUS-syn can greatly enhance the performance of traditional BLEUS evaluation. It makes the Baidu BLEUS value improve 6.81%, Youdao improve 6.98%, Google 7.82%, and Bing 7.55%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Assessment Specifications of Machine Translation Systems. GF 2006
Yu, S.: Automatic evaluation of output quality for Machine Translation systems. Mach. Transl. 8, 117–126 (1993)
Papineni, K., Roukos, S., Ward, T., et al.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics 2002, pp. 311–318 (2002)
Lin, C.Y., Och, F.J.: ORANGE: a method for evaluating automatic evaluation metrics for machine translation. In: Proceedings of COLING-2004 (2004)
Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Proceedings of Workshop on Text Summarization Branches Out, Post-conference Workshop of ACL 2004 (2004)
Banerjee, S., Lavie, A.: Meteor: an automatic metric for MT evaluation with improved correlation with human judgments. In: ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization (2005)
Mei, J., Zhu, Y., Gao, Y., et al.: Tongyici Cilin (Extended). HIT IR-Lab (1996)
Chen, B., Cherry, C.: A systematic comparison of smoothing techniques for sentence-level BLEU. In: Proceedings of the Ninth Workshop on Statistical Machine Translation 2014, pp. 362–367 (2014)
Du, W., Liu, W., Yu, J., et al.: Russian-Chinese sentence-level aligned news corpus. In: EAMT 2015 (2015)
Koehn, P.: Moses-statistical machine translation system- user manual and code guide (2015)
Acknowledgments
The research is supported by the Key Project of State Language Commission of China (Resource Construction and Application of Low-Resource Languages for the 21st Century Maritime Silk Road) and the Featured Innovation Project of Guangdong Province (No. 2015KTSCX035).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Yu, J., Liu, W., He, H., Yi, M. (2016). BLEUS-syn: Cilin-Based Smoothed BLEU. In: Yang, M., Liu, S. (eds) Machine Translation. CWMT 2016. Communications in Computer and Information Science, vol 668. Springer, Singapore. https://doi.org/10.1007/978-981-10-3635-4_9
Download citation
DOI: https://doi.org/10.1007/978-981-10-3635-4_9
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-3634-7
Online ISBN: 978-981-10-3635-4
eBook Packages: Computer ScienceComputer Science (R0)