A Dependency-Inspired Semantic Evaluation of Machine Translation Systems

  • Mohammad Reza Mirsarraf
  • Nazanin Dehghani
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8138)


The goal of translation is to preserve the original text meaning. However, lexical-based machine translation (MT) evaluation metrics count the similar terms in MT output with the human translated reference rather than measuring the similarity in meaning. In this paper, we developed an MT evaluation metric to assess the output of MT systems, semantically. Inspiring by the dependency grammar, we consider to what extent the headword and its dependents contribute in preserving the meaning of the original input text. Our experimental results show that this metric is significantly better correlated with human judgment.


Machine Translation Evaluation Metrics Human Judgment Semantic Role Machine Translation System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: A Method for Automatic Evaluation of Machine Translation. In: Proc. of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, pp. 311–318 (2002)Google Scholar
  2. 2.
    Snover, M., Dorr, B., Schwartz, L., Makhoul, J.: A Study of Translation Edit Rate with Targeted Human Annotation. In: Proc. of the 7th Conference of the Association for Machine Translation in the Americas, Cambridge, MA, pp. 223–231 (2006)Google Scholar
  3. 3.
    Denkowski, M., Lavie, A.: METEOR-NEXT and the METEOR Paraphrase Tables: Improve Evaluation Support for Five Target Languages. In: Proc. of the ACL WMT/MetricsMATR 2010 (2010)Google Scholar
  4. 4.
    Koehn, P.: What is a Better Translation? Reflections on Six Years of Running Evaluation Campaigns. Tralogy (2011)Google Scholar
  5. 5.
    Lo, C., Wu, D.: MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility via semantic frames. In: Proc. of the 49th Annual Meeting of the Association for Computational Linguistics, pp. 220–229 (2011)Google Scholar
  6. 6.
    Lo, C., Wu, D.: Evaluating Machine Translation Utility via Semantic Role Labels (2010)Google Scholar
  7. 7.
    Kingsbury, P., Palmer, M.: Propbank: the next level of treebank. In: Proc. of Treebanks and Lexical Theories (2003)Google Scholar
  8. 8.
    Baker, C.F., Fillmore, C.J., Lowe, J.B.: The Berkeley FrameNet project. In: Proc. of the COLING-ACL Conference, Montreal, Canada, pp. 86–90 (1998)Google Scholar
  9. 9.
  10. 10.
    Farazin, Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Mohammad Reza Mirsarraf
    • 1
  • Nazanin Dehghani
    • 1
    • 2
  1. 1.CyberSpace Research InstituteTehranIran
  2. 2.ECE DepartmentUniversity of TehranTehranIran

Personalised recommendations