Language Independent Evaluation of Translation Style and Consistency: Comparing Human and Machine Translations of Camus’ Novel “The Stranger”

  • Mahmoud El-Haj
  • Paul Rayson
  • David Hall
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8655)


We present quantitative and qualitative results of automatic and manual comparisons of translations of the originally French novel “The Stranger” (French: L’Étranger). We provide a novel approach to evaluating translation performance across languages without the need for reference translations or comparable corpora. Our approach examines the consistency of the translation of various document levels including chapters, parts and sentences. In our experiments we analyse four expert translations of the French novel. We also used Google’s machine translation output as baselines. We analyse the translations by using readability metrics, rank correlation comparisons and Word Error Rate (WER).


Machine Translation Readability Score Statistical Machine Translation Computational Linguistics Word Error Rate 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Altamimi, A.K., Jaradat, M., Aljarrah, N., Ghanem, S.: Aari: Automatic Arabic readability index iajit first online publication (2013)Google Scholar
  2. 2.
    Anderson, J.: Analysing the readability of English and non-English texts in the classroom with lix. In: The Australian Reading Association Conference. ERIC Institute of Education Sciences (1981)Google Scholar
  3. 3.
    Anderson, J.: Lix and rix: Variations on a little-known readability index. Journal of Reading 26(6), 490–496 (1983)Google Scholar
  4. 4.
    Benjamin, W.: Illuminations. Houghton Mifflin Harcourt (1968),
  5. 5.
    Björnsson, C.H.: Läsbarhet. In: Stockholm: Liber (1968)Google Scholar
  6. 6.
    Egbert, J.: Student perceptions of stylistic variation in introductory university textbooks. Linguistics and Education 25, 64–77 (2014)CrossRefGoogle Scholar
  7. 7.
    Goldhammer, A.: Translating subtexts: What the translator must know. In: Talk Delivered to Brandeis University Translation Seminar. CUNY Conference on Translation. Association for Computational Linguistics, Waltham (1994),
  8. 8.
    Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. Tech. Rep. 8 (1966)Google Scholar
  9. 9.
    Padó, S., Galley, M., Jurafsky, D., Manning, C.: Robust machine translation evaluation with entailment features. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, ACL 2009, vol. 1, pp. 297–305. Association for Computational Linguistics, Stroudsburg (2009)Google Scholar
  10. 10.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: A method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL 2002, pp. 311–318. Association for Computational Linguistics, Stroudsburg (2002)Google Scholar
  11. 11.
    Popović, M., Ney, H.: Word error rates: Decomposition over pos classes and applications for error analysis. In: Proceeding of the Second Workshop on Statistical Machine Translation Held within ACL 2007, pp. 48–55 (2007)Google Scholar
  12. 12.
    Smith, E., Senter, R.: Automated Readability Index. AMRL-TR-66-220, Aerospace Medical Research Laboratories (1967)Google Scholar
  13. 13.
    Stymne, S., Tiedemann, J., Hardmeier, C., Nivre, J.: Statistical machine translation with readability constraints. In: Proceedings of the 19th Nordic Conference of Computational Linguistics (NODALIDA 2013). NEALT Proceedings Series 16, pp. 375–386. LiU Electronic Press (2013)Google Scholar
  14. 14.
    Uitdenbogerd, A.L.: Readability of french as a foreign language and its uses. In: Proceedings of the 10th Australasian Document Computing Symposium (2005)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Mahmoud El-Haj
    • 1
  • Paul Rayson
    • 1
  • David Hall
    • 2
  1. 1.School of Computing and CommunicationsLancaster UniversityUK
  2. 2.UK Data ArchiveThe University of EssexUK

Personalised recommendations