Machine Translation

, Volume 28, Issue 1, pp 1–17

A conjoint analysis framework for evaluating user preferences in machine translation

  • Katrin Kirchhoff
  • Daniel Capurro
  • Anne M. Turner

DOI: 10.1007/s10590-013-9140-x

Cite this article as:
Kirchhoff, K., Capurro, D. & Turner, A.M. Machine Translation (2014) 28: 1. doi:10.1007/s10590-013-9140-x


Despite much research on machine translation (MT) evaluation, there is surprisingly little work that directly measures users’ intuitive or emotional preferences regarding different types of MT errors. However, the elicitation and modeling of user preferences is an important prerequisite for research on user adaptation and customization of MT engines. In this paper we explore the use of conjoint analysis as a formal quantitative framework to assess users’ relative preferences for different types of translation errors. We apply our approach to the analysis of MT output from translating public health documents from English into Spanish. Our results indicate that word order errors are clearly the most dispreferred error type, followed by word sense, morphological, and function word errors. The conjoint analysis-based model is able to predict user preferences more accurately than a baseline model that chooses the translation with the fewest errors overall. Additionally we analyze the effect of using a crowd-sourced respondent population versus a sample of domain experts and observe that main preference effects are remarkably stable across the two samples.


Machine translationEvaluationUser modeling Preference elicitation

Copyright information

© Springer Science+Business Media Dordrecht 2013

Authors and Affiliations

  • Katrin Kirchhoff
    • 1
  • Daniel Capurro
    • 2
  • Anne M. Turner
    • 3
    • 4
  1. 1.Department of Electrical EngineeringUniversity of WashingtonSeattleUSA
  2. 2.Department of Internal MedicinePontificia Universidad Católica de ChileSantiagoChile
  3. 3.Department of Health ServicesUniversity of WashingtonSeattleUSA
  4. 4.Department of Biomedical Informatics and Medical EducationUniversity of WashingtonSeattleUSA