Skip to main content
SpringerLink
Log in

Reassessing the proper place of man and machine in translation: a pre-translation scenario

  • Published:
Machine Translation

Abstract

Traditionally, human–machine interaction to reach an improved machine translation (MT) output takes place ex-post and consists of correcting this output. In this work, we investigate other modes of intervention in the MT process. We propose a Pre-Edition protocol that involves: (a) the detection of MT translation difficulties; (b) the resolution of those difficulties by a human translator, who provides their translations (pre-translation); and (c) the integration of the obtained information prior to the automatic translation. This approach can meet individual interaction preferences of certain translators and can be particularly useful for production environments, where more control over output quality is needed. Early resolution of translation difficulties can prevent downstream errors, thus improving the final translation quality “for free”. We show that translation difficulty can be reliably predicted for English for various source units. We demonstrate that the pre-translation information can be successfully exploited by an MT system and that the indirect effects are genuine, accounting for around 16% of the total improvement. We also provide a study of the human effort involved in the resolution process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. This problem no longer occurs when PE is complemented with adaptive learning. However, in such scenarios the human is not usually asked explicitly to provide generalized translations for all the corresponding contexts. Therefore, the corrections of recurrent translations may still be required.

  2. http://statmt.org/wmt14/medical-task.

  3. http://opennlp.apache.org/index.html.

  4. Grid search with 5-fold cross-validation was used to tune the following parameters towards F-score: the optimizing criterion, the number of estimators, the maximum depth and the minimum number of leaf samples. All other parameters are those provided by default.

  5. We use a Radial Basis Function (RBF) kernel. Grid search with 5-fold cross-validation was used to tune \(\gamma \) and C towards F-score. All other parameters are those provided by default.

  6. We used the l-bfgs algorithm as the optimization algorithm. All other parameters are those provided by default. All the hyperparameters were tuned on the development set.

  7. The first hidden layer contained the quantity of units equal to the total quantity of features (the embedding dimension was taken into account, e.g. 1822 hidden units for word-seg), the second layer used half that number of units. The following features served as inputs to embedding layers: word \(f_i\), its left and right context \(f_{i-1}\) and \(f_{i+1}\), its aligned word \(e^1_{j}\), its left and right context \(e^1_{j-1}\) and \(e^1_{j+1}\) for word-seg; sequence \(f_{[k:t]}\), its aligned translation \(e_{[r:g]}\) and its the left (\(f_{k-2}, f_{k-1}\)) and right (\(f_{t+1}, f_{t+2}\)) contexts for phrase-level segmentations. The length of a segment sequence was limited to 10 words, and masking was used for shorter segments.

  8. https://github.com/fchollet/keras.

  9. We used the DummyClassifier with default parameters.

  10. http://nlp.stanford.edu/software/tokenizer.shtml.

  11. Translations of such nouns tend to vary greatly depending on the context, even when natural translation variability is taken into account. Some of them are also homonymous to verbs, which contributes further to their translation difficulty.

  12. We artificially balanced the quantity of examples in both classes for the language pairs where we found an unbalanced proportion of ET and DT (EN-ES, EN-FR) by removing the least frequent examples of ET. This resulted in a reduction of around 34% of the initial training data and a prediction improvement of about 0.07 in \(F_{\texttt {DT}}\).

  13. In all the experiments, we will translate complete texts rather than isolated random sentences. Our reference translations are thus likely to be somewhat “normalized”, i.e. to contain less translation variety than random sentences.

  14. In a more elaborate version, the user could select these translations from the variants proposed by an MT system (Cheng et al. 2016), or/and from the cache of past translations of DT segments, thus potentially saving many keystrokes.

  15. For this experiment, unaligned words in the reference \(\hat{e}\) are aligned (recursively) to the same word(s) as their syntactic heads. Dependencies were identified with the help of the Stanford Parser toolkit.

  16. Note that similar technical solutions also exist for other MT architecture, see e.g. Chatterjee et al. (2017), Hokamp and Liu (2017) for constrained decoding in NMT.

  17. We round word averages to nearest integer, which may result in several TER values per each rounded value. We report average TER values for such cases.

  18. Recall that DT segments represent approximately 50% of a sentence, cf. Sect. 3.1.1.

  19. \(\varDelta \) TER is computed as the absolute difference between the initial MT quality score (a point with a pair of coordinates (0,n)) and a quality score resulting from a pre-translation experiment (any other point). A maximum quality gain is computed as the absolute difference of the y-coordinate values for points (0,n) and (max(x),m).

References

  • Alabau V, González-Rubio J, Ortiz-Martínez D, Sanchis Trilles G, Casacuberta F, García-Martínez M, Mesa-Lao B, Petersen DC, Dragsted B, Carl M (2014) Integrating online and active learning in a computer-assisted translation workbench. In: Proceedings of the AMTA 2014 workshop on interactive and adaptive machine translation, Vancouver, BC, Canada, pp 1–9

  • Aronson AR, Lang FM (2010) An overview of MetaMap: historical perspective and recent advances. J Am Med Inform Assoc 17(3):229–236

    Article  Google Scholar 

  • Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473

  • Blanchon H (1992) A solution for the problem of interactive disambiguation. In: COLING 1992 volume 4: the 15th international conference on computational linguistics. Nantes, France, pp 1234–1238

  • Bojar O, Chatterjee R, Federmann C, Graham Y, Haddow B, Huck M, Jimeno Yepes A, Koehn P, Logacheva V, Monz C, Negri M, Neveol A, Neves M, Popel M, Post M, Rubino R, Scarton C, Specia L, Turchi M, Verspoor K, Zampieri M (2016) Findings of the 2016 conference on machine translation. In: Proceedings of the first conference on machine translation: volume 2, shared task papers, Berlin, Germany, pp 131–198. https://doi.org/10.18653/v1/W16-2301

  • Bojar O, Chatterjee R, Federmann C, Graham Y, Haddow B, Huang S, Huck M, Koehn P, Liu Q, Logacheva V, Monz C, Negri M, Post M, Rubino R, Specia L, Turchi M (2017) Findings of the 2017 conference on machine translation (WMT17). In: Proceedings of the second conference on machine translation, Copenhagen, Denmark, pp 169–214

  • Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    Article  Google Scholar 

  • Brown RD, Nirenburg S (1990) Human-computer interaction for semantic disambiguation. COLING 1990 volume 3: papers presented to the 13th international conference on computational linguistics. Helsinki, Finland, pp 42–47

  • Bubeck S, Cesa-Bianchi N (2012) Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Found Trends Mach Learn 5(1):1–122

    Article  Google Scholar 

  • Chatterjee R, Negri M, Turchi M, Federico M, Specia L, Blain F (2017) Guiding neural machine translation decoding with external knowledge. In: Proceedings of the eighth workshop on statistical machine translation, Copenhagen, Denmark, pp 157–168

  • Cheng S, Huang S, Chen H, Dai XY, Chen J (2016) PRIMT: A pick-revise framework for interactive machine translation. In: Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, San Diego, California, pp 1240–1249, https://doi.org/10.18653/v1/N16-1148

  • Cherry C, Foster G (2012) Batch tuning strategies for statistical machine translation. In: Proceedings of the 2012 conference of the North American chapter of the association for computational linguistics: human language technologies, Montrèal, Canada, pp 427–436

  • Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297

    MATH  Google Scholar 

  • Denkowski M, Dyer C, Lavie A (2014) Learning from post-editing: Online model adaptation for statistical machine translation. In: Proceedings of the 14th conference of the European chapter of the association for computational linguistics, Gothenburg, Sweden, pp 395–404

  • Du J, Srivastava A, Way A, Maldonado-Guerra A, Lewis D (2015) An empirical study of segment prioritization for incrementally retrained post-editing-based SMT. In: Proceedings of the XV machine translation summit, Miami, Florida, pp 172–185

  • Dyer C, Chahuneau V, Smith NA (2013) A simple, fast, and effective reparameterization of IBM model 2. In: Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: human language technologies, Atlanta, Georgia, pp 644–648

  • Eisele A, Chen Y (2010) MultiUN: A multilingual corpus from United Nation documents. In: Proceedings of the seventh conference on international language resources and evaluation (LREC’10), Valletta, Malta, pp 2868–2872

  • Federico M, Koehn P, Schwenk H, Trombetti M (2013) Matecat: machine translation enhanced computer assisted translation. In: Proceedings of the XIV machine translation summit, Nice, France, p 425

  • Gao Q, Vogel S (2008) Parallel implementations of word alignment tool. In: Proceedings of the conference on software engineering, testing, and quality assurance for natural language processing, Columbus, Ohio, pp 49–57

  • Germann U (2015) Sampling phrase tables for the Moses statistical machine translation system. Prague Bull Math Linguist 104(1):39–50

    Article  Google Scholar 

  • Gong L, Max A, Yvon F (2014) Incremental development of statistical machine translation systems. In: Proceedings of the 11th international workshop on spoken language translation (IWSLT), South Lake Tahoe, California, Nevada, pp 214–222

  • González-Rubio J, Ortiz-Martínez D, Casacuberta F (2012) Active learning for interactive machine translation. In: Proceedings of the 13th conference of the European chapter of the association for computational linguistics, Avignon, France, pp 245–254

  • Green S, Manning CD (2010) Better Arabic parsing: baselines, evaluations, and analysis. In: Proceedings of the 23rd international conference on computational linguistics, Beijing, China, pp 394–402

  • Hokamp C, Liu Q (2017) Lexically constrained decoding for sequence generation using grid beam search. In: Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), Vancouver, Canada, pp 1535–1546. https://doi.org/10.18653/v1/P17-1141

  • Ive J, Max A, Yvon F, Ravaud P (2016) Diagnosing high-quality statistical machine translation using traces of post-edition operations. In: Proceedings of the LREC 2016 workshop on translation evaluation—from fragmented tools and data sets to an integrated ecosystem, Portorož, Slovenia, pp 55–62

  • Kay M (1973) The MIND system. In: Rustin R (ed) Natural language processing: courant computer science symposium 8: December 20–21, 1971, Algorithmics Press, pp 155–188

  • Kay M (1997) The proper place of men and machines in language translation. Mach Transl 12(1):3–23

    Article  Google Scholar 

  • Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. CoRR abs/1412.6980

  • Kneser R, Ney H (1995) Improved backing-off for m-gram language modeling. In: Proceedings of the international conference on acoustics, speech, and signal processing (ICASSP’95), Detroit, MI, USA, pp 181–184

  • Koehn P, Hoang H, Birch A, Callison-Burch C, Federico M, Bertoldi N, Cowan B, Shen W, Moran C, Zens R, Dyer C, Bojar O, Constantin A, Herbst E (2007) Moses: open source toolkit for statistical machine translation. In: Proceedings of the 45th annual meeting of the association for computational linguistics companion volume proceedings of the demo and poster sessions, Prague, Czech Republic, pp 177–180

  • Koehn P, Carl M, Casacuberta F, Marcos E (2013) CASMACAT: cognitive analysis and statistical methods for advanced computer aided translation. In: Proceedings of the XIV machine translation summit, Nice, France, p 411

  • Lafferty JD, McCallum A, Pereira FCN (2001) Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of the 18th international conference on machine learning, Morgan Kaufmann Publishers Inc., Williamstown, MA, USA, pp 282–289

  • Lavergne T, Cappé O, Yvon F (2010) Practical very large scale CRFs. In: Proceedings of the 48th annual meeting of the Association for Computational Linguistics, Uppsala, Sweden, pp 504–513

  • Levenshtein VI (1966) Binary codes capable of correcting deletions, insertions and reversals. Sov Phys Doklady 10(8):707–710

    MathSciNet  Google Scholar 

  • Logacheva V, Specia L (2015) Phrase-level quality estimation for machine translation. In: Proceedings of the 12th international workshop on spoken language translation (IWSLT), Da Nang, Vietnam, pp 143–150

  • Logacheva V, Hokamp C, Specia L (2016) MARMOT: a toolkit for translation quality estimation at the word level. In: Proceedings of the tenth international conference on language resources and evaluation (LREC-2016), Portorož, Slovenia, pp 3671–3674

  • Mathur P, Mauro C, Federico M (2013) Online learning approaches in computer assisted translation. In: Proceedings of the eighth workshop on statistical machine translation, Sofia, Bulgaria, pp 301–308

  • Mathur P, Cettolo M, Federico M, de Souza JGC (2014) Online multi-user adaptive statistical machine translation. In: Proceedings of the 11th conference of the association for machine translation in the Americas (Volume 1: Research Papers), Vancouver, BC, Canada, pp 152–166

  • Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. CoRR abs/1301.3781

  • Mohit B, Hwa R (2007) Localization of difficult-to-translate phrases. In: Proceedings of the second workshop on statistical machine translation, Prague, Czech Republic, pp 248–255

  • Monroe W, Green S, Manning CD (2014) Word segmentation of informal Arabic with domain adaptation. In: Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: short papers), Baltimore, Maryland, pp 206–211. https://doi.org/10.3115/v1/P14-2034

  • Nelder JA, Mead R (1965) A simplex method for function minimization. Comput J 7(4):308–313

    Article  MathSciNet  Google Scholar 

  • Papineni K, Roukos S, Ward T, Zhu WJ (2002) Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the association for computational linguistics, Philadelphia, Pennsylvania, USA, pp 311–318

  • Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in Python. J Mach Learn Res 12:2825–2830

    MathSciNet  MATH  Google Scholar 

  • Schmid H (1995) Improvements in part-of-speech tagging with an application to German. In: Proceedings of the ACL SIGDAT-workshop, Dublin, Ireland, pp 47–50

  • Sharoff S, Nivre J (2011) The proper place of men and machines in language technology: processing Russian without any linguistic knowledge. In: Proceedings of the dialogue 2011, Russian conference on computational linguistics, Bekasovo, Russia, pp 657–670

  • Snover M, Dorr B, Schwartz R, Micciulla L, Makhoul J (2006) A study of translation edit rate with targeted human annotation. In: Proceedings of the 7th conference of the association for machine translation in the Americas, Cambridge, Massachusetts, USA, pp 223–231

  • Specia L, Shah K, de Souza JG, Cohn T (2013) QuEst – A translation quality estimation framework. In: Proceedings of the 51st annual meeting of the association for computational linguistics: system demonstrations, Sofia, Bulgaria, pp 79–84

  • Stolcke A (2002) SRILM—an extensible language modeling toolkit. In: Proceedings of the 7th international conference on spoken language processing (ICSLP), Denver, Colorado, pp 901–904

  • Tiedemann J (2012) Parallel data, tools and interfaces in OPUS. In: Proceedings of the eighth international conference on language resources and evaluation (LREC-2012), Istanbul, Turkey, pp 2214–2218

  • Tomita M (1985) Feasibility study of personal/interactive machine translation systems. In: Proceedings of the conference on theoretical and methodological issues in machine translation of natural languages, Colgate University, Hamilton, New York, pp 289–297

  • Toutanova K, Klein D, Manning CD, Singer Y (2003) Feature-rich part-of-speech tagging with a cyclic dependency network. In: Proceedings of the 2003 human language technology conference of the North American Chapter of the Association for Computational Linguistics, Edmonton, Canada, pp 173–180

  • Way A (2013) Traditional and emerging use-cases for machine translation. In: Proceedings of translating and the computer 35, London, UK

  • Wuebker J, Green S, DeNero J (2015) Hierarchical incremental adaptation for statistical machine translation. In: Proceedings of the 2015 conference on empirical methods in natural language processing, Lisbon, Portugal, pp 1059–1065

Download references

Funding

Funding was provided by ANRT (partially).

Author information

Authors and Affiliations

  1. LIMSI, CNRS, University of Paris-Sud, Université Paris-Saclay, 91 403, Orsay, France

    Julia Ive & Aurélien Max

  2. LIMSI, CNRS, Université Paris-Saclay, 91 403, Orsay, France

    François Yvon

Authors
  1. Julia Ive
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aurélien Max
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. François Yvon
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Julia Ive.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ive, J., Max, A. & Yvon, F. Reassessing the proper place of man and machine in translation: a pre-translation scenario. Machine Translation 32, 279–308 (2018). https://doi.org/10.1007/s10590-018-9223-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10590-018-9223-9

Keywords

Search

Navigation