Skip to main content
SpringerLink
Log in

Human Post-editing in Hybrid Machine Translation Systems: Automatic and Manual Analysis and Evaluation

  • Conference paper
Book cover Trends and Advances in Information Systems and Technologies (WorldCIST'18 2018)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 745))

Included in the following conference series:

Abstract

This study assesses, automatically and manually, the performance of two hybrid machine translation (HMT) systems, via a text corpus of questions in the Spanish and English languages. The results show that human evaluation metrics are more reliable when evaluating HMT performance. Further, there is evidence that MT can streamline the translation process for specific types of texts, such as questions; however, it does not yet rival the quality of human translations, to which post-editing is key in this process.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 259.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 329.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Available at http://asiya.cs.upc.edu/ (Last visit 05/01/2018)

References

  1. Gutiérrez Artacho, J.: Recursos y herramientas lingüísticos para los sistemas de búsqueda de respuestas monolingües y multilingües (2015)

    Google Scholar 

  2. Zhou, D., Truran, M., Brailsford, T., Wade, V., Ashman, H.: Translation techniques in cross-language information retrieval. ACM Comput. Surv. 45(1), 1–44 (2012)

    Article  Google Scholar 

  3. Olvera-Lobo, M., Gutierrez-Artacho, J.: Language resources used in multi-lingual question-answering systems. Online Inf. Rev. 35, 543–557 (2011)

    Article  Google Scholar 

  4. Olvera-Lobo, M.D., Garcia-Santiago, L.: Analysis of errors in the automatic translation of questions for translingual QA systems. J. Doc. 66, 434–455 (2010)

    Article  Google Scholar 

  5. García-Santiago, L., Olvera-Lobo, M.-D.: Analysis of automatic translation of questions for question-answering systems. Inf. Res. 15 (2010). http://www.informationr.net/ir/15-4/paper450.html

  6. Madankar, M., Chandak, M.B., Chavhan, N.: Information retrieval system and machine translation: a review. Procedia Comput. Sci. 78, 845–850 (2016)

    Article  Google Scholar 

  7. Gupta, M., Bendersky, M.: Information retrieval with verbose queries. Found. Trends Inf. Retrieval 9(3–4), 200–354 (2015)

    Google Scholar 

  8. Olvera-Lobo, M.D., Castro-Prieto, M.R., Quero-Gervilla, E., Munoz-Martin, R., Munoz-Raya, E., Murillo-Melero, M., Robinson, B., Senso-Ruiz, A., Vargas-Quesada, B., Dominguez-Lopez, C.: Translator training and modern market demands. Perspect. Transl. 13, 132–142 (2005)

    Google Scholar 

  9. Koponen, M.: Is machine translation post-editing worth the effort? a survey of research into post-editing and effort. J. Spec. Transl 23, 131–148 (2016)

    Google Scholar 

  10. Mesa-lao, B.: Introduction to post-editing – The CasMaCat GUI 1 (2013)

    Google Scholar 

  11. Allen, J.: Post-editing. In: Somers, H.L. (ed.) Computers and Translation: A Translators Guide. John Benjamins, Amsterdam/Philadelphia (2003). 297–317

    Google Scholar 

  12. Lagarda, A.L., Ortiz-Martinez, D., Alabau, V., Casacuberta, F.: Translating without in-domain corpus: machine translation post-editing with online learning techniques. Comput. Speech Lang. 32, 109–134 (2015)

    Article  Google Scholar 

  13. Shaw, F., Gros, X.: Survey of Machine Translation Evaluation, Saarbrücken (2007)

    Google Scholar 

  14. Mauser, A., Hasan, S., Ney, H.: Automatic evaluation measures for statistical machine translation system optimization. In: Proceedings of the Sixth International Language Resources and Evaluation (LREC 2008), Marraketch, Morocco (2008)

    Google Scholar 

  15. Hutchins, W.J., Somers, H.L.: An Introduction to Machine Translation. Academic Press, London (1992). 57, 377

    MATH  Google Scholar 

  16. Arnold, D., Balkan, L., Meijer, S., Humphreys, R.L., Sadler, L.: An Introductory Guide. NCC Blackwell, London (1994)

    Google Scholar 

  17. Costa-Jussa, M.R., Fonollosa, J.A.R.: Latest trends in hybrid machine translation and its applications. Comput. Speech Lang. 32, 3–10 (2015)

    Article  Google Scholar 

  18. Labaka, G., España-Bonet, C., Màrquez, L., Sarasola, K.: A hybrid machine translation architecture guided by syntax. Mach. Transl. 28, 91–125 (2014)

    Article  Google Scholar 

  19. Hunsicker, S., Yu, C., Federmann, C.: Machine learning for hybrid machine translation. In: Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, pp. 312–316 (2012)

    Google Scholar 

  20. Tambouratzis, G.: Conditional random fields versus template-matching in MT phrasing tasks involving sparse training data. PATTERN Recognit. Lett. 53, 44–52 (2015)

    Article  Google Scholar 

  21. Kamran, A.: Hybrid Machine Translation Panel (2013)

    Google Scholar 

  22. Leusch, G.: Evaluation Measures in Machine Translation (2005)

    Google Scholar 

  23. Niessen, S., Och, F.J., Leusch, G., Ney, H.: An evaluation tool for machine translation: Fast evaluation for MT research. ACM Trans. Inf. Syst. 20, 39–45 (2000)

    Google Scholar 

  24. Mayor, A., Alegria, I., Díaz Ilarraza, A., Labaka, G., Lersundi, M., Sarasola, K.: Evaluación de un sistema de traducción automática basado en reglas o por qué BLEU sólo sirve para lo que sirve. Proces. leng. nat. 43, 197–205 (2009)

    Google Scholar 

  25. Homola, P., Kubon, V., Pecina, P., Pecina, P.: A simple automatic MT evaluation metric. In: Association for Computational Linguistics (ed.) Proceedings of the 4th EACL Workshop on Statistical Machine Translation, Athens, pp. 33–36 (2009)

    Google Scholar 

  26. Turian, J.P., Shen, L., Melamed, I.D.: Evaluation of machine translation and its evaluation. In: Proceedings of MT Summit IX, New Orleans (2003)

    Google Scholar 

  27. Boitet, C., Bey, Y., Tomokiyo, M., Cao, W., Blanchon, H.: IWSLT-06 : experiments with commercial MT systems and lessons from subjective evaluations (2006)

    Google Scholar 

  28. Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311–318. Association for Computational Linguistics, Stroudsburg (2002)

    Google Scholar 

  29. Bouillon, P. (Coord.): Analysis of existing metrics and proposal for a task-oriented metric. In: ACCEPT- Automated Community Content Editing PorTal (2012)

    Google Scholar 

  30. Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, pp. 65–72 (2005)

    Google Scholar 

  31. Goutte, C.: Learning Machine Translation. MIT Press, Cambridge (2009)

    Google Scholar 

  32. Ferrández, O., Micol, D., Muñoz, R., Palomar, M.: Técnicas léxico- sintácticas para el reconocimiento de Implicación Textual. Proces. del Leng. Nat. 38, 53–60 (2007)

    Google Scholar 

  33. Popovic, M., Ney, H.: Word error rates: decomposition over POS classes and applications for error analysis. In: Proceedings of the Second Workshop on Statistical Machine Translation, pp. 48–55. Association for Computational Linguistics, Prague (2007)

    Google Scholar 

  34. Helle, A.: Hibridación en lenguas distantes (2013)

    Google Scholar 

  35. Olvera‐Lobo, M.D., Quero‐Gervilla, E., Robinson, B., Senso‐Ruiz, J.A., Castro-Prieto, M.R., Muñoz‐Martín, R., Muñoz‐Raya, E., Murillo‐Melero, M: Presentation of a distance training model for introduction into the practice of teaching translation according to the requirements of the Bologna declaration. Vestn. MGU. Ser. (26) (2010). Pedagog. Sci. 605, 196–208

    Google Scholar 

  36. Olvera-Lobo, M.-D., Gutiérrez-Artacho, J.: Evaluación de los sistemas QA de dominio abierto frente a los de dominio especializado en el ámbito biomédico. In: I Congreso Español de Recuperación de Información (CERI 2010), Madrid, pp. 161–169 (2010)

    Google Scholar 

  37. Olvera-Lobo, M.-D., Gutierrez-Artacho, J.: Open - vs. restricted-domain QA systems in the biomedical field. J. Inf. Sci. 37, 152–162 (2011)

    Article  Google Scholar 

  38. Olvera-Lobo, M.-D., Gutierrez-Artacho, J.: Multilingual question-answering system in biomedical domain on the web: an evaluation. In: Forner, P., Gonzalo, J., Kekalainen, J., Lalmas, M., DeRijke, M. (eds.) Multilingual and Multimodal Information Access Evaluation, pp. 83–88 (2011)

    Chapter  Google Scholar 

  39. Olvera-Lobo, M.-D., Gutierrez-Artacho, J.: Performance analysis in web-based question answering systems. Rev. Esp. Doc. Cient. 36, e009 (2013)

    Article  Google Scholar 

  40. Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Searching health information in question-answering systems. In: Cruz-Cunha, M.M., Miranda, I.M., Gonçalves, P. (eds.) Handbook of Research on ICTs for Human-Centered Healthcare and Social Care Services, pp. 474–490. IGI Global, Hershey (2013)

    Google Scholar 

  41. Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Nuevas tendencias en recuperación de información: la búsqueda de respuestas desde la perspectiva de la traducción. In: VI Congreso Internacional de la Asociación Ibérica de Estudios de Traducción e Interpretación: ‘Traducimos desde el sur’. Asociación Ibérica de Estudios de Traducción e Interpretación (AIETI), Las Palmas de Gran Canaria (2013)

    Google Scholar 

  42. Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Satisfacción de los usuarios en la búsqueda multilingüe de respuestas como recursos de información terminológica. In: Vargas-Sierra, C. (ed.) TIC, trabajo colaborativo e interacción en terminología y traducción, Comares, Granada, pp. 191–200 (2014)

    Google Scholar 

  43. Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Overview of translation techniques in cross-language question answering during the last decade. In: Khosrow-Pour, M. (ed.) Encyclopedia of Information Science and Technology, pp. 4747–4755. IGI Global, Hershey (2015)

    Google Scholar 

  44. Olvera-Lobo, M.-D., Gutierrez-Artacho, J.: Question answering track evaluation in TREC, CLEF and NTCIR. In: Rocha, A., Correia, A.M., Costanzo, S., Reis, L.P. (ed.) New Contributions in Information Systems and Technologies, vol. 1, pp. 13–22 (2015)

    Chapter  Google Scholar 

  45. Koehn, P.: Europarl: a parallel corpus for statistical machine translation. MT Summit. 11, 79–86 (2005)

    Google Scholar 

Download references

Acknowledgements

This work was supported by the University of Granada Special Research Programme - Starting Research Grants for Master Students.

Author information

Authors and Affiliations

  1. Department of Translation and Interpreting, Faculty of Translation and Interpreting, University of Granada, C/Buensuceso, 11, 18003, Granada, Spain

    Juncal Gutiérrez-Artacho

  2. Department of Information and Communication, Colegio Máximo de Cartuja, University of Granada, Campus Cartuja s/n, 18071, Granada, Spain

    María-Dolores Olvera-Lobo

  3. CSIC, Unidad Asociada Grupo SCImago, Madrid, Spain

    María-Dolores Olvera-Lobo

  4. University of Granada, Granada, Spain

    Irene Rivera-Trigueros

Authors
  1. Juncal Gutiérrez-Artacho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. María-Dolores Olvera-Lobo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Irene Rivera-Trigueros
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Irene Rivera-Trigueros .

Editor information

Editors and Affiliations

  1. Departamento de Engenharia Informática, Universidade de Coimbra, Coimbra, Portugal

    Álvaro Rocha

  2. College of Engineering, The Ohio State University, Columbus, OH, USA

    Hojjat Adeli

  3. DSI/EEUM, Universidade do Minho, Guimarães, Portugal

    Luís Paulo Reis

  4. DIMES, Università della Calabria, Arcavacata di Rende, Italy

    Sandra Costanzo

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Cite this paper

Gutiérrez-Artacho, J., Olvera-Lobo, MD., Rivera-Trigueros, I. (2018). Human Post-editing in Hybrid Machine Translation Systems: Automatic and Manual Analysis and Evaluation. In: Rocha, Á., Adeli, H., Reis, L.P., Costanzo, S. (eds) Trends and Advances in Information Systems and Technologies. WorldCIST'18 2018. Advances in Intelligent Systems and Computing, vol 745. Springer, Cham. https://doi.org/10.1007/978-3-319-77703-0_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-77703-0_26

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-77702-3

  • Online ISBN: 978-3-319-77703-0

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation