Skip to main content
SpringerLink
Log in

Assessing the impact of real-time machine translation on multilingual meetings in global software projects

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Communication in global software development is hindered by language differences in countries with a lack of English speaking professionals. Machine translation is a technology that uses software to translate from one natural language to another. The progress of machine translation systems has been steady in the last decade. As for now, machine translation technology is particularly appealing because it might be used, in the form of cross-language chat services, in countries that are entering into global software projects. However, despite the recent progress of the technology, we still lack a thorough understanding of how real-time machine translation affects communication. In this paper, we present a set of empirical studies with the goal of assessing to what extent real-time machine translation can be used in distributed, multilingual requirements meetings instead of English. Results suggest that, despite far from 100 % accurate, real-time machine translation is not disruptive of the conversation flow and, therefore, is accepted with favor by participants. However, stronger effects can be expected to emerge when language barriers are more critical. Our findings add to the evidence about the recent advances of machine translation technology and provide some guidance to global software engineering practitioners in regarding the losses and gains of using English as a lingua franca in multilingual group communication, as in the case of computer-mediated requirements meetings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. http://www.apertium.org

  2. http://translate.google.com

  3. http://json.org

  4. http://www.statmt.org/moses

  5. http://www.bing.com/translator

  6. http://logos-os.dfki.de

  7. In English: “It’s not so complicated”

  8. www.cambridge.org/us/esl/venturesadulted/placement_test.html

  9. SVN revision 19832

  10. www.molto-project.eu

References

  • Altman DG (1991) Practical statistics for medical research. Chapman and Hall, London

    Google Scholar 

  • Arnold D (2003) Why translation is difficult for computers. In Computers and translation: a translator’s guide. Benjamins Translation Library

  • Arnold D, Balkan L, Meijer S, Humphreys RL, Sadler L (1994) Machine translation: an introductory guide. NCC Blackwell, London

  • Aziz W, Sousa SCM, Specia L (2012) PET: a tool for post-editing and assessing machine translation. Proc 8th Int’l Conf Lang Resour Eval (LREC’12):3982–3987

  • Berander P (2004) Using students as subjects in requirements prioritization. Int’l Symp Empir Softw Eng (ISESE’04):167–176. doi:10.1109/ISESE.2004.34

  • Brazil IT-BPO Book 2010–2011 (2013) Published by Brasscom, Brazilian Association of Information Technology and Communication Companies, São Paulo, Brazil

  • Burchardt A, Tscherwinka C, Eleftherios A, Uszkoreit H (2013) Machine translation at work. Computational Linguistics, Studies in Computational Intelligence, Springer, 458:241–261. doi:10.1007/978-3-642-34399-5_13

  • Calefato F, Lanubile F (2007) Using frameworks to develop a distributed conferencing system: an experience report. Softw Pract Exper 39(15):1293–1311. doi:10.1002/spe.937

    Article  Google Scholar 

  • Calefato F, Damian D, Lanubile F (2007) An empirical investigation on text-based communication in distributed requirements engineering. Proc 2nd Int’l Conf Glob Softw Eng (ICGSE’07):27–30. doi:10.1109/ICGSE.2007.9

  • Calefato F, Lanubile F, Minervini P (2010) Can real-time machine translation overcome language barriers in distributed requirements engineering? Proc 5th Int’l Conf Glob Softw Eng (ICGSE’10):257–264. doi:10.1109/ICGSE.2010.37

  • Calefato F, Lanubile F, Prikladnicki R (2011) A controlled experiment on the effects of machine translation in multilingual requirements meetings. Proc 6th Int’l Conf Glob Softw Eng (ICGSE’11):94–102. doi:10.1109/ICGSE.2011.14

  • Calefato F, Damian D, Lanubile F (2012a) Computer-mediated communication to support distributed requirements elicitations and negotiations tasks. Empir Softw Eng J 17(6):640–674. doi:10.1007/s10664-011-9179-3

    Article  Google Scholar 

  • Calefato F, Lanubile F, Conte T, Prikladnicki R (2012b) Assessing the impact of real-time machine translation on requirements meetings: a replicated experiment. Proc of the 6th Int’l Symp Empir Softw Eng and Meas (ESEM’12):251–260. doi:10.1145/2372251.2372299

  • Carmel E, Agarwal R (2001) Tactical approaches for alleviating distance in global software development. IEEE Softw 18(2):22–29. doi:10.1109/52.914734

    Article  Google Scholar 

  • Clark HH, Brennan SE (1991) Grounding in communication, in perspectives on socially shared cognition. American Psychological Association, Washington DC, pp 127–149

    Book  Google Scholar 

  • Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46. doi:10.1177/001316446002000104

    Article  Google Scholar 

  • Cohen J (1992) A power primer. Psychol Bull 112(1):155–159. doi:10.1037/0033-2909.112.1.155

    Article  Google Scholar 

  • Cohn M (2005) Agile estimating and planning. Prentice Hall

  • Conover WJ (1980) Practical nonparametric statistics. Wiley, New York

    Google Scholar 

  • Cronbach LJ (1951) Coefficient alpha and the internal structure of tests. Psychometrika 16:297–334

    Article  Google Scholar 

  • Damian D (2007) Stakeholders in global requirements engineering: lessons learned from practice. IEEE Softw 24(2):21–27. doi:10.1109/MS.2007.55

    Article  Google Scholar 

  • Damian D, Zowghi D (2003) Requirements engineering challenges in multi-site software development organizations. Requir Eng J 8(3):149–160. doi:10.1007/s00766-003-0173-1

    Article  Google Scholar 

  • DeSanctis D, Gallupe RB (1987) A foundation for the study of group decision support systems. Manag Sci 33(5):589–609. doi:10.1287/mnsc.33.5.589

    Article  Google Scholar 

  • Fleiss JL (1981) Statistical methods for rates and proportions, 2nd edn. Wiley, New York, pp 38–46

    MATH  Google Scholar 

  • Gao G, Wang H-C, Cosley D, Fussell SR (2013) Same translation but different experience: the effects of highlighting on machine-translated conversations. In Proc SIGCHI Conf Hum Factors Comput Syst (CHI ‘13):449–458. doi:10.1145/2470654.2470719

  • Garland R (1991) The mid-point on a rating scale: is it desirable? Mark Bull 2:66–70

    Google Scholar 

  • Gottesdiener E (2002) Requirements by collaboration: workshops for defining needs. Addison-Wesley Longman Publishing Co., Inc.

  • Herbsleb JD, Atkins DL, Boyer DG, Handel M, Finholt TA (2002) Introducing Instant messaging and chat into the workplace. Proc Int’l Conf Comput Hum Interact (CHI ‘02):171–178. doi: 10.1145/503376.503408

  • Höst M, Regnell B, Wohlin C (2000) Using students as subjects - a comparative study of students and professionals in lead-time impact assessment. Empir Softw Eng 5(3):201–214. doi:10.1023/A:1026586415054

    Article  MATH  Google Scholar 

  • Hsieh Y (2006) Culture and shared understanding in distributed requirements engineering. 1st Int’l Conf Glob Softw Eng (ICGSE’06):101–108. doi:10.1109/ICGSE.2006.261221

  • Johns R (2005) One size doesn’t fit all: selecting response scales for attitude items. J Elections Public Opin Parties 15(2):237–264. doi:10.1080/13689880500178849

    Article  MathSciNet  Google Scholar 

  • Jurafsky D, Martin JH (2008) Speech and language processing 2nd ed. Prentice Hall Series in Artificial Intelligence, Prentice Hall

  • Kearney AT (2007) Destination Latin America: a nearshore alternative, technical report

  • KPMG (2009) Nearshore attraction: Latin America Beckons as a global outsourcing destination. Technical Report

  • Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174. doi:10.2307/2529310

    Article  MathSciNet  MATH  Google Scholar 

  • Lehtola L, Kauppinen M, Kujala S (2004) Requirements prioritization challenges in practice. LNCS: Product Focused Software Process Improvement 3009:497–508. doi:10.1007/978-3-540-24659-6_36

  • Lutz B (2009) Linguistic challenges in global software development: lessons learned in an International SW Development Division. Proc 4th Int’l Conf Glob Softw Eng (ICGSE’09):249–253. doi:10.1109/ICGSE.2009.33

  • Macaulay LA (1996) Requirements engineering, Springer-Verlag Telos

  • Mitkov R (2003) The Oxford handbook of computational linguistics. Oxford Handbooks in Linguistics, Oxford University Press

  • Montgomery DC (1996) Design and analysis of experiments. Wiley, New York

    Google Scholar 

  • Mudge RS (2009) After the deadline – language checking technology. Automattic. http://open.afterthedeadline.com

  • Nunnally JC (1978) Psychometric theory (2nd ed.). McGraw-Hill, New York

  • Ogden W, Zacharski Z, An S, Ishikawa Y (2009) User choice as an evaluation metric for web translation in cross language instant messaging applications. Proc Mach Transl Summit VII

  • Paulson LD (2001) Translation technology tries to hurdle the language barrier. Computer 34(9):12–15. doi:10.1109/MC.2001.947080

    Article  Google Scholar 

  • Prikladnicki R, Carmel E (2013) Is time zone proximity an advantage for software development? The case of the Brazilian I.T. industry. Int’l Conf Softw Eng (ICSE’13):973–981. doi:10.1109/ICSE.2013.6606647

  • Rautio, J, Koponen, M (2013) “MOLTO evaluation and assessment report.” Technical Report

  • Raybaud S, Langlois D, Smaïli K (2011) ‘This sentence is wrong’. Detecting errors in machine-translated sentences. Mach Transl 25(11):1–35. doi:10.1007/s10590-011-9094-9

    Article  Google Scholar 

  • Shah YH, Raza M, UlHaq S (2012) Communication issues in GSD. Int’l J Adv Sci Technol 40:69–76

    Google Scholar 

  • Stemler S (2001) An overview of content analysis. Practical assessment, research & evaluation, 7(17)

  • Wisniewski G, Kumar Singh A, Yvon F (2012) Quality estimation for machine translation: some lessons learned. Mach Transl 27(3–4):213–238. doi:10.1007/s10590-013-9141-9

    Google Scholar 

  • Wohlin C, Runesson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2000) Experimentation in software engineering, an introduction. Kluwer Academic Publishers

  • Yamashita N, Ishida T (2006) Effects of machine translation on collaborative work. Proc 20th Int’l Conf Comput Supported Coop Work (CSCW ‘06):515–524. doi:10.1145/1180875.1180955

  • Yamashita N, Inaba R, Kuzuoka H, Ishida T (2009) Difficulties in establishing common ground in multiparty groups using machine translation. Proc 27th Int’l Conf Hum Factors Comput Syst (CHI ‘09):679–688. doi:10.1145/1518701.1518807

Download references

Acknowledgments

This research is partially funded by the Rio Grande do Sul State funding agency (FAPERGS) under projects 11/2022-3 and 002061-2551/13, and by CNPq (483125/2010-5, and 309000/2012-2). We would like to thank all the participants who took part in the experiment and Pasquale Minervini for the support he provided with the management of the Apertium service.

Author information

Authors and Affiliations

  1. Dipartimento di Informatica, University of Bari, Bari, Italy

    Fabio Calefato & Filippo Lanubile

  2. Instituto de Computação, Universidade Federal do Amazonas, Manaus, Brazil

    Tayana Conte

  3. Pontifícia Universidade Católica do Rio Grande do Sul, Porto Alegre, Brazil

    Rafael Prikladnicki

Authors
  1. Fabio Calefato
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Filippo Lanubile
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tayana Conte
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rafael Prikladnicki
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fabio Calefato.

Additional information

Communicated by: Nachiappan Nagappan

Appendix: Experimental Data

Appendix: Experimental Data

Table 13 Categories of the coding schema (thematic units) in content analysis
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Calefato, F., Lanubile, F., Conte, T. et al. Assessing the impact of real-time machine translation on multilingual meetings in global software projects. Empir Software Eng 21, 1002–1034 (2016). https://doi.org/10.1007/s10664-015-9372-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-015-9372-x

Keywords

Search

Navigation