Empirical Software Engineering

, Volume 19, Issue 3, pp 501–557 | Cite as

Replication of empirical studies in software engineering research: a systematic mapping study

  • Fabio Q. B. da SilvaEmail author
  • Marcos Suassuna
  • A. César C. França
  • Alicia M. Grubb
  • Tatiana B. Gouveia
  • Cleviton V. F. Monteiro
  • Igor Ebrahim dos Santos


In this article, we present a systematic mapping study of replications in software engineering. The goal is to plot the landscape of current published replications of empirical studies in software engineering research. We applied the systematic review method to search and select published articles, and to extract and synthesize data from the selected articles that reported replications. Our search retrieved more than 16,000 articles, from which we selected 96 articles, reporting 133 replications performed between 1994 and 2010, of 72 original studies. Nearly 70 % of the replications were published after 2004 and 70 % of these studies were internal replications. The topics of software requirements, software construction, and software quality concentrated over 55 % of the replications, while software design, configuration management, and software tools and methods were the topics with the smallest number of replications. We conclude that the number of replications has grown in the last few years, but the absolute number of replications is still small, in particular considering the breadth of topics in software engineering. We still need incentives to perform external replications, better standards to report empirical studies and their replications, and collaborative research agendas that could speed up development and publication of replications.


Replications Experiments Empirical studies Mapping study Systematic literature review Software engineering 



Fabio Q. B. da Silva holds a research grant from the Brazilian National Research Council (CNPq), process #314523/2009-0. This article was written while Prof. Fabio Silva was in a sabbatical leave at the University of Toronto, receiving a CAPES research grant process # 6441/10-6. A. César C. França is a doctoral student at the Center of Informatics of the Federal University of Pernambuco where he receives a scholarship from the Brazilian National Research Council (CNPq), process #141156/2010-4. We would like to thank Prof. Steve Easterbrook, Jonathan Lung, and Elizabeth Patitsas for many discussions, comments, and criticisms that lead to important improvements in the content and structure of this article. We also thank Prof. André Santos, Rodrigo Lopes, João Paulo Oliveira, and Leonardo Oliveira, for their participation in the earlier version of this study published at RESER’2011. Finally, we are grateful for the partial support of the Samsung Institute for Development of Informatics (Samsung/SIDI) for this research.


  1. Abran A, Moore J, Bourque P, Dupuis T (Eds.) (2004) Guide to software engineering body of knowledge, IEEE Computer Society. 204Google Scholar
  2. Almqvist JPF (2006) Replication of controlled experiments in empirical software engineering —a survey. Master’s Thesis, Department of Computer Science, Faculty of Science, Lund University, Sweden. 129Google Scholar
  3. Arksey H, O’Malley L (2005) Scoping studies: towards a methodological framework. Int J Soc Res Meth 8:19–32CrossRefGoogle Scholar
  4. Basili V et al (1999) Building knowledge through families of experiments. IEEE Trans Software Eng 25:456–473. doi: 10.1109/32.799939 CrossRefGoogle Scholar
  5. Brooks A et al. (1995) Replication of Experimental Results in Software Engineering. Technical Report, EFoCS-17-95 [RR/95/193], Dept. of Computer Science, Univ. of Strathclyde. 38Google Scholar
  6. Brooks A et al. (2007) Replication’s role in software engineering. In F Shull, J Singer, and DIK Sjøberg (eds) Guide to Advanced Empirical Software Engineering. Springer, pp 365–379Google Scholar
  7. Carver JC. (2010) Towards Reporting Guidelines for Experimental Replications: A Proposal. In RESER’2010: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research, Cape Town, South Africa. 4Google Scholar
  8. Carver JC et al. (2003) Issues in using students in empirical studies in soft —ware engineering education. In Proceedings of the 9th International Software Metrics Symposium (METRICS2003), pp239–249Google Scholar
  9. Ciolkowski M et al. (2004) Using academic courses for empirical validation of software development processes. In Proceedings of the 30th Euromicro Conference, pp 354–361Google Scholar
  10. Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46. doi: 10.1177/001316446002000104 CrossRefGoogle Scholar
  11. da Silva FQB et al (2011a) Six years of systematic literature reviews in software engineering: an updated tertiary study. Inform Software Tech 53(9):899–913. doi: 10.1016/j.infsof.2011.04.004 CrossRefGoogle Scholar
  12. da Silva FQB et al. (2011b) Replication of empirical studies in software engineering: Preliminary findings from a systematic mapping study. Proceedings of the 2nd International Workshop on Replication in Empirical Software Engineering Research RESER’2011, pp 61–70Google Scholar
  13. Daly J, Brooks A, Miller J, Roper M, Wood M (1994) Verification of Results in Software Maintenance Through External Replication. IEEE International Conference on Software Maintenance, pp. 50–57Google Scholar
  14. Davidsen MK, Krogstie J (2010) A longitudinal study of development and maintenance. Inform Software Tech 52(7):707–719CrossRefGoogle Scholar
  15. Dybå T, Dingsøyr T (2008) Empirical studies of agile software development: a systematic review. Inform Software Tech 50:833–859CrossRefGoogle Scholar
  16. Easterbrook SM et al. (2007) Selecting Empirical Methods for Software Engineering Research.. In: F Shull, J Singer and D Sjøberg (eds.) Guide to Advanced Empirical Software Engineering. Springer, pp 285–311Google Scholar
  17. França A César C et al. (2010) The Effect of Reasoning Strategies on Success in Early Learning of Programming: Lessons Learned from an External Experiment Replication. In EASE’2010: 14th International Conference on Evaluation and Assessment in Software Engineering, Keele University, UK. 10Google Scholar
  18. Gómez G, Omar S, Juristo N, Vegas N (2010a) Replication, Reproduction and Re-analysis: Three ways for verifying experimental findings. In RESER’2010: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research. Cape Town, South Africa. pp 42–44Google Scholar
  19. Gómez G, Omar S, Juristo N, Vegas N (2010b) Replications Types in Experimental Disciplines. In ESEM’2010: Proceedings of the ACM/IEEE 4th International Symposium on Empirical Software Engineering and Measurement, September 16–17, Bolzano-Bozen, Italy. pp. 1–10Google Scholar
  20. Gould J, Kolb WL (eds) (1964) A dictionary of the social sciences. Tavistock Publications, London, 761Google Scholar
  21. Holgeid KK, Krogstie J, Sjøberg DIK (2000) A study of development and maintenance in Norway: assessing the efficiency of information systems support using functional maintenance. Inform Software Tech 42:687–700CrossRefGoogle Scholar
  22. Juristo N, Vegas S (2009) Using differences among replications of software engineering experiments to gain knowledge. In ESEM’09: Proceedings of the ACM/IEEE 3rd International Symposium on Empirical Software Engineering and Measurement. IEEE Computer Society, Washington, DC, USA, pp 356–366Google Scholar
  23. Kitchenham B (2008) The role of replications in empirical software engineering—a word of warning. Empir Software Eng 13:219–221CrossRefGoogle Scholar
  24. Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering, Technical Report EBSE-2007-01, School of Computer Science and Mathematics, Keele UniversityGoogle Scholar
  25. Kitchenham BA, Pfleeger SL (2007) Personal Opinion Surveys. In: F Shull, J Singer, D. Sjøberg (eds), pp. 63–92, Guide to Advanced Empirical Software Engineering, SpringerGoogle Scholar
  26. Kitchenham B, Dybå T, Jørgensen M (2004) Evidence-based Software Engineering. In ICSE’2004: Proceedings of the 26th International Conference on Software Engineering, Washington DC, USA. pp 273–281Google Scholar
  27. Kitchenham B et al (2010) Literature reviews in software engineering—a tertiary study. Inform Software Tech 52:792–805CrossRefGoogle Scholar
  28. Krein Jonathan L, Knutson Charles D (2010) A Case for Replication: Synthesizing Research Methodologies in Software Engineering. In RESER’2010: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research, Cape Town, South Africa. 10Google Scholar
  29. Krogstie J, Sølvberg A (1994) Software Maintenance in Norway: a survey investigation. In ICSM’1994: Proceedings of the International Conference on Software Maintenance. pp 304–313Google Scholar
  30. Krogstie J, Jahr A, Sjøberg DIK (2006) A longitudinal study of development and maintenance in Norway: report from the 2003 investigation. Inform Softw Technol 48:993–1005CrossRefGoogle Scholar
  31. La Sorte MA (1972) Replication as a verification technique in survey research: a paradigm. Socio Q 13(2):219–227CrossRefGoogle Scholar
  32. Lindsay RM, Ehrenberg A (1993) The design of replicated studies. Am Stat 47(3):217–228Google Scholar
  33. Lung J et al. (2008) On the difficulty of replicating human subjects studies in software engineering. In ICSE’2008: Proceedings of the 13th international conference on Software engineering, New York, USA: ACM Press. pp 191–201Google Scholar
  34. Petticrew M, Roberts H (2006) Systematic Reviews in the Social Sciences. Blackwell Publishing. 336Google Scholar
  35. Popper K (1959) The Logic of Scientific Discovery. Hutchinson & Co. 513Google Scholar
  36. Schmidt S (2009) Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Rev Gen Psychol 13:90–100. doi: 10.1037/a0015108 CrossRefGoogle Scholar
  37. Shull F, Basili V, Carver J, Maldonado JC, Travassos GH, Mendonça M, Fabbri S (2002) Replicating software engineering experiments: Addressing the tacit knowledge problem. In ISESE’2002: Proc. Int. Symp. on Empirical Softw. Eng., Washington, DC, USA, IEEE Computer Society. 10Google Scholar
  38. Shull F, Carver J, Vegas S, Juristo N (2008) The Role of Replications in Empirical Software Engineering. Empir Software Eng 13:211–218CrossRefGoogle Scholar
  39. Sjøberg D (2010) Confronting the myth of rapid obsolescence in computing research. Commun ACM 53(9):62–67CrossRefGoogle Scholar
  40. Sjøberg D et al (2005) A survey of controlled experiments in software engineering. IEEE Trans Software Eng 31:733–753CrossRefGoogle Scholar
  41. Vegas S et al. (2006) Analysis of the Influence of Communication between Researchers on Experiment Replication. In ISESE’2006: Proceedings of the 5th International Symposium on Empirical Software Engineering. September 20–21, Rio de Janeiro, Brazil. pp 28–37Google Scholar
  42. Yin RK (2009) Case study research: Design and methods, 4th edn. Sage Publications, London, 240Google Scholar
  43. Zhang H, Babarb MA, Tell P (2010) Identifying relevant studies in software engineering. Inform Software Tech 53(6):625–637, CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  • Fabio Q. B. da Silva
    • 1
    Email author
  • Marcos Suassuna
    • 1
  • A. César C. França
    • 1
  • Alicia M. Grubb
    • 2
  • Tatiana B. Gouveia
    • 1
  • Cleviton V. F. Monteiro
    • 1
  • Igor Ebrahim dos Santos
    • 1
  1. 1.Centre for InformaticsFederal University of PernambucoRecifeBrazil
  2. 2.Department of Computer ScienceUniversity of TorontoTorontoCanada

Personalised recommendations