Empirical Software Engineering

, Volume 13, Issue 2, pp 211–218 | Cite as

The role of replications in Empirical Software Engineering

  • Forrest J. Shull
  • Jeffrey C.  CarverEmail author
  • Sira Vegas
  • Natalia Juristo


Replications play a key role in Empirical Software Engineering by allowing the community to build knowledge about which results or observations hold under which conditions. Therefore, not only can a replication that produces similar results as the original experiment be viewed as successful, but a replication that produce results different from those of the original experiment can also be viewed as successful. In this paper we identify two types of replications: exact replications, in which the procedures of an experiment are followed as closely as possible; and conceptual replications, in which the same research question is evaluated by using a different experimental procedure. The focus of this paper is on exact replications. We further explore them to identify two sub-categories: dependent replications, where researchers attempt to keep all the conditions of the experiment the same or very similar and independent replications, where researchers deliberately vary one or more major aspects of the conditions of the experiment. We then discuss the role played by each type of replication in terms of its goals, benefits, and limitations. Finally, we highlight the importance of producing adequate documentation for an experiment (original or replication) to allow for replication. A properly documented replication provides the details necessary to gain a sufficient understanding of the study being replicated without requiring the replicator to slavishly follow the given procedures.


Experimental replications Experimental infrastructure Lab package 


  1. Basili V, Shull F et al (1999) Building knowledge through families of experiments. IEEE Trans Softw Eng 25(4):456–473CrossRefGoogle Scholar
  2. Brooks A, Roper M et al (2007) Replication's role in software engineering. In: Shull F, Singer J, Sjoberg DIK (eds) Guide to advanced empirical software engineering. Springer, London, pp 365–379Google Scholar
  3. Cozby (2007) Methods in behavioral research. McGraw-Hill, New YorkGoogle Scholar
  4. Lindvall M, Rus I et al (2005) An evolutionary testbed for software technology evolution. Innovations in Systems and Software Engineering—A NASA Journal 1(1):3–11CrossRefGoogle Scholar
  5. Miller J (2000) Applying meta-analytical procedures to software engineering experiments. J Syst Softw 54(1):29–39CrossRefGoogle Scholar
  6. Shull F, Mendonca M et al (2004) Knowledge-sharing issues in experimental software engineering. Empirical Software Engineering—An International Journal 9(1):111–137CrossRefGoogle Scholar
  7. Vegas S, Juristo N et al (2006) Analysis of the influence of communication between researchers on an experiment replication. Proceedings of 5th International Symposium on Empirical Software Engineering, Rio de Janeiro, Brazil, pp 28–37Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Forrest J. Shull
    • 1
  • Jeffrey C.  Carver
    • 2
    Email author
  • Sira Vegas
    • 3
  • Natalia Juristo
    • 3
  1. 1.Fraunhofer Center for Experimental Software EngineeringCollege ParkUSA
  2. 2.Mississippi State UniversityMississippi StateUSA
  3. 3.Universidad Politécnica de MadridMadridSpain

Personalised recommendations