The role of replications in Empirical Software Engineering
Replications play a key role in Empirical Software Engineering by allowing the community to build knowledge about which results or observations hold under which conditions. Therefore, not only can a replication that produces similar results as the original experiment be viewed as successful, but a replication that produce results different from those of the original experiment can also be viewed as successful. In this paper we identify two types of replications: exact replications, in which the procedures of an experiment are followed as closely as possible; and conceptual replications, in which the same research question is evaluated by using a different experimental procedure. The focus of this paper is on exact replications. We further explore them to identify two sub-categories: dependent replications, where researchers attempt to keep all the conditions of the experiment the same or very similar and independent replications, where researchers deliberately vary one or more major aspects of the conditions of the experiment. We then discuss the role played by each type of replication in terms of its goals, benefits, and limitations. Finally, we highlight the importance of producing adequate documentation for an experiment (original or replication) to allow for replication. A properly documented replication provides the details necessary to gain a sufficient understanding of the study being replicated without requiring the replicator to slavishly follow the given procedures.
KeywordsExperimental replications Experimental infrastructure Lab package
- Brooks A, Roper M et al (2007) Replication's role in software engineering. In: Shull F, Singer J, Sjoberg DIK (eds) Guide to advanced empirical software engineering. Springer, London, pp 365–379Google Scholar
- Cozby (2007) Methods in behavioral research. McGraw-Hill, New YorkGoogle Scholar
- Vegas S, Juristo N et al (2006) Analysis of the influence of communication between researchers on an experiment replication. Proceedings of 5th International Symposium on Empirical Software Engineering, Rio de Janeiro, Brazil, pp 28–37Google Scholar