Empirical Software Engineering

, Volume 13, Issue 2, pp 219–221 | Cite as

The role of replications in empirical software engineering—a word of warning

  • Barbara Kitchenham

There are many issues raised by Shull et al. that I would whole-heartedly endorse. For example, replication is a basic component of the scientific method, so it hardly needs to be justified. Furthermore, a full comprehensive report of any empirical study is a requirement of good science. When restrictions on the length of conference or journal paper prevent such reporting, it is an extremely good idea to record all the experimental details somewhere, whether in a “laboratory package” or a technical report. However, is such supporting information essential for subsequent replications? The authors give examples such as insufficient training, or systems not having defects appropriate for the technique, causing replications to wrongly contradict the original experiment. I believe issues such as the criticality of training or the type of relevant defects should be reported in the published study not hidden in ancillary data. It may be preferable to advise empirical researchers to make very...


Software Engineering Systematic Literature Review Original Experiment Independent Replication Ancillary Data 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Kampenes V (2007) Quality of design, analysis and reporting of software engineering experiments a systematic review. Ph.D. Thesis, Department of Informatics, University of Oslo, NorwayGoogle Scholar
  2. Kitchenham BA, Mendes E, Travassos GH (2007) Cross versus within-company cost estimation studies: a systematic review. IEEE Trans Softw Eng 33(5):316–329CrossRefGoogle Scholar
  3. Shaddish WR, Cook TD, Campbell TD (2002) Experimental and quasi-experimental designs for generalized casual inference. Houghton Mifflin Company, Boston, MAGoogle Scholar
  4. Sjøberg DIK, Hannay JE, Hansen O, By Kampenes V, Karahasanovic A, Liborg N-K, Rekdal AC (2005) A survey of controlled experiments in software engineering. IEEE Trans Softw Eng 31(9):733–753CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  1. 1.School of Computing and MathematicsKeele UniversityKeeleUK

Personalised recommendations