The role of replications in empirical software engineering—a word of warning
- 341 Downloads
There are many issues raised by Shull et al. that I would whole-heartedly endorse. For example, replication is a basic component of the scientific method, so it hardly needs to be justified. Furthermore, a full comprehensive report of any empirical study is a requirement of good science. When restrictions on the length of conference or journal paper prevent such reporting, it is an extremely good idea to record all the experimental details somewhere, whether in a “laboratory package” or a technical report. However, is such supporting information essential for subsequent replications? The authors give examples such as insufficient training, or systems not having defects appropriate for the technique, causing replications to wrongly contradict the original experiment. I believe issues such as the criticality of training or the type of relevant defects should be reported in the published study not hidden in ancillary data. It may be preferable to advise empirical researchers to make very...
KeywordsSoftware Engineering Systematic Literature Review Original Experiment Independent Replication Ancillary Data
- Kampenes V (2007) Quality of design, analysis and reporting of software engineering experiments a systematic review. Ph.D. Thesis, Department of Informatics, University of Oslo, NorwayGoogle Scholar
- Shaddish WR, Cook TD, Campbell TD (2002) Experimental and quasi-experimental designs for generalized casual inference. Houghton Mifflin Company, Boston, MAGoogle Scholar