Creating Real Value in Software Engineering Experiments
Undertaking empirical work to understand the software engineering process is an increasingly common activity. However, due to the nature of most software engineering studies, drawing reliable conclusions from a single study is inherently dangerous. A single study undertaken by a small group of practitioners, in a single location, with a single group of subjects, will possess a large number of parameters, some controlled, some completely unconstrained. Any one of which can potentially cause a significant change in the result of the study; and hence their effect must be controlled, eliminated or understood, if reliable results are to be derived. The normal method of achieving this objective is to repeat the study, often while changing some of the parameters, to see if the original result is stable with regard to repetition and alteration of some of its components. Without this confirming power, empirically based claims in software engineering should be regarded as having at best limited value and at worst with suspicion and mistrust. Hence, despite their now significant volume, software engineering experiments impact has been rather limited upon industrial software processes. The potential problems with single software engineering empirical studies are three-fold: low statistical power; large number of potential covariates with the treatment variable; and the verification of the process and products of the study.