Skip to main content
Log in

Experimental design and analysis in software engineering

  • Published:
Annals of Software Engineering


The paper presents key activities necessary for designing and analyzing an experiment in software engineering. After explaining how to choose an appropriate research technique to fit project goals, the paper shows how to state a hypothesis and determine how much control is needed over the variables involved. If control is not possible, then a formal experiment is not possible; a case study may be a better approach. Next, the six stages of an experiment (conception, design, preparation, execution, analysis and dissemination) are described, with design examined in detail. Design considerations such as replication, randomization and local control are discussed, and design techniques such as crossing and nesting are explained. Finally, data analysis is shown to be a function both of the experimental design and the distribution of the data. Throughout, examples are given to show how the techniques are interpreted and used in software engineering.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others


  • Caulcutt, R. (1991),Statistics in Research and Development, Chapman and Hall, London, England.

    Google Scholar 

  • Chatfield, C. (1993),Statistics for Technology, Chapman and Hall, England.

    Google Scholar 

  • Fenton, N. S.L. Pfleeger, and R.L. Glass (1994), “Science and Substance: A Challenge to Software Engineers”,IEEE Software 11, 4, 86–95.

    Google Scholar 

  • Green, J. and M. d'Oliveira (1990),Units 16 & 21 Methodology Handbook (part 2), Open University, Milton Keynes, England.

    Google Scholar 

  • Kitchenham, B. (1992),DESMET Handbook of Data Collection and Metrication Book 3: Analysing Software Data, National Computer Centre, Manchester, England.

    Google Scholar 

  • Lee, W. (1975),Experimental Design and Analysis, W.H. Freeman and Company, San Francisco, CA.

    Google Scholar 

  • McCabe, T.J. (1976), “A Complexity Measure”,IEEE Transactions on Software Engineering SE-2, 4, 308–320.

    Google Scholar 

  • Ostle, B. and L.C. Malone (1988),Statistics in Research, Fourth Edition, Iowa State University Press, Ames, Iowa.

    Google Scholar 

  • Pfleeger, S.L. (1993),DESMET Experimental Design and Analysis Procedures (EXPDA), National Computing Centre, Manchester, England.

    Google Scholar 

Download references

Author information

Authors and Affiliations


Rights and permissions

Reprints and permissions

About this article

Cite this article

Pfleeger, S.L. Experimental design and analysis in software engineering. Ann Software Eng 1, 219–253 (1995).

Download citation

  • Issue Date:

  • DOI: