We provide motivation for researchers to replicate experiments in software engineering. The ideology of replication is discussed. We address the question: Is an experiment worth repeating? The current lack of replication studies is highlighted. We make clear that exact replication is unattainable and we draw on our first experience of performing an external replication. To categorise various kinds of replication, we propose a simple extension to Basili et al.’s framework for experimentation in software engineering. We present guidance as to the level of reported detail required to enable others perform a replication. Our conclusion is that there is only one route for empirical software engineering to follow: to make available laboratory packages of experimental materials to facilitate internal and external replications, especially the latter, which have greater confirming power.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
I Amato. Pons and fleischmann redux? Science, 260:895, 1993.
JJ Baroudi and WJ Orlikowski. The problem of statistical power in MIS research. MIS Quarterly, 13:87–106, 1989.
VR Basili, RW Selby, and DH Hutchens. Experimentation in software engineering. IEEE Transactions in Software Engineering, 12(7):733–743, 1986.
VR Basili, F Shull, and F Lanubile. Building knowledge through families of experiments. IEEE Transactions on Software Engineering, 25(4):456–473, 1999.
J Brewer and A Hunter. Multimethod Research: A Synthesis of Styles. SAGE Publications, Newbury Park, CA, 1989.
W Broad and N Wade. Betrayers of the Truth, page 17 and 81. Oxford University Press, New York, 1986.
RE Brooks. Studying programmer behavior experimentally: the problems of proper methodology. Communications of the ACM, 23(4):207–213, 1980.
A Brooks and P Vezza. Inductive analysis applied to the evaluation of a CAL tutorial. Interacting with Computers, the Interdisciplinary Journal of Human-Computer Interaction, 1(2):159–170, 1989.
DN Card. Software quality engineering. Information and Software Technology, 32(1):3–10, 1990.
F Close. Too Hot to Handle The Story of the Race for Cold Fusion. W H Allen Publishing, London, 1990.
HM Collins. Changing Order Replication and Induction in Scientific Practice, pages 19, 35, 43. SAGE Publications, London, 1985.
B Curtis. Measurement and experimentation in software engineering. Proceedings of the IEEE, 68(9):1144–1157, 1980.
J Daly, A Brooks, J Miller, M Roper, and M Wood. An external replication of korson’s experiment. Research report EFoCS-4–94, Department of Computer Science, University of Strathclyde, Glasgow, 1994a.
J Daly, A Brooks, J Miller, M Roper, and M Wood. Verification of results in software maintenance through external replication. In Proceedings of the IEEE International Conference on Software Maintenance, pages 50–57. IEEE, Los Alamitos, CA, 1994b. ICSM’94.
M Goldstein and Inge F Goldstein. HOW WE KNOW An Exploration of the Scientific Process, page 207. Plenum Press, New York and London, 1978.
SM Henry and M Humphrey. A controlled experiment to evaluate maintainability of object-oriented software. In Proceedings of the IEEE Conference on Software Maintenance, pages 258–265, 1990.
TH Huxley. We are all scientists. In H Shapley, S Rapport, and H Wright, editors, The New treasury of Science, page 14. Collins, London and Glasgow, 1965.
A Jedlitschka and D Pfahl. Reporting Guidelines for Controlled Experiments in Software Engineering. Verification of results in software maintenance through external replication. In International Symposium on Empirical Software Engineering, pages 95–104. IEEE, Los Alamitos, CA, 2005. ISESE 2005.
JPJ Kelly, TI McVittie, and WI Yamamoto. Implementing design diversity to achieve fault tolerance. IEEE Software, 8(4):61–71, 1991.
TD Korson. An Empirical Study of the Effects of Modularity on Program Modifiability. PhD thesis, College of Business Administration, Georgia State University, 1986.
TD Korson and VK Vaishnavi. An empirical study of the effects of modularity on program modifiability. In E Soloway and Iyengar S S, editors, Empirical Studies of Programmers: First Workshop, pages 168–186. Ablex Publishing Corporation, Norwood, NJ, 1986. A Volume in the Ablex Human/Computer Interaction Series.
J Lewis, S Henry, D Kafura, and R Schulman. An empirical study of the object-oriented paradigm and software reuse. OOPSLA, 184–196, 1991.
B Lientz and E Swanson. Software Maintenance Management. Addison-Wesley, Reading, MA, 1st edition, 1980.
SG MacDonnell. Rigor in software complexity measurement experimentation. Journal of Systems and Software, 16:141–149, 1991.
DR Moreau and WD Dominick. A programming environment evaluation methodology for object-oriented systems: part ii–test case application. Journal of Object-Oriented Programming, 3(3):23–32, 1990.
KR Popper. The Logic of Scientific Discovery. Hutchinson, London, revised edition, 1968.
M Roper. Software testing: a selected annotated bibliography. Software Testing, Verification and Reliability, 2:113–132, 1992.
DA Scanlan. Structured flowcharts outperform pseudocode: an experimental comparison. IEEE Software, 6(5):28–36, September 1989.
S Sharpe, DA Haworth, and D Hale. Characteristics of empirical software maintenance studies: 1980–1989. Journal of Software Maintenance: Research and Practice, 3:1–15, 1991.
B Shneiderman, R Mayer, D McKay, and P Heller. Experimental investigations of the utility of detailed flowcharts in programming. Communications of the ACM, 20(6):373–381, 1977.
DIK Sjoberg, JE Hannay, O Hansen, VB Kampenes, A Karahasanovíc, N-K Liborg, and AC Rekdal. A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering, 31(9):733–752, 2005.
GP Smith. The problems of reduction and replication in the practice of the scientific method. Annals of the New York Academy of Sciences, 406:1–4, 1983.
University of Maryland Experimental Software Engineering Group. Lab packages. http://www.cs.umd.edu/projects/SoftEng/ESEG/downloads.html.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag London Limited
About this chapter
Cite this chapter
Brooks, A., Roper, M., Wood, M., Daly, J., Miller, J. (2008). Replication's Role in Software Engineering. In: Shull, F., Singer, J., Sjøberg, D.I.K. (eds) Guide to Advanced Empirical Software Engineering. Springer, London. https://doi.org/10.1007/978-1-84800-044-5_14
Download citation
DOI: https://doi.org/10.1007/978-1-84800-044-5_14
Publisher Name: Springer, London
Print ISBN: 978-1-84800-043-8
Online ISBN: 978-1-84800-044-5
eBook Packages: Computer ScienceComputer Science (R0)