Empirical Paradigm – The Role of Experiments
Abstract
This article discusses the role of formal experiments in empirical software engineering. I take the view that the role of experiments has been overemphasised. Laboratory experiments are not representative of industrial software engineering tasks, so do not provide us with a reliable assessment of the effect of our techniques and tools. I suggest we need to concentrate a larger proportion of our research effort on industrial quasi-experiments and case studies. Methodologies for these empirical methods are well-understood in the social science and would appear to be appropriate mechanisms for investigating many software engineering research questions. In addition, I believe we need to make the results of empirical software engineering more visible and relevant to practitioners. To influence practitioners I suggest that we need to produce evidence-based text books and evidence-based software engineering standards.
Preview
Unable to display preview. Download preview PDF.
References
- 1.Jørgensen, M., Carelius, G.J.: An Empirical Study of Software Project Bidding. IEEE Transactions of Software Engineering 30(12), 953–969 (2004)CrossRefGoogle Scholar
- 2.Petticrew, M., Roberts, H.: Systematic Reviews in the Social Sciences. A practical Guide. Blackwell Publishing, Oxford (2006)Google Scholar
- 3.Shaddish, W.R., Cook, T.D., Campbell, D.T.: Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin Company, Boston (2002)Google Scholar
- 4.El Emam, K., Birk, A.: Validating ISO/IEC 15504 measures of software development process capability. The Journal of Systems and Software 51, 119–149 (2000a)CrossRefGoogle Scholar
- 5.El Emam, K., Birk, A.: Validating ISO/IEC 15504 measures of software Requirements Analysis Process Capability. IEEE Transactions on Software Engineering 26(6), 541–566 (2000b)CrossRefGoogle Scholar
- 6.Dion, R.: Process Improvement and the Corporate Balance Sheet. IEEE Software 10(4), 28–35 (1993)CrossRefGoogle Scholar
- 7.Steen, H.U.: Reporting framework-based software process improvement. A quantitative and qualitative review of 71 experience reports of CMM-based SPI. Master Thesis, Simula Research Laboratory & Department of Informatics University of Oslo, 29th October (2004)Google Scholar
- 8.McGarry, F., Burke, S., Decker, B.: Measuring the Impacts Individual process Maturity Attributes have on Software Products. In: Proceedings Fifth International Software Metrics Symposium, IEEE Computer Society, pp. 52–60. IEEE Computer Society Press, Los Alamitos (1998)CrossRefGoogle Scholar
- 9.Yin, R.K.: Case Design and Methods, 3rd edn. Sage Publications, Thousand Oaks (2004)Google Scholar
- 10.Kitchenham, B., Budgen, D., Breton, P., Turner, M., Charters, S., Linkman, S.: Large Scale Software Engineering Questions – Expert Opinion or Empirical Evidence? Experience and Methods from Integrating Evidence-based Software Engineering into Education. In: Proceedings 4th International Workshop WSESE2003, Fraunhofer IESE-Report No., 068.06/E (2006)Google Scholar
- 11.Rainer, A., Jagielska, D., Hall, T.: Software Practice versus evidence-based software engineering research. In: Proceedings of the Workshop on Realising Evidence-based Software Engineering, ICSE-2005 (2005), http:/cfm.portal.acm.org/dlGoogle Scholar
- 12.Endres, A., Rombach, H.D.: Empirical Software and Systems Engineering: A Handbook of Observations, Laws and Theories. Addison Wesley, Reading (2003)Google Scholar
- 13.Kitchenham, B.: Procedures for Undertaking Systematic Reviews. Joint Technical Report, Computer Science Department, Keele University (TR/SE-0401) and National ICT Australia Ltd. (0400011T.1) (2004)Google Scholar
- 14.ISO/IEC 25000. International Standard. Software Engineering – Software product Quality Requirements and Evaluation (SQuaRE) – Guide to SQuaRE (2005-08-01)Google Scholar
- 15.Koyani, S.J., Bailey, R.W., Nall, J.R.: Research based web design and usability guidelines. National Cancer Institute (2003), Available for download at http://usability.gov/pdfs/guidelines.html