Selecting Empirical Methods for Software Engineering Research

  • Steve Easterbrook
  • Janice Singer
  • Margaret-Anne Storey
  • Daniela Damian

Selecting a research method for empirical software engineering research is problematic because the benefits and challenges to using each method are not yet well catalogued. Therefore, this chapter describes a number of empirical methods available. It examines the goals of each and analyzes the types of questions each best addresses. Theoretical stances behind the methods, practical considerations in the application of the methods and data collection are also briefly reviewed. Taken together, this information provides a suitable basis for both understanding and selecting from the variety of methods applicable to empirical software engineering.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bratthall, L. and Jørgensen, M. (2002) Can you trust a single data source exploratory software engineering case study? Journal of Empirical Software Engineering, 7(1), 9–26.MATHCrossRefGoogle Scholar
  2. Calhoun, C. (1995) Critical Social Theory: Culture, History, and the Challenge of Difference. Blackwell, Oxford, UK.Google Scholar
  3. Chalmers, A. (1999) What Is This Thing Called Science? 3rd Edition, Hackett Publishing Co, Indianapolis.Google Scholar
  4. Creswell, J.W. (2002) Research Design: Qualitative, Quantitative and Mixed Methods Approaches. 2nd Edition, Sage Publications, Thousand Oaks, CA.Google Scholar
  5. Damian, D. and Chisan, J. (2006) An empirical study of the complex relationships between requirements engineering processes and other processes that lead to payoffs in productivity, quality and risk management, IEEE Transactions on Software Engineering, 32(8), 433–453.CrossRefGoogle Scholar
  6. Damian, D.E., Eberlein, A., Shaw, M., and Gaines, B. (2000) Using different communication media in requirements negotiation, IEEE Software, 17(3), 28–36.CrossRefGoogle Scholar
  7. Davison, R.M., Martinsons, M.G., and Kock, N. (2004) Principles of canonical action research, Information Systems Journal, 14(1), 65–86.CrossRefGoogle Scholar
  8. Dittrich, Y. (2002) Doing Empirical Research on Software Development: Finding a Path Between Understanding, Intervention, and Method Development. In Social Thinking: Software Practice, Y. Dittrich, C. Floyd, and R. Klischewski, Eds. MIT Press.Google Scholar
  9. Dittrich, Y., John, M., Singer, J., and Tessem, B. (2007) Editorial for the Special Issue on Qualitative Software Engineering Research, Information and Software Technology, 49(6), 531–539.CrossRefGoogle Scholar
  10. Flyvbjerg, B. (2006) Five misunderstandings about case study research, Qualitative Inquiry, 12(2), 219–245.CrossRefGoogle Scholar
  11. Glaser, B.G. and Strauss, A. (1967) Discovery of Grounded Theory: Strategies for Qualitative Research. Sociology Press, Mill Valley, CA.Google Scholar
  12. Gregor, S. (2006) The Nature of Theories in Information Systems, MIS Quarterly, 30(3), 611–642.Google Scholar
  13. Jørgensen, M. and Sjøberg, D.I.K. (2004) Generalization and Theory-Building in Software Engineering Research. IEE Proceedings, Workshop on Empirical Assessment in Software Engineering (EASE’04), at ICSE’04, pp. 29–36.Google Scholar
  14. Kitchenham, B., Pickard, L., and Pfleeger, S.L. (1995) Case studies for method and tool evaluation, IEEE Software, 12(4), 52–62.CrossRefGoogle Scholar
  15. Klein, H.K. and Myers, M.D. (1999) A set of principles for conducting and evaluating interpretive field studies in information systems, MIS Quarterly, 23(1), 67–93.CrossRefGoogle Scholar
  16. Lau, F. (1999).Towards a framework for action research in information systems studies, Information Technology and People, 12(2), 148–175.CrossRefGoogle Scholar
  17. Lincoln, Y.S. and Guba, E.G. (1985) Naturalistic Inquiry. Sage, Beverly Hills, CA.Google Scholar
  18. Littlejohn, S.W. and Foss, K.A. (2004) Theories of Human Communication. 8th Edition, Wadsworth Publishing, Belmont, CA.Google Scholar
  19. McGrath, J.E. (1995) Methodology matters: doing research in the behavioral and social sciences. In Human–Computer Interaction: Toward the Year 2000, R.M. Baecker, J. Grudin, W. Buxton, A., and Greenberg, S., Eds. Morgan Kaufmann Publishers, San Francisco, CA, pp. 152–169.Google Scholar
  20. Meltzoff, J. (1998) Critical Thinking About Research: Psychology and Related Fields. American Psychological Association, Washington DC.Google Scholar
  21. Menand, L. (1997) Pragmatism: A Reader. Vintage Press, New York.Google Scholar
  22. Morse, J.M., Barrett, M., Mayan, M., Olson, K. and Spiers, J. (2002) Verification strategies for establishing reliability and validity in qualitative research, International Journal of Qualitative Methods, 1(2), 1–19.Google Scholar
  23. Robinson, H., Segal, J. and Sharp, H. (2007) Ethnographically-informed empirical studies of software practice, Information and Software Technology, 49(6), 540–551.CrossRefGoogle Scholar
  24. Sandelowski, M. (1993) Rigor or rigor mortis: the problem of rigor in qualitative research revisited, Advances in Nursing Science, 16(2), 1–8.Google Scholar
  25. Simon, H. (1996) The Sciences of the Artificial. 3rd Edition, MIT Press, Cambridge, MA.Google Scholar
  26. Singer, J.A. and Vinson, N.G. (2002) Ethical issues in empirical studies of software engineering, IEEE Transactions on Software Engineering, 28(12), 1171–1180.CrossRefGoogle Scholar
  27. Varkevisser, C.M., Pathmanathan, I., and Brownlee, A. (2003) Designing and Conducting Health Systems Research Projects: Volume 1–Proposal Development and Fieldwork. Chapter 10: Data Collection Techniques. Available online at http://www.idrc.ca/en/ev-56605–201-1-DO_TOPIC.html.
  28. Vinson, N.G. and Singer, J.A. (2004) Consent issues raised by observational research in organisations, NCEHR Communique, 12(2), 35–36.Google Scholar
  29. Wieringa, R.J. and Heerkens, J.M.G. (2006) The methodological soundness of requirements engineering papers: a conceptual framework and two case studies, Requirements Engineering Journal, 11, 295–307.CrossRefGoogle Scholar
  30. Wohlin, C., Runesson, P., Höst, M., Ohlsson, M.C., Regnell, B., and Wesslén, A. (2000) Introduction to Experimentation in Software Engineering. Kluwer Academic Publishers, Boston, MA.Google Scholar
  31. Yin, R.K. (2002) Case Study Research: Design and Methods. Sage, Thousand Oaks, CA.Google Scholar

Copyright information

© Springer-Verlag London Limited 2008

Authors and Affiliations

  • Steve Easterbrook
    • 1
  • Janice Singer
    • 2
  • Margaret-Anne Storey
    • 3
  • Daniela Damian
    • 3
  1. 1.Department of Computer ScienceUniversity of TorontoTorontoCanada
  2. 2.Institute for Information TechnologyNational Research Council CanadaOttawaCanada
  3. 3.Dept. of Computer ScienceUniversity of VictoriaVictoriaCanada

Personalised recommendations