Towards a decision-making structure for selecting a research design in empirical software engineering

Abstract

Several factors make empirical research in software engineering particularly challenging as it requires studying not only technology but its stakeholders’ activities while drawing concepts and theories from social science. Researchers, in general, agree that selecting a research design in empirical software engineering research is challenging, because the implications of using individual research methods are not well recorded. The main objective of this article is to make researchers aware and support them in their research design, by providing a foundation of knowledge about empirical software engineering research decisions, in order to ensure that researchers make well-founded and informed decisions about their research designs. This article provides a decision-making structure containing a number of decision points, each one of them representing a specific aspect on empirical software engineering research. The article provides an introduction to each decision point and its constituents, as well as to the relationships between the different parts in the decision-making structure. The intention is the structure should act as a starting point for the research design before going into the details of the research design chosen. The article provides an in-depth discussion of decision points in relation to the research design when conducting empirical research.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Notes

  1. 1.

    This is an example of the different opinions related to terminology. Others may refer to this as “research type”.

References

  1. Adolp A, Hall W, Kruchten P (2011) Using grounded theory to study the experience of software development. J Empir Softw Eng 16(4):487–513

    Article  Google Scholar 

  2. Allison I, Merali Y (2007) Software process improvement as emergent change: a structurational analysis. Inf Softw Technol 49(6):668–681

    Article  Google Scholar 

  3. Barney S, Mohankumar V, Chatzipetrou P, Aurum A, Wohlin C, Angelis L (2014) Software quality across borders: three case studies on company internal alignment. Inf Softw Technol 56(1):20–38

    Article  Google Scholar 

  4. Basili V (1993) The experimental paradigm in software engineering. Proceedings of the International Workshop on Experimental Software Engineering Issues: Critical Assessment and Future Directions. Springer-Verlag, LNCS 706, London, UK, pp 3–12 Link: http://link.springer.com/content/pdf/10.1007/3-540-57092-6_91.pdf

  5. Baskerville R (2008) What design science is not. Eur J Inf Syst 17:441–443

    Article  Google Scholar 

  6. Benbasat I, Goldstein DK, Mead M (1987) The case research strategy in studies of information systems. MIS Q 11(3):369–386

    Article  Google Scholar 

  7. Bertelsen OW (1997) Towards a unified field of se research and practice. IEEE Softw 14:87–88

    Article  Google Scholar 

  8. Bhattacherjee A (2012) Social science research: principles, methods, and practices. USF Open Access Textbooks Collection. Book 3 University of South Florida Link http://scholarcommons.usf.edu/oa_textbooks/3

  9. Birks DF, Fernandez W, Levina N, Nasirin S (2013) Grounded theory method in information systems research: its nature, diversity and opportunities. Guest editorial. Eur J Inf Syst 22:1–8

    Article  Google Scholar 

  10. Boell S, Cecez-Kecmanivic D (2010) Literature reviews and the hermeneutic circle. Aust Acad Res Libr 41(2):129–144

    Article  Google Scholar 

  11. Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3(2):77–101

    Article  Google Scholar 

  12. Brooke C (2002) What does it mean to be ‘critical’ in IS research? J Inf Technol 17(2):49–57

    Article  Google Scholar 

  13. Butler T (1998) Towards a hermeneutic method for interpretive research in information systems. J Inf Technol 13:285–300

    Article  Google Scholar 

  14. Carver J, Seaman C, Jeffery R (2004) Using qualitative methods in software engineering. International Advanced School of Empirical Software Engineering (IASESE04), August 18, 2004, LA CA Link: http://chess.cs.umd.edu/class/fall2004/cmsc735/CMSC735%2011a%20Qualitative%20Analysis.pdf

  15. Cecez-Kecmanovic D (2011) Doing critical information systems research–arguments for a critical research methodology. Eur J Inf Syst 20(4):440–455

    Article  Google Scholar 

  16. Checkland P (1981) Systems thinking, systems practice. Wiley, UK

    Google Scholar 

  17. Chen WS, Hirschheim R (2004) A paradigmatic and methodological examination of information systems research from 1991 to 2001. Inf Syst J 14(3):197–235

    Article  Google Scholar 

  18. Coleman G, O’Connor R (2007a) Investigating software process in practice: a grounded theory perspective. J Syst Softw 81:772–784

    Article  Google Scholar 

  19. Coleman G, O’Connor R (2007b) Using grounded theory to understand software process improvement: a study of Irish software product companies. Inf Softw Technol 49(6):654–667

    Article  Google Scholar 

  20. Collis J, Hussey R (2009) Business research. Palgrave MacMillan, UK

    Google Scholar 

  21. Creswell J (2013) Research design: qualitative, quantitative and mixed methods approach. Sage Publication, Thousand Oaks

    Google Scholar 

  22. Davison RM, Martinsons MG, Kock N (2004) Principles of canonical action research. Inf Syst J 14(1):65–86

    Article  Google Scholar 

  23. Dunne C (2011) The place of the literature review in grounded theory research. Int J Soc Res Methodol 14(2):111–124

    MathSciNet  Article  Google Scholar 

  24. Dybå T, Prikladnicki R, Rönkkö K, Seaman CB, Sillito J (2011) Qualitative research in software engineering. Empir Softw Eng 16(4):425–429

    Article  Google Scholar 

  25. Easterbrook S, Singer J, Storey MA, Damian D (2008) Selecting empirical methods for software engineering research. In: Shull F, Singer J, Sjøberg DIK (eds) Guide to advanced empirical software engineering, Springer Germany, pp 285–311

  26. Eisenhardt KM (1989) Building theories from case study research. Academy of management. Acad Manag Rev 14(4):532–550

    Google Scholar 

  27. Engel RJ, Schutt RK (2010) The practice of research in social work. Sage Publication, Thousand Oaks

    Google Scholar 

  28. Ghanam Y, Maurer F, Abrahamsson P (2012) Making a leap to a software platform strategy: issues and challenges. Inf Softw Technol 54(9):968–984

    Article  Google Scholar 

  29. Ghapanchi AH (2011) Dynamic capabilities and project characteristics contributing to the success of open source software projects. PhD Dissertation Thesis. The University of New South Wales, Sydney, NSW Australia

  30. Gilbert GN (1995) Dagstuhl seminar on social science microsimulation: a challenge to computer science, SchjoB, May 1–5, 1995

  31. Glaser BG (1992) Emergence vs. forcing: basics of grounded theory analysts. Sociology Press, California

    Google Scholar 

  32. Glaser BG, Strauss AL (1967) The discovery of grounded theory: strategies for qualitative research. Aldine, New York

    Google Scholar 

  33. Gregg DG, Kulkarni UR, Vinze AS (2001) Understanding the philosophical underpinnings of software engineering research in information systems. Inf Syst Front 3(2):169–183

    Article  Google Scholar 

  34. Grix J (2002) Introducing students to the generic terminology of social research. Politics 22(3):175–186

    Article  Google Scholar 

  35. Hannah DR, Lautsch BA (2011) Counting in qualitative research: why to conduct it, when to avoid it, and when to closet it. J Manag Inq 20(1):14–22

    Google Scholar 

  36. Hansen S, Rennecker J (2010) Getting on the same page: collective hermeneutics in a systems development team. Inf Organ 20(1):44–63

    Article  Google Scholar 

  37. Harrison RJ, Lin Z, Caroll GR, Carley KM (2007) Simulation modeling in organization and management research. Acad Manag Rev 32(4):1229–1245

    Article  Google Scholar 

  38. Hevner AR, March ST, Park J, Ram S (2004) Design science in information systems research. MIS Q 28(1):75–105

    Google Scholar 

  39. Jafarzadeh H, Aurum A, D’Ambra J (2011) Factors affecting the success of businesses in effective utilization of search engine advertising. International Conference on Information Systems, 3–7 December, Shanghai, China, 2011, sigIQ pre-ICIS workshop, pp 1 Link: http://www.northeastern.edu/iqworld/papers/Papers/Factors%20Affecting%20the%20Success%20of%20Businesses%20in%20Effective%20Utilization%20of%20Search%20Engine%20Advertising_abstract.pdf

  40. Jafarzadeh H, Aurum A, D’ambra J, Abedin B (2013) Determinant of intention to use search engine advertising: a conceptual model. Int J Enterp Inf Syst 9(3):22–38

    Article  Google Scholar 

  41. Johnson CF (1996) Deductive versus inductive reasoning: a closer look at economics. Soc Sci J 33(3):287–299

    Article  Google Scholar 

  42. Johnson RB, Onwuegbuzie AJ (2004) Mixed methods research: a research paradigm whose time has come. Educ Res 33(7):14–26

    Article  Google Scholar 

  43. Kachigan SK (1986) Statistical analysis: an interdisciplinary introduction to univariate & multivariate methods. Radius Press, New York

    Google Scholar 

  44. Kitchenham BA, Pfleeger SA (2002) Principles of survey research part 2: designing a survey. ACM SIGSOFT Softw Eng Notes 27(1):18–20

    Article  Google Scholar 

  45. Kitchenham BA, Pickard LM, Pfleeger SL (1995) Case studies for method and tool evaluation. IEEE Softw 12(4):52–62

    Article  Google Scholar 

  46. Kitchenham BA, Pfleeger SL, Pickard LM, Jones PW, Hoaglin DC, El Emam K, Rosenberg J (2002) Preliminary guidelines for empirical research in software engineering. IEEE Trans Softw Eng 28(8):721–734

    Article  Google Scholar 

  47. Klein HK, Myers MD (1999) A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Q 23(1):67–89

    Article  Google Scholar 

  48. Kline RB (2011) Principles and practice of structural equation modeling. Guilford Press, New York

    Google Scholar 

  49. Kontio J, Lehtola L, Bragge J (2004) Using the focus group method in software engineering: obtaining practitioner and user experience. Proceedings of International Symposium of Empirical Software Engineering, Los Angeles, CA, USA, August 2004, IEEE Computer Society Washington, DC, USA, pp 271–280

  50. Kyburz-Graber R (2007) Does case–study methodology lack rigour? The need for quality criteria for sound case–study research, as illustrated by a recent case in secondary and higher education. Environ Educ Res 10(1):53–65

    Article  Google Scholar 

  51. Law AM (2007) Simulation modeling and analysis, volume 4. McGraw-Hill, New York

    Google Scholar 

  52. Lee AS (1989) A scientific methodology for MIS case studies. MIS Q 13(1):33–54

    Article  Google Scholar 

  53. Lethbridge TC, Sim SE, Singer J (2005) Studying software engineers: data collection techniques for software field studies. J Empir Softw Eng 10(3):311–341

    Article  Google Scholar 

  54. Lyons H (2009) Case Study research methodology for publishing developments in ICT-facilitated learning in higher education—a perspective approach. Innov Educ Teach Int 46(1):27–39

    Article  Google Scholar 

  55. Mack N, Woodsong C, MacQueen K, Guest G, Namey E (2005) Qualitative research methods: a data collector’s field guide. Family Health International, Research Triangle Park

    Google Scholar 

  56. Marascuilo LA, Serlin RC (1988) Statistical methods for the social and behavioral sciences. W.H. Freeman and Company, New York

    Google Scholar 

  57. March ST, Smith GF (1995) Design and natural science research on information technology. Decis Support Syst 15(4):251–266

    Article  Google Scholar 

  58. McKernan J (1996) Curriculum action research: a handbook of methods and resources for the reflective practitioner. Kogan Page, London

    Google Scholar 

  59. McLeod L, MacDonell SG, Doolin B (2011) Qualitative research on software development: a longitudinal case study methodology. J Empir Softw Eng 16(4):430–459

    Article  Google Scholar 

  60. Mingers J (2001) Combining IS research methods: towards a pluralist methodology. Inf Syst Res 12(3):240–259

    Article  Google Scholar 

  61. Mkansi M, Acheampong EA (2012) Research philosophy debates and classification: students’ dilemma. Electron J Bus Res Methods 10(2):132–140

    Google Scholar 

  62. Moe NB, Aurum A, Dybå T (2012) Challenges of shared decision-making: a multiple case study of agile software development. Inf Softw Technol 54(8):853–865

    Article  Google Scholar 

  63. Müller M, Pfahl D (2008) Simulation methods. In: Shull F, Singer J, Sjøberg DIK (eds) Guide to advanced empirical software engineering. Springer, Germany

    Google Scholar 

  64. Myers MD (1995) Dialectical hermeneutics: a theoretical framework for the implementation of information systems. Inf Syst J 5(1):51–70

    Article  Google Scholar 

  65. Myers MD (1997) Qualitative research in information systems. MIS Q 21(2):241–242

    Article  Google Scholar 

  66. Myers MD, Klein HK (2011) A set of principles for conducting critical research in information systems. MIS Q 35(1):17–36

    Google Scholar 

  67. Nunamaker JF Jr, Chen M, Purdin TDM (1991) Systems development in information systems research. J Manag Inf Syst 7(3):89–106

    Google Scholar 

  68. Orlikowski WJ, Baroudi JJ (1991) Studying information technology in organizations: research approaches and assumptions. Inf Syst Res 2(1):1–28

    Article  Google Scholar 

  69. Ostrowski L, Helfert M (2011) Commonality in various design science methodologies. Proceedings of the Federated Conference on Computer Science and Information Systems, pp 317–320. ISBN 978-83-60810-39-2

  70. Perry C (1998) Processes of a case study methodology for postgraduate research in marketing. Eur J Mark 32(9/10):785–802

    Article  Google Scholar 

  71. Perry DE, Porter AA, Votta LG (2000) Empirical studies of software engineering: a roadmap. In: Finkelstein A (ed) The future of software engineering. ACM Press, New York

    Google Scholar 

  72. Pinsonneault A, Kraemer K (1993) Survey research methodology in management information systems: an assessment. J Manag Inf Syst 10(2):75–105

    Google Scholar 

  73. Rossi P, Freeman HF (1993) Evaluation: a systematic approach. Sage Publication, USA

    Google Scholar 

  74. Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. J Empir Softw Eng 14(2):131–164

    Article  Google Scholar 

  75. Runeson P, Höst M, Rainer A, Regnell B (2012) Case study research in software engineering: guidelines and examples. Wiley, USA

    Google Scholar 

  76. Seaman CB (1999) Qualitative methods in empirical studies of software engineering. IEEE Trans Softw Eng 25(4):557–573

    Article  Google Scholar 

  77. Shanks G, Parr A (2003) Positivist, single case study research in information systems: a critical analysis. Proceedings of the 11th European Conference on Information Systems, Naples, Italy 16-21 June 2003, pp 1760–1774 Link: http://is2.lse.ac.uk/asp/aspecis/20030140.pdf

  78. Shaw M (2002) What makes good research in software engineering? Int J Softw Tools Technol Transfer 4(1):1–7

    Article  Google Scholar 

  79. Shaw M (2003) Writing good software engineering research papers. Proceedings of 25th Int Conference on Software Engineering, Portland, Oregon, USA, May 2003, IEEE Computer Society Washington, DC, USA, pp 726–736

  80. Shull F, Singer J, Sjøberg DIK (2008) Guide to advanced empirical software engineering. Springer, Germany

    Google Scholar 

  81. Shye S (1988) Inductive and deductive reasoning: a structural reanalysis of ability tests. J Appl Psychol 73(2):308–311

    MathSciNet  Article  Google Scholar 

  82. Siegel S, Castellan NJ Jr (1988) Nonparametric statistics for the behavioral sciences. Mcgraw-Hill Book Company, New York

    Google Scholar 

  83. Sjøberg DIK, Dybå T, Jørgensen M (2007) The future of empirical methods in software engineering. IEEE Proceedings of Future of Software Engineering (FOSE)

  84. Smite D, Wohlin C, Gorschek T, Feldt R (2010) Empirical evidence in global software engineering: a systematic review. Empir Softw Eng Int J 15(1):91–118

    Article  Google Scholar 

  85. Smite D, Wohlin C, Aurum A, Jabangwe R, Numminen E (2013) Offshore insourcing in software development: structuring the decision-making process. J Syst Softw 86(4):1054–1067

    Article  Google Scholar 

  86. Staron M, Kuzniarz L, Wohlin C (2006) Empirical assessment of using stereotypes to improve comprehension of UML models: a set of experiments. J Syst Softw 79(5):727–742

    Article  Google Scholar 

  87. Strauss AL, Corbin JM (1998) Basics of qualitative research: techniques and procedures for developing grounded theory. Sage Publication, Thousand Oaks

    Google Scholar 

  88. Susman G, Evered R (1978) An assessment of the scientific merits of action research. Adm Sci Q 23(4):582–603

    Article  Google Scholar 

  89. Tichy WF, Lukowicz P, Prechelt L, Heinz EA (1995) Experimental evaluation in computer science: a quantitative study. J Syst Softw 28(1):9–18

    Article  Google Scholar 

  90. Tom E, Aurum A, Vidgen R (2013) An exploration of technical debt. J Syst Softw 86(6):1498–1516

    Article  Google Scholar 

  91. Urquhart C (2002) Regrounding grounded theory–or reinforcing old prejudices? a brief reply to bryant. J Inf Technol Theory Appl 4(3):43–54

    Google Scholar 

  92. Walker D, Myrick F (2006) Grounded theory: an exploration of process and procedure. Qual Health Res 16(4):547–559

    Article  Google Scholar 

  93. Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer, ISBN 978-3-642-29043-5

  94. Wynekoop JL, Russo NL (1997) Studying system development methodologies: an examination of research methods. Inf Syst J 7(1):47–65

    Article  Google Scholar 

  95. Yin RK (2002) Case study research. Sage Publication, CA

    Google Scholar 

  96. Zelkowitz MV, Wallace D (1997) Experimental validation in software engineering. Inf Softw Technol 31(5):23–31

    Google Scholar 

Download references

Acknowledgments

The Knowledge Foundation in Sweden partially funded this work under a research grant for the Blekinge Engineering Software Qualities (BESQ+) research environment.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Claes Wohlin.

Additional information

Communicated by: Jeffrey C. Carver

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wohlin, C., Aurum, A. Towards a decision-making structure for selecting a research design in empirical software engineering. Empir Software Eng 20, 1427–1455 (2015). https://doi.org/10.1007/s10664-014-9319-7

Download citation

Keywords

  • Research methods
  • Empirical software engineering research
  • Selecting research method
  • Research design