Empirical Software Engineering

, Volume 20, Issue 6, pp 1427–1455 | Cite as

Towards a decision-making structure for selecting a research design in empirical software engineering

  • Claes WohlinEmail author
  • Aybüke Aurum


Several factors make empirical research in software engineering particularly challenging as it requires studying not only technology but its stakeholders’ activities while drawing concepts and theories from social science. Researchers, in general, agree that selecting a research design in empirical software engineering research is challenging, because the implications of using individual research methods are not well recorded. The main objective of this article is to make researchers aware and support them in their research design, by providing a foundation of knowledge about empirical software engineering research decisions, in order to ensure that researchers make well-founded and informed decisions about their research designs. This article provides a decision-making structure containing a number of decision points, each one of them representing a specific aspect on empirical software engineering research. The article provides an introduction to each decision point and its constituents, as well as to the relationships between the different parts in the decision-making structure. The intention is the structure should act as a starting point for the research design before going into the details of the research design chosen. The article provides an in-depth discussion of decision points in relation to the research design when conducting empirical research.


Research methods Empirical software engineering research Selecting research method Research design 



The Knowledge Foundation in Sweden partially funded this work under a research grant for the Blekinge Engineering Software Qualities (BESQ+) research environment.


  1. Adolp A, Hall W, Kruchten P (2011) Using grounded theory to study the experience of software development. J Empir Softw Eng 16(4):487–513CrossRefGoogle Scholar
  2. Allison I, Merali Y (2007) Software process improvement as emergent change: a structurational analysis. Inf Softw Technol 49(6):668–681CrossRefGoogle Scholar
  3. Barney S, Mohankumar V, Chatzipetrou P, Aurum A, Wohlin C, Angelis L (2014) Software quality across borders: three case studies on company internal alignment. Inf Softw Technol 56(1):20–38CrossRefGoogle Scholar
  4. Basili V (1993) The experimental paradigm in software engineering. Proceedings of the International Workshop on Experimental Software Engineering Issues: Critical Assessment and Future Directions. Springer-Verlag, LNCS 706, London, UK, pp 3–12 Link:
  5. Baskerville R (2008) What design science is not. Eur J Inf Syst 17:441–443CrossRefGoogle Scholar
  6. Benbasat I, Goldstein DK, Mead M (1987) The case research strategy in studies of information systems. MIS Q 11(3):369–386CrossRefGoogle Scholar
  7. Bertelsen OW (1997) Towards a unified field of se research and practice. IEEE Softw 14:87–88CrossRefGoogle Scholar
  8. Bhattacherjee A (2012) Social science research: principles, methods, and practices. USF Open Access Textbooks Collection. Book 3 University of South Florida Link
  9. Birks DF, Fernandez W, Levina N, Nasirin S (2013) Grounded theory method in information systems research: its nature, diversity and opportunities. Guest editorial. Eur J Inf Syst 22:1–8CrossRefGoogle Scholar
  10. Boell S, Cecez-Kecmanivic D (2010) Literature reviews and the hermeneutic circle. Aust Acad Res Libr 41(2):129–144CrossRefGoogle Scholar
  11. Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3(2):77–101CrossRefGoogle Scholar
  12. Brooke C (2002) What does it mean to be ‘critical’ in IS research? J Inf Technol 17(2):49–57CrossRefGoogle Scholar
  13. Butler T (1998) Towards a hermeneutic method for interpretive research in information systems. J Inf Technol 13:285–300CrossRefGoogle Scholar
  14. Carver J, Seaman C, Jeffery R (2004) Using qualitative methods in software engineering. International Advanced School of Empirical Software Engineering (IASESE04), August 18, 2004, LA CA Link:
  15. Cecez-Kecmanovic D (2011) Doing critical information systems research–arguments for a critical research methodology. Eur J Inf Syst 20(4):440–455CrossRefGoogle Scholar
  16. Checkland P (1981) Systems thinking, systems practice. Wiley, UKGoogle Scholar
  17. Chen WS, Hirschheim R (2004) A paradigmatic and methodological examination of information systems research from 1991 to 2001. Inf Syst J 14(3):197–235CrossRefGoogle Scholar
  18. Coleman G, O’Connor R (2007a) Investigating software process in practice: a grounded theory perspective. J Syst Softw 81:772–784CrossRefGoogle Scholar
  19. Coleman G, O’Connor R (2007b) Using grounded theory to understand software process improvement: a study of Irish software product companies. Inf Softw Technol 49(6):654–667CrossRefGoogle Scholar
  20. Collis J, Hussey R (2009) Business research. Palgrave MacMillan, UKGoogle Scholar
  21. Creswell J (2013) Research design: qualitative, quantitative and mixed methods approach. Sage Publication, Thousand OaksGoogle Scholar
  22. Davison RM, Martinsons MG, Kock N (2004) Principles of canonical action research. Inf Syst J 14(1):65–86CrossRefGoogle Scholar
  23. Dunne C (2011) The place of the literature review in grounded theory research. Int J Soc Res Methodol 14(2):111–124MathSciNetCrossRefGoogle Scholar
  24. Dybå T, Prikladnicki R, Rönkkö K, Seaman CB, Sillito J (2011) Qualitative research in software engineering. Empir Softw Eng 16(4):425–429CrossRefGoogle Scholar
  25. Easterbrook S, Singer J, Storey MA, Damian D (2008) Selecting empirical methods for software engineering research. In: Shull F, Singer J, Sjøberg DIK (eds) Guide to advanced empirical software engineering, Springer Germany, pp 285–311Google Scholar
  26. Eisenhardt KM (1989) Building theories from case study research. Academy of management. Acad Manag Rev 14(4):532–550Google Scholar
  27. Engel RJ, Schutt RK (2010) The practice of research in social work. Sage Publication, Thousand OaksGoogle Scholar
  28. Ghanam Y, Maurer F, Abrahamsson P (2012) Making a leap to a software platform strategy: issues and challenges. Inf Softw Technol 54(9):968–984CrossRefGoogle Scholar
  29. Ghapanchi AH (2011) Dynamic capabilities and project characteristics contributing to the success of open source software projects. PhD Dissertation Thesis. The University of New South Wales, Sydney, NSW AustraliaGoogle Scholar
  30. Gilbert GN (1995) Dagstuhl seminar on social science microsimulation: a challenge to computer science, SchjoB, May 1–5, 1995Google Scholar
  31. Glaser BG (1992) Emergence vs. forcing: basics of grounded theory analysts. Sociology Press, CaliforniaGoogle Scholar
  32. Glaser BG, Strauss AL (1967) The discovery of grounded theory: strategies for qualitative research. Aldine, New YorkGoogle Scholar
  33. Gregg DG, Kulkarni UR, Vinze AS (2001) Understanding the philosophical underpinnings of software engineering research in information systems. Inf Syst Front 3(2):169–183CrossRefGoogle Scholar
  34. Grix J (2002) Introducing students to the generic terminology of social research. Politics 22(3):175–186CrossRefGoogle Scholar
  35. Hannah DR, Lautsch BA (2011) Counting in qualitative research: why to conduct it, when to avoid it, and when to closet it. J Manag Inq 20(1):14–22Google Scholar
  36. Hansen S, Rennecker J (2010) Getting on the same page: collective hermeneutics in a systems development team. Inf Organ 20(1):44–63CrossRefGoogle Scholar
  37. Harrison RJ, Lin Z, Caroll GR, Carley KM (2007) Simulation modeling in organization and management research. Acad Manag Rev 32(4):1229–1245CrossRefGoogle Scholar
  38. Hevner AR, March ST, Park J, Ram S (2004) Design science in information systems research. MIS Q 28(1):75–105Google Scholar
  39. Jafarzadeh H, Aurum A, D’Ambra J (2011) Factors affecting the success of businesses in effective utilization of search engine advertising. International Conference on Information Systems, 3–7 December, Shanghai, China, 2011, sigIQ pre-ICIS workshop, pp 1 Link:
  40. Jafarzadeh H, Aurum A, D’ambra J, Abedin B (2013) Determinant of intention to use search engine advertising: a conceptual model. Int J Enterp Inf Syst 9(3):22–38CrossRefGoogle Scholar
  41. Johnson CF (1996) Deductive versus inductive reasoning: a closer look at economics. Soc Sci J 33(3):287–299CrossRefGoogle Scholar
  42. Johnson RB, Onwuegbuzie AJ (2004) Mixed methods research: a research paradigm whose time has come. Educ Res 33(7):14–26CrossRefGoogle Scholar
  43. Kachigan SK (1986) Statistical analysis: an interdisciplinary introduction to univariate & multivariate methods. Radius Press, New YorkGoogle Scholar
  44. Kitchenham BA, Pfleeger SA (2002) Principles of survey research part 2: designing a survey. ACM SIGSOFT Softw Eng Notes 27(1):18–20CrossRefGoogle Scholar
  45. Kitchenham BA, Pickard LM, Pfleeger SL (1995) Case studies for method and tool evaluation. IEEE Softw 12(4):52–62CrossRefGoogle Scholar
  46. Kitchenham BA, Pfleeger SL, Pickard LM, Jones PW, Hoaglin DC, El Emam K, Rosenberg J (2002) Preliminary guidelines for empirical research in software engineering. IEEE Trans Softw Eng 28(8):721–734CrossRefGoogle Scholar
  47. Klein HK, Myers MD (1999) A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Q 23(1):67–89CrossRefGoogle Scholar
  48. Kline RB (2011) Principles and practice of structural equation modeling. Guilford Press, New YorkzbMATHGoogle Scholar
  49. Kontio J, Lehtola L, Bragge J (2004) Using the focus group method in software engineering: obtaining practitioner and user experience. Proceedings of International Symposium of Empirical Software Engineering, Los Angeles, CA, USA, August 2004, IEEE Computer Society Washington, DC, USA, pp 271–280Google Scholar
  50. Kyburz-Graber R (2007) Does case–study methodology lack rigour? The need for quality criteria for sound case–study research, as illustrated by a recent case in secondary and higher education. Environ Educ Res 10(1):53–65CrossRefGoogle Scholar
  51. Law AM (2007) Simulation modeling and analysis, volume 4. McGraw-Hill, New YorkGoogle Scholar
  52. Lee AS (1989) A scientific methodology for MIS case studies. MIS Q 13(1):33–54CrossRefGoogle Scholar
  53. Lethbridge TC, Sim SE, Singer J (2005) Studying software engineers: data collection techniques for software field studies. J Empir Softw Eng 10(3):311–341CrossRefGoogle Scholar
  54. Lyons H (2009) Case Study research methodology for publishing developments in ICT-facilitated learning in higher education—a perspective approach. Innov Educ Teach Int 46(1):27–39CrossRefGoogle Scholar
  55. Mack N, Woodsong C, MacQueen K, Guest G, Namey E (2005) Qualitative research methods: a data collector’s field guide. Family Health International, Research Triangle ParkGoogle Scholar
  56. Marascuilo LA, Serlin RC (1988) Statistical methods for the social and behavioral sciences. W.H. Freeman and Company, New YorkzbMATHGoogle Scholar
  57. March ST, Smith GF (1995) Design and natural science research on information technology. Decis Support Syst 15(4):251–266CrossRefGoogle Scholar
  58. McKernan J (1996) Curriculum action research: a handbook of methods and resources for the reflective practitioner. Kogan Page, LondonGoogle Scholar
  59. McLeod L, MacDonell SG, Doolin B (2011) Qualitative research on software development: a longitudinal case study methodology. J Empir Softw Eng 16(4):430–459CrossRefGoogle Scholar
  60. Mingers J (2001) Combining IS research methods: towards a pluralist methodology. Inf Syst Res 12(3):240–259CrossRefGoogle Scholar
  61. Mkansi M, Acheampong EA (2012) Research philosophy debates and classification: students’ dilemma. Electron J Bus Res Methods 10(2):132–140Google Scholar
  62. Moe NB, Aurum A, Dybå T (2012) Challenges of shared decision-making: a multiple case study of agile software development. Inf Softw Technol 54(8):853–865CrossRefGoogle Scholar
  63. Müller M, Pfahl D (2008) Simulation methods. In: Shull F, Singer J, Sjøberg DIK (eds) Guide to advanced empirical software engineering. Springer, GermanyGoogle Scholar
  64. Myers MD (1995) Dialectical hermeneutics: a theoretical framework for the implementation of information systems. Inf Syst J 5(1):51–70CrossRefGoogle Scholar
  65. Myers MD (1997) Qualitative research in information systems. MIS Q 21(2):241–242CrossRefGoogle Scholar
  66. Myers MD, Klein HK (2011) A set of principles for conducting critical research in information systems. MIS Q 35(1):17–36Google Scholar
  67. Nunamaker JF Jr, Chen M, Purdin TDM (1991) Systems development in information systems research. J Manag Inf Syst 7(3):89–106Google Scholar
  68. Orlikowski WJ, Baroudi JJ (1991) Studying information technology in organizations: research approaches and assumptions. Inf Syst Res 2(1):1–28CrossRefGoogle Scholar
  69. Ostrowski L, Helfert M (2011) Commonality in various design science methodologies. Proceedings of the Federated Conference on Computer Science and Information Systems, pp 317–320. ISBN 978-83-60810-39-2Google Scholar
  70. Perry C (1998) Processes of a case study methodology for postgraduate research in marketing. Eur J Mark 32(9/10):785–802CrossRefGoogle Scholar
  71. Perry DE, Porter AA, Votta LG (2000) Empirical studies of software engineering: a roadmap. In: Finkelstein A (ed) The future of software engineering. ACM Press, New YorkGoogle Scholar
  72. Pinsonneault A, Kraemer K (1993) Survey research methodology in management information systems: an assessment. J Manag Inf Syst 10(2):75–105Google Scholar
  73. Rossi P, Freeman HF (1993) Evaluation: a systematic approach. Sage Publication, USAGoogle Scholar
  74. Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. J Empir Softw Eng 14(2):131–164CrossRefGoogle Scholar
  75. Runeson P, Höst M, Rainer A, Regnell B (2012) Case study research in software engineering: guidelines and examples. Wiley, USACrossRefGoogle Scholar
  76. Seaman CB (1999) Qualitative methods in empirical studies of software engineering. IEEE Trans Softw Eng 25(4):557–573CrossRefGoogle Scholar
  77. Shanks G, Parr A (2003) Positivist, single case study research in information systems: a critical analysis. Proceedings of the 11th European Conference on Information Systems, Naples, Italy 16-21 June 2003, pp 1760–1774 Link:
  78. Shaw M (2002) What makes good research in software engineering? Int J Softw Tools Technol Transfer 4(1):1–7CrossRefGoogle Scholar
  79. Shaw M (2003) Writing good software engineering research papers. Proceedings of 25th Int Conference on Software Engineering, Portland, Oregon, USA, May 2003, IEEE Computer Society Washington, DC, USA, pp 726–736Google Scholar
  80. Shull F, Singer J, Sjøberg DIK (2008) Guide to advanced empirical software engineering. Springer, GermanyCrossRefGoogle Scholar
  81. Shye S (1988) Inductive and deductive reasoning: a structural reanalysis of ability tests. J Appl Psychol 73(2):308–311MathSciNetCrossRefGoogle Scholar
  82. Siegel S, Castellan NJ Jr (1988) Nonparametric statistics for the behavioral sciences. Mcgraw-Hill Book Company, New YorkGoogle Scholar
  83. Sjøberg DIK, Dybå T, Jørgensen M (2007) The future of empirical methods in software engineering. IEEE Proceedings of Future of Software Engineering (FOSE)Google Scholar
  84. Smite D, Wohlin C, Gorschek T, Feldt R (2010) Empirical evidence in global software engineering: a systematic review. Empir Softw Eng Int J 15(1):91–118CrossRefGoogle Scholar
  85. Smite D, Wohlin C, Aurum A, Jabangwe R, Numminen E (2013) Offshore insourcing in software development: structuring the decision-making process. J Syst Softw 86(4):1054–1067CrossRefGoogle Scholar
  86. Staron M, Kuzniarz L, Wohlin C (2006) Empirical assessment of using stereotypes to improve comprehension of UML models: a set of experiments. J Syst Softw 79(5):727–742CrossRefGoogle Scholar
  87. Strauss AL, Corbin JM (1998) Basics of qualitative research: techniques and procedures for developing grounded theory. Sage Publication, Thousand OaksGoogle Scholar
  88. Susman G, Evered R (1978) An assessment of the scientific merits of action research. Adm Sci Q 23(4):582–603CrossRefGoogle Scholar
  89. Tichy WF, Lukowicz P, Prechelt L, Heinz EA (1995) Experimental evaluation in computer science: a quantitative study. J Syst Softw 28(1):9–18CrossRefGoogle Scholar
  90. Tom E, Aurum A, Vidgen R (2013) An exploration of technical debt. J Syst Softw 86(6):1498–1516CrossRefGoogle Scholar
  91. Urquhart C (2002) Regrounding grounded theory–or reinforcing old prejudices? a brief reply to bryant. J Inf Technol Theory Appl 4(3):43–54Google Scholar
  92. Walker D, Myrick F (2006) Grounded theory: an exploration of process and procedure. Qual Health Res 16(4):547–559CrossRefGoogle Scholar
  93. Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer, ISBN 978-3-642-29043-5Google Scholar
  94. Wynekoop JL, Russo NL (1997) Studying system development methodologies: an examination of research methods. Inf Syst J 7(1):47–65CrossRefGoogle Scholar
  95. Yin RK (2002) Case study research. Sage Publication, CAGoogle Scholar
  96. Zelkowitz MV, Wallace D (1997) Experimental validation in software engineering. Inf Softw Technol 31(5):23–31Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Blekinge Institute of TechnologyKarlskronaSweden
  2. 2.SydneyAustralia

Personalised recommendations