Challenges in Survey Research



While being an important and often used research method, survey research has been less often discussed on a methodological level in empirical software engineering than other types of research. This chapter compiles a set of important and challenging issues in survey research based on experiences with several large-scale international surveys. The chapter covers theory building, sampling, invitation and follow-up, statistical as well as qualitative analysis of survey data and the usage of psychometrics in software engineering surveys.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



We are grateful to all collaborating researchers in the NaPiRE initiative.


  1. AERA, APA, NCME (2014) Standards for educational and psychological testing. American Educational Research Association, WashingtonGoogle Scholar
  2. Amrhein V, Greenland S, McShane B (2019) Retire statistical significance. Nature 567:305–307Google Scholar
  3. Baltes S, Diehl S (2016) Worse than spam: issues in sampling software developers. In: Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement, ESEM ’16. ACM, New York, pp 52:1–52:6.
  4. Binning JF (2016) Construct.
  5. Birks M, Mills J (2011) Grounded theory: a practical guide. Sage, Thousand OaksGoogle Scholar
  6. Bourque P, Fairley RE et al (2014) Guide to the software engineering body of knowledge (SWEBOK): version 3.0. IEEE Computer Society Press, WashingtonGoogle Scholar
  7. Campbell DT, Fiske DW (1959) Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull 56(2):81–105Google Scholar
  8. Charmaz K (2014) Constructing grounded theory. Sage, Thousand OaksGoogle Scholar
  9. Ciolkowski M, Laitenberger O, Vegas S, Biffl S (2003) Practical experiences in the design and conduct of surveys in empirical software engineering. In: Conradi R, Wang AI (eds) Empirical methods and studies in software engineering, experiences from ESERNET, vol 2765. Lecture notes in computer science. Springer, Berlin, pp 104–128. Google Scholar
  10. Coaley K (2014) An introduction to psychological assessment and psychometrics. Sage, Thousand OaksGoogle Scholar
  11. Cochran WG (1977) Sampling techniques. Wiley, New YorkzbMATHGoogle Scholar
  12. Cohen RJ, Swerdlik ME, Phillips SM (1995) Psychological testing and assessment: an introduction to tests and measurement. Mayfield Publishing, CaliforniaGoogle Scholar
  13. Cruz S, da Silva FQ, Capretz LF (2015) Forty years of research on personality in software engineering: a mapping study. Comput Hum Behav 46:94–113Google Scholar
  14. de Mello RM, da Silva PC, Travassos GH (2015) Investigating probabilistic sampling approaches for large-scale surveys in software engineering. J Softw Eng Res Dev 3(1):8. Google Scholar
  15. DiCiccio TJ, Efron B (1996) Bootstrap confidence intervals. Stat Sci 11(3):189–228MathSciNetzbMATHGoogle Scholar
  16. Feldt R, Torkar R, Angelis L, Samuelsson M (2008) Towards individualized software engineering: empirical studies should collect psychometrics. In: Cheng L, Sillito J, Storey MD, Tessem B, Venolia G, de Souza CRB, Dittrich Y, John M, Hazzan O, Maurer F, Sharp H, Singer, J, Sim SE (eds) Proceedings of the 2008 international workshop on cooperative and human aspects of software engineering, CHASE 2008, Leipzig. ACM, New York, pp 49–52. Google Scholar
  17. Fowler FJ (2013) Survey research methods. Sage, Thousand OaksGoogle Scholar
  18. Freedman D, Pisani R, Purves R (2007). Statistics. Norton, New YorkzbMATHGoogle Scholar
  19. Ghazi AN, Petersen K, Reddy SS, Nekkanti H (2019) Survey research in software engineering: problems and mitigation strategies. IEEE Access 7:24703–24718Google Scholar
  20. Glaser BG (1992) Basics of grounded theory analysis: emergence vs. forcing. Sociology Press, Mill ValleyGoogle Scholar
  21. Glaser BG, Strauss AL (1967) Discovery of grounded theory: strategies for qualitative research. Aldine de Gruyter, New YorkGoogle Scholar
  22. Graziotin D, Fagerholm F (2019) Happiness and the productivity of software engineers. In: Rethinking productivity in software engineering. Apress, Berkeley, pp 109–124Google Scholar
  23. Graziotin D, Wang X, Abrahamsson P (2015) Understanding the affect of developers: theoretical background and guidelines for psychoempirical software engineering. In: Proceedings of the 7th international workshop on social software engineering, SSE 2015. ACM, New York, pp 25–32. Google Scholar
  24. Graziotin D, Fagerholm F, Wang X, Abrahamsson P (2017) On the unhappiness of software developers. In: Mendes E, Counsell S, Petersen K (eds) Proceedings of the 21st international conference on evaluation and assessment in software engineering. ACM Press, New York, pp 324–333Google Scholar
  25. Graziotin D, Fagerholm F, Wang, Abrahamsson P (2018) What happens when software developers are (un)happy. J Syst Softw 140:32–47Google Scholar
  26. Gregor S (2006) The nature of theory in information systems. MIS Q 30(3):611–642. Google Scholar
  27. Gren L (2018) Standards of validity and the validity of standards in behavioral software engineering research. In: Standards of validity and the validity of standards in behavioral software engineering research. ACM Press, New YorkGoogle Scholar
  28. Hannay JE, Sjøberg DI, Dybå T (2007) A systematic review of theory use in software engineering experiments. IEEE Trans Softw Eng 33(2):87–107Google Scholar
  29. Hogan R (2017) Personality and the fate of organizations. Erlbaum, MahwahGoogle Scholar
  30. Inayat I, Salim SS, Marczak S, Daneva M, Shamshirband S (2015) A systematic literature review on agile requirements engineering practices and challenges. Comput Hum Behav 51:915–929Google Scholar
  31. Kalinowski M, Card DN, Travassos GH (2012) Evidence-based guidelines to defect causal analysis. IEEE Softw 29(4):16–18Google Scholar
  32. Kalinowski M, Curty P, Paes A, Ferreira A, Spínola RO, Fernández DM, Felderer M, Wagner S (2017) Supporting defect causal analysis in practice with cross-company data on causes of requirements engineering problems. In: Proceedings of the 39th IEEE/ACM international conference on software engineering: software engineering in practice track, ICSE-SEIP 2017, Buenos Aires. IEEE Computer Society, Silver Spring, pp 223–232.
  33. Kass RE (2011) Statistical inference: the big picture. Stat Sci Rev J Inst Math Stat 26(1):1MathSciNetGoogle Scholar
  34. Kasunic M (2005) Designing an effective survey. Technical report, Carnegie-Mellon University, Pittsburgh, PA and Software Engineering InstituteGoogle Scholar
  35. Kitchenham BA, Pfleeger SL (2002a) Principles of survey research part 2: designing a survey. ACM SIGSOFT Softw Eng Notes 27(1):18–20. Google Scholar
  36. Kitchenham BA, Pfleeger SL (2002b) Principles of survey research: part 3: constructing a survey instrument. ACM SIGSOFT Softw Eng Notes 27(2):20–24. Google Scholar
  37. Kitchenham, BA, Pfleeger SL (2002c) Principles of survey research part 4: questionnaire evaluation. ACM SIGSOFT Softw Eng Notes 27(3):20–23. Google Scholar
  38. Kitchenham BA, Pfleeger SL (2002d) Principles of survey research: part 5: populations and samples. ACM SIGSOFT Softw Eng Notes 27(5):17–20. Google Scholar
  39. Kitchenham BA, Pfleeger SL (2008) Personal opinion surveys. In: Guide to advanced empirical software engineering. Springer, Berlin, pp 63–92Google Scholar
  40. Kline P (2015) A handbook of test construction (psychology revivals): introduction to psychometric design. Routledge, LondonGoogle Scholar
  41. Lenberg P, Feldt R, Wallgren LG (2015) Behavioral software engineering: a definition and systematic literature review. J Syst Softw 107:15–37Google Scholar
  42. Levine TR, Weber R, Hullett C, Park HS, Massi Lindsey LL (2008) A critical assessment of null hypothesis significance testing in quantitative communication research. Hum Commun Res 34:171–187Google Scholar
  43. Malhotra MK, Grover V (1998) An assessment of survey research in POM: from constructs to theory. J Oper Manag 16(4):407–425Google Scholar
  44. Mannio M, Nikula U (2001) Requirements elicitation using a combination of prototypes and scenarios. Technical report, Telecom Business Research Center LappeenrantaGoogle Scholar
  45. Méndez Fernández D, Passoth J-H (2018) Empirical software engineering: from discipline to interdiscipline. J Syst Softw 148:170–179Google Scholar
  46. Méndez Fernández D, Wagner S (2015) Naming the pain in requirements engineering: a design for a global family of surveys and first results from Germany. Inform Softw Tech 57:616–643Google Scholar
  47. Méndez Fernández D, Wagner S, Kalinowski M, Schekelmann A, Tuzcu A, Conte T, Spinola R, Prikladnicki R (2015) Naming the pain in requirements engineering: comparing practices in Brazil and Germany. IEEE Softw 32(5):16–23Google Scholar
  48. Méndez Fernández D, Wagner S, Kalinowski M, Felderer M, Mafra P, Vetrò A, Conte T, Christiansson M-T, Greer D, Lassenius C et al. (2017) Naming the pain in requirements engineering—contemporary problems, causes, and effects in practice. Empir Softw Eng 22(5):2298–2338Google Scholar
  49. Méndez Fernández D, Tießler M, Kalinowski M, Felderer M, Kuhrmann M (2018) On evidence-based risk management in requirements engineering. In: International conference on software quality. Springer, Berlin, pp 39–59Google Scholar
  50. Molléri JS, Petersen K, Mendes E (2019) CERSE-catalog for empirical research in software engineering: a systematic mapping study. Inform Softw Tech 105:117–149Google Scholar
  51. Pfleeger SL, Kitchenham BA (2001) Principles of survey research: part 1: turning lemons into lemonade. ACM SIGSOFT Softw Eng Notes 26(6):16–18. Google Scholar
  52. Pinsonneault A, Kraemer K (1993) Survey research methodology in management information systems: an assessment. J Manag Inform Syst 10(2):75–105Google Scholar
  53. Pittenger DJ (1993) Measuring the MBTI…and coming up short. J Career Plan Employ 54(1):48–52Google Scholar
  54. Runeson P, Höst M, Rainer A, Regnell B (2012) Case study research in software engineering. Wiley, LondonGoogle Scholar
  55. Rust J (2009) Modern psychometrics: the science of psychological assessment. Routledge, Hove, East Sussex New YorkGoogle Scholar
  56. Sjøberg DI, Dybå T, Anda BC, Hannay JE (2008) Building theories in software engineering. In: Guide to advanced empirical software engineering. Springer, Berlin, pp 312–336Google Scholar
  57. Sommerville I, Sawyer P, Viller S (1998) Viewpoints for requirements elicitation: a practical approach. In: Proceedings of the 3rd international conference on requirements engineering (ICRE ’98), Putting requirements engineering to practice, Colorado Springs. IEEE Computer Society, Silver Spring, pp 74–81.
  58. Stol K-J, Fitzgerald B (2015) Theory-oriented software engineering. Sci Comput Program 101:79–98Google Scholar
  59. Stol K, Ralph P, Fitzgerald B (2016) Grounded theory in software engineering research: a critical review and guidelines. In: Dillon LK, Visser W, Williams L (eds) Proceedings of the 38th international conference on software engineering, ICSE 2016, Austin. ACM, New York, pp 120–131. Google Scholar
  60. Strauss A, Corbin J (1990) Basics of qualitative research. Sage, Thousand OaksGoogle Scholar
  61. Torchiano M, Fernández DM, Travassos GH, de Mello RM (2017) Lessons learnt in conducting survey research. In: Proceedings of the 5th IEEE/ACM international workshop on conducting empirical studies in industry, CESI@ICSE 2017, Buenos Aires. IEEE, Piscataway, pp 33–39. Google Scholar
  62. Usman M, Britto R, Börstler J, Mendes E (2017) Taxonomies in software engineering: a systematic mapping study and a revised taxonomy development method. Inform Softw Tech 85:43–59Google Scholar
  63. Wagner S, Méndez Fernández D, Felderer M, Vetrò A, Kalinowski M, Wieringa R, Pfahl D, Conte T, Christiansson M-T, Greer D, Lassenius C, Männistö T, Nayebi M, Oivo M, Penzenstadler B, Prikladnicki R, Ruhe G, Schekelmann A, Sen S, Spínola R, Tuzcu A, De La Vara JL, Winkler D (2019) Status quo in requirements engineering: a theory and a global family of surveys. ACM Trans Softw Eng Methodol. 28(2):9:1–9:48Google Scholar
  64. Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer, BerlinzbMATHGoogle Scholar
  65. Yamane T (1973) Statistics: an introductory analysis. Longman, New YorkzbMATHGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.University of StuttgartStuttgartGermany
  2. 2.Technical University of MunichMunichGermany
  3. 3.Blekinge Institute of TechnologyKarlskronaSweden
  4. 4.fortiss GmbHMunichGermany
  5. 5.Department of Computer ScienceUniversity of InnsbruckInnsbruckAustria
  6. 6.Blekinge Institute of TechnologyKarlskronaSweden
  7. 7.Pontifical Catholic University of Rio de JaneiroRio de JaneiroBrazil

Personalised recommendations